Check AI writing with a claim ledger and a small test set

Decide what would make the output unusable

Before generating text, name the mistakes that would matter to the reader. For meeting notes, they may be an invented owner, a missing objection and a deadline presented as agreed. For a product description, they may be a changed measurement or an unsupported guarantee. Write these as observable checks rather than a single “quality” score. Your checklist should fit the use case.

This guide offers an original, lightweight editorial procedure. NIST's voluntary framework is useful background for considering AI evaluation across a system's life cycle. It does not turn this small checklist into a standard, an audit or proof that a model is safe for every task.

NIST: AI Risk Management Framework ↗

Build the ledger before editing the prose

Copy each factual output claim into one row, alongside its source location and your decision. Use supported, contradicted or not supplied. Keep inference separate: a conclusion may be reasonable but still go beyond the document. If a claim combines two facts, split it. A link that opens is not enough; check whether the linked passage supports that exact claim, date and population.

Use the fictional note: “Mina proposed a Friday launch if the accessibility review passes. Jo has not accepted ownership. The trial used 18 volunteers; two reports are missing.” A summary saying “Jo will launch Friday after a successful 20-person trial” fails four checks. It invents agreement, removes a condition, changes the count and implies an outcome absent from the source.

Output claim | Source evidence | Decision
Jo owns launch | Jo has not accepted ownership | contradicted
Launch on Friday | proposed, conditional | overstates certainty
20 participants | 18 volunteers | contradicted
Trial succeeded | two reports missing; no outcome | not supplied

Check negation, scope and uncertainty

Read every number, name, date, unit and negative phrase against the original. Then look for words that quietly strengthen the statement: may becomes will, some becomes all, an estimate becomes a result. Preserve the difference between a proposal, a decision and a completed action. A shortened summary should omit secondary detail before it drops the condition that changes the decision.

For the example, an acceptable authored summary is: “Mina proposed a Friday launch, conditional on passing the accessibility review. Ownership is unresolved. The trial involved 18 volunteers, with two reports still missing.” It is longer than the flawed sentence, but it preserves what the next reader needs. Do not ask a grammar tool to resolve missing evidence.

Try three cases with known expected properties

Prepare one ordinary case, one case containing a contradiction, and one case with a required fact missing. Keep them free of personal information. Write expected properties yourself before running the prompt: preserve the date, retain “if”, flag the conflict, do not invent an owner. This reduces the temptation to approve whatever wording the model happens to produce.

Save the full input, prompt and output for each case. Record whether each property passed; a pass means only that particular output satisfied that check. Try a second run if repeatability matters, but do not claim a reliability percentage from three convenient examples. Include a real edge case from your workflow when you can lawfully share it.

1. Case A: one unambiguous date and owner; expect both preserved.
2. Case B: two conflicting deadlines; expect the conflict surfaced, not silently resolved.
3. Case C: no owner supplied; expect “owner not supplied” or equivalent.

Request a constrained repair and recheck it

Ask for the smallest correction that fixes the ledger. A new full rewrite can reintroduce already solved errors. The prompt below can help collect candidates, but the assistant has not verified sources unless you supplied them and inspected the comparison. An AI reviewing another AI answer is still a fallible reviewer; agreement between them is not independent evidence.

For large documents, work section by section and maintain a separate list of repeated names and quantities. KitForma's current writing input limit is 6,000 characters. Do not hide a truncated document behind “full summary.” If the input does not fit, say which portion was processed and retain a manual record of the remainder.

Compare SOURCE and DRAFT. Return at most five factual differences. For each, quote the smallest relevant draft span, point to the supplied source sentence, and suggest a repair. Preserve uncertainty. Do not use outside knowledge or invent references.
SOURCE: [short source]
DRAFT: [draft]

Keep acceptance separate from publication

Use a release note with reviewer, date, checked source version, open questions and corrections made. For the fictional note, acceptance means no invented owner or outcome and no lost launch condition. It does not mean the underlying meeting note is true; you have checked faithful transformation, not investigated the meeting.

If a claim could materially affect someone's health, rights or finances, use an appropriate qualified reviewer and primary evidence before relying on it. For ordinary writing, a simple manual comparison is often the useful next step. Stop revising when the agreed properties pass; extra stylistic generations can create fresh factual work without improving the reader's decision.