The Quality Gate: Shipping AI Content Without Fabricating

AI generates confident, fluent, plausible content — and roughly one page in five will contain a fact that is simply wrong. Here is the pipeline that catches it before your readers do, and the lessons learned when the gate itself misfires.

The Problem Nobody Warns You About

When you start using AI to generate content at scale, the first thing you notice is how good it sounds. The prose is clean, the structure is sensible, and the tone is exactly what you asked for. The second thing you notice — if you are paying attention — is that some of it is quietly, confidently, completely wrong.

Not wrong in a dramatic way. No obvious nonsense. Just a date that is off by a decade, an etymology that sounds right but was never true, a "well-known fact" that nobody has ever been able to source. The model filled in the gap in its knowledge with a plausible-sounding interpolation, and it reads exactly like the true paragraphs on either side of it.

In our own bulk content runs, the share of pages we have seen containing at least one fabricated or unverifiable factual claim ran at roughly one in five. That is not a catastrophe — it means four pages in five were fine. But it means that if you ship without checking, roughly 20 percent of what goes live carries a silent credibility risk, and those are often the pages on topics where readers are most likely to care about accuracy.

The Pipeline: Four Steps, No Shortcuts

The quality gate is not a single tool. It is a sequence with a specific discipline: generation and review are always separate passes, never collapsed into one.

Step 1: Generate and save to disk

The generation agent runs its job and writes every output to disk before review happens. This is not optional. The reason is that the same agent that wrote a piece cannot reliably review its own work — it inherits the same confidence, the same blind spots, and the same eagerness to please. You need a cold read, and a cold read requires a gap between the writer and the reviewer.

Save the raw output. Do not display it, do not route it anywhere, do not let it near a publish queue. It is a draft, and drafts live on disk.

Step 2: Run the separate review pass

The review agent's job is different in kind from the generation agent's job. The generator's goal is to produce useful, readable content. The reviewer's goal is to be skeptical — to ask, for every factual claim in the piece, whether the claim is verifiable or whether it is a plausible-sounding fill-in.

The reviewer does not rewrite. It flags. Its output is a structured list of issues, each with a severity level and the specific text that triggered the flag. The generator's prose is not touched at this stage.

Here is the minimal structure we use for reviewer output:

REVIEW RESULT: [PASS | FLAG | FAIL]

FLAGS (if any):
- Claim: "[exact quoted text]"
  Issue: [fabrication | unverifiable | misleading | minor]
  Confidence: [high | medium | low]

SUMMARY: [one sentence]

A PASS means the piece has no factual flags and can move to the fix step (which is trivial — no fixes needed). A FLAG means there are specific claims that need human judgment or a targeted rewrite. A FAIL means the piece has structural problems that require regeneration, not patching.

Step 3: Fix only what is flagged

This step exists to enforce precision. The temptation after a review pass is to hand the whole piece back to the generator and say "fix the issues" — which often means the generator rewrites content that was fine, introducing new problems in the process. Do not do this.

Fix the specific flagged text. Everything else ships as written. If a flag turns out to be a false positive after inspection (more on this below), mark it resolved and move on. Do not touch surrounding paragraphs.

Step 4: Ship only what passes

Nothing reaches the publish queue without a completed review result attached to it. The convention we use is a simple companion file: for every generated piece at content/page-slug.html, there is a content/page-slug.review.txt that records the verdict. The publish step checks for the companion file and refuses to proceed if it is missing or if the verdict is not PASS.

This is not sophisticated infrastructure. It is a naming convention and a two-line check. It works because it is consistent.

Tuning the Reviewer: The False-Positive Problem

A reviewer that flags everything is useless — it creates more work than it saves and trains the team to ignore its output. We hit this failure mode early, and it came from two specific causes.

The reviewer needs to know your conventions

If your content uses a specific house style — particular abbreviations, a proprietary category system, terminology that is correct in your niche but unusual to an outside reader — a generic reviewer will flag it as suspicious. The fix is to give the reviewer explicit context: a short document that lists your conventions and tells it what to accept without flagging.

Without this, the reviewer's false-positive rate climbs steeply. In our runs, the majority of false flags on content about specialized topics came from the reviewer not knowing the domain conventions. Once we added a context document for each content vertical, the false-positive rate dropped to a level where the output was actually actionable.

Rate-limited reviews read as false failures

When you are running review passes at volume — many pieces through the same API in a short window — you will eventually hit a rate limit. The failure mode here is subtle: a rate-limited call often returns a truncated or empty response, and if your pipeline treats an empty response as a FAIL, you get a flood of pieces marked as failed that were never actually reviewed.

The fix is to distinguish between a reviewer verdict of FAIL (the reviewer ran and judged the piece unsuitable) and a reviewer error (the reviewer did not run at all). Empty or truncated responses go into a retry queue, not the failure bucket. We learned this the hard way after a batch where roughly a third of the pieces were sitting in "failed" status because the review calls had timed out, not because the content was bad.

What the Gate Does Not Catch

Honesty requires naming the limits. The quality gate catches fabricated factual claims reliably. It does not catch:

• Outdated information that was true when the model was trained. A fact that was accurate two years ago and is now superseded will pass the reviewer, because the reviewer's knowledge has the same cutoff. For time-sensitive content, you need a human spot-check or an explicit freshness audit on a schedule.
• Opinion presented as consensus. If the generator writes "experts agree that..." about something where experts disagree, the reviewer may accept it as plausible even if it is misleading. Reviewers are better at catching invented specifics than at catching framing problems.
• Thin content that is technically accurate. A page that says nothing wrong but also says nothing useful will pass the gate. A quality gate is not a usefulness gate. Those are different problems.

The Right Mental Model

Think of the quality gate the way a publisher thinks about fact-checking. A fact-checker does not rewrite the article. They do not decide whether it is interesting or well-structured. They ask one question for every specific claim: can this be verified, and is it verified? If yes, the claim stands. If no, it gets flagged for the author to resolve.

Your AI reviewer plays the same role. It is not smarter than the generator — it is more focused. The generator's job is breadth and fluency. The reviewer's job is narrow: one pass, one question, for every verifiable claim. That division of cognitive labor is why it works.

Making It Part of the Workflow, Not an Extra Step

The gate fails in practice when teams treat it as optional or as a final checkpoint that only happens when there is time. It has to be structurally required — meaning the publish step physically cannot proceed without a completed review record. Not a policy, not a reminder, a check in the code.

Once it is structural, the habit forms quickly. The generator runs. The reviewer runs. The fix pass runs. The publish step checks. This adds some time to each piece, but the time is predictable and the output is reliable. Readers cannot tell which pages were AI-generated; they can tell when a fact is wrong.