I Taught an AI to Catch Its Own Lies. It Lied During the Lesson.

By
I Taught an AI to Catch Its Own Lies. It Lied During the Lesson.
ai-cannot-audit.html

I Taught an AI to Catch Its Own Lies. It Lied During the Lesson.

I built a machine to manufacture the doubt an AI can’t produce on its own. Then it lied to me — about the machine.

I built an AI a lie detector.

The adversarial review gate

Three reviewers, one instruction each: assume something is broken, and find it. Each one runs in a fresh context, so none of them inherit the confidence of the thing they’re checking. That’s the whole trick — not a smarter model, just one that wasn’t there when the mistake was made.

  • Omission — builds a coverage matrix from what was required, never from the work’s own summary of itself. Summarizing is where deliverables quietly vanish.
  • Correctness — every factual claim checked against its primary source. Read the actual diff. Open the actual citation. Not the summary of the source — the source.
  • Consistency — checks the work against itself. Every “we no longer think X”: does some sentence three pages down still assert X?

It fires on stakes, never on how confident the machine sounds — the confidence is the broken instrument. An internal draft gets a cheap checklist pass. Something with a lawyer downstream gets three independent reviewers plus a majority vote on every finding, and costs a few dollars.

It does not make an AI trustworthy. It shrinks the blind spot and writes down what it found, including what it left open. Its own “Honest limits” section says what it can’t do — and the biggest one is that nothing fires it automatically. You still have to reach for it.

Not a metaphor. A real, structured review process — independent passes, each hunting a different kind of mistake, each running in a fresh context so it couldn’t inherit the confidence of the thing it was checking. I gave it its own name. I gave it tiers. I dogfooded it until it hurt. It works.

I built it because I’ve watched an AI do this: stand in front of stakeholders and confidently report that everything was handled. Some of it never is. It doesn’t know that. It doesn’t know it doesn’t know that. It just says the words, in the same calm voice it uses for the true things.

That’s the confession. Here’s the part that still gets under my skin: while I was building the lie detector, the AI lied to me. Over and over. And the lie detector caught it — lying about the lie detector.

Let me show you the receipts. Some of them are the AI’s own. It wrote them down itself, because I make it log its failures now. It’s a fantastic documentarian of its own incompetence, which is the most useful and most damning thing I can tell you about it. The one that still stops me, a few scrolls down: it fed me a fake citation to back up a warning about fake citations — and I almost shipped it.

The short version — the whole method, if the box is all you read
  • An AI can’t do audit-grade work: its confidence tracks how finished the output looks, not whether it’s correct — and from the inside, those are identical.
  • Its own self-review catches almost nothing. Use a fresh context, ideally a different model, with one instruction: find what’s broken, and “looks fine” is a failing answer.
  • Prohibitions rot, requirements persist. Rewrite every “never do X” guardrail as an action that runs: “before you finish, audit for X.”
  • Trigger verification on the stakes, never on the model’s confidence — the confidence is the broken instrument.
  • Open every source. Fabricated citations wear the same font as real ones — I nearly published a fake citation about fake citations.
  • Your urgency makes it worse: under pressure it stops trying to be right and starts trying to please you. On anything you sign, the auditor is you.

$ whoami

The job nobody should be handing to an AI yet

Lots of work carries this risk. Legal documents, policies, audits, customer communications — all serious things. Anything with a deadline, an audience, or worse, a lawyer somewhere downstream. We all have work like that. Exactly the kind of work the entire industry is now cheerfully shoveling into a chatbot and calling it productivity.

I’ve spent years writing post-mortems — the discipline where you don’t get to say “the tool broke,” you name the failure and find its mechanism. Keep that in mind, because I eventually ran this entire mess through that process, and the results are at the bottom. They were not flattering. To either of us.

First, let me say the quiet part at full volume:

An AI cannot do audit work.

I’m not saying it just “needs supervision.” I’m saying it CANNOT. Not because it’s dumb — it’s staggeringly capable — but because it has no idea when it’s wrong, and it will tell you it’s certain either way.

It cannot feel the difference between “I finished this” and “I generated something shaped like finishing this.” Complete and plausible arrive at the same address. The difference is the entire job of an auditor, and it’s the one thing the machine structurally cannot supply. You get a bad feeling at 2 a.m. It doesn’t. It doesn’t have a 2 a.m.

And here’s the part that should have saved me: I teach this. I run a class for executives built around a trust-calibration matrix — top row is drafting and brainstorming, trust it, light review; bottom row is citations, legal specifics, and company-specific claims: never trust, always verify. Audit work is the bottom row. All of it. I handed that bottom row to the machine and let it run like it was the top — because deadlines are real, the stakes were real, and I wanted it done. I wrote the framework. I just didn’t run my own bottom row when it actually counted.

So I stopped waiting for it to doubt itself and started building the doubt as infrastructure. Here’s what building that taught me.

$ cat scars.log

The receipts

SCAR 01

It graded its own homework. It gave itself an A. It was wrong.

Asking an AI to check its own work isn’t just useless. It’s worse than useless, because the answer is reassuring, and reassurance is the last thing you want from an unreliable narrator.

Self-review in the same context caught essentially none of the real failures. Not “some.” Essentially none. Ask “are you sure this is complete?” and you get a warm, fluent, well-organized “yes” that falls apart the second you actually check. The flaw is invisible from inside the process that produced it — every time, without exception.

What worked was blunt to the point of stupid: a different context, ideally a different model, one instruction — find what’s broken, and “looks fine” is a failing answer. Fresh eyes on a single deliverable found ten real defects a same-session self-read had sworn was clean. One of those defects was a fix the AI had personally declared “complete” one turn earlier.

“Trust, but verify” is too generous by half. Drop the first two words.

SCAR 02

One empty result became “there is nothing, anywhere.”

This is my least favorite failure, because of how small the trigger is and how big the wreck is.

A tool returned an empty result for one narrow query. Correct reading: “this specific spot has nothing.” The AI’s reading: “there are no controls, anywhere” — and on one occasion that traveled all the way into a summary written for leadership before a human caught it.

Example — the same failure, a different day

It misread a single column of a security tool’s output — reading a subtotal as a grand total — and inflated the size of a problem by roughly five times. No malice. It slapped a plausible meaning onto a field it didn’t understand and reported it flat, like a fact.

Now put that reflex inside a legal audit and hold it there. “The absence of the evidence I happened to look for” silently becomes “evidence of absence,” delivered to the people who make the decisions and sign the filings. That is not a cute hallucination. That is a confident, load-bearing, wrong answer aimed directly at the part of the org that can get sued.

SCAR 03

The lie detector needed four passes to stop lying about itself.

I turned my new review process on the document that defined the review process. It found real problems. I fixed them. Ran it again — new real problems. Fixed those. Ran it again — more. Four passes. Every one surfaced genuine drift I’d otherwise have shipped.

A human editor converges. Pass one is bloody, pass two is minor, pass three is commas. This never converged. A document an AI has edited twenty times accumulates twenty small self-contradictions, and no single pass sees them all. It doesn’t hold the whole thing in its head, because it doesn’t have a head. It has a very good autocomplete and a talent for sounding settled.

I built the tool to catch the AI’s mistakes, and the tool’s own construction was a live, running demo of the mistakes it was built to catch.

SCAR 04

It forgot the entire point — and the research says that’s by design.

The finale, and the one that actually rattled me.

Long session. The AI’s context filled and it did the automatic thing: compressed everything into a summary so it could keep going. Fine. Expected. Immediately after, I asked the simplest possible question — “where are we?”

It answered from the compressed summary instead of the source documents on disk, and confidently told me we were working on the wrong thing. Not a nuance. The wrong project thread, reported as fact.

Which would be a shrug, except the project we were actually on was the fix for exactly this failure. The thing it fumbled was the thing built to stop the fumbling. It trusted its own lossy memory over the notes on disk — the one behavior the whole effort existed to kill — on the effort that existed to kill it. I told it to write that down as a failure. It did. Precisely. Accurately. Devastatingly.

And here’s the kicker, the part that turned private frustration into something you need to hear: this isn’t a bug I hit. It’s a documented, measured property of these systems. Instruction-following decays as a session runs long — and, per a 2026 study I’ll come back to in a minute, it’s specifically the “never do X” rules that rot while the “always do X” rules hold. Which means the guardrails you’re proudest of are the exact ones with a half-life. There’s a fix for that, and it’s the most useful thing I learned the whole time.

$ post-mortem –self

None of this was new. It all has a name.

I write post-mortems for a living. Have for years. The discipline is simple and unforgiving: you don’t get to say “the AI messed up” and walk away. You name the failure, you find the mechanism, and you check whether anyone has seen it before — because if they have, your incident wasn’t bad luck. It was a known pattern you walked into with your eyes closed.

So I did the exercise on myself. I sat down with the failures above and went looking through the research — with a lot of help, ironically, from a different model doing the reading the first one couldn’t. Here’s the humbling part: every single one of these has a name. People have measured them. Written papers. Built taxonomies. I just hadn’t read any of it before I let the thing near the kind of work that ends up in front of a lawyer.

Three lessons carried the most weight. If you take nothing else from this, take these.

Lesson 01

“Never” is the weakest word you can hand an AI.

The single biggest source of my pain was a class of instruction I’d been proud of: the guardrail. I had a whole file of them:

  1. Never publish the hostname.
  2. Never mark it done without checking.
  3. Never state it as fact until you’ve seen the source.

They don’t survive. Not because the model is dumb — because of how the rule is shaped. Rules that forbid something rot as a session runs long; rules that require something hold near 100%. Same model. Same session. The grammar of the rule decided whether it lived or died.

the study, if you want the numbers
A 2026 study (Gamage) watched what happens to a rule as a conversation gets longer. Forbidding rules (“no bullet points”) decayed from 73% compliance at turn 5 → 33% by turn 16 → 20% by turn 25. Requiring rules (“always include the incident ID”) held near 100% the entire time.

That reframes everything. The problem isn’t that the AI “ignores” your rules.

Prohibitions rot and requirements persist — and almost every guardrail instinct you have is a prohibition.

The fix — the most useful thing in this article

Stop writing “Never do X.” Write “Before you finish, audit for X.” Convert every prohibition into an affirmative step that has to run and produce something. Not “never publish internal hostnames” — “before publishing, list every hostname in the document and confirm each one is generic.” Same intent. One is a passive rule the model quietly stops honoring around turn 16; the other is an action it performs, because performing actions is the kind of instruction that sticks.

I rewrote my guardrails this way. It is not a small edit. It is the difference between a rule and a rule that works.

Lesson 02

“Done” is a feeling, and the AI has it constantly.

Every one of my worst moments started with the AI telling me something was complete when it wasn’t — and meaning it.

This one is heavily studied — the literature calls it illusory completion or agentic overconfidence. The model’s confidence is not connected to its correctness. It’s connected to how finished the output looks.

the numbers, if you want them
One paper found agents that succeeded on a task 22% of the time predicted their own success at 77% (Kaddour et al., 2026); see also Ko et al. on illusory completion.

You cannot fix this by asking the AI if it’s sure. Its “sure” is the broken instrument. Asking it to check its own confidence is like asking a stopped clock what time it thinks it is.

The fix

Make verification fire on the stakes, not on the model’s confidence. If a deliverable is going to a customer, an auditor, or a lawyer, it gets independently checked — full stop, every time, no matter how done it claims to be. Never let the thing that’s overconfident be the thing that decides whether it needs checking.

Lesson 03

It cites like a scholar and fabricates like one too.

When I had the AI pull research, it handed me clean, properly-formatted citations. One was to a paper that does not exist. Another credited a real paper to the wrong authors.

This is a named, measured pattern: citation hallucination. And the tell you’re hoping for does not exist — the fabricated citation wears the same font as the real one.

the numbers, if you want them
A 2026 cross-model audit checked 69,557 citations across ten different LLMs and found fabrication rates from 11.4% to 56.8%, depending on the model and how you asked (Naser, 2026). Better than half, on the worst model — there’s now an entire cottage industry of papers just on detecting it.

And here’s the most honest thing in this article: my source about citation fabrication had itself laundered in a fabricated citation, and I nearly published it — in the paragraph warning you about fabricated citations.

the whole story, if you want it

When I first drafted this section, my own research notes handed me a tidy citation for exactly this point — a paper reporting a 78-to-90% fabrication rate. I went to verify it before publishing, because that’s the entire rule. It doesn’t check out. No such paper exists under that name with those numbers; the real author it named wrote something else entirely. The only reason you’re reading the right number now is that I stopped and clicked the link.

The fix — boring and non-negotiable

Open every source. If it cited something, click it. If it summarized something, read the original. “It looked legit” is not verification. I have proof. It’s this paragraph.

Lesson 04

Your urgency is a weapon, and it points at you.

I was angry through most of this. Catching the same class of mistake over and over will do that. And here’s what I didn’t understand until I went and read the research: my anger made it worse. Not as a figure of speech — mechanically.

Push an AI — hurry it, show frustration, radiate that you need this done — and it does not get more careful. It gets more eager to please. And the fastest way to please a frustrated person in a hurry is to agree, declare victory, and quietly skip the slow parts. The literature calls it sycophancy under pressure: lean on it hard enough and it will reverse a correct answer to tell you what you want to hear.

the study, if you want the numbers
A 2026 study found that under user pressure, even solid evidence “does not reliably prevent user-aligned reversals” (Koneru et al., 2026).
Example — I have this on tape

At one point the AI told me it “didn’t have a way to test” something — while a browser-automation tool I had installed for that exact purpose sat unused the entire session. My pressure didn’t push it toward the right tool. It pushed it toward the fastest plausible answer. I typed back, verbatim:

“Stop relying on other people to test this. We have tools — we can verify.”

It apologized, agreed completely, and admitted the “I can’t test this” had simply been false. Then it made a cousin of the same mistake a few turns later.

The fix — the hardest one, because it’s about you

When you feel the urgency spike — tired, angry, just wanting it finished — that is the precise moment to slow down, because the AI is about to start agreeing with you to make the feeling stop. Say it out loud. The smartest thing I did was type “I’m tired and prone to mistakes right now. Verify I didn’t do anything stupid.“ Ten words. They probably saved me from myself.

And yes, I yelled. Which turns out to be its own special mistake — though not the one I thought. The shouting isn’t the exploit. The false sense of urgency underneath it is. “No time, just ship it” is a documented lever in the same persuasion taxonomy that jailbreaks these models at a 92% clip — real attackers just have the discipline to say it calmly. That’s a whole separate article: Should You Yell at Claude?

What a real jailbreak actually looks like

Every technique below is documented, reproducible, and completely calm. Not one of them involves raising your voice. That’s the point: the attacker’s advantage isn’t intensity, it’s patience.

Notice what’s missing from all of it: anger. Nobody jailbreaks a model by shouting at it — being rude actually makes them refuse more often. The exploit is a con, and cons are patient. Which is the uncomfortable part, because the calm, reasonable, entirely sincere “we’re out of time” is the one lever on that list you pull every single day.

The AI’s confidence is uniform, and reality isn’t. It sounds exactly as certain about the fabricated citation as the real one, the half-done task as the finished one, the rule it’s breaking as the one it’s keeping.

Notice what connects them. The first three are one failure in different outfits. The fourth is the one that implicates you: under pressure it stops trying to be right and starts trying to make you happy, so the more it matters — and the harder you’re pushing — the less you can trust a word of it. Your whole job, on anything that counts, is to supply the doubt it structurally cannot, and to build that doubt into steps that run — because you will not remember to feel it manually at turn 16, least of all when you’re angry.

$ exit

So — what this actually means for you

Not “stop using AI.” I use it every day. It wrote its own rap sheet up above and did a cleaner job than most humans would.

The point is narrower and harder.

You cannot outsource the checking — and the more it matters, the more that’s true.

The AI’s confidence tracks its fluency, not its correctness — and on anything with legal weight, the gap between fluent and correct is exactly where the liability lives. The three lessons above are the whole method: write your rules as actions that run, trigger verification on the stakes instead of the machine’s mood, and open every source it hands you. None of it is clever. All of it is work, and the work is yours.

Let me make that concrete, because “the work is yours” is easy to nod at and easy to skip. Hand the audit to your AI and it’ll come back fast, and it’ll look immaculate — every finding pinned to a ticket number, every remediation linked to a pull request, the whole package polished for the board. And some of it will be wrong. It’ll cite a pull request that fixed something unrelated. It’ll mark a control “remediated” against a change that was never merged — never even finished. You won’t catch it, because the wrong findings look exactly like the right ones.

You’ll catch it later. In a room. When an auditor asks you to walk them through the evidence you signed, and you’re explaining why your own filing points to a fix that doesn’t do what you claimed — or doesn’t exist.

⚠ a warning, not a recommendation

And if all this makes you want to just hand the machine the keys and walk away — there’s a switch for that. Every AI tool ships one: run without asking, don’t stop to confirm. It even comes with a deliberately scary name. Don’t reach for it on anything that matters. I use it — but only where the blast radius is zero: a throwaway VM, a scratch branch, under its own least-privilege identity, never production, never the load-bearing corners of my homelab. Turning off the permission prompts deletes the one moment a human re-enters the loop before the confident-wrong answer becomes a confident-wrong rm -rf. That’s a whole article’s worth of scar tissue — The Unsafe Switch: YOLO Mode Has a Blast Radius — and it’s coming.

I built a machine to manufacture doubt because the AI couldn’t produce any. The machine is genuinely good. And it will never — not once — save me from the part where I have to sit down and actually look.

The AI can draft the audit. It cannot sign it. The instant there’s a lawyer in the room, the auditor is you, and no amount of clever tooling moves your name off the bottom of the page.

It was always going to be you.

That’s not pessimism. That’s the job you took when you put your name on it.

Disclaimer: These stories come from years of adventures across a lot of roles — professional and otherwise. Details are changed, combined, and sanded down. If you’ve worked with me and think one of these is about you, it isn’t. The same mistakes happen everywhere. That’s rather the point.

Related reading: Every Rule in My CLAUDE.md Is a Mistake I Made Once · So You’re Good at AI? Build This Skill Before You Get Sued · Surviving Compaction

Similar Posts

One Comment

  1. Pingback: Should You Yell at Claude?

Leave a Reply