Beware of the Illusion · Clief Notes

Beware of the Illusion

AI can make you think your smarter and more knowledgeable than you really are.

The base phenomenon Psychologist Frank Keil found people rate their understanding of how things work (toilets, zippers, engines) much higher than they actually can explain — until you ask them to explain it step by step, and the confidence collapses. Fluent exposure to an explanation gets mistaken for possessing the explanation yourself.

What AI adds on top

Fluency as a proxy for truth/competence. When output reads clean, structured, and confident, your brain uses “this is easy to follow” as a stand-in for “I understand this.” AI text is optimized for exactly that fluency, so the proxy misfires more than it does with a messy textbook or a mumbling human expert.
Co-authorship inflation. You typed the prompts, made choices, steered the direction — so you feel ownership over the output the same way you’d feel ownership over code you wrote. But steering isn’t the same as generating the underlying logic. You built the map; the AI built the terrain.
No friction, no failure signal. Normally, hitting the edge of your understanding feels like friction — you get stuck, you have to look something up, you notice the gap. AI smooths that friction away by answering immediately, so the gap never announces itself. You only find it later, when something breaks or someone asks you to defend a step.
Automation bias. Once a system has been right a few times, you extend it trust it hasn’t earned on the next output, especially in domains (like algorithmic finance) where verifying correctness is itself hard.
The practical tell, If you can use the model and get the right answer but can’t predict when it’ll be wrong or explain a step to someone else without the AI’s help — that’s the gap between “I operated it” and “I understand it.” Not a character flaw, just how fluency-based confidence works.

The fix isn’t “understand everything.” It’s knowing which parts of the map you’re taking on faith vs. which parts you could rebuild from scratch. If you find what your working on leads to this, —pick the 2-3 load-bearing assumptions and force yourself to derive them without the AI, even roughly. That’s usually enough to convert vibes-confidence into real confidence, or to find out it doesn’t hold up.

12 comments

Beware of the Illusion