Second Eyes: does a second AI actually catch more? I'm testing it properly

Hayley Cullen-Xiang asked a few days ago whether checking Claude's work with ChatGPT or Codex beats just opening a fresh Claude. Most of us in that thread said yes from experience (me too). Nobody had tested it. So I'm running the test.

The full design is here: https://claude.ai/artifact/8awrFEqk4d1pnd8qZGh5CK

The question has two parts. Does a second AI catch more real errors? And does the work actually come out better after the fixes, or just different?

Most comparisons have a trap in them. They put one Claude review against Claude plus ChatGPT, and two reviews beat one whatever the models are. So this one compares two Claude reviews against Claude plus another company. Same number of looks, different eyes.

What's in it:

12 pieces of realistic work for a made-up renovation company (code, an SOP, a client proposal, cost estimates, schedules) with 96 errors planted on purpose. Plus 3 pieces with no errors at all, to see who invents problems.

ChatGPT, Gemini, Perplexity, Claude Desktop and Claude Code each review every piece 3 times (270 reviews).

Claude then fixes each piece from different mixes of those reviews. Every review it gets has 2 fake findings slipped in, to see if it takes bad advice. Sometimes it's told the review came from "a different company's AI" when it didn't.

People who haven't seen any of this rate the fixed versions blind.

Grading is blind too. I grade, and so does Jev (TypeSafe), which isn't one of the AIs being tested.

The questions, the scoring and the analysis code were locked in before a single result exists. If it comes out against what I expect, I still post it.

One thing already happened. I ran the design itself past fresh AI reviewers, 3 rounds against 22 written checks. 8 passed, then 13, then 13. Every round found something new and smaller, which is exactly what Shaun Lawton described in that thread. I stopped at 3 rounds (thanks

@Heith Murray

Pilot results tomorrow. Full results when all of it (about 400 sessions) is done and graded.

7 comments

Second Eyes: does a second AI actually catch more? I'm testing it properly