427 code-review loops with different models analyzed

Blake Sims

8h • General discussion

One aspect of my current graph-engineering project morphisms is that every run is captured.

I can define which model is used to write the code, and which one will review it.

I often change models to see the impact it has over many runs and different tasks.

The results are interesting and attached in images.

What I learned:

Fable is NOT a good reviewer: it's not thorough
Gpt-5.6-sol and Astra are thorough and good reviewers - and also DON'T (!) favor their own work over other models (which is surprising, hats off to OpenAI)
Opus 5 sucks

It's too early to say anything meaningful about deepseek v4.1 flash, though it's not looking great. Time will tell. It's cheap, fast, but I'm not yet confident in doing deep complex code-reviews in 100-200k token tasks.

What I am now doing: