How to Split the Loop Between Deterministic and Interpretive?

The cheapest model I run does most of the work in my agent loop. The most expensive one only wakes up when a deterministic check has already decided it is warranted.

- Deterministic harvest - the mechanical half. Routes each job's report, updates the index, moves the finished record from pending to completed. Zero judgement calls, so it runs on the cheapest model available.

- The verdict - a plain rule, computed at the end of every run, that decides whether the interpretive step is even needed this cycle.

- Judgement pass - the one genuinely interpretive step. Runs on the strongest model, and only fires when the verdict says it is warranted.

The loop keeps the state of my system current after every job runs. Most of what that involves is not thinking, it is filing - taking what each job reported, sending errors to one place and state updates to another, then moving the finished record out of the pending folder. None of that needs a judgement call, so I run it as a plain deterministic script on Haiku. It behaves identically every time, which is exactly what you want from the mechanical half.

The interpretive work is a different animal. Merging conflicting updates by hand, pruning a list down, deciding what is genuinely resolved and safe to archive - that needs real judgement, and it is where a weak model quietly makes a mess you only find two laps later. So it is a separate stage and it runs on Opus. The part that keeps it cheap is that it does not run every cycle. At the end of the deterministic stage the script computes one thing: is the judgement pass warranted right now. The rule is plain - is there a queue of unresolved items, has a file grown past its size line, have enough jobs landed since the last judgement pass. If none of that is true, the expensive stage never fires.

The two stages never collide because a single-writer lock hands off between them. The deterministic stage takes the lock, does its filing, computes the verdict, and if the verdict is warranted it keeps the lock held and spawns the strong model to run the judgement pass under that same lock. If the verdict is not warranted it just releases and the cycle ends cheap. One held lock, never two writers touching the same record.

The wider point is that model choice is a per-step decision, not a system-wide one. The instinct is to reach for the best model everywhere and eat the cost, or pick a cheap one everywhere and eat the quality drop. Neither is the real move. Split the work by whether the step makes a judgement call, run the mechanical majority on the cheapest thing that will do it reliably, and reserve the strong model for the interpretive minority - then gate even that behind a check so it only fires when there is real work for it. Same discipline as the rest of ICM: the cost problem is structural, solved by architecture, not by switching to a bigger model and hoping.

What I have not closed yet is running this fully unattended. Right now the loop lives in a window I keep open. Handing the timer to the OS so it fires with nobody watching is the last piece, and I have deliberately left it unwired until the auto-firing between the two stages has more laps behind it.

How are you splitting model choice across your own loops - one model for the whole thing, or by the step?

2 comments

How to Split the Loop Between Deterministic and Interpretive?