Make your ICM battle tested

Two weekends ago, I had to run not one but two audits on the fly of my system.

One because Opus 4.8 hallucinated and said something concerning and I wanted to trace the work.

The other because a CloudKit error ate some of my files (in NotePlan) and I needed to be sure my governance was firing and it wasn’t AI gone rogue.

In both cases, traceability that I had insisted on in the architecture - against the advice of the model - made the audits possible and comprehensive. (And reassuring. Except for the CloudKit issue which means I have a migration coming up because my provider isn’t as resilient as I require).

The result of that - a feedback memory captured and enhanced to “preserve the margin” and the proof point of where items the system/model had seen as “bloat”, I had fought for. It saw the finding as so important it wrote and drafted a memory for my review without me asking for it - AI’s version of an epiphany.

The core truth underlying that - I’ve seen things go wrong. I’ve been around, I’ve had plenty of unhappy path moments. I have been part of the rescue crew. I’m not unique in that - almost everyone here will have their professional and personal equivalents. Those experiences, I carry the scars of those. And each of those I have learnt from. Where the margin is worth it. Why this is a compromise that isn’t worth the cost. Where to hold the line.

AI can give you the textbook theory. It has never been burned. It has never been battle tested. You have. Give it those scars in context in your ICM system and see what happens. It’s game changing.

7 comments