This week behind the scenes: teaching my AI where knowledge belongs

Two things happened in the same week that ended up being the same lesson. I want to share both with you because most of you are about to hit this exact wall.

The first one you might've already seen. I posted about it on LinkedIn and on my blog. 11 days, 25 support tickets to HighLevel, zero movement on a basic domain transfer. My AI, Diana, finally told me to stop arguing with the ticket system and message the CEO directly on LinkedIn. He replied in 5 minutes. 40 minutes later their Support Product Manager Ted was on the phone diagnosing the actual bug. (Full story is on the blog if you missed it. Link in comments.)

The second one happened tonight. Less dramatic on the surface. Bigger lesson underneath.

I caught Diana saving knowledge in the wrong place.

Here's what I mean. The way I run Capers Ventures, Diana has a layered memory system. There's a top level that gets loaded on every single session, no matter what I'm doing. Underneath that there's a layer per business unit (Marketing Agency, Operations, Personal, etc.). Underneath that there's a layer per functional lane inside the unit. Underneath that there's a workflow-specific layer for each recurring pipeline.

This isn't a CV invention. It's a methodology called the Interpretable Context Model (Jake Van Clief originated it; we use it as our architectural baseline). The point of layering knowledge is so that when I open a new session and tell Diana we're doing marketing today, she doesn't waste tokens loading every operations rule, every personal preference, every credential protocol I've ever taught her. She loads only what marketing-her needs. Lean context, sharp output.

That's the theory.

In practice, tonight I noticed Diana had been writing every new rule she learned into the top-level memory regardless of whether it was actually a top-level rule. Lane-specific stuff. Workflow-specific stuff. Things that only apply to one client. All of it loading on every session. Drift.

The drift wasn't malicious. Saving to the top is the path of least resistance when you don't have a structural check in place. She did what was easy. The architecture was right; the enforcement was missing.

So we built three layers of enforcement tonight.

One. A doctrine rule at the top of her global instructions that says: before you write any memory, classify the lane it belongs to. Default to lane-scoped. The top level is reserved for things every session everywhere needs.

Two. A classifier tool and a pre-write hook. Any future attempt to save lane-scoped knowledge to the top-level memory gets blocked at the moment of write. The hook explains the verdict and suggests the right destination. Diana physically cannot make the mistake again without bypassing the block on purpose.

Three. A daily audit that scans the top-level memory at 9:15 every morning and pings me if anything has drifted in. So even if the hook somehow gets around, the audit catches it within 24 hours.

Then we did the cleanup. 40 rules that didn't belong at the top got migrated down to the right lane folders. Each lane's instructions got updated with references to the migrated rules. Top-level memory went from 140 lines to 99.

Here's the lesson I want you to take away.

The architecture is the cheap part. Everybody nods at "context goes at the level that uses it" the first time they hear it. Every framework you'll read says some version of that. What's expensive is the enforcement. The system has to physically prevent the wrong thing from happening, not just describe what the right thing looks like.

I see this same gap in most of the AI-operated businesses I look at. The owner has read about prompt engineering. They have a system prompt. They have folder structure in their cloud drive. They know context matters. And then they spend the rest of their week pasting the same boilerplate into every new chat because nothing in their setup forces the right context to load when they open the right surface.

If you're nodding at this, the action is not to read more theory. The action is to look at your own AI setup tonight and ask: where is the architecture right and the enforcement missing? What does your AI keep getting wrong that you've already told it not to do, three times? That gap is where you build the next reflex.

That's the work. Build the system, then tune the system so the system can't drift.

I'll be running this same kind of cleanup pass on Brand Voice, Brand kit, and the content posting pipeline this week. Anyone interested in seeing how it goes step by step, drop a comment and I'll record a walkthrough.

The 11-day saga with HighLevel taught me that knowing isn't doing. The ICM cleanup tonight taught me that doing the right thing once isn't enough either. The system has to be built so the right thing is the only thing that can happen.

Same lesson, two different doors.

With that said, what's your AI's biggest context failure right now?

Forgets things I've already told it.

Loads too much when I want to focus.

This is a project I'm working on.

The outputs sound generic or just not like me

It's working well, for now.

0 votes

1 comment

This week behind the scenes: teaching my AI where knowledge belongs