I'm wondering if anyone here has experimented with self-evolving strategies for ICM contexts to achieve better results. I'm thinking of something similar to model fine-tuning, but at the context/workflow level. The idea would be: 1. Start with a training dataset (inputs and expected outputs). 2. Run a sample of dataset through an ICM flow. 3. Collect the outputs, compare them against the expected results, and assign a score. 4. Have an LLM modify the ICM flow based on the evaluation. 5. Run the updated flow against the same training dataset. 6. Compare the new outputs with the expected results and calculate a new score. 7. Run both againt remaining of the dataset compare with expected outputs, scores. 8. Compare both scores and keep the better-performing version. 9. Repeat until the score stops improving. In other words, the LLM would iteratively optimize the ICM workflow itself instead of just optimizing prompts. Has anyone tried something like this? If so, what worked well, and what were the biggest challenges?

And no, I’m not delusional writing that title - and it was hand-typed by a human, not hallucinated by AI. A model isn’t going to take that kind of a risk. Let me explain the actual connection. The company I work for sells stuff to supermarkets, food service and so on. Among the things we sell are instant noodles. Our instant noodles are very popular - without saying too much, we dominate the NZ market based on the data. What that meant was that I got to listen to real people talk about noodles - how they cook them, what they like, what they don’t like - for work this week. What I found fascinating was a few trends that I also see coming to play in the AI space as well: 1. Personalisation - people who are heavily involved in the category are way beyond the average consumer. They would wax lyrical about turning it into a meal base, using it as a hangover cure and even how you can fake a full hot pot using noodles as the base to get them (and the rest of the food) to go further. But the taste was *personal* - it wasn’t shared. Just like a system or knowledge base or ICM that is customised to your needs. 2. Taste is a thing - which flavour, whether the noodles are soupy or dry - people know what they like (or dislike) and they’ll tell you. Model, ICM, code as a sidebar, local vs git - I am sure that everyone here also has their own taste that works for their needs too. 3. Consistency - one of the key things people love about the product is you always know what you’re going to get. High consistency and with consistency comes comfort, familiarity, repeat consumption. You can take risks around it, because of the strong base. Sounds an awful lot like ICM done right to me. And then we get into the other stuff - which brands noodle is thicker and more filling. Who has discontinued a flavour the market loved (thankfully, a competitor!). But ultimately - their favourites were their favourites that worked for them and they stuck with them. Sounds a bit like some files and folders where the new model of the month doesn’t phase you and you can just continue on. Because you have a system that works.

Hi everyone, I'm posting this because I'm curious to hear your perspective on harness engineering within the ICM method. If you have any experience or opinions, please share :)

TLDR: An agent was running an entire data workflow and performing work that a script should have been doing. Splitting the steps into code and keeping the agent for real judgement made the workflow faster, cheaper, more deterministic. Anyone else splitting workflows into code? ## Context I recently read the argument we need to use code more in our AI workflows since its fast, deterministic, and costs 0 tokens to run 🥳. I revisited a slow and token expensive workflow to see if any improvement can be made (Hint: there was), here is my story. ## The workflow I built a workflow to enrich data on a directory site I am building. 1. Find records to enrich 2. Discover the official source ( e.g. company website) 3. Extract information 4. Decide what needs adding or fixing 5. Write it back into the directory 6. Open a PR My first instinct was to throw (possibly too hard) an agent at the whole thing, one smart loop. It was slow and had some performance issues (skill issue I know). Some steps were obvious to switch: - Deciding which records to process is a filter, not a decision. - Writing changes to a database is an API call, not a choice. I was tolerating token cost and hallucination risk for steps a script does perfectly, every time, for free. Side note: I didn't run token cost comparison or analysis of many tokens were used for orchestration vs actual work. ## Reframing the workflow The workflow has 3 actors: - Code - You - Agents Thinking about it this way, the agent only touches the part a script structurally can't do. Some problems it solved: 1. The agent didn't try loading the entire database of options into context 2. It kept extracting a value the schema had no column for, and improvising a workaround every time ## Insights The big insight for me is: if an agent is solving the same structural problem repeatedly instead of once, it's not doing judgment work, it's doing the job code should've done from the start. The LLM brain works on the decision steps.

Last month I was speaking with a friend who works for an engineering firm in Australia about ai, and all the cool things we can do with it these days, and he mentioned that he was trying to push to get a monthly newsletter out to their team to inform them of upcoming professional development courses and workshops. Of course it takes a lot of time to manually search the relevant websites, and put together a newsletter, etc. so no one has done it. I asked a few more questions about the tools they use, and then went and built out a small, structured ICM folder system, with the exact same blueprint that we have been learning in here and using for the competition building. I then made a loom video showing how it worked, and then emailed it to him along with the fully company branded email that it output. It was near the end of financial year at the time, so they were a bit busy, and he said he'd get back to me. Today, 1 month later, he came back and accepted my quote of $600, which includes 2 rounds of revision. To get the draft up and tested, it took me probably about 4 hours, and then there will be another few hrs in finalising it (with the revisions). In reality I probably undersold myself, but this is a side project at the moment (I run a cafe, not an ai consulting business...yet!), and I was excited to have the opportunity at a real client to test against. Let me be clear, I'm not selling a fancy Ai loaded website or app or anything... It is literally 5 folders, with a top level claude.md file, instructing claude co-work (or claude ai or code - or any other Ai tool that can follow structured instructions with a renaming of the claude.md file) on exactly what to do. This is exactly what Jake has been teaching here, and it will stand the test of time. As Anthropic updates their interface, and Open Ai starts to take over claude or another better tool comes along, this ICM folder system will continue to do its thing. It may need some tweaks along the way, but it's not going to need it's whole codebase updated or anything, because it's all plain language, english instructions.