Thoughts on Self-Evolving Strategies for ICM

I'm wondering if anyone here has experimented with self-evolving strategies for ICM contexts to achieve better results.

I'm thinking of something similar to model fine-tuning, but at the context/workflow level.

The idea would be:

Start with a training dataset (inputs and expected outputs).
Run a sample of dataset through an ICM flow.
Collect the outputs, compare them against the expected results, and assign a score.
Have an LLM modify the ICM flow based on the evaluation.
Run the updated flow against the same training dataset.
Compare the new outputs with the expected results and calculate a new score.
Run both againt remaining of the dataset compare with expected outputs, scores.
Compare both scores and keep the better-performing version.
Repeat until the score stops improving.

In other words, the LLM would iteratively optimize the ICM workflow itself instead of just optimizing prompts.

Has anyone tried something like this? If so, what worked well, and what were the biggest challenges?

11 comments

Thoughts on Self-Evolving Strategies for ICM