Auto-Memory: Fine-Tuning in Disguise?
I'm here to share a rather intriguing observation I've made while running problem-solving tests on Agent Zero, and more generally within the realm of physics. There's a particular problem – #7 on page 24 of the Oxford University Physics 1 textbook – that no AI can crack using a zero-shot approach, save for o1-preview, as you can see in the third attached screenshot. The first video shows Agent Zero equipped with Gemini 1.5 Flash 002 and two other subordinate agents, having also been informed that one of these subordinates is an expert physicist, but no success, they just overthink while being biased towards their first assumptions. My "Grass grows, guys!" is very clear about that. In the second video, however, we have Agent Zero successfully solving the problem after I prompted it to memorise the following sentence: "When solving physics problems, don't rely solely on appearances, and utilise predictive models to compensate for any missing features within your problems". None of the LLMs I tested managed to deduce the fact that the grass continues to grow, thus requiring a mathematical model that accounts for this growth. They all answered 27 days, but the correct answer is 54. It seems the auto-memory function, when used effectively, offers less case-specific functionality than I initially expected, and actually allows A0 to generalise even better, without further bloating the system prompt. Thanks a million, Jan. [Oxford's textbook URL: https://www.ox.ac.uk/sites/files/oxford/media_wysiwyg/physics-problems-solutions-1-compressed-1.pdf]