16d • 💬 General discussion
filesystems or databases ??
I, recently, came across an article argues the debate around “filesystem vs. database” for AI agent memory.
The real architecture used by leading teams is increasingly:
Filesystem interface for what agents interact with, database storage for what persists.
Agents often perform well with filesystem-style operations because LLMs already understand actions like ls, cat, grep, read, write, and search from training data. This reduces the need to teach the model many custom APIs or schemas, saving tokens and simplifying interaction.
However, the filesystem is mainly useful as an interface layer, especially for:
- Coding agents
- Document navigation
- DevOps/configuration workflows
- Knowledge bases with clear folder-like structure
It works well because code and documents naturally map to files and directories.
But the article stresses that filesystem interfaces are not universal. They break down when dealing with:
- Structured data queries
- Joins, aggregations, and analytics
- Large-scale document collections
- Serverless/cloud-native environments
- Customer service or business workflows that need direct database access
For these cases, databases, SQL, MongoDB-style query APIs, MCP tools, or direct APIs may be more effective.
The key architectural distinction is:
Layer Meaning
----------------------------------------------------------------------------------------------------------
Interface What the agent sees and uses
Storage Where the data actually lives and persists
----------------------------------------------------------------------------------------------------------
Leading teams often expose files or folders to the agent, but store the data underneath in databases because databases provide:
- Indexing and scale
- Transactions and ACID guarantees
- Multi-agent coordination
- Audit trails and governance
- Vector search, full-text search, and structured queries
The article concludes that production agent memory systems are converging on a decoupled architecture:
Choose the interface based on the agent type, and choose the storage based on system requirements.
So the answer is not “filesystem or database.” It is:
Use both, but in the right layers.
The link to the article: The “files are all you need” debate misses what's actually happening in agent memory architecture - The New Stack
, what's take on this?
1
1 comment
filesystems or databases ??
skool.com/cliefnotes
Jake Van Clief, giving you the Cliff notes on the new AI age.
Leaderboard (30-day)
1
+858
2
+763
3
+653
4
+571
5
+429
Powered by