⬅️ 𝗣𝗥𝗘𝗩.: 𝟬𝟱 — 𝗗𝗼𝗰𝘂𝗺𝗲𝗻𝘁 𝗜𝗻𝗴𝗲𝘀𝘁𝗶𝗼𝗻 │🏠 𝟬𝟬 𝗦𝗧𝗔𝗥𝗧 𝗛𝗘𝗥𝗘│ 𝗡𝗘𝗫𝗧: 𝟬𝟳 — 𝗖𝗼𝗺𝗶𝗻𝗴 𝗦𝗼𝗼𝗻 ➡️ Note: Press on the links above to navigate to each series or home. At the end of Post 05, my Knowledge System was able to process documents, extract their contents and recognise unchanged files. A great achievement and another step forward. However, there was a bit of a problem. Just because you have the information you need doesn't mean you can find it. Imagine thousands of documents. Somewhere in there is the information you are looking for. But which file does that information sit in and where exactly within that file? It's the needle in the haystack. I certainly didn't want to ask an LLM to read everything each time. Remember, I don’t have tokens burning a hole in pocket. 😅 🔎 𝗦𝗼, 𝗵𝗼𝘄 𝗱𝗼 𝘄𝗲 𝗳𝗶𝗻𝗱 𝘁𝗵𝗲 𝗿𝗶𝗴𝗵𝘁 𝘀𝘁𝘂𝗳𝗳? From Post 05, I already had the extracted content stored in JSON files on disk. Those files contained the information extracted from the original Word, Excel and PDF documents, along with details of where it came from. Technically, those files could be searched directly. However, with potentially thousands of documents in mind, I wanted a better way to organise and search that information. That's where SQLite came in. Essentially a lightweight database that could act as a searchable catalogue. The idea was simple: 📁 Original documents (Word, Excel, PDF) —> ⚙️ Extracted JSON files (stored on disk) —> 🗂️ SQLite catalogue (searchable index) —> 🔎 Matching passages and their source locations. Instead of going through all the extracted files for every search, SQLite could help locate matching information through its search index. Most importantly though, this was still deterministic processing, not AI reasoning.