The Unfiltered Guide to LLM Tracking
Today I want to give you the unfiltered reality of what's going on in the market. Why there are so many LLM rank trackers, some are cheap, some are expensive, and some are going to the extreme level now. Let me break it down. THE TWO TYPES OF LLM TRACKING Whenever someone says they're going to "track LLMs," there are really only two ways anyone in the world can do it: 1. API Calls: with web search enabled, or without web search 2. UI-Based (Frontend): which has two versions: logged-in session and non-logged-in session That's it. Whatever tools you hear about on the internet, they're just selecting one of these and building their infrastructure on top of it. And the cheaper tools? They're almost always making API calls. So you might ask why are API calls different from the UI-based versions? HOW LLMs ACTUALLY WORK (CORE PRINCIPLES) To understand the difference, we need to understand how LLMs work at the core. Let's say you're building an LLM from scratch. What you'd do is find a massive amount of data, push it into your model, and process it. Simplest way I can explain it, imagine you turned off your internet and hit the search button on your Windows file finder. You're just trying to find a file on your own machine. That's essentially what an LLM does with its training data. Offline data, whenever a non-web API call is made, this offline/training data is what gets triggered. Web-enabled data, whenever a web-enabled API call is made, the LLM has the capability to go search the live internet. "SEARCH THE WEB" BUT WHERE ARE THEY SEARCHING? You and I have Google or Bing to search. But for LLMs, where are they actually looking? Think of it like this imagine the entire internet is your PC, and all that data has been converted into an index. Like the index pages of a book. You look at the index, it says "if you want to learn about X, go to page 247." That's how it extends its knowledge. The LLM starts with its initial training data. Let's say it has 10 documents. Those 10 documents contain links, external references, bits of information. From those, it pulls more. Now think about the compounding effect 10 documents with 10 links each becomes 100, then 1,000, then millions, then billions of connections. An entire network gets created.