Feb '24 • 💡 Help
Tokenizer for open source LLMs?
When you send messages to an LLM, you want to make sure they fit in the context window. So before sending them, being able to calculate their size is useful.
Not every LLM calculates tokens the same way. How do I find out what tokenizer to use for a particular model?
9
8 comments
Tokenizer for open source LLMs?
skool.com/data-alchemy
Your Community to Master the Fundamentals of Working with Data and AI — by Datalumina®
Leaderboard (30-day)
1
+203
2
+75
3
+33
4
+32
5
+28
Powered by