Prerequisite: TOKENIZATION & TEXT PROCESSING (How AI breaks down text to understand and generate language) WHAT IS TOKENIZATION → Tokenization is the process of splitting text into smaller units called tokens → Tokens can be words, parts of words (subwords), individual characters, or even punctuation → Every AI model starts by analyzing tokens, this is how it reads and processes text → Clean tokenization = better accuracy, faster processing, and fewer mistakes TOKENIZATION TECHNIQUES Different ways to split text, depending on the language and use case WORD TOKENIZATION → Breaks text based on spaces → Example: "AI is transforming industries" → ["AI", "is", "transforming", "industries"] SUBWORD TOKENIZATION → Splits words into smaller meaningful parts like prefixes and suffixes → Handles rare or made-up words better → Example: "unhappiness" → ["un", "happiness"] CHARACTER TOKENIZATION → Treats each character (including spaces and punctuation) as a token → Useful for very detailed processing or languages without clear word boundaries → Example: "AI"→ ["A", "I"] BYTE-PAIR ENCODING (BPE) → Merges the most frequent pairs of bytes in a word → Balances efficiency with vocabulary coverage → Example: "lower" → ["low", "er"] SENTENCE TOKENIZATION → Splits large chunks of text into sentences → Helps AI preserve structure and meaning → Example: "AI is evolving. It impacts many sectors." → ["AI is evolving.", "It impacts many sectors."] TOKENIZATION EXAMPLE How the sentence “n8n is such a powerful tool!!” gets split → Word Tokenization: ["n8n", "is", "such", "a", "powerful", "tool!!"] → Subword Tokenization (BPE): ["n", "8", "n", "is", "such", "a", "powerful", "tool", "!!"] → Character Tokenization: Every letter, space, and punctuation is separate → OpenAI Tokenization: ["n8n", " is", " such", " a", " powerful", " tool", "!!"] IMPACT ON EMBEDDINGS Why token choice affects how text becomes numbers GRANULARITY → Word-level: Fewer tokens, but struggles with rare words