Tokenization

Definition

The process of breaking down text into smaller units called tokens that AI models can process.

In-Depth Explanation

Tokens can be words, subwords, or characters depending on the tokenizer. Most LLMs use subword tokenization (like BPE) which balances vocabulary size with the ability to handle unknown words. Token count affects both cost (API pricing) and context window usage.

Real-World Example

The word "unhappiness" might be tokenized as ["un", "happiness"] or ["un", "happi", "ness"].

4 views0 found helpful

Related Terms

Context Window

The maximum amount of text (measured in tokens) that a language model can process in a single interaction.

Large Language Model(LLM)

A type of AI model trained on massive amounts of text data to understand, generate, and manipulate human language.

Natural Language Processing(NLP)

A field of AI focused on enabling computers to understand, interpret, and generate human language.