Launching soon — get early access:
Back to Glossary
Language Models

Context Length

Definition

The maximum number of tokens a language model can process in a single prompt and response combined.

In-Depth Explanation

Context length determines how much text a model can "remember" during a conversation. Early models had 2K-4K tokens, while modern models support 128K (GPT-4 Turbo) or even 1M tokens (Gemini). Longer contexts enable processing entire books or codebases but require more memory and compute. Context length is a key differentiator between models.

Real-World Example

Claude 3 with 200K context can analyze an entire novel in one prompt, while GPT-3 with 4K context could only handle a few pages.

4 views0 found helpful

Related Terms

Attention Mechanism

A neural network component that allows models to focus on relevant parts of the input when producing outputs.

Context Window

The maximum amount of text (measured in tokens) that a language model can process in a single interaction.

Tokenization

The process of breaking down text into smaller units called tokens that AI models can process.