Token

A token is the fundamental unit of text that large language models use to read and write language. Before a model does anything with your input, it splits the text into tokens, a process called tokenisation. Only then does it begin to process the meaning of what you have written. Understanding tokens helps explain several of the practical limitations you encounter when using AI tools.

Tokens do not map neatly onto words. The way a model splits text depends on its tokeniser, a component trained to divide language into units that balance vocabulary coverage with processing efficiency. Common short words become a single token. Long words, technical terms, and words from languages that were less well represented in the training data often become two, three, or more tokens. Punctuation marks are typically their own tokens too.

How it works#

When you submit a prompt, the model converts it into a sequence of numeric identifiers, one per token. These numbers are fed into the transformer architecture, which uses attention to weigh the relationships between every token in the sequence. The model then generates a response one token at a time, selecting the most probable next token at each step until the output is complete.

The context window is the total number of tokens the model can hold in working memory at once: your prompt, any background instructions, the conversation history, and the response it is generating. If the total exceeds the window, something has to give. Older parts of the conversation may be dropped, or the model may cut off a long response partway through. Context windows vary widely between models and have been growing steadily as the technology matures.

Tokens also matter for cost. Most commercial AI APIs charge by the number of tokens processed, both in (your prompt) and out (the model's response). Longer, chattier prompts cost more to run than short, precise ones.

Examples#

Token splitting in practice. The word "understanding" is likely to be split into two tokens, something like "under" and "standing", because the tokeniser has seen each sub-string frequently enough to treat it as a unit. By contrast, "the" is so common that it is always a single token. A technical term like "laryngopharyngeal" might become four or five tokens because it is rare and long. You can paste any text into a tokeniser tool to see exactly how it gets divided.

Hitting the context window. You paste a lengthy meeting transcript into an AI chatbot and ask for a summary. The chatbot returns an error or simply stops mid-response. The transcript, plus the instruction you wrote, has exceeded the model's token limit. The solution is to break the document into shorter sections and summarise each in turn, then combine the summaries, or to use a model with a larger context window.

Why it matters#

Tokens are the unit of capacity for AI language tools. Knowing roughly how many tokens your content uses helps you avoid context-window errors, write more efficient prompts, and manage costs if you are using a paid API. It also explains why AI tools sometimes seem to "forget" the start of a long conversation: once earlier messages fall outside the context window, they are simply no longer visible to the model.

For a fuller picture of how tokens flow through a language model, the tutorial How LLMs Work walks through the whole process step by step. You can also explore related concepts such as natural language processing and prompt engineering in the glossary.