Transformer

A transformer is a neural network architecture introduced in 2017 that changed the course of AI research. Unlike older models that processed text one word at a time, a transformer reads an entire sequence at once, which lets it understand the relationships between words no matter how far apart they appear in a sentence. This parallel approach is both faster and more accurate, particularly for the kind of nuanced language tasks that large language models need to handle.

The transformer's success comes down to a single core idea: instead of treating every word equally, it learns to pay more attention to the words that matter most in a given context. This mechanism, called self-attention, is what allows the model to disambiguate meaning, follow long chains of reasoning, and translate between languages with impressive accuracy.

How it works#

A transformer is made of stacked layers, each containing two key components: an attention block and a feed-forward block.

The attention block scans all the tokens in the input and assigns each pair a relevance score. Tokens that are strongly related, such as a pronoun and the noun it refers to, receive high scores. The model then uses these scores to weight how much influence each token has on the others. This happens across multiple parallel "heads" simultaneously, allowing the model to track several types of relationship at once.

The feed-forward block processes the output of the attention step and adds further transformations, helping the model learn complex patterns. After many such layers, the final output is a rich numerical representation of each token that captures its meaning in context.

During training, millions of examples are fed through the network and the parameters are adjusted until the outputs become reliably accurate. At inference time, the process runs in reverse: the model uses its learned parameters to generate the most appropriate next token, one at a time.

Examples#

Resolving ambiguous meaning. Consider the sentence "The bank by the river was covered in wildflowers." A transformer uses attention to notice that "river" is nearby and highly relevant to "bank". It assigns a high relevance score to that pairing and correctly interprets "bank" as a riverbank rather than a financial institution. An older, word-by-word model would have no mechanism to look back at "river" while processing "bank".

Long-document summarisation. When asked to summarise a 10-page report, a transformer can hold the whole document in view at once, rather than reading it in chunks and losing earlier context. This is why modern AI tools can produce coherent, accurate summaries of lengthy material.

Why it matters#

The transformer architecture is the reason AI language tools improved so dramatically after 2017. Every major model you are likely to use, from ChatGPT to Claude to Google's Gemini, is built on this design. Understanding it helps explain why these tools are good at some things (holding context across a long conversation) and less reliable at others (precise arithmetic, which requires a different kind of reasoning).

For anyone building with AI or simply trying to use it well, knowing that transformers exist and why they work gives useful intuition about what to expect. If you want to go deeper, the tutorial How LLMs Work walks through the transformer without the jargon, and the broader AI vs ML vs Deep Learning tutorial puts it in context alongside other approaches.