Training Data

Training data is the raw material from which an AI model learns. Before a model can make a single useful prediction or generate a coherent sentence, it must be exposed to thousands, millions, or even billions of examples. The model studies those examples, finds patterns, and builds up an internal representation of the world, at least as reflected in that data.

For a large language model, training data is primarily text: books, websites, news articles, and academic papers. For an image recognition system, it is labelled photographs. For a fraud detection model, it is historical transaction records marked as legitimate or fraudulent. The type of data must match the task the model is built to handle.

How it works#

During training, the model processes each example in the training dataset and adjusts its internal parameters, the millions or billions of numerical values that define how it behaves. For supervised learning tasks, each example includes a label, a correct answer the model should learn to produce. The model compares its output against that label and makes small adjustments to reduce the error. This process is repeated across the entire dataset, often many times over, until the model reaches an acceptable level of accuracy.

The training dataset is typically split into three parts. The training set is what the model actually learns from. The validation set is used to check performance during training and to adjust settings. The test set is held back entirely and used only at the end to measure how well the model performs on data it has genuinely never seen.

Data preparation, collecting, cleaning, deduplicating, and labelling examples, commonly takes up more time than building or training the model itself. Raw data is messy: web-scraped text contains spam and errors, images are mislabelled, records are duplicated. Each problem left unaddressed teaches the model something wrong.

Examples#

Skin condition diagnosis. A model designed to help clinicians identify skin conditions must be trained on images covering a wide range of skin tones. Early medical imaging datasets were heavily skewed towards lighter skin tones, meaning models trained on them performed significantly worse for patients with darker skin. Researchers and health organisations have invested considerably in building more representative datasets to correct this disparity.

Language translation. Translation models are trained on pairs of sentences, the same sentence in two different languages. The quality of these pairs matters greatly. If the training data contains mistranslations or outdated phrasing, the model learns those errors and reproduces them. Models trained on more recent, carefully reviewed translation pairs produce noticeably more natural output.

Why it matters#

Understanding training data helps you evaluate any AI tool more critically. When a vendor claims their model is accurate, the right follow-up questions are: accurate on what data, collected when, from which sources, and labelled by whom? A model that performs brilliantly on its own test set may still fail in your specific context if your data looks different from what it was trained on.

It also explains why fine-tuning has become such a popular technique. Rather than training a model from scratch on a general dataset, businesses can take an existing model and fine-tune it on their own domain-specific data, giving them a more relevant tool without the enormous cost of starting from scratch.

To understand how training data feeds into the broader process of building language models, see the tutorial How LLMs Work, or explore related topics through our /glossary.