Learn
Pretraining: where a model’s knowledge comes from
Every large language model starts with Jargon busterPretraining: The first and biggest stage of training, where a model learns from vast amounts of text by predicting the next token.. It reads a vast amount of text, such as web pages, books and code, and practises one task: predicting the next Jargon busterToken: A chunk of text, often part of a word, that a language model reads and writes one at a time.. After each guess, its internal numbers are nudged towards the right answer.
At enormous scale, this builds grammar, facts, styles and reasoning patterns, plus the text’s errors and stereotypes. Its knowledge stops where the data ends.
It’s so costly that few organisations do it: you’ll almost always start from someone else’s model.
- Read“Put the kettle on and make a cup of…”
- PredictIt guesses “coffee”
- CheckThe real text says “tea”
- AdjustIts numbers shift towards “tea”