Skip to main content

AI glossary Models & Architecture

What is GPT (generative pre-trained transformer)?

GPT (Generative Pre-trained Transformer)

← All glossary terms
In short

Generative pre-trained transformer: a type of language model that is pre-trained on large text corpora to predict the next token and then generates text.

Explained

GPT stands for generative pre-trained transformer, an approach to building language models introduced by OpenAI. The model uses the transformer architecture, is first pre-trained on a very large body of text to predict the next token, and can then be adapted to specific tasks. Because it generates text one token at a time, each conditioned on the previous ones, it is called autoregressive. GPT is also the name of OpenAI's model family behind ChatGPT, but the same paradigm underlies many other large language models.

What each word means

  • Generative: the model produces new text rather than only classifying it.
  • Pre-trained: it first learns general language patterns from a very large amount of unlabelled text, before any task-specific adaptation.
  • Transformer: the neural network architecture it uses, based on attention, which lets each word take into account all the others in its context.

How it works

During pre-training, the model reads text and learns to predict the next token, a word or part of a word. Doing this across a huge corpus forces it to capture grammar, facts, styles and patterns of reasoning.

To generate, it takes a prompt, predicts the most likely next token, adds it and repeats. Later models showed that, at sufficient scale, the same model can perform new tasks from a few examples in the prompt, without retraining. Chat models such as ChatGPT add further training with human feedback so that they follow instructions and respond helpfully.

Limits

  • It predicts plausible text, so it can produce fluent statements that are false.
  • Its knowledge is fixed at training time unless it is connected to current data.
  • It works within a limited context window.
  • Cost and latency grow with model size and the length of input and output.

Frequently asked questions

What does GPT stand for?

GPT stands for generative pre-trained transformer. The name describes a language model that generates text, is pre-trained on large amounts of text and uses the transformer architecture.

Is GPT the same as ChatGPT?

No. GPT refers to the model architecture and to OpenAI's family of models. ChatGPT is a chat application built on GPT models, with extra training to follow instructions and hold a conversation.

Is GPT a large language model?

Yes. GPT models are large language models (LLMs). Many other LLMs, from various providers and open-source projects, follow the same decoder-only, autoregressive transformer design.

Who invented GPT?

OpenAI introduced the approach in a 2018 paper by Radford and colleagues, "Improving Language Understanding by Generative Pre-Training". It built on the transformer architecture published by researchers at Google in 2017.

Where it shows up at Thinkia

Further reading

Sources

Updated: 2026-10-21

Want to see how this applies to your organisation?

Talk to an AI Expert