Skip to main content

AI glossary Models & Architecture

What is BERT in NLP?

BERT (Bidirectional Encoder Representations from Transformers)

← All glossary terms
In short

A Transformer language model from Google, pre-trained to read context in both directions; widely used for search, classification and embeddings.

Explained

BERT (Bidirectional Encoder Representations from Transformers) is a language model introduced by Google researchers in 2018. It is pre-trained on large amounts of unlabelled text by hiding some words and learning to predict them from the words on both sides, which gives it a strong grasp of context. A pre-trained BERT can then be fine-tuned with one extra output layer for tasks such as question answering, classification or entity recognition. It reads and represents text rather than generating it, and BERT-style encoders remain the basis of many semantic search and embedding systems.

How BERT works

BERT uses only the encoder part of the Transformer architecture. During pre-training, some of the words in each sentence are masked and the model learns to fill them in using the words before and after. The original paper describes this masked language modelling objective as inspired by the Cloze task, the classic fill-in-the-blank exercise.

The result is a model that produces a contextual representation for every word: the same word gets a different vector depending on the sentence around it. Those representations are the starting point for downstream tasks.

BERT vs GPT-style LLMs

  • BERT is an encoder: it reads text and produces representations. GPT-style models are decoders: they generate text one token at a time.
  • BERT looks at context in both directions at once; generative models predict the next token from what came before.
  • BERT-style models are small and fast enough for high-volume classification and retrieval; generative LLMs are more flexible but heavier.
  • In many RAG systems both appear: an encoder for embeddings and re-ranking, a generative model for the answer.

What it is still used for

Encoders descended from BERT are still common where speed, cost and predictability matter: semantic search and embeddings, re-ranking retrieved passages, classifying tickets or emails, detecting entities in documents and filtering content. Smaller distilled versions make them practical to run on modest hardware.

Frequently asked questions

What does BERT stand for?

BERT stands for Bidirectional Encoder Representations from Transformers. Bidirectional refers to using context on both sides of each word; encoder refers to the part of the Transformer architecture it uses.

Is BERT a large language model?

It is a pre-trained language model and an important ancestor of today's LLMs, but it is much smaller than current generative models and does not generate free text. It is usually described as an encoder model rather than an LLM in the modern sense.

Is BERT still used today?

Yes. BERT and its descendants are widely used for embeddings, semantic search, re-ranking and classification, because they are efficient and well understood. For open-ended generation, teams use generative LLMs instead.

What is the difference between BERT and ChatGPT?

BERT reads text and turns it into representations that other components use for search or classification. ChatGPT is built on generative models that write answers. They solve different parts of a problem and are often used together.

How is BERT used in search?

BERT-style models turn queries and documents into vectors so that results can be matched by meaning, not just keywords. They are also used to re-rank a short list of results by how well each one answers the query.

Where it shows up at Thinkia

Further reading

Sources

Updated: 2026-10-21

Want to see how this applies to your organisation?

Talk to an AI Expert