NLP & Transformers
Modern NLP rests on Transformers, and those rest on a few precise ideas. Embeddings turn words into vectors, self-attention mixes context without recurrence, positional encoding restores the order attention cannot see by itself, and the encoder/decoder and BERT are concrete ways to assemble this into a working model.
The topic is deep, and the traps here are mechanical. Self-attention is permutation-invariant — without positional encoding it is blind to order. The causal mask in the decoder sets −∞ before softmax, it does not zero the weights afterward. BERT is an encoder, it does not generate text, it encodes it. word2vec vectors are static, while BERT is contextual. The full map is in the layers below.
Topic map
- Word embeddings — from sparse BoW/TF-IDF to dense word2vec/GloVe/FastText and contextual ELMo/BERT.
- Self-attention — the Query/Key/Value mechanism, scaled softmax, and the decoder's causal mask.
- Positional encoding — why attention is order-blind and how sinusoidal, learned embeddings, and RoPE fix it.
- Transformer — the encoder and decoder blocks, their sub-layers, and the cross-attention between them.
- BERT — an encoder model for embeddings, rare words, whole texts, and extractive question answering.
Common traps
| Mistake | Consequence |
|---|---|
| Thinking self-attention is order-sensitive without encoding | Without positional encoding the model is blind to word order |
| Forgetting the √dₖ scaling inside the softmax | Dot products grow, softmax saturates, the gradient vanishes |
| Masking past instead of future positions in the decoder | Autoregression breaks — the model peeks at the answer |
| Applying the causal mask after softmax | The weights no longer sum to one; set −∞ before softmax |
| Thinking word2vec vectors are contextual | They are static — one word, one vector, regardless of context |
| Thinking BERT is a text generator | BERT is an encoder, it encodes, it does not produce a sequence |
Interview relevance
The NLP part selects those who understand the Transformer as a mechanism. A candidate who explains why √dₖ is needed and why the causal mask is −∞ before softmax sounds an order of magnitude stronger than "well, attention looks at all the words".
Typical checks:
- How static embeddings (word2vec) differ from contextual ones (BERT).
- How self-attention works — Query/Key/Value, scaled softmax, output as a weighted sum of Values.
- Why positional encoding is needed and what variants exist.
- How a decoder block differs from an encoder and what BERT does with a rare word or a whole text.
Common wrong answer: "self-attention accounts for word order by itself". In fact it is permutation-invariant: without positional encoding, permuting the words would not change who attends to whom.