NLP & Transformers

Cosine compares only the angle between two vectors and ignores magnitude, so it captures the direction of meaning. Euclidean distance grows with the norm, and the norm tracks text length and word frequency, so two texts on one topic can look far apart merely because one is longer.

It adds position information to each token's embedding so the same token at different positions differs. It is needed because self-attention is permutation-invariant — order-blind by itself. Variants: original sinusoidal, learned embeddings, and RoPE (rotary positional embeddings).

Each token attends to every other token to mix context. A token is projected into Query, Key, and Value vectors. The attention weight between two tokens is softmax(Q·Kᵀ / √dₖ) — a normalized Query–Key similarity. Each token's output is the weighted sum of all Value vectors.

An encoder block has two sub-layers: self-attention and a feed-forward network. A decoder block adds a THIRD — encoder-decoder cross-attention over the encoder outputs — and its self-attention is causally masked, unlike the encoder's bidirectional one. The decoder ends in Linear + Softmax.

A word-level vocabulary is finite, so unseen words collapse to a single [UNK] token and lose all meaning. Subword schemes keep frequent fragments instead and split any unknown word into known pieces, so OOV vanishes. BPE learns its merges by greedily joining the most frequent adjacent pair.

Sparse/count-based: Bag-of-Words and TF-IDF. Dense static: word2vec and GloVe (one vector per word), FastText (word2vec plus subword n-grams, handling rare/OOV words). Contextual: ELMo and BERT, where a word's vector depends on its context, so the same word varies.

Stemming chops affixes by rules and may emit non-words, but it is cheap; lemmatization uses a dictionary and part of speech to return a real base form at a higher cost. Both mainly help sparse count models by collapsing forms; with subword contextual models they add little.

Bag-of-words counts terms inside one document. IDF scales each count by a factor that falls as the number of documents containing the term rises, so corpus-wide terms are damped. A word repeated often in one document still scores high; only spread across documents is penalised.

CBOW predicts the centre word from its averaged context, which trains fast and favours frequent words. Skip-gram reverses it and predicts each context word from the centre one, creating one pair per neighbour, so rare words get more updates and small corpora fare better.

Training-free: pool all token embeddings (mean/max), or take the [CLS] embedding — though raw [CLS] is a weak similarity vector. For a strong text embedding, fine-tune SentenceBERT-style: add pooling plus a feed-forward head and train with metric learning (siamese/triplet loss).

A triangular causal mask is applied: before the softmax, scores for future positions (j > i) are set to −∞, so position i attends only to itself and earlier tokens. This enforces autoregression — the model can't peek at the token it must predict. An encoder (e.g. BERT) uses no such mask.

A static model such as word2vec stores one vector per word type, so every sense of a word is averaged into a single point that fits none of them well. Contextual models compute the vector from the surrounding sentence, so each occurrence gets its own representation.

Yes. BERT adds positional embeddings, so the same token at a different position starts from a different input vector. Also, self-attention mixes each token with its context, which reordering changes. Both effects make the contextual embeddings differ after a permutation.

Yes, for two reasons. First, WordPiece tokenization splits the rare word into frequent subword tokens, so there is no out-of-vocabulary failure. Second, BERT is contextual, so surrounding words inform it. To get a single word vector, mean- or max-pool its subword embeddings.

One word becomes several pieces, so the word tag under the BIO scheme goes on its first piece; the remaining pieces are marked as continuations and masked out of the loss. At inference you map predictions back to words by reading the tag of each word first piece.

Each distinct n-gram becomes its own column, so the vocabulary grows far faster than the corpus and most columns fire in a few documents. Prune with a minimum document frequency, cap the vocabulary size, or hash into a fixed width; the matrix is sparse, so cost tracks nonzeros.

Frame the answer as two numbers — the start and end token positions. Feed the question and text together and add two output heads over the tokens: one predicting P(start), one P(end), each a feed-forward layer plus softmax. The span runs from the argmax start to the argmax end (SQuAD-style).

Adding tokens grows the tokenizer vocabulary but not the model embedding matrix, so the new ids index past its rows and training crashes. Call resize_token_embeddings after add_tokens; the appended rows start random, so budget continued pretraining or the new terms carry no meaning.