Domain terms were added to the tokenizer and fine-tuning now fails — what is wrong?
A pharma text classifier degraded because the tokenizer shreds drug names into eight or more pieces. Four hundred domain terms were added to the tokenizer, and fine-tuning now crashes inside the embedding lookup.
Constraints:
- the
bert-base-uncasedcheckpoint must be reused, training is fine-tuning only; - the extended tokenizer vocabulary must be kept as is.
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
tok.add_tokens(DRUG_TERMS) # 400 domain terms
model = AutoModelForSequenceClassification.from_pretrained(
"bert-base-uncased", num_labels=12
)
batch = tok(texts, padding=True, truncation=True, return_tensors="pt")
loss = model(**batch, labels=labels).loss # IndexError in the embeddingFind and fix the bug.
Adding tokens grows the tokenizer vocabulary but not the model embedding matrix, so the new ids index past its rows and training crashes. Call resize_token_embeddings after add_tokens; the appended rows start random, so budget continued pretraining or the new terms carry no meaning.
- ✗Calling add_tokens without resizing the model embedding matrix
- ✗Expecting new token rows to be meaningful without further training
- ✗Blaming the optimizer or learning rate for an index-range crash
- →How would you initialise the new rows better than at random?
- →When is it cheaper to leave the terms fragmented instead?
Growing the tokenizer vocabulary and growing the model embedding matrix are two independent steps. add_tokens only touches the vocabulary, so the new ids (30522, 30523, …) fall outside the checkpoint's nn.Embedding and the first forward pass raises IndexError. One line right after adding the tokens fixes it:
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
added = tok.add_tokens(DRUG_TERMS)
model = AutoModelForSequenceClassification.from_pretrained(
"bert-base-uncased", num_labels=12
)
model.resize_token_embeddings(len(tok)) # <- the missing line
# The new rows are random: seed them with the mean of the
# original subword pieces, or they need a long training run.
emb = model.get_input_embeddings().weight.data
for term in DRUG_TERMS:
pieces = tok(term, add_special_tokens=False)["input_ids"]
emb[tok.convert_tokens_to_ids(term)] = emb[pieces].mean(dim=0)The cost of extending the vocabulary is added extra rows of hidden_size parameters each, all untrained. With few domain terms and a small fine-tuning corpus it is cheaper to leave them fragmented — the subword representation is often better than a randomly initialised token.
