The container scores the same rows differently than the notebook — how do you bisect the cause?
A model is loaded from the same artifact file in a notebook and inside the serving container. For one identical input row the two predictions differ. Both environments were built from the same repository revision, and the artifact checksum matches on both sides.
Constraints: you may add logging, but you cannot retrain and you cannot change the artifact.
row_id=88431
notebook : proba=0.8134 n_features=41 dtypes={float64:38, int64:3}
container: proba=0.2977 n_features=41 dtypes={float64:35, int64:3, object:3}
artifact : sha256 match lib(train)=1.4.2 lib(serve)=1.6.0Name the ordered checks that isolate the cause, and say which line above already narrows it.
Compare the environments, not the model. Log the exact feature vector each side builds for one row and diff it — order, dtype, missing-value fill and skipped preprocessing show up there. If the vectors match, diff the library versions that load the artifact.
- ✗Debugging the model artifact when the divergence lives in the feature vector that reaches it
- ✗Reading a matching feature count as proof that the two feature vectors are identical
- ✗Blaming nondeterminism for a large, reproducible gap that a dtype or version diff explains
- →How does pinning transitive dependencies with a lock file remove this class of bug?
- →Why does an untyped column read as
objectchange a tree model's split decisions?
The dtypes line already narrows it: three columns arrive as object inside the container, so the input diverges, not the weights. Work from the input towards the model.
failed and the categorical encoder received something it never saw in training.
- Log the exact feature vector on both sides for one row and diff it element by element.
- Compare column names and order — a model over an array sees positions, not names.
- Compare dtypes and missing-value fill:
objectinstead offloat64means numeric parsing - Only if the vectors match, diff the library versions (
1.4.2against1.6.0).
feature[12] loan_amount notebook=float64 12500.0 container=object '12 500'
feature[27] region_code notebook=float64 3.0 container=object '3'The cause here is CSV parsing under a different locale and separator, not the model itself.
