MLOps & Production ML
Training-serving skew, data vs concept drift, feature stores, retraining triggers, reproducibility, model registries and production monitoring.
11 questions
JuniorTheoryVery commonWhat is training-serving skew, and how do the two feature pipelines diverge?
What is training-serving skew, and how do the two feature pipelines diverge?
Training-serving skew is a mismatch in how a feature is computed offline for training versus online at serving. It appears when the two pipelines are separate code paths that drift apart, or when serving uses different windows, defaults or units.
Common mistakes
- ✗Equating skew with distribution drift — skew is a computation mismatch, not a shift in the data
- ✗Assuming a shared library removes skew while training and serving still read different data snapshots
- ✗Retraining to fix skew, which re-bakes the same wrong feature definition into the new model
Follow-up questions
- →How does logging the exact features seen at serving time let you measure the skew directly?
- →Why does a nightly batch job that backfills a feature often introduce skew for online traffic?
JuniorDesignCommonYour team ships two or three model versions a week. Today the artifacts sit in an object-storage bucket, the metrics live in a spreadsheet, and the deployed version is whatever the last engineer copied onto the serving host. Last month a rollback restored the wrong checkpoint and nobody noticed for two days. Explain what a model registry is, what metadata each registered version must carry so that any engineer can answer which model is live and how it was produced, and what gates a version has to pass before it is promoted from staging to production.
Your team ships two or three model versions a week. Today the artifacts sit in an object-storage bucket, the metrics live in a spreadsheet, and the deployed version is whatever the last engineer copied onto the serving host. Last month a rollback restored the wrong checkpoint and nobody noticed for two days. Explain what a model registry is, what metadata each registered version must carry so that any engineer can answer which model is live and how it was produced, and what gates a version has to pass before it is promoted from staging to production.
A model registry is the catalogue of model versions and their stage. A version records the artifact, data and code revision, hyperparameters, metrics, environment and owner. Promotion is gated on metric thresholds, a sign-off and a predecessor kept for rollback.
Common mistakes
- ✗Treating the registry as storage for weights instead of the source of truth for stage and lineage
- ✗Registering metrics without the data and code revision that produced them, making runs unexplainable
- ✗Promoting on a metric alone, with no recorded predecessor to roll back to
Follow-up questions
- →How does the registry stage transition drive the deployment pipeline instead of a manual copy?
- →What breaks if two versions claim the production stage for the same model name?
JuniorTheoryCommonHow is a trained model actually served — in-process artifact, model server, or portable runtime?
How is a trained model actually served — in-process artifact, model server, or portable runtime?
The artifact can be loaded in-process by the application, hosted behind a dedicated model server with its own endpoint, or exported to a portable format such as ONNX. In-process is simplest but pins the app to the exact training library versions.
Common mistakes
- ✗Assuming a serialised artifact is self-contained and portable across library versions
- ✗Treating a model server purely as infrastructure overhead rather than as version decoupling
- ✗Forgetting that preprocessing code, not just weights, must travel with the model
Follow-up questions
- →When does the open exchange format
ONNXstop being an option for your model? - →What does an inference server give you that an in-process load cannot, besides isolation?
MiddleTheoryCommonData drift versus concept drift — what changes in each, and which one forces a retrain?
Data drift versus concept drift — what changes in each, and which one forces a retrain?
Data drift is a change in P(x), the input distribution, while the input-to-label rule holds. Concept drift is a change in P(y|x) — the same inputs now imply different labels. Concept drift forces a retrain; data drift matters only when a metric moves.
Common mistakes
- ✗Swapping the definitions — putting P(y|x) under data drift and P(x) under concept drift
- ✗Retraining on every detected input shift even when the production metric is unchanged
- ✗Assuming drift and training-serving skew are the same failure and share one fix
Follow-up questions
- →Which detector fits which drift — the population stability index
PSIon inputs, or a metric decay curve? - →Why can label drift be visible in the target rate long before any input feature moves?
MiddleTheoryCommonWhat does a feature store actually solve, and when is standing one up overkill?
What does a feature store actually solve, and when is standing one up overkill?
A feature store holds one definition per feature and serves it from an offline store for training and an online store for inference, so both paths share one transformation. It adds reuse and point-in-time joins; for a single batch-only model it is overkill.
Common mistakes
- ✗Reducing the feature store to an online cache and ignoring the shared feature definition
- ✗Computing training features with a naive join that leaks values from after the event time
- ✗Adopting a feature store for one model, paying the operational cost with no reuse to amortise it
Follow-up questions
- →What exactly does a point-in-time correct join prevent when you build a training table?
- →How do the offline and online stores stay consistent when a feature definition changes?
SeniorDesignCommonA credit model goes live tomorrow, but a loan is only confirmed good or bad after thirty days, so the first honest accuracy number arrives a month from now. Leadership still wants to know on day one whether the deployment is healthy. Design the monitoring you can actually run without labels: which distributions and operational signals you track, which of them you page an on-call engineer for versus merely chart, what proxy or business metric stands in for accuracy meanwhile, and how you avoid drowning the team in alerts from harmless daily variation.
A credit model goes live tomorrow, but a loan is only confirmed good or bad after thirty days, so the first honest accuracy number arrives a month from now. Leadership still wants to know on day one whether the deployment is healthy. Design the monitoring you can actually run without labels: which distributions and operational signals you track, which of them you page an on-call engineer for versus merely chart, what proxy or business metric stands in for accuracy meanwhile, and how you avoid drowning the team in alerts from harmless daily variation.
Without labels you monitor inputs and outputs — feature distributions, the score distribution and approval rate, plus null rates, latency and errors. Page on breakage and abrupt score shifts; only chart slow drift. Manual-review outcomes proxy accuracy until labels land.
Common mistakes
- ✗Concluding that nothing is measurable until labels arrive, leaving the launch unmonitored
- ✗Alerting on every feature-level shift instead of on breakage and abrupt output changes
- ✗Confusing a proxy metric with accuracy and declaring success on it once labels do arrive
Follow-up questions
- →Why is a sudden shift in the approval rate a stronger day-one signal than any single feature moving?
- →How does an anchored reference window keep seasonal variation from firing your drift alerts?
SeniorDesignCommonA fraud model has been live for a year on a weekly manual retrain that an analyst kicks off by hand. Fraud patterns shift in bursts, so some weeks the retrain is wasted compute and other weeks the model is three weeks behind an attack. Design the retraining policy: choose between scheduled, triggered and continuous retraining, state exactly what fires a trigger and on which measurement window, define who or what approves the candidate before it reaches production, and describe the rollback path when the fresh model turns out worse than the one it replaced.
A fraud model has been live for a year on a weekly manual retrain that an analyst kicks off by hand. Fraud patterns shift in bursts, so some weeks the retrain is wasted compute and other weeks the model is three weeks behind an attack. Design the retraining policy: choose between scheduled, triggered and continuous retraining, state exactly what fires a trigger and on which measurement window, define who or what approves the candidate before it reaches production, and describe the rollback path when the fresh model turns out worse than the one it replaced.
Keep the schedule as a floor and add triggers — metric decay, drift past a threshold, or new label volume — each on a window long enough to ignore noise. Promotion compares the candidate against the incumbent on a recent held-out slice, with the incumbent kept for rollback.
Common mistakes
- ✗Treating retraining cadence as a calendar question and ignoring what actually changed in the data
- ✗Firing triggers off a window so short that ordinary noise causes constant retrains
- ✗Promoting a candidate without comparing it to the incumbent on the same recent evaluation slice
Follow-up questions
- →How would you keep a burst of fraud from poisoning the labels the retrain learns from?
- →What makes rollback a configuration change rather than a rebuild in your design?
MiddleDebuggingOccasionalThe container scores the same rows differently than the notebook — how do you bisect the cause?
The container scores the same rows differently than the notebook — how do you bisect the cause?
Compare the environments, not the model. Log the exact feature vector each side builds for one row and diff it — order, dtype, missing-value fill and skipped preprocessing show up there. If the vectors match, diff the library versions that load the artifact.
Open full question →Common mistakes
- ✗Debugging the model artifact when the divergence lives in the feature vector that reaches it
- ✗Reading a matching feature count as proof that the two feature vectors are identical
- ✗Blaming nondeterminism for a large, reproducible gap that a dtype or version diff explains
Follow-up questions
- →How does pinning transitive dependencies with a lock file remove this class of bug?
- →Why does an untyped column read as
objectchange a tree model's split decisions?
MiddlePerformanceOccasionalBatch, online and streaming inference — how do latency, volume and cost decide between them?
Batch, online and streaming inference — how do latency, volume and cost decide between them?
Batch scoring precomputes predictions on a schedule — cheapest per record, but stale. Online inference answers each request under a p99 budget and pays for idle capacity. Streaming scores events on arrival, giving seconds-level freshness at high throughput.
Common mistakes
- ✗Choosing online inference by default and paying for idle capacity the traffic never needs
- ✗Ignoring that batch predictions are stale by the length of the scheduling interval
- ✗Sizing an online service on mean latency instead of the p99 the product actually promises
Follow-up questions
- →Which mode fits a daily churn score for ten million users, and why on cost grounds?
- →How does request batching on an online service trade p99 latency against throughput?
MiddleTheoryOccasionalShadow, canary and A/B rollout of a new model — what does each catch, and what does each cost?
Shadow, canary and A/B rollout of a new model — what does each catch, and what does each cost?
Shadow mode mirrors traffic and discards the output, catching crashes and latency at no user risk but proving nothing about impact. A canary routes a small real share and catches regressions early. An A/B test measures business impact but costs traffic and time.
Common mistakes
- ✗Expecting shadow mode to answer a business question it structurally cannot answer
- ✗Calling a canary an experiment and reading its uplift as a statistically valid result
- ✗Rolling out without keeping the previous version addressable, so rollback needs a rebuild
Follow-up questions
- →What signals would you watch on a canary to trigger an automatic rollback?
- →Why does shadow mode still surface training-serving skew even though its output is discarded?
SeniorDesignOccasionalA regulator asks your team to reproduce a scoring model that was trained six months ago and to demonstrate that the rebuilt artifact behaves identically. The training notebook still exists, the raw tables have since been updated in place, and the person who ran it has left. Design a training pipeline that would make this request routine: state exactly what has to be versioned and how each item is addressed, how you guarantee the training table is rebuilt as of the original point in time, and what artefact you would hand the regulator as evidence that the rerun matches the original.
A regulator asks your team to reproduce a scoring model that was trained six months ago and to demonstrate that the rebuilt artifact behaves identically. The training notebook still exists, the raw tables have since been updated in place, and the person who ran it has left. Design a training pipeline that would make this request routine: state exactly what has to be versioned and how each item is addressed, how you guarantee the training table is rebuilt as of the original point in time, and what artefact you would hand the regulator as evidence that the rerun matches the original.
Version five things immutably — code revision, data snapshot, feature definitions, config with seeds, and environment image. Rebuild the training table by point-in-time queries over append-only data. Evidence is a run manifest tying them to the artifact hash and metrics.
Common mistakes
- ✗Versioning code but not the data snapshot, so a rerun silently trains on different rows
- ✗Filtering current tables by date and calling it a point-in-time rebuild after in-place updates
- ✗Omitting the environment image and seeds, leaving the rerun close but not identical
Follow-up questions
- →Which sources of nondeterminism survive a fixed seed, and how do you bound them?
- →How do shared feature definitions make an old run explainable as well as repeatable?
