The model works. It scored well on the test set, the demo went fine, and the notebook is on a data scientist's laptop. Six months later it is still there. This is where a large share of machine learning projects stall – not because the model was bad, but because nobody built the path from notebook to production. MLOps is the discipline of building that path, and of keeping a model healthy once it is live.
If DevOps is about shipping code reliably, MLOps is about shipping code, data and models reliably – three things that change independently and can each break the system.
Why machine learning in production is harder than normal software
A conventional service behaves the same until someone changes the code. A machine learning system can degrade with no code change at all:
- The data changes. A new product line, a marketing campaign or a change in customer mix shifts what the model sees.
- The world changes. Fraudsters adapt; buying habits shift seasonally; a competitor changes prices.
- Upstream pipelines change. Someone renames a column or switches a unit from euros to cents, and the model quietly produces nonsense.
There is also a reproducibility problem. To recreate a model you need the exact code, the exact training data snapshot, the hyperparameters and the library versions. Miss any one and "retrain the model from March" becomes guesswork.
The building blocks of MLOps
You don't need every tool on day one, but you need each of these jobs covered somehow.
Versioning: code, data and models
Code goes in git, as usual. Training data needs a versioned snapshot or at least an immutable reference (a dated table, a hashed file). Every trained model should be stored in a model registry with metadata: which code commit, which data version, which parameters, which evaluation scores. When something goes wrong, this is how you answer "what changed?".
Training pipelines instead of notebooks
Notebooks are great for exploration and poor for production. Turn the steps – data extraction, validation, feature engineering, training, evaluation – into a pipeline that runs end to end from a single command or a scheduler. Then retraining is a button, not a project.
Feature consistency
A classic production bug: features are computed one way in training (in SQL over historical data) and another way at prediction time (in application code). The model sees slightly different inputs live, and its accuracy drops. This training-serving skew is why teams share feature code between training and serving, or use a feature store.
Automated testing and validation
Beyond unit tests, ML pipelines need:
- Data validation – schema checks, null rates, value ranges, row counts.
- Model validation – the new model must beat the current one on a fixed evaluation set, and must not get worse on important segments.
- Behavioural tests – known cases with known expected outputs.
Model deployment patterns
How a model reaches users depends on the use case:
| Pattern | How it works | Good for |
|---|---|---|
| Batch scoring | Run the model on a schedule, write predictions to a table | Churn lists, daily forecasts, lead scores |
| Online API | Model behind an HTTP endpoint, scores each request | Fraud checks, pricing, recommendations |
| Streaming | Model consumes events from a queue in near real time | Sensor monitoring, transaction streams |
| Edge / on-device | Model shipped inside an app or device | Offline features, privacy, low latency |
Batch is underrated. If a prediction is needed once a day, a scheduled job writing to a database is far simpler to operate than a low-latency service.
For riskier changes, roll out gradually: run the new model in shadow mode (scoring live traffic without acting on it), then a canary release to a small share of users, then a full rollout. Keep the previous model one switch away.
Model monitoring and data drift
Once live, watch three layers:
- Service health – latency, error rates, throughput. Standard observability.
- Data health – are input distributions shifting compared to training? This is data drift. Population stability index and simple distribution comparisons catch most of it.
- Model quality – are predictions still accurate? This needs ground truth, which often arrives late (you only know whether a loan defaulted months later). Track proxy metrics in the meantime, such as the distribution of predicted scores.
Alerts should go to a channel people actually read, with enough context to act: which feature drifted, by how much, since when. A drift alert that nobody understands gets muted within a week.
A pragmatic MLOps maturity path
You don't need a platform team to start. A sensible progression:
- Level 0 – manual, but documented. Training in a script, not a notebook. Model files stored with their data version and scores. A README on how to retrain.
- Level 1 – automated pipeline. Scheduled or triggered training, automatic evaluation against the current model, a registry, one-click deployment.
- Level 2 – continuous training and monitoring. Drift and quality monitoring trigger retraining; deployments go through shadow and canary stages automatically.
Most companies with a handful of models do perfectly well at level 1. Jumping straight to an elaborate platform before you have three models in production is a common, expensive mistake.
MLOps checklist before go-live
- Training is reproducible from code + data version + config
- Model and metadata stored in a registry
- Same feature code used in training and serving
- Data validation runs before training and before scoring
- New model must beat the current one to be deployed
- Rollback is one step
- Monitoring covers service health, input drift and prediction quality
- Alerts reach a named owner
The bottom line
MLOps is less about any particular tool and more about treating models as living software with data dependencies. Version everything, automate the path to production, and watch the model after launch as carefully as you watched it during training. Do that, and machine learning in production becomes routine rather than heroic.
