Building Production ML Pipelines: MLOps Best Practices
A minimal MLOps stack that works: experiment tracking, a model registry with aliases, a CI gate for promotion, canary rollouts and a simple drift check you can run today.

Model v7 went to production on a Friday afternoon. By Monday its conversion numbers were worse than v6, and I wanted to roll back. That is when I learned that v6 existed as a .pkl file on a colleague's laptop and a notebook that no longer ran from top to bottom. We eventually rebuilt it, but the week taught me what MLOps actually is: not a platform purchase, but a set of habits that make "go back to the previous model" a one-minute operation.
MLOps applies DevOps discipline to models, and models add two things code does not have: data that changes under you and quality that decays without any deploy. This post describes the smallest setup I trust in production. For the data side, see Data Engineering Best Practices for ML Projects.
Quick takeaways
- Version everything that defines a model: code commit, data version, configuration and the resulting artifact.
- Use a registry with aliases such as
candidateandchampionso promotion and rollback are metadata changes, not file copies. - Gate promotion in CI: a model must beat the current champion on a held-out set and pass fairness and regression checks.
- Roll out gradually (shadow, then canary) and keep the old model one alias flip away.
- Monitor input drift as well as accuracy, because labels often arrive days or weeks late.
What does a production ML pipeline consist of?
Every pipeline I have maintained follows the same six stages, whatever the tooling:
ingest -> validate -> features -> train -> evaluate -> register
|
champion alias <----- CI gate <-------------+
|
canary -> full rollout -> monitor (data drift, quality)
Each arrow should be an automated, rerunnable step. Validation catches schema and quality problems before training. Feature code must be shared between training and serving, otherwise you create training-serving skew, the most common silent bug in ML systems. Evaluation produces numbers that a machine, not a person, compares against a threshold.
How do I track experiments and register models?
Experiment tracking answers "which run produced this model, and how?". MLflow does this with a few lines, and the tags are where reproducibility actually lives:
import mlflow
import mlflow.sklearn
from mlflow import MlflowClient
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import roc_auc_score
NAME = "churn-model"
def train_and_register(X_train, y_train, X_valid, y_valid, git_sha: str, data_version: str):
mlflow.set_experiment(NAME)
params = {"n_estimators": 300, "learning_rate": 0.05, "max_depth": 3}
with mlflow.start_run() as run:
model = GradientBoostingClassifier(**params, random_state=42).fit(X_train, y_train)
auc = roc_auc_score(y_valid, model.predict_proba(X_valid)[:, 1])
mlflow.log_params(params)
mlflow.log_metric("valid_auc", auc)
mlflow.set_tags({"git_sha": git_sha, "data_version": data_version})
mlflow.sklearn.log_model(model, "model", registered_model_name=NAME)
client = MlflowClient()
version = client.search_model_versions(f"run_id = '{run.info.run_id}'")[0].version
client.set_registered_model_alias(NAME, "candidate", version)
return version
The training function never decides what is in production. It only produces a candidate. Weights & Biases or Neptune cover the same ground if you prefer them, and the pattern is identical: run, metrics, tags, artifact, registry entry.
How do I gate model promotion in CI?
Promotion is a separate, automated decision. This script compares the candidate with the current champion on the same validation metric and refuses to promote a model that is not clearly better:
import sys
from mlflow import MlflowClient
from mlflow.exceptions import MlflowException
NAME, MIN_GAIN = "churn-model", 0.002
client = MlflowClient()
def auc(version) -> float:
return client.get_run(version.run_id).data.metrics["valid_auc"]
candidate = client.get_model_version_by_alias(NAME, "candidate")
try:
champion = client.get_model_version_by_alias(NAME, "champion")
except MlflowException:
champion = None # first model ever: nothing to beat
if champion is not None and auc(candidate) < auc(champion) + MIN_GAIN:
print(f"candidate v{candidate.version} does not beat champion v{champion.version}")
sys.exit(1)
client.set_registered_model_alias(NAME, "champion", candidate.version)
print(f"promoted v{candidate.version}")
One metric is a start, not a finish. Add the checks that protect users: per-segment metrics so an average does not hide a regression, a fairness comparison as described in Responsible AI: Bias Detection and Mitigation Pipelines, and for LLM-based systems a regression suite like the one in Building Evaluation Frameworks for LLM Applications. Rolling back is then set_registered_model_alias(NAME, "champion", <older version>).
How do I deploy a new model safely?
Offline metrics describe yesterday's data. Start in shadow mode, where the new model scores live traffic but its output is only logged. Then send a small share of real requests to it, compare business metrics and latency against the champion, and widen the share step by step. The same canary logic used for application releases applies here; the mechanics are described in Reliable CI/CD Pipeline Automation. Keep the previous model loaded or quickly loadable so rollback does not depend on a rebuild.
How do I detect drift in production?
Ground-truth labels often arrive late, so watch the inputs and outputs in the meantime. The population stability index (PSI) compares a feature's current distribution with the training one:
import numpy as np
def psi(expected, actual, bins: int = 10) -> float:
edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
e = np.histogram(expected, edges)[0] / len(expected)
a = np.histogram(actual, edges)[0] / len(actual)
e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
return float(np.sum((a - e) * np.log(a / e)))
A common rule of thumb: below 0.1 is stable, 0.1 to 0.25 deserves a look, above 0.25 needs investigation. Treat these thresholds as starting points and tune them to your data, and note that features with few distinct values produce duplicate bin edges and need a different treatment. For streaming systems and alerting on unusual values, see Real-Time Anomaly Detection with Streaming ML.
Which tools should I pick?
Tooling matters less than the habits above. Common choices: MLflow, Weights & Biases or Neptune for tracking; Feast, Tecton or Hopsworks for feature stores once several models share features; the MLflow, Vertex AI or SageMaker registries; Airflow, Kubeflow or Prefect for orchestration. Start with tracking plus a registry. Add a feature store only when you feel the pain of duplicated feature code.
What should I do first?
Take your most important model and make rollback boring. Log the git SHA and data version on every training run, register the artifact, and serve whatever the champion alias points to. That single change turns "which model is live and where is the old one?" into a command, and every other practice in this post builds on it.
Uğur Kaval
AI/ML Engineer & Full Stack Developer specializing in building innovative solutions with modern technologies. Passionate about automation, machine learning, and web development.


