MLOps concepts for data engineers

ML workflows sit on top of your data pipelines, and the data engineering layer determines how well the models can actually perform. What follows covers the concepts that come up when first joining a team running ML in production — not model theory, but the infrastructure abstractions a data engineer needs to reason about.

How ML differs from traditional pipelines

The practical differences that affect your work:

	Traditional pipeline	ML pipeline
Output	Table or file	Model artifact (.pkl, .onnx, etc.)
Failure mode	Loud — exception, empty table	Silent — wrong predictions, no error
Debugging	Read the code	Analyze data and metrics
Versioning	Code	Code + data + model artifact
Testing	Unit/integration tests	Holdout datasets and evaluation metrics

The silent failure mode is the hardest shift. A model with degraded accuracy still returns valid-looking floats. There’s no exception to catch, no row count to check. This is why monitoring matters in a way it doesn’t for deterministic pipelines.

The versioning requirement is also genuinely different. The same code trained on different data produces a different model. Reproducing a model means pinning the code version and the exact training dataset — both need to be logged.

What the data engineer owns: the data that feeds training and inference. Bad features lead to a bad model regardless of the algorithm. Feature freshness, correctness, and consistency between training and production are data engineering problems.

Feature engineering

Features are the columns your model trains on. Raw events — individual transactions, login timestamps, support tickets — can’t be used directly. Models need aggregated, normalized signals:

Transformation	Raw	Feature	Why
Aggregation	individual transactions	total_spend_30d	models need summarized behavior
Recency	last login timestamp	days_since_last_login	time since last activity is predictive
Frequency	login events	login_count_7d	usage intensity
Ratio	tickets ÷ logins	tickets_per_login	relative signals often outperform absolutes
Encoding	plan = "premium"	plan_premium = 1	models require numeric inputs
Lag	current spend	spend_30d_prev_month	trend requires comparison

The critical constraint that doesn’t exist in traditional DE: no feature can use information that wouldn’t be available at prediction time. Features derived from future events are data leakage — the model trains on a cheat sheet and fails in production. See the drift section for details.

Feature Stores

A feature store is a centralized repository for computed features, designed to solve one specific problem: training and serving must use identically computed features.

Without a feature store, teams typically rewrite feature logic in the serving layer, introduce subtle differences, and end up with a model that performs worse in production than in testing. That gap is called training-serving skew.

Offline store

Stores the full history of feature values, one row per entity per timestamp. Built on Delta Lake or Parquet. Query latency in seconds to minutes. Used for model training and batch inference.

The full history is necessary because training requires features as they existed at past points in time — not current values. If a customer churned on February 15th, training needs their features as of February 14th, not today.

Online store

Stores only the latest feature values per entity. Built on a key-value store (DynamoDB, Redis, Cosmos DB). Query latency in milliseconds. Used for real-time inference.

A nightly job (or streaming job) syncs from the offline store to the online store. Both are populated by the same feature computation code, which is the guarantee of consistency.

Point-in-time lookups

The mechanism that prevents data leakage during training. When creating a training dataset, instead of joining features at their current values, you specify a timestamp_lookup_key — the observation date for each label. The feature store retrieves feature values that existed at or before that timestamp.

Without this: training data for a customer who churned on Feb 15 gets their current features (computed today), not their pre-churn features. The model learns from data it would never have at prediction time.

With point-in-time: the same customer gets features as they existed on Feb 14, which is what the model would see in production.

Summary

	Offline store	Online store
Storage	Delta Lake / Parquet	DynamoDB / Redis
Latency	Seconds to minutes	Milliseconds
Data	Full history	Latest values only
Used for	Training, batch inference	Real-time inference

Drift and data quality

Data drift

Input feature distributions change over time. The model was trained on data where average order value was $50; production data now shows $75. The model encounters inputs it wasn’t trained on and produces less reliable predictions.

Detection: compare current input distributions to a baseline using statistical tests. PSI (Population Stability Index) > 0.2 is the conventional threshold for significant drift. Fix: retrain on recent data.

Concept drift

The relationship between features and the target changes. In 2023, low login frequency predicted churn. In 2024, low login frequency predicts mobile app usage. The model’s learned pattern is now wrong even though the feature values themselves are fine.

This is harder to detect than data drift because you need ground truth outcomes, which are often delayed. Detection requires monitoring prediction accuracy over time, not just input distributions. Fix: retrain — and possibly rethink the feature set.

Schema drift

Upstream schema changes break feature computation or silently corrupt feature values. Column renamed, type changed, new category values added. This is the familiar failure mode from traditional data engineering. Detection: schema validation and contract tests on feature pipelines.

Data leakage

Using a feature that wouldn’t be available at prediction time. The model trains on a cheat sheet, achieves excellent evaluation metrics, and fails entirely in production.

Common patterns:

Pattern	Example	Why it’s wrong
Future feature	days_until_churn	Unknown at prediction time
Target leakage	cancellation_reason	Only exists after the event
Temporal leakage	Feb features to predict Jan churn	Wrong time alignment
Post-processing leakage	Normalize before splitting train/test	Test set statistics leak into training

Point-in-time lookups in the feature store prevent the temporal variant. The others require review of feature definitions.

Experiment tracking

Data scientists run many training iterations — different algorithms, hyperparameters, feature subsets, data cuts. Without tracking, there’s no way to answer: which run produced the model currently in production? What data was it trained on? Can we reproduce it?

Experiment tracking logs the context of each training run:

• Parameters — hyperparameters set before training (max_depth=10, learning_rate=0.01)
• Metrics — evaluation results (accuracy=0.88, auc=0.91)
• Artifacts — output files (model file, plots, feature importance charts)
• Tags — metadata (git commit hash, data version, team name)

The hierarchy: an experiment groups related runs (e.g., all iterations on the churn model). Each run is a single training execution.

What the data engineer should ensure is logged alongside model artifacts: the version of the training data (table name + snapshot date or Delta version). MLflow doesn’t log this automatically — it has to be explicit. Without it, the model can’t be reproduced.

Model registry

The registry is distinct from experiment tracking. Tracking is for iteration — many runs, most of them failures. The registry is for production — a curated set of versioned models that have been reviewed and approved.

Versions and aliases

Every model registered gets a version number. Version numbers are immutable pointers to a specific model artifact. But production code should not reference version numbers directly — when you promote a new model, you’d have to update every consumer’s code.

Aliases solve this. An alias is a named pointer to a version: @champion points to the current production model, @challenger points to the candidate being evaluated. Production code loads models:/my-model@champion. When you promote a new model, you move the alias — no code changes in consumers.

Promotion workflow

train → register as new version → set @challenger alias
→ A/B test or shadow mode
→ if metrics improve: move @champion alias to new version
→ previous version automatically loses the alias

The alias move is atomic. Production instantly sees the new model without a deployment.

Inference patterns

Batch inference

A scheduled Spark job scores many records at once and writes predictions to a table. The pattern is familiar — it’s a Spark job that reads from the feature store and writes to a gold-layer table.

Use batch when:

• Predictions don’t need to be instant (scoring all customers nightly is fine)
• Volume is high (scoring millions of records cost-effectively)
• Downstream consumers read from tables (CRM exports, dashboards)

Always use the feature store’s score_batch method (or equivalent) rather than loading features manually. This guarantees the same feature definitions used in training are used at inference time.

Real-time inference

A REST endpoint scores a single record in milliseconds. The application calls the endpoint at request time and uses the prediction immediately.

Use real-time when:

• Predictions must inform a user-facing decision instantly (fraud scoring, recommendation)
• Latency requirements are in the tens of milliseconds

Real-time serving is more expensive operationally: the model runs as a persistent service, and features must come from the online store. Batch is the default; real-time only when the latency requirement justifies it.

Classification metrics

These come up in conversations with data scientists and in monitoring dashboards. The reference model: a binary classifier (churn yes/no) with 1,000 predictions — 90 predicted churn, 910 predicted no churn.

Confusion matrix

	Actual: churn	Actual: no churn
Predicted: churn	TP = 80	FP = 20
Predicted: no churn	FN = 10	TN = 890

Metrics

Accuracy: (TP + TN) / total = 970 / 1000 = 97%. Misleading when classes are imbalanced. If only 5% of customers churn, predicting “no churn” for everyone gives 95% accuracy — and catches zero churners.

Precision: TP / (TP + FP) = 80 / 100 = 80%. Of customers the model flagged as churning, 80% actually churned. High precision = fewer false alarms.

Recall: TP / (TP + FN) = 80 / 90 = 89%. Of customers who actually churned, the model caught 89%. High recall = fewer missed positives.

F1: Harmonic mean of precision and recall = 2 × (0.80 × 0.89) / (0.80 + 0.89) = 0.84. Use when you need a single number and both false positives and false negatives matter.

AUC-ROC: How well the model ranks positives above negatives, independent of threshold. 0.5 = random, 1.0 = perfect. Useful for comparing models and for imbalanced classes.

Precision-recall tradeoff

The threshold controls the tradeoff. Lower the threshold (e.g., predict churn if probability ≥ 0.3): higher recall, lower precision — catch more churners, more false alarms. Raise it (≥ 0.8): higher precision, lower recall — fewer false alarms, more missed churners.

Which to optimize depends on the cost asymmetry. Missing a churner who loses $1000 vs. a wasted $10 retention offer → optimize recall. Annoying loyal customers with aggressive outreach → optimize precision.

Monitoring

Models degrade silently. Accuracy at deployment does not stay constant — data distributions shift, concept relationships change, upstream pipelines introduce bugs. Without monitoring, degradation is only noticed when business impact becomes visible.

What to monitor

Input drift — compare current feature distributions to a training baseline. Statistical distance metrics: PSI (> 0.2 = significant), KL divergence, Kolmogorov-Smirnov. Watch null rates separately — a sudden spike in nulls is usually a pipeline problem, not drift.

Prediction drift — distribution of model outputs. If the fraction of high-risk predictions doubles, either the population changed or something is wrong upstream. Doesn’t require ground truth.

Accuracy vs ground truth — the most meaningful signal, but delayed. For churn, you know whether someone actually churned 30 days after prediction. Join predictions to outcomes and compute accuracy weekly or monthly. A sustained decline triggers retraining.

Monitoring layers

Signal	Latency	Requires ground truth
Input feature distributions	Immediate	No
Prediction distribution	Immediate	No
Accuracy vs actual outcomes	Days to weeks (label delay)	Yes

The first two are leading indicators. Accuracy against outcomes is the real measure, but you have to wait for outcomes to arrive.

When to retrain

No universal rule. Common triggers:

• PSI > 0.2 on a key feature sustained for more than a week
• Prediction distribution shifts more than 20% from baseline
• Accuracy drops more than 5% week-over-week
• Known external event (market shock, product change, seasonal shift)

Retraining restores accuracy if the cause is data drift. Concept drift may require rethinking the feature set entirely.

---

This document was produced by Claude based on David Morel’s instructions and questions, drawing on the Databricks and MLflow documentation. Treat it as a starting point, not a canonical reference — verify against official docs before relying on specific API signatures.