Data Engineering Best Practices for ML Projects
Reliable data pipelines for machine learning: schema validation at the boundary, idempotent loads, dataset versioning with DVC, and freshness checks that alert before a retrain goes wrong.

Our churn model lost accuracy after Monday's scheduled retrain. The training code had not changed, the hyperparameters had not changed, and the model looked healthy by every metric except the one that mattered. Finding the cause took half a day: an upstream team had switched event_time from local time to UTC. In Turkey that is a three-hour shift, so every "activity in the last 7 days" feature slid, and a few hundred users landed in the wrong daily partition.
Nothing crashed, and that is the point. In ML projects, bad data rarely raises an exception; it quietly produces a slightly worse model. The practices below are the ones that would have caught that incident on Monday morning instead of Wednesday afternoon. For the model side of the story, see Building Production ML Pipelines.
Quick takeaways
- Validate data where it enters your pipeline, with a schema that fails loudly on unexpected columns, types, ranges and nulls.
- Make every load idempotent: running the same day twice must give the same table, not duplicates.
- Version datasets the way you version code, and record the data version next to every trained model.
- Measure freshness and volume, not just correctness, and alert before the next scheduled run consumes the data.
- Keep orchestration simple: small tasks, explicit dependencies, safe retries.
How do I validate data before training?
Validate at the boundary, in code, on every run. A schema library makes the rules readable and the failure report useful. Here is a version using pandera:
import pandas as pd
import pandera as pa
events_schema = pa.DataFrameSchema(
{
"user_id": pa.Column(str, nullable=False),
# naive timestamps fail this dtype check, which is exactly the point
"event_time": pa.Column(pd.DatetimeTZDtype(tz="UTC"), nullable=False),
"amount": pa.Column(float, pa.Check.in_range(0, 100_000), nullable=True),
"country": pa.Column(str, pa.Check.isin(["TR", "DE", "US", "GB"])),
},
strict=True, # unexpected columns fail the run
unique=["user_id", "event_time"],
)
def load_events(path: str) -> pd.DataFrame:
df = pd.read_parquet(path)
try:
return events_schema.validate(df, lazy=True)
except pa.errors.SchemaErrors as err:
print(err.failure_cases.head(20)) # which column, which check, which rows
raise
lazy=True collects all failures instead of stopping at the first one, so one run shows the full picture. Timestamp precision depends on how the file was written; if yours use microseconds, declare pd.DatetimeTZDtype("us", "UTC") instead. Schema rules cover the checks that never change. For distribution shifts, compare summary statistics (null rate, mean, percentiles) against the previous run and fail when they move more than a threshold you chose on purpose.
How do I make a pipeline idempotent?
Idempotent means a rerun after a failure, a backfill or an accidental double trigger leaves the data exactly as one successful run would. The simplest pattern is to compute a bounded window and upsert by a natural key:
-- Re-running the same day replaces that day's rows instead of duplicating them.
-- Requires: UNIQUE (event_date, user_id) on features_daily
INSERT INTO features_daily (event_date, user_id, sessions, revenue_cents)
SELECT (event_time AT TIME ZONE 'UTC')::date,
user_id,
count(DISTINCT session_id),
coalesce(sum(amount_cents), 0)
FROM raw_events
WHERE event_time >= TIMESTAMPTZ '2026-10-20 00:00:00+00'
AND event_time < TIMESTAMPTZ '2026-10-21 00:00:00+00'
GROUP BY 1, 2
ON CONFLICT (event_date, user_id)
DO UPDATE SET sessions = EXCLUDED.sessions,
revenue_cents = EXCLUDED.revenue_cents;
Parameterize the window instead of hard-coding it, and process incrementally by day or hour so a failure costs one window, not the full history. The upsert does not remove rows that disappeared from the source; if that matters, delete the partition and insert it again inside a single transaction. Give every task a retry policy with backoff and log the window it processed, because "what did this run touch?" is the first question after any incident.
How should I version datasets?
Reproducibility needs three things pinned together: code, data and configuration. Git handles the first and third. For data, DVC keeps small pointer files in Git and the real data in object storage:
dvc init
dvc remote add -d storage s3://my-ml-bucket/dvc
dvc add data/train.parquet
git add data/train.parquet.dvc data/.gitignore
git commit -m "Training snapshot 2026-10-20"
dvc push
Checking out an old commit and running dvc pull restores the exact training data it used. If your data already lives in a lakehouse, Delta Lake table versions or LakeFS branches give you the same guarantee without separate files. Whichever tool you use, log the data version as a tag on every training run so a model can always be traced back to its inputs.
Data lake or warehouse, and which file format?
Keep raw data in a lake as an immutable record, and keep cleaned, modeled data in a warehouse or lakehouse tables where schemas are enforced. A raw layer you never edit is what lets you fix a transformation bug and rebuild history. For analytics and training sets, Parquet is the default: columnar, compressed and typed. Delta adds ACID transactions and time travel on top of Parquet. JSON is fine for ingestion and a poor format for repeated reads.
How do I monitor freshness and quality?
Correctness checks do not notice a table that simply stopped updating. Track at least four numbers per table: freshness, row count versus the usual range, null rate on key columns, and duplicate rate on the primary key. Freshness is the cheapest and catches the most:
SELECT 'raw_events' AS table_name,
now() - max(loaded_at) AS lag,
(now() - max(loaded_at)) > interval '2 hours' AS is_stale
FROM raw_events;
Run it on a schedule and alert on is_stale, ideally before the training job reads the table. Row-count and null-rate checks follow the same pattern. If you want to flag unusual values automatically, see Real-Time Anomaly Detection.
How should I orchestrate the pipeline?
Use whichever orchestrator your team can operate: Airflow, Prefect or Dagster all work. Keep each task small enough to rerun alone, make dependencies explicit, and separate ingestion, transformation and serving so a failure in one does not hide in another. Scheduling and retry trade-offs are discussed in Automating Data Pipelines Without Silently Corrupting Your Data, and a reporting-oriented example is in Scheduled Reports That Survive Retries.
What should I do this week?
Pick the one table your best model depends on most. Write a schema for it with strict=True, run it in the pipeline, and let it fail a build once. Add a freshness query with an alert. Those two changes take an afternoon and cover most of the incidents I have seen where bad data reached a model without anyone noticing.
Uğur Kaval
AI/ML Engineer & Full Stack Developer specializing in building innovative solutions with modern technologies. Passionate about automation, machine learning, and web development.


