PySpark data quality checks that belong in production pipelines

Bad data rarely arrives with a loud failure.

Most production incidents start quietly: a missing key, a duplicated file, a stale partition, a changed enum, a metric that no longer balances. The pipeline still runs. The dashboard still refreshes. The decision is just wrong.

The useful unit is not "25 checks." The useful unit is a validation gate at each pipeline boundary.

Summary

Use PySpark checks as production control points: validate the source contract, stop duplicate loads, test business meaning, monitor drift, and make every failure route to an owner and recovery path.

Validate the contract before transformation

Start before the first expensive transform. If the source contract is broken, downstream joins and aggregations only make the failure harder to explain.

Contract checks should answer simple questions:

• Are required columns present?
• Are required values non-null?
• Are values in the expected type, length, format, and domain?
• Did the schema change since the last accepted run?

A small PySpark gate is enough for many pipelines:

from pyspark.sql import functions as F

invalid_contract = df.filter(
    F.col("customer_id").isNull()
    | F.col("event_date").isNull()
    | ~F.col("status").isin("active", "paused", "closed")
)

If invalid_contract.count() is greater than zero, the pipeline should not silently continue. Either quarantine the records with evidence or fail the run before the data becomes harder to repair.

Catch duplicate and replay failures early

Duplicate problems are common because pipelines retry, files are replayed, source systems resend batches, and orchestration tools run backfills.

Check identity before publish:

duplicate_keys = (
    df.groupBy("order_id")
    .count()
    .filter(F.col("count") > 1)
)

For file-based ingestion, track file name, load id, row count, and hash. A duplicate row and a duplicate file are different failure modes. The first may point to source data. The second may point to ingestion control.

Test business meaning, not only column shape

A clean schema can still produce bad decisions.

Business checks protect meaning:

• quantities should not be negative unless the process allows returns or corrections,
• totals should equal component amounts,
• start dates should not be after end dates,
• percentages should stay between expected limits,
• aggregated values should reconcile with trusted control totals.

These checks need product ownership. A platform team can provide the pattern, but the business rule must come from the people who own the data contract.

Monitor freshness, volume, and drift

Some failures only appear over time.

A table can pass schema checks and still be stale. A partition can be missing. Row counts can drop by 90 percent because an upstream source stopped sending one market. A distribution can shift because a code mapping changed.

Use PySpark for the direct checks:

• max event date against the expected freshness window,
• expected partitions or files for the processing date,
• row count against recent history,
• summary statistics for important measures.

Treat these checks carefully. Not every anomaly should fail production. Some should warn, some should open investigation, and some should block publish. The distinction matters.

Reconcile relationships and reference data

Referential integrity is not only a database feature. Lakehouse pipelines often need to enforce it in the job.

Check that foreign keys resolve, hierarchies are valid, reference data is current, and counts reconcile across systems. These checks are especially important before publishing curated tables, semantic models, or data products used by finance, operations, or leadership.

Compact checklist for pipeline owners

The table is simple by design. A production validation plan should be easy to explain under pressure.

Turn checks into operational controls

Every check needs four fields:

1. Severity: fail, quarantine, warn, or observe.
2. Owner: who decides whether the data is acceptable.
3. Evidence: rejected records, counts, thresholds, and run id.
4. Recovery: rerun, replay, backfill, source fix, or controlled exception.

Without those fields, a check is only a code snippet. With them, it becomes part of the operating model.

The goal is not to make every pipeline strict. The goal is to make the strictness intentional. A landing table may quarantine imperfect rows. A finance metric may block publish. A monitoring table may warn and keep history. A recovery pipeline may allow replay but require hash and audit controls.

Practical pattern

Keep the first version boring:

1. Define the contract for the source and the curated output.
2. Add the checks that catch the most expensive failure modes.
3. Store failed rows and summary evidence.
4. Route failures to an owner.
5. Connect validation failures to recovery and replay behavior.

Do not start with a large framework debate. Start with the failure modes your pipeline already knows how to create.

PySpark gives enough primitives for the first control layer: isNull, isin, cast, rlike, groupBy, count, distinct, summary, datediff, hashes, joins, and aggregations. A dedicated expectations framework can help later, but it should formalize the operating pattern rather than replace it.

Related tools

• Plan the recovery side with the Pipeline Recovery Planner.
• Review Spark execution behavior with Spark Explained.
• Turn quality rules into an explicit contract with the Data Product Contract Builder.

References

• Apache Spark PySpark SQL functions
• Apache Spark SQL null semantics
• Databricks Lakeflow Declarative Pipelines expectations

Disclosure

This article was drafted with AI assistance and reviewed by Rujikorn Ngoensaard before publication.