Data Quality Pipeline Lab

I created Data Quality Pipeline Lab as a local PySpark reference pipeline on fully synthetic B2B wholesale fixtures. The goal is to make contracts, quarantine decisions, reconciliation and CI evidence easy to inspect without exposing private client systems.

Data-quality controls are hard to discuss when schemas are inferred, rejected rows are opaque and source-to-target differences are not reconciled. I wanted a small public example where every accepted or quarantined row can be accounted for.

• Five explicit source contracts with typed schemas and primary keys.
• Referential, domain, temporal and arithmetic quality rules with stable rule IDs.
• Row-level quarantine records for rejected inputs.
• Idempotent runs with overwrite-safe Parquet outputs.
• Source-to-target row accounting and monetary reconciliation.
• JSON evidence artefacts, automated tests and GitHub Actions CI.

• The data, business rules and architecture are entirely synthetic.
• The lab does not reproduce employer code, data or systems.
• It is a local batch pipeline and does not include Databricks, cloud infrastructure, streaming, orchestration, an API or a user interface.
• It does not claim production scale or business impact.

1. 01

   Declare contracts instead of inferring schemas

   Each source has an explicit Spark schema, primary key and typed money fields so the pipeline never relies on inference.

2. 02

   Apply inspectable quality rules

   Rejected rows are written once with every applicable rule ID, the source name and a deterministic run ID.

3. 03

   Keep runs idempotent

   The run ID is derived from the configuration and input bytes. Outputs are written safely so repeated runs produce the same logical tables and counts.

4. 04

   Reconcile accepted outputs

   The pipeline checks that every source row is accepted or quarantined and reconciles accepted order-line totals against the curated order summary.

5. 05

   Package evidence and CI

   Manifest, quality and reconciliation reports are written as JSON, and the repository gates format, types, tests, build and the end-to-end demo in CI.

• A local PySpark batch pipeline with synthetic fixtures.
• Explicit source contracts and quarantine outputs.
• Reconciliation and run-manifest evidence reports.
• Automated tests and a locked development environment.
• GitHub Actions CI and a versioned public release.

• Python
• PySpark
• uv
• pytest
• Ruff
• mypy
• GitHub Actions
• Parquet

• Synthetic lab package fact

  Explicit source contracts

  5

  The published lab declares five source contracts with explicit schemas instead of inference.

• Synthetic lab package fact

  Automated tests

  17

  The published package includes seventeen automated tests covering contracts, quality behaviour and pipeline evidence.

• Synthetic lab package fact

  Reconciliation checks

  2

  The pipeline performs two reconciliation checks: source-row accounting and accepted line-total reconciliation against the curated summary.

• This is a synthetic reference lab, not a client delivery or production system.
• It does not demonstrate Databricks, cloud operations, streaming or orchestration.
• Fixture counts and demo totals are not business KPIs.
• There is no evidence of scale, latency or cost under realistic workloads.

• Orchestration, retries and failure recovery.
• Access control, cataloguing and lineage.
• Schema-evolution policy and incremental processing.
• Operational metrics, alerting and partitioning under real workloads.

• ETL and data-quality design.
• Contract-first pipeline review.
• Quarantine and reconciliation design.
• Technical assessment of analytical-process modernisation.

• View the public repository

  Source code, synthetic fixtures, tests, evidence reports and CI for the independent lab. It uses synthetic data, rules and architecture and does not reproduce employer code, data or systems.

• View release v0.1.0

  Versioned public release of the synthetic lab package and documented local demo expectations.