A data batch in which every row is accounted for

I wrote Data Quality Pipeline Lab as a local PySpark pipeline on synthetic wholesale orders. I wanted the contracts, the quarantine decisions, the reconciliation, and the CI evidence to be readable without opening a private system.

Quality checks are hard to discuss when the schema is guessed, a rejected row does not say why, and nobody reconciles what went in with what came out. I wanted a small public example where each row is either accepted or quarantined, and the totals can be checked.

• Five explicit source contracts with typed schemas and primary keys.
• Referential, domain, temporal and arithmetic quality rules with stable rule IDs.
• Row-level quarantine records for rejected inputs.
• Idempotent runs with overwrite-safe Parquet outputs.
• Source-to-target row accounting and monetary reconciliation.
• JSON evidence artefacts, automated tests and GitHub Actions CI.

• The data, business rules and architecture are entirely synthetic.
• The lab does not reproduce employer code, data or systems.
• It is a local batch pipeline and does not include Databricks, cloud infrastructure, streaming, orchestration, an API or a user interface.
• It does not claim production scale or business impact.

1. 01

   Declare contracts instead of inferring schemas

   Each source has an explicit Spark schema, primary key and typed money fields so the pipeline never relies on inference.

2. 02

   Apply inspectable quality rules

   Rejected rows are written once with every applicable rule ID, the source name and a deterministic run ID.

3. 03

   Keep runs idempotent

   The run ID is derived from the configuration and input bytes. Outputs are written safely so repeated runs produce the same logical tables and counts.

4. 04

   Reconcile accepted outputs

   The pipeline checks that every source row is accepted or quarantined and reconciles accepted order-line totals against the curated order summary.

5. 05

   Package evidence and CI

   Manifest, quality and reconciliation reports are written as JSON, and the repository gates format, types, tests, build and the end-to-end demo in CI.

• A local PySpark batch pipeline with synthetic fixtures.
• Explicit source contracts and quarantine outputs.
• Reconciliation and run-manifest evidence reports.
• Automated tests and a locked development environment.
• GitHub Actions CI and a versioned public release.

• Python
• PySpark
• uv
• pytest
• Ruff
• mypy
• GitHub Actions
• Parquet

• Synthetic lab package fact

  Explicit source contracts

  customers 4/3/1, orders 5/2/3, order lines 5/2/3, payments 3/1/2, shipments 3/0/3

  Pinned demo run 6afdf1566acc8178. Counts are read, accepted and quarantined.

• Synthetic lab package fact

  Automated tests

  customer_id.required and email.format

  The pinned test fixes the customer quarantine set as exactly these two ids.

• Synthetic lab package fact

  Reconciliation checks

  PASS, both line totals "400.00"

  Public synthetic lab. Not an employer system. Not Databricks, cloud, streaming, or orchestration.

• This is a synthetic reference lab, not a client delivery or production system.
• It does not demonstrate Databricks, cloud operations, streaming or orchestration.
• Fixture counts and demo totals are not business KPIs.
• There is no evidence of scale, latency or cost under realistic workloads.

• Orchestration, retries and failure recovery.
• Access control, cataloguing and lineage.
• Schema-evolution policy and incremental processing.
• Operational metrics, alerting and partitioning under real workloads.

• ETL and data-quality design.
• Contract-first pipeline review.
• Quarantine and reconciliation design.
• Technical assessment of analytical-process modernisation.

• View the public repository

  The pipeline, the synthetic fixtures, the tests, the evidence reports, and CI. The data and the rules are synthetic, and they do not copy an employer's system.

• View release v0.1.0

  The v0.1.0 release of that lab, with the local demo the tests pin.