Data Quality and Testing

A pipeline can finish with a green checkmark and still produce garbage: a source API silently changed a field name, duplicate rows crept in, or a currency conversion used a stale rate. Data quality is the practice of catching these problems automatically, before a stakeholder notices a wrong number in a dashboard.

Layers of data quality checks

1. Schema tests: does this column exist, is it the right type, is it non-null where required?
2. Freshness tests: did this table update recently, or is it silently stale?
3. Uniqueness and referential integrity: are primary keys actually unique, do foreign keys resolve?
4. Distribution/volume tests: did row counts or key metrics suddenly shift far outside historical norms?
5. Business-logic tests: domain-specific rules, e.g. 'order total should never be negative'

# A dbt-style schema test
models:
  - name: orders
    columns:
      - name: order_id
        tests:
          - unique
          - not_null
      - name: amount_usd
        tests:
          - not_null

Where tests should live

Push checks as close to the source as possible. Catching a bad record at ingestion is far cheaper than discovering it after it has propagated through five downstream tables and three dashboards. Many teams also add a 'quarantine' step: records that fail validation are routed to a separate location for investigation rather than silently dropped or allowed to break the pipeline.

As pipelines mature, teams often adopt a formal contract between producers and consumers of a dataset — a 'data contract' specifying schema, freshness, and quality guarantees — so that breaking changes are caught at the source before they ship downstream at all.