Interview Prep/Architecture & System Design

Advanced11 min read

Data Quality, Reliability and Observability

How to make a platform trustworthy — the layers of testing, data contracts, SLAs, incident response, and how to answer 'how do you know your data is correct?'

At senior level, correctness is the job. A pipeline that runs on time with wrong numbers is worse than one that fails loudly — because nobody knows to stop trusting it.

The dimensions of data quality

DimensionQuestion it answersHow you check it
CompletenessIs anything missing?Row counts vs baseline; null rate on required fields
UniquenessAre there duplicates?Unique tests on the declared grain
ValidityDo values make sense?Type checks, accepted ranges, enum tests
ConsistencyDo related datasets agree?Referential integrity; cross-system reconciliation
TimelinessIs it fresh enough?Freshness SLA on max ingestion timestamp
AccuracyDoes it match reality?Reconcile against the source of truth or a known control total

Where checks belong

  1. At ingestion — schema validation and required fields. Cheapest place to catch problems; quarantine failures with a reason code
  2. At transformation — dbt tests gating promotion between layers, so bad data never reaches marts
  3. At serving — reconciliation against control totals and business-rule assertions before consumers see it
  4. Continuously — freshness and volume anomaly detection independent of whether a job ran

Watch out — 'The job succeeded' is not a quality signal

Pipeline status and data correctness are different axes. A job can succeed while loading zero rows, duplicating everything, or writing nulls. Monitor the data, not just the process — this distinction alone marks a senior answer.

Data contracts

A data contract is an explicit agreement between a producer and its consumers: schema, semantics, freshness, quality guarantees, and a versioning/deprecation policy. It moves breakage from *discovered downstream in a dashboard* to *caught at the source in CI*.

# A simplified data contract
dataset: network_events
owner: network-platform-team
sla:
  freshness: 15 minutes
  availability: 99.9%
schema:
  - name: subscriber_id
    type: string
    required: true
    pii: true            # drives masking + access policy
  - name: event_ts
    type: timestamp
    required: true
  - name: volume_bytes
    type: integer
    required: false
guarantees:
  - unique: [subscriber_id, event_ts]
  - not_null: [subscriber_id, event_ts]
change_policy:
  additive_changes: allowed
  breaking_changes: 30 days notice + consumer sign-off

Incident response for data

  1. Detect — automated alert on freshness, volume or a failed test, ideally before a consumer notices
  2. Communicate early — tell consumers the data is suspect before they make decisions on it
  3. Contain — stop downstream propagation; consider marking the dataset stale rather than serving wrong numbers
  4. Diagnose — use lineage to find the blast radius and the root cause
  5. Fix and backfill — correct the logic, replay the affected partitions from raw
  6. Post-mortem — add the test that would have caught it. Every incident should produce a new automated check

Practice questions

QA stakeholder says yesterday's numbers look wrong. What do you do?show

Confirm and scope before touching anything — half of these turn out to be a definition mismatch rather than a pipeline bug.

  • Reproduce — get the exact number, filters and time window they used; confirm it is genuinely wrong versus differently defined
  • Scope — is it one metric, one table, or everything? One dashboard or all of them?
  • Check the obvious — did the pipeline run, was the source late, did volumes shift, did a deploy land yesterday?
  • Trace lineage upstream to the first layer where the number breaks
  • Communicate a holding message quickly, even before the root cause is known
  • Fix, backfill, then add the test that would have caught it automatically

Likely follow-up: How do you prevent this class of issue systematically?

QHow do you set an SLA for a data pipeline?show

Derive it from the business decision the data supports, not from how long the pipeline currently takes.

  • Ask what decision depends on it and by when — 'the ops team reviews KPIs at 08:00' gives you a hard deadline
  • Define freshness (data no older than X), availability (percentage of days met) and quality (tests pass)
  • Build in buffer for retries — if the job takes 2 hours, do not promise delivery 2 hours after the source lands
  • Publish it, monitor it, and report against it; an unmeasured SLA is just a wish
QWhat is data lineage and why does it matter operationally?show

Lineage is the map of how data flows from source through transformations to consumers, at table and ideally column level.

  • Impact analysis — before changing a model, see exactly which dashboards and downstream tables break
  • Root cause — when a mart is wrong, walk upstream to find where it first went wrong
  • Compliance — prove where PII travelled and who consumed it
  • Retirement — safely delete assets once you can prove nothing reads them
  • dbt gives model-level lineage automatically; Dataplex/Data Catalog extends it across non-dbt systems
← PreviousHadoop to Cloud Migration: The Full StoryNext →Orchestration: Airflow, Composer and Dagster