CI/CD, Docker and Kubernetes for Data Engineers

The platform-engineering side of the role — testing data pipelines in CI, environment strategy, containerisation, secrets, and infrastructure as code.

Senior data engineers are expected to ship like software engineers: version control, automated tests, reproducible environments, and deployments that can be rolled back.

What CI should actually run for a data project

# Illustrative CI pipeline for a dbt + PySpark repo
stages:
  - lint:
      - ruff / black          # Python style
      - sqlfluff              # SQL style
      - dbt parse             # does the project even compile?

  - unit:
      - pytest                # transformation logic on small fixtures
      - dbt compile           # Jinja resolves, refs exist

  - data-tests:
      - dbt build --select state:modified+ --target ci
      #   runs ONLY changed models and everything downstream,
      #   against a CI dataset seeded with sampled data

  - cost-check:
      - bq query --dry_run    # fail if a changed model would scan > N TB

  - deploy:
      - dbt run --target prod        # on merge to main
      - terraform apply              # infra changes, reviewed separately

Tip — The two details that impress

state:modified+ (only build what changed plus its downstream, instead of the whole project) and a dry-run cost gate that fails a PR which would scan terabytes. Both show you have run this at scale where a full rebuild is neither fast nor cheap.

Environments

Environment	Data	Purpose
Dev	Sampled or synthetic subset	Fast iteration; each engineer gets their own dataset/schema
CI	Small deterministic fixtures	Automated tests on every PR; must be cheap and fast
Staging	Production-like volume or a recent copy	Performance and cost validation before release
Prod	Real data	Deploy only via CI from main; no manual changes

Docker and Kubernetes in a data context

Docker — pin the exact Python, Spark and library versions so a job behaves identically on a laptop, in CI, and in production. It removes the entire class of 'works on my machine' pipeline bugs
Kubernetes — run Spark on K8s (instead of YARN), run Airflow workers as pods with KubernetesPodOperator, and isolate a heavy job's resources from everything else
Why it matters for data — dependency conflicts between pipelines are common; containers give each job its own isolated environment
Keep images small and version-tagged; never deploy :latest to production, because you lose the ability to roll back to a known build

Secrets and infrastructure as code

Never commit credentials; use Secret Manager and inject at runtime
Workload Identity so pods and jobs assume a service account without a key file existing at all
Least privilege — a per-pipeline service account scoped to the datasets it needs
Terraform for datasets, IAM, buckets, Composer environments and reservations — so infrastructure is reviewable, reproducible and auditable

Practice questions

QHow do you test a data pipeline?show

Test at several levels — logic in isolation, data as it flows, and the pipeline end to end.

Unit tests — pure transformation functions against small fixtures; fast, deterministic, no cluster required
Data tests — dbt tests for uniqueness, not-null, referential integrity, accepted ranges
Integration tests — run the DAG end to end in CI against sampled data
Contract tests — assert incoming source schemas so upstream changes fail loudly
Regression checks — compare key aggregates before and after a change, and require sign-off on unexpected differences

Likely follow-up: How do you test something that depends on a huge production table?

QHow do you roll back a bad data deployment?show

Code rollback and data rollback are separate problems; you need a plan for both.

Code — revert the merge and redeploy; keep deployments small so blast radius is small
Data — replay affected partitions from RAW with the corrected code, since every layer is rebuildable
Fast mitigation — BigQuery time travel lets you restore a table to a point within the last 7 days
Blue/green for critical marts — build the new version alongside, validate, then swap the view consumers read
Communicate — tell consumers the window of affected data, do not silently repair it

QHow do you manage schema changes safely across many consumers?show

Make changes additive by default and give consumers a migration window for anything breaking.

Additive (new nullable column) — safe, deploy freely
Breaking (rename, drop, type change) — announce, run both versions in parallel, migrate consumers, then remove
Versioned models — publish v2 alongside v1 rather than mutating v1 under people's feet
Lineage-driven impact analysis to know exactly who is affected before you ship
Encode the policy in the data contract so it is a rule, not a favour

← PreviousOrchestration: Airflow, Composer and Dagster Next →Behavioral and Leadership Questions