Data Engineering

How data moves: pipelines, storage systems, processing paradigms, orchestration, and quality.

Overall progress

0 / 6

Progress by topic area

Fundamentals0/2 · 0%

Storage0/1 · 0%

Processing0/1 · 0%

Orchestration & Quality0/2 · 0%

Fundamentals

Beginner6 min read

Introduction to Data Engineering

What data engineers actually build, how the role fits alongside analytics and data science, and the core responsibilities you'll see in almost every job description.

Beginner7 min read

ETL vs ELT: Where Transformation Happens

The classic Extract-Transform-Load pattern and its modern successor, Extract-Load-Transform — what changed, why it changed, and when you'd still choose each one.

Storage

Intermediate8 min read

Data Warehouses, Data Lakes, and Lakehouses

Three storage paradigms with different tradeoffs on schema, cost, and query performance — and how the lakehouse tries to get the best of both worlds.

Processing

Intermediate7 min read

Batch vs Streaming Processing

Two fundamentally different ways to move and process data — running on a schedule versus reacting to events as they happen — and how to decide which one a problem needs.

Orchestration & Quality

Intermediate7 min read

Orchestrating Pipelines with DAGs

Why pipelines are modeled as directed acyclic graphs, what tools like Airflow and Dagster actually coordinate, and the scheduling concepts that trip up beginners.

Advanced8 min read

Data Quality and Testing

Why 'the pipeline ran successfully' doesn't mean the data is correct, and the layered approach — schema checks, freshness, and business-logic tests — that catches problems before dashboards do.