Spark Architecture Explained Simply

Driver, executors, jobs, stages, tasks, and why the shuffle is the thing that decides whether your job takes 4 minutes or 4 hours.

Almost every Spark interview starts here. You do not need to recite internals — you need to explain, in plain words, who does what and where the time goes.

The moving parts

The driver plans; the executors do the work. The cluster manager just hands out resources.

Component	What it does	One-line answer
Driver	Runs your `main`, builds the DAG, splits it into stages, schedules tasks, collects results	'The brain — it plans but does not process data'
Cluster manager	YARN, Kubernetes, or standalone — allocates containers	'Just the resource landlord'
Executor	A JVM process on a worker node; runs tasks and holds cached data	'The hands — where the data actually gets processed'
Task	One unit of work on one partition of data	'The smallest unit — one task per partition per stage'

Job → Stage → Task

Job — created by one action (write, count, collect)
Stage — a group of tasks that can run without moving data between machines. A new stage begins wherever a shuffle is needed
Task — one stage's work applied to one partition; tasks in a stage run in parallel

Tip — The sentence that lands well

'The number of stages equals the number of shuffles plus one. So when I tune a job, I am really asking: can I remove a shuffle, or make the shuffled data smaller?'

Narrow vs wide transformations

A shuffle writes to disk and sends data across the network — it is the single most expensive thing Spark does.

Narrow (map, filter, select, withColumn) — each output partition needs only one input partition. No data movement
Wide (groupBy, join, distinct, repartition, orderBy) — output partitions need data from many input partitions. Requires a shuffle

Lazy evaluation and Catalyst

Transformations only build a plan. Nothing runs until an action. That delay is what lets the Catalyst optimizer rewrite your query: pushing filters down to the scan, pruning unused columns, and reordering joins. It is why a DataFrame written badly can still run well — and why a Python UDF hurts, since Catalyst cannot see inside it.

API	Type safety	Optimised by Catalyst	Verdict
RDD	Compile-time	No	Legacy — only for low-level control
DataFrame	Runtime	Yes	Default choice in PySpark
Dataset	Compile-time	Yes	Scala/Java only; nice when you want type safety

Practice questions

QWhat exactly happens when you call an action on a DataFrame?show

The driver turns the logical plan into a physical plan, splits it at shuffle boundaries into stages, and submits tasks to executors stage by stage.

Catalyst optimises the logical plan (predicate pushdown, column pruning, join reorder)
The physical plan is cut into stages at each shuffle
Each stage becomes tasks — one per partition — scheduled onto executor cores
A stage cannot start until the previous stage's shuffle files are fully written

QWhat is the difference between repartition and coalesce?show

repartition(n) does a full shuffle and can increase or decrease partitions, producing evenly sized ones. coalesce(n) only decreases, and avoids a full shuffle by merging neighbouring partitions — cheaper but can leave uneven sizes.

Use coalesce to reduce small output files at the end of a job
Use repartition when data is skewed, or before a big join/write where evenness matters
coalesce(1) on a huge DataFrame is a classic mistake — it funnels everything through one task

Likely follow-up: How would you control the number of output files without hurting parallelism?

Qcache() vs persist() — and when should you avoid both?show

cache() is just persist(MEMORY_AND_DISK). persist() lets you pick a storage level. Both only take effect after the next action.

Cache when a DataFrame is reused by two or more actions — otherwise it is pure overhead
Do not cache something you read once and write straight out
Caching consumes executor memory that tasks need; over-caching causes spills and GC pressure
Always unpersist() when done in a long-running job

QWhat are shuffle partitions and what should you set them to?show

spark.sql.shuffle.partitions controls how many partitions are produced after a wide transformation. The default of 200 is almost never right.

Too few → huge partitions, spilling to disk, OOM
Too many → thousands of tiny tasks, scheduler overhead dominates
Rule of thumb: aim for partitions of roughly 128–256 MB, and a partition count that is a small multiple of total executor cores
With Adaptive Query Execution (AQE) enabled, Spark coalesces these at runtime — so the setting matters much less on Spark 3+

← PreviousPython and PySpark Coding Rounds Next →Spark Performance Tuning and Data Skew