Interview Prep/GCP & Cloud
GCP Data Services: Choosing the Right Tool
Dataflow vs Dataproc vs BigQuery, Pub/Sub, Cloud Composer, GCS, and the architecture questions a Google Certified Professional Data Engineer is expected to answer instantly.
Cloud interviews are mostly 'which service and why'. The winning answer names the service, gives one reason, and names the trade-off you accepted.
The service map
| Service | What it is | Reach for it when |
|---|---|---|
| BigQuery | Serverless columnar warehouse | SQL analytics, dbt models, the default home for modelled data |
| Cloud Storage (GCS) | Object storage | Raw landing zone, data lake, backups, staging for loads |
| Pub/Sub | Managed messaging | Decoupled event ingestion, fan-out, no partitions to size |
| Dataflow | Managed Apache Beam | Unified batch + stream with autoscaling and no cluster to run |
| Dataproc | Managed Hadoop/Spark | Lift-and-shift of existing Spark/Hive jobs; you want Spark specifically |
| Cloud Composer | Managed Airflow | Scheduling and dependency management across services |
| Datastream | Managed CDC | Replicating changes out of operational databases |
| Dataplex / Data Catalog | Governance & metadata | Discovery, lineage, classification across the estate |
Dataflow vs Dataproc vs BigQuery — the classic question
- BigQuery — if it can be expressed as SQL over warehouse data, do it here. No infrastructure, best cost profile for analytics
- Dataflow — for streaming or complex transformation logic, especially when you want one Beam codebase for batch and stream and true autoscaling
- Dataproc — when you already have Spark/Hive code and want the fastest, lowest-risk migration path, or need specific Spark libraries
Tip — The migration-flavoured answer
'In a Hadoop-to-GCP migration I would move Spark jobs to Dataproc first to de-risk the lift, then progressively rewrite the SQL-shaped workloads as BigQuery-native models. Dataproc buys speed of migration; BigQuery buys long-term cost and operational simplicity.'
GCS storage classes and lifecycle
| Class | Min duration | Good for |
|---|---|---|
| Standard | None | Active raw zone, frequently read data |
| Nearline | 30 days | Data read about once a month |
| Coldline | 90 days | Quarterly access, DR copies |
| Archive | 365 days | Compliance retention, rarely read |
Set lifecycle rules to transition objects automatically (for example Standard → Nearline at 30 days → Archive at 365) and to delete temp/staging prefixes. This is a cheap, very visible cost win worth mentioning.
Security and governance basics
- IAM at the right level — grant on dataset or table, not project-wide
bigquery.admin - Service accounts per pipeline, with least privilege; no shared human credentials
- Column-level security with policy tags for PII (subscriber identifiers, MSISDNs)
- Row-level security when one table serves multiple tenants or regions
- CMEK when regulation requires customer-managed keys; VPC Service Controls to prevent exfiltration
- Data residency — for EU telecom data, pin datasets to an EU region and know that BigQuery regions are not automatically multi-region
Practice questions
QDesign an ingestion pipeline for network events arriving at 2 million records per minute.show
Separate ingestion from processing so a spike never blocks producers, then land raw before transforming.
- Ingest — Pub/Sub (or Kafka if strict per-key ordering and replay are required) as the durable buffer
- Process — Dataflow streaming, or Spark Structured Streaming if the team's skills are Spark-centric
- Land raw — write immutable raw events to GCS/BigQuery raw first, so any transform bug is replayable
- Model — dbt on BigQuery builds stage and core layers incrementally
- Serve — partitioned + clustered marts feeding Tableau; pre-aggregate for dashboard speed
- Operate — monitor subscription backlog/consumer lag, dead-letter bad messages, alert on freshness SLAs
Likely follow-up: How would you handle a 10× traffic spike during a major incident?
QHow does Pub/Sub guarantee delivery, and what do you do about duplicates?show
Pub/Sub is at-least-once by default: a message is redelivered until acknowledged, so duplicates are normal and must be designed for.
- Set an ack deadline longer than worst-case processing, and extend it for slow messages
- Use a dead-letter topic after N delivery attempts so one poison message cannot block progress
- Make the sink idempotent — deterministic keys plus
MERGE, or dedupe onmessage_idwithin a window - Pub/Sub also offers exactly-once delivery within a region and ordering keys when strict order is required
QHow do you monitor and alert on a GCP data platform?show
Monitor the data, not only the infrastructure — a green pipeline that produced zero rows is still an outage.
- Freshness — assert max ingestion timestamp per table; alert when it exceeds the SLA
- Volume — row counts per partition versus a historical baseline; alert on sudden drops or spikes
- Quality — dbt tests for uniqueness, not-null, referential integrity, accepted values
- Pipeline health — task failures/retries in Composer or Dagster, Pub/Sub backlog, Dataflow system lag
- Cost — budget alerts and per-query byte monitoring so a runaway query is caught the same day
- Surface all of it on Grafana dashboards and route pages to an on-call rotation with clear runbooks
QWhat would you do differently building this platform on AWS?show
The architecture stays the same; only the service names change. Showing you can map concepts across clouds is a plus.
- BigQuery → Redshift or Athena/Trino over S3 (or Snowflake on AWS)
- GCS → S3; Pub/Sub → Kinesis or MSK
- Dataflow → Kinesis Data Analytics / Flink; Dataproc → EMR
- Cloud Composer → MWAA (managed Airflow)
- The invariant: durable buffer → immutable raw landing → layered transformation → partitioned serving layer