Hadoop to Cloud Migration: The Full Story

If migration is on your résumé, expect 20 minutes on it. Interviewers want the *decision-making*: how you sequenced it, how you proved correctness, and what you would do differently.

Why organisations migrate

• Coupled storage and compute — scaling one forces you to scale (and pay for) the other
• Operational burden — cluster upgrades, YARN queue tuning, hardware refresh cycles
• Elasticity — on-prem is sized for peak and idle the rest of the time
• Capability gap — managed streaming, ML and BI integration are far ahead in cloud
• Say the business reason, not just the technical one: faster reporting cycles and lower TCO are what funded the project

The service mapping

The four migration patterns

The phases

1. Discovery — inventory every table, job, dependency and consumer. Most teams discover they cannot fully explain their own estate; lineage tooling pays for itself here
2. Prioritise — score by business value, technical risk and cost. Pick a first domain that is meaningful but survivable
3. Foundation — landing zone, IAM model, network, cost controls, CI/CD, orchestration, monitoring. Build this before pipeline work
4. Migrate by domain — one domain end to end, including its consumers and dashboards
5. Run in parallel — old and new produce output simultaneously; reconcile daily
6. Cut over — switch consumers, keep the old path warm as rollback
7. Decommission — only after a defined quiet period with no regressions

Validation: how you prove it is correct

-- Parallel-run reconciliation: compare legacy vs cloud output daily
with legacy as (
  select event_date, cell_site_id,
         sum(total_gb) as gb, sum(dropped_calls) as drops
  from legacy_export.network_kpi_daily
  where event_date = @run_date
  group by 1, 2
),
cloud as (
  select event_date, cell_site_id,
         sum(total_gb) as gb, sum(dropped_calls) as drops
  from core.fct_network_kpi_daily
  where event_date = @run_date
  group by 1, 2
)
select
  coalesce(l.cell_site_id, c.cell_site_id) as cell_site_id,
  l.gb    as legacy_gb,
  c.gb    as cloud_gb,
  abs(coalesce(l.gb, 0) - coalesce(c.gb, 0)) as gb_diff,
  case
    when l.cell_site_id is null then 'MISSING_IN_LEGACY'
    when c.cell_site_id is null then 'MISSING_IN_CLOUD'
    when abs(l.gb - c.gb) > 0.01 then 'VALUE_MISMATCH'
    else 'OK'
  end as status
from legacy l
full outer join cloud c
  on l.event_date = c.event_date
 and l.cell_site_id = c.cell_site_id
where l.cell_site_id is null
   or c.cell_site_id is null
   or abs(l.gb - c.gb) > 0.01;

• Row counts per partition — the coarse first check
• Aggregate reconciliation on key business metrics, as above
• Row-level hashing for critical tables when aggregates are not enough
• Consumer sign-off — the report owner confirms their numbers match before cutover
• Automate this as a daily job during the parallel run, not a one-off manual check

Practice questions