Data Engineering Core Interview Prep
Test & Track Your Readiness

What Data Engineering actually is (vs Data Science), the data lifecycle (generation → storage → ingestion → serving), responsibilities, and how to choose between batch and streaming, lake and warehouse, OLTP and OLAP.

Star vs snowflake schemas, fact and dimension tables, Slowly Changing Dimensions (SCD types 1/2/3), normalisation (1NF–3NF), denormalisation trade-offs, Kimball vs Inmon, surrogate keys.

Relational vs document vs column vs key-value vs graph databases — when to use each, the CAP theorem, consistency models, write/read trade-offs, and how columnar layouts (Parquet/ORC) win for analytics.

Modern data warehouses (Snowflake, BigQuery, Redshift), MPP architecture, columnar storage, partitioning and clustering, separation of storage and compute, slot/credit cost models.

Object storage as the foundation, file formats (Parquet/Avro), table formats (Delta Lake, Apache Iceberg, Hudi), the Lakehouse model, schema evolution, ACID on S3.

Why Hadoop existed, HDFS internals, MapReduce mental model, YARN, Hive's role, and where Hadoop still matters today vs where it's been replaced by cloud-native warehouses and Spark.

Spark architecture (driver, executors, cluster manager), the RDD → DataFrame → Dataset evolution, transformations vs actions, lazy evaluation, the Catalyst optimizer, Tungsten execution.

Shuffles and why they're expensive, data skew handling, broadcast joins, partitioning strategies, AQE (Adaptive Query Execution), caching when it helps vs hurts, executor sizing, spill diagnosis.

ETL vs ELT, designing idempotent pipelines, watermarking for incremental loads, backfill strategies, the dbt mental model (model + test + docs), medallion architecture (bronze/silver/gold).

Batch vs streaming, event time vs processing time, windowing (tumbling, sliding, session), watermarks for handling late data, exactly-once vs at-least-once semantics, the lambda and kappa architectures.

Kafka architecture (topics, partitions, offsets, consumer groups), brokers and ZooKeeper/KRaft, exactly-once semantics, log compaction, Kafka Connect and Streams, Flink vs Spark Structured Streaming.

Airflow's DAG model (operators, tasks, dependencies), scheduling and SLA, retries and idempotency, sensors and triggers, the dbt + Airflow pattern, when to choose Prefect, Dagster, or Step Functions.

Categories of data quality (completeness, freshness, uniqueness, distribution, schema), test-driven pipelines with dbt tests and Great Expectations, anomaly detection, the data contract pattern.

Data catalogues (Amundsen, DataHub, Unity Catalog), end-to-end lineage with OpenLineage, metadata management, ownership and stewardship, why data mesh is more an org pattern than a tech stack.

Encryption at rest and in transit, PII identification and tokenisation, column- and row-level masking, RBAC and ABAC, audit logging, GDPR essentials (right to be forgotten, data residency), India DPDP.

Log-based CDC internals: snapshot vs incremental phases, Debezium envelope, WAL/binlog mechanics, ordering and delivery semantics, schema evolution across CDC boundaries, downstream patterns (warehouse MERGE, lakehouse UPSERT, cache invalidation), the outbox pattern, and operational failure modes.

The transformation framework most modern warehouses use: mental model, materializations (view/table/incremental/ephemeral), incremental strategies, ref-driven DAG, sources and freshness, tests, snapshots (SCD Type 2), Jinja macros, project structure, packages, dbt Cloud vs Core vs Fusion, SQLMesh comparison, and common anti-patterns.

Micro-batch streaming on the Spark engine: streaming DataFrames, sources and sinks, watermarks and late data, output modes (append/update/complete), checkpointing, triggers (processingTime/once/availableNow/continuous), stream-stream joins, RocksDB state store, backpressure controls, and when to prefer Flink instead.

Where data engineering meets ML: offline vs online store, training/serving skew, point-in-time joins, feature reuse across teams, freshness SLA per feature, streaming feature materialization, serving latency budget, Feast/Tecton/Databricks Feature Store landscape, common anti-patterns, and when a team actually needs one.