Scenario Catalog¶
This page catalogs all 19 scenarios in the data-eng-lab lakehouse, organized by functional category. Each scenario is a self-contained folder with Scala Spark (Zeppelin), PySpark (Jupyter), and optional Airflow DAG implementations. Architecture diagrams are linked from each scenario's README.
Categories¶
The scenarios demonstrate the end-to-end data engineering lifecycle: raw data ingestion through medallion transformation, quality validation, streaming, BI, and advanced Iceberg features.
Batch Ingestion¶
Batch processing of raw Parquet data into Iceberg bronze. Demonstrates the fundamental lakehouse path from MinIO landing zone through Spark to cataloged Iceberg tables.
Medallion Pipeline¶
Multi-layer transformation: bronze → silver (deduplication) → gold (daily aggregation). Demonstrates the medallion architecture with a single dataset flowing through all three tiers.
Data Quality¶
Validation, quarantine, and quality gates. Demonstrates filtering invalid data into a quarantine table while preserving it rather than silently dropping it.
Schema & Maintenance¶
Schema evolution (ADD/RENAME columns with backward-compatible NULL filling), time travel (VERSION AS OF, rollback, WAP branches), and table maintenance (compaction, expire snapshots, remove orphan files). These are pure Iceberg capabilities that require only the delivered Spark/Iceberg runtime.
- schema_evolution-gh_archive-spark-iceberg
- time_travel-nyc_taxi-spark-iceberg
- table_maintenance-nyc_taxi-spark-iceberg
Streaming¶
Four streaming scenarios covering file-source and Kafka-source structured streaming, windowed aggregation with watermarks, and CDC (Change Data Capture) with MERGE INTO. These require the enhanced-Atlas stack with Redpanda enabled.
- streaming_ingest-events-spark-iceberg
- streaming_ingest-gh_archive-spark-iceberg
- streaming_windows-events-spark-iceberg
- cdc_streaming-online_retail-spark-iceberg
BI & Queries¶
Trino SQL queries over Iceberg tables via the REST connector. Demonstrates multi-engine interoperability: Spark writes, Trino reads (and writes via CTAS), with both engines accessing the same Iceberg tables on shared storage.
Join Optimization¶
Broadcast join hints and Adaptive Query Execution (AQE) on TPCH data. Demonstrates query performance optimization: broadcasting a dimension table to avoid shuffle, plus AQE repartitioning.
Dimensional Modeling¶
Star schema construction from TPCH source tables. Fact and dimension gold marts are written as Iceberg tables for BI consumption, demonstrating traditional data warehouse patterns on the lakehouse.
Feature Engineering¶
ML feature marts from MovieLens ratings. Per-user and per-movie aggregated features stored as Iceberg tables, bridging to Spark MLlib pipelines for collaborative filtering.
SCD Type 2¶
Slowly Changing Dimension Type 2 with effective_from/to tracking and is_current flags. Demonstrates full history preservation through UPDATE + INSERT patterns on Iceberg.
JSON Processing¶
Nested JSON semi-structured data from the GitHub Archive flattened into typed relational columns using Spark's from_json and explode functions. Demonstrates schema enforcement on semi-structured data.
Session Analysis¶
Window-based sessionization on timestamped events. Gap-based session boundary detection (>30s gap = new session) with cumsum-assigned session IDs.
Running Scenarios¶
Each scenario follows a dual-notebook pattern: a Scala Spark notebook for Zeppelin and a PySpark notebook for JupyterHub. Both must produce equivalent output, validated by the parity harness (tests/scenarios/parity.py). Airflow DAGs provide an orchestration alternative for batch scenarios.
Streaming scenarios require Redpanda to be running and the appropriate Kafka topic to be created (via REDPANDA_DEMO_TOPICS in .env or producer.py). Checkpoint state is stored in MinIO's checkpoints bucket; re-running a streaming notebook without a checkpoint cleanup will resume from the last offset.