Skip to content

5.1. Catalog

This page catalogs all 19 scenarios in the data-eng-lab lakehouse, organized by functional category. Each scenario is a self-contained folder with paired Zeppelin and Jupyter implementations. The execution-mode matrix distinguishes the eight production DAGs covering nine scenarios, notebook-only demonstrations, and intentionally unscheduled streams. Seventeen Spark scenarios form Scala/PySpark parity pairs; the two Trino scenarios instead pair %trino Zeppelin SQL with a Jupyter client. Architecture diagrams are linked from each scenario's README.

1. Categories

The scenarios demonstrate the end-to-end data engineering lifecycle: raw data ingestion through medallion transformation, quality validation, streaming, BI, and advanced Iceberg features.

Batch Ingestion

Batch processing of raw Parquet data into Iceberg bronze. Demonstrates the fundamental lakehouse path from MinIO landing zone through Spark to cataloged Iceberg tables.

Medallion Pipeline

Multi-layer transformation: bronze → silver (deduplication) → gold (daily aggregation). Demonstrates the medallion architecture with a single dataset flowing through all three tiers.

Data Quality

Validation, quarantine, and quality gates. Demonstrates filtering invalid data into a quarantine table while preserving it rather than silently dropping it.

Schema & Maintenance

Schema evolution (ADD/RENAME columns with backward-compatible NULL filling), time travel (VERSION AS OF, rollback, WAP branches), and table maintenance (compaction, expire snapshots, remove orphan files). These are pure Iceberg capabilities that require only the delivered Spark/Iceberg runtime.

Streaming

Four Structured Streaming scenarios cover broker ingestion, incremental file ingestion, windowed aggregation with watermarks, and CDC (Change Data Capture) with MERGE INTO. Three scenarios require Redpanda: streaming_ingest-events-spark-iceberg, streaming_windows-events-spark-iceberg, and cdc_streaming-online_retail-spark-iceberg. streaming_ingest-gh_archive-spark-iceberg polls files incrementally and requires no Kafka broker.

BI & Queries

Trino SQL queries over Iceberg tables via the REST connector. The notebooks demonstrate multi-engine interoperability, including educational CTAS writes. Production uses the read-only tpch_bi_query and nyc_taxi_trino_daily DAGs and stores bounded canonical artifacts as metadata-DB XCom records, not Iceberg tables.

Join Optimization

Broadcast join hints and Adaptive Query Execution (AQE) on TPCH data. Demonstrates query performance optimization: broadcasting a dimension table to avoid shuffle, plus AQE repartitioning.

Dimensional Modeling

Star schema construction from TPCH source tables. Fact and dimension gold marts are written as Iceberg tables for BI consumption, demonstrating traditional data warehouse patterns on the lakehouse.

Feature Engineering

ML feature marts from MovieLens ratings. Per-user and per-movie aggregated features stored as Iceberg tables, bridging to Spark MLlib pipelines for collaborative filtering.

SCD Type 2

Slowly Changing Dimension Type 2 with effective_from/to tracking and is_current flags. Demonstrates full history preservation through UPDATE + INSERT patterns on Iceberg.

JSON Processing

Nested JSON semi-structured data from the GitHub Archive flattened into typed relational columns using Spark's from_json and explode functions. Demonstrates schema enforcement on semi-structured data.

Session Analysis

Window-based sessionization on timestamped events. Per-actor gap-based session boundary detection (>30-minute gap = new session) uses deterministic (created_at, id) ordering and cumulative session IDs.

2. Running Scenarios

Each scenario follows a dual-notebook pattern across Zeppelin and JupyterHub. The 17 Spark pairs must produce equivalent output, validated by tests/scenarios/parity.py; the two Trino pairs are execution-gated but excluded from Scala/PySpark parity. Airflow runs eight production DAGs covering nine scenarios: nyc_taxi_etl, nyc_taxi_medallion, nyc_taxi_data_quality, tpch_star_schema, movielens_feature_pipeline, gh_archive_flatten_sessionization, tpch_bi_query, and nyc_taxi_trino_daily. The two Trino tasks return durable metadata-DB XCom records for their DagRuns; those artifacts are not Iceberg tables and follow the Airflow metadata retention policy. Retrieve them from the task-instance XCom view/API before retention cleanup.

The three broker-backed scenarios require Redpanda and their Kafka topics (created through REDPANDA_DEMO_TOPICS or a scenario producer). The GH Archive streaming-ingest scenario instead requests one resolver-verified immutable file set. All four store checkpoint state in MinIO's checkpoints bucket, so a rerun resumes from the last processed offset or file unless its checkpoint is intentionally reset.

The checkpoint retention runbook owns those paths and keeps active or uncertain state fail-closed. Issue #86 supplies writer leases, scoped credentials, immutable tombstones/audits, manual-only exact-leaf apply, and disposable live evidence. Its separate support DAG is paused, dry-run-only, and has schedule=None; automatic and scheduled destructive apply remain disabled pending stronger MinIO capabilities.

3. See Also