Skip to content

data-eng-lab

An Iceberg-lakehouse data-engineering lab built on the Atlas platform.

Curated Spark scenarios in Scala (Zeppelin) and PySpark (Jupyter), orchestrated with Airflow, plus Maven Scala Spark apps built by Jenkins — all over Apache Iceberg on MinIO, cataloged by the Atlas Iceberg REST catalog.

1.1 Architecture

Full-stack Lakehouse Architecture

The lab implements a medallion lakehouse with four layers:

s3a://landing/   →   bronze   →   silver   →   gold
  (raw Parquet)       (clean)     (enriched)   (aggregated/modelled)

Every table is an Apache Iceberg table, accessed through the Atlas Iceberg REST catalog (lakehouse). Compute is provided by a Spark cluster; Trino handles ad-hoc SQL and federated queries. Redpanda (Kafka-compatible) drives all streaming scenarios. Airflow schedules production DAGs; Jenkins builds and publishes Maven Spark apps to MinIO artifacts.

1.2 Quick navigation

  • Scenario catalog

    19 end-to-end Spark and Trino scenarios across bronze, silver, and gold layers.

    Browse scenarios

  • Spark apps

    2 CI-verified Maven Scala Spark apps built by Jenkins and run by Airflow.

    Browse apps

  • Datasets

    5 curated datasets (NYC Taxi, TPC-H, Online Retail, GH Archive, Events) with make datasets.

    Dataset guide

  • Lakehouse design

    Medallion layout, Iceberg namespaces, MinIO buckets, and the bronze smoke test.

    Lakehouse guide

  • Atlas platform

    A1–A9 Atlas enablement checklist, expectations, and go-live runbook.

    Atlas enablement

  • Getting started

    Prerequisites, make datasets, starting the stack, and running notebooks.

    Quick start

1.3 By the numbers

What Count
Scenario notebooks (Scala + PySpark pairs) 19 (14 batch, 4 streaming, 1 hybrid)
CI-verified Maven Spark apps 2
Curated datasets 5
Atlas enablement items (A1–A9) 9
Iceberg medallion layers 3 (bronze / silver / gold)

1.4 Scenario catalog

Scenario Engine Layer Dataset
batch_ingest-nyc_taxi-spark-iceberg Spark Bronze NYC Taxi
medallion-nyc_taxi-spark-iceberg Spark Bronze→Silver→Gold NYC Taxi
data_quality-nyc_taxi-spark-iceberg Spark Silver NYC Taxi
schema_evolution-gh_archive-spark-iceberg Spark Silver GH Archive
time_travel-nyc_taxi-spark-iceberg Spark Silver NYC Taxi
table_maintenance-nyc_taxi-spark-iceberg Spark Silver NYC Taxi
streaming_ingest-events-spark-iceberg Spark (stream) Bronze Events
streaming_ingest-gh_archive-spark-iceberg Spark (stream) Bronze GH Archive
streaming_windows-events-spark-iceberg Spark (stream) Silver Events
cdc_streaming-online_retail-spark-iceberg Spark (stream) Silver Online Retail
federated_query-nyc_taxi-trino-iceberg Trino Gold NYC Taxi
bi_query-tpch-trino-iceberg Trino Gold TPC-H
join_optimization-tpch-spark-iceberg Spark Gold TPC-H
star_schema-tpch-spark-iceberg Spark Gold TPC-H
feature_engineering-movielens-spark-iceberg Spark Gold MovieLens
scd2-online_retail-spark-iceberg Spark Silver Online Retail
json_flatten-gh_archive-spark-iceberg Spark Silver GH Archive
sessionization-gh_archive-spark-iceberg Spark Silver GH Archive
incremental_upsert-online_retail-spark-iceberg Spark Silver Online Retail

1.5 Scenarios by Category

Batch Ingestion — batch_ingest

Medallion Pipeline — medallion

Data Quality — data_quality

Schema & Maintenance — schema_evolution, time_travel, table_maintenance

Streaming — streaming_ingest (events + gh_archive), streaming_windows, cdc_streaming

BI & Queries — federated_query, bi_query

Join Optimization — join_optimization

Dimensional Modeling — star_schema

Feature Engineering — feature_engineering

SCD — scd2

JSON Processing — json_flatten

Session Analysis — sessionization

New here?

Start with Getting started to get the stack running, then pick a scenario from the catalog or dive into the lakehouse design.

Atlas platform

The Atlas platform underpins this lab. See Atlas enablement for the full A1–A9 checklist and Go-live runbook for production readiness steps.