data-eng-lab
An Iceberg-lakehouse data-engineering lab built on the Atlas platform.
Build, orchestrate, stream, and query production-shaped lakehouse pipelines from paired notebooks and deployable Spark applications.
data-eng-lab consumes Atlas as its pinned infra/ git submodule through atlas.consumer.yml, so make up launches the default development profile as the Data Engineering workspace. The lab integrates storage, compute, orchestration, delivery, and observability instead of leaving users to wire independent services together: Iceberg tables live on MinIO, Spark runs batch and streaming workloads, Airflow coordinates production DAGs, Jenkins publishes six CI-built Maven applications, Trino serves analytical SQL, and Prometheus and Grafana monitor the Iceberg REST boundary. Nineteen paired Zeppelin and Jupyter scenarios provide 17 Scala/PySpark implementations plus two Trino client pairs, while Redpanda supplies three broker-backed streams. The same locked datasets and catalog contracts support notebook exploration and deployable application paths.
1. Quick start¶
git submodule update --init --recursive infra
uv sync --all-groups
make up
make datasets
make verify
The launcher validates atlas.consumer.yml, materializes the Atlas environment, starts the data-engineering track, registers the Iceberg namespaces, and runs the repository preflight. See Getting started for prerequisites, endpoints, and the complete walkthrough.
2. Architecture¶
The landing zone and the three Iceberg medallion layers are distinct storage stages:
Every curated table is an Apache Iceberg table accessed through the Atlas Iceberg REST catalog (lakehouse). Spark supplies compute, Trino handles ad-hoc and federated SQL, and Airflow schedules production DAGs. Jenkins builds Maven Scala Spark apps and publishes their JARs to MinIO. Redpanda (Kafka-compatible) backs the event-ingest, windowing, and CDC streaming scenarios. streaming_ingest-gh_archive-spark-iceberg uses an incremental file source and requires no Kafka broker.
3. Explore the lab¶
| Destination | What it contains |
|---|---|
| Scenario catalog | All 19 end-to-end scenarios, dependencies, execution modes, and source datasets |
| Execution-mode matrix | Eight production DAGs covering nine scenarios, seven notebook-only scenarios, and three unscheduled streams |
| Notebook walkthroughs | Side-by-side docs for 17 Scala/PySpark pairs and two Trino SQL/client pairs |
| Spark apps | Six CI-built Maven applications published by Jenkins and submitted through Airflow |
| Datasets | NYC Taxi, TPC-H, Online Retail, GH Archive, and MovieLens data plus synthetic events |
| Lakehouse design | Landing, bronze, silver, gold, namespaces, buckets, and catalog behavior |
| Atlas enablement | Accepted consumer configuration and the A1–A9 infrastructure record |
| Go-live runbook | Reproducible platform, notebook, DAG, and Spark-application acceptance |
| Checkpoint retention | Manual-only exact-leaf planning/deletion, writer leases, immutable recovery evidence, and the disabled automatic-scheduling boundary from issue #86 |
| Security automation | Dependency updates, exact-manifest vulnerability audits, CodeQL scope, verified repository protections, private reporting, and remediation workflow |
4. Scenario catalog¶
5. By the numbers¶
| Inventory | Count |
|---|---|
| Paired Zeppelin and Jupyter scenario implementations | 19 |
| Dual-language Scala/PySpark parity pairs | 17 |
| Redpanda-backed Structured Streaming scenarios | 3 |
| Incremental file-source Structured Streaming scenarios | 1 |
| CI-built Maven Spark apps | 6 |
| Curated downloaded datasets | 5 |
| Iceberg medallion layers | 3 |
New here? Run the Getting started walkthrough, then choose a scenario from the catalog.