5.21. Execution Modes

This table is generated from scenarios/execution-modes.yaml. It is the reviewed execution contract for all 19 paired-notebook scenarios; edit the YAML and run uv run python -m scripts.scenario_execution --render.

nyc_taxi_etl, nyc_taxi_medallion, nyc_taxi_data_quality, tpch_star_schema, movielens_feature_pipeline, gh_archive_flatten_sessionization, tpch_bi_query, and nyc_taxi_trino_daily are eight production DAGs today. The separate checkpoint_retention support DAG is paused, manual, and dry-run-only with schedule=None. An approved child issue is a delivery boundary, not a runnable DAG. Notebook-only and continuous-stream scenarios run from their paired Zeppelin or Jupyter notebooks.

Scenario	Classification	Owner	Runtime and schedule	Entrypoint / child	Dependencies	Justification	Acceptance contract
batch_ingest-nyc_taxi-spark-iceberg	existing production DAG	Data Engineering	Atlas Airflow and Spark standalone via the nyc-taxi-etl JAR; @daily with manual trigger and explicit dataset_scale supported	spark-apps/nyc-taxi-etl/dag.py	#78 operator-owned Spark submission; #81 verified dataset resolver.	The reviewed NYC Taxi ETL application performs the same resolver-gated batch ingest and replaces the Bronze table; the former scenario-local DAG was a superseded no-op.	The Airflow task resolves one expected-scale immutable NYC Taxi generation and passes only those URIs to Spark; The run replaces lakehouse.bronze.nyc_taxi_trips and ends with Airflow success plus Spark FINISHED and success=true.
bi_query-tpch-trino-iceberg	existing production DAG	Analytics Engineering	Atlas Airflow PythonOperator using the bounded consumer-owned Trino HTTP statement client; Daily at 01:00 UTC; max_active_runs=1 and one retry after two minutes	airflow-dags/trino_bi/dag.py	#82 execution-mode matrix; #83 Trino Airflow execution contract; #107 production TPC-H star schema.	The daily read-only Airflow task validates both TPC-H tables' exact five-key provenance and stable snapshots before accepting a bounded canonical segment-revenue XCom artifact.	Fail closed before BI SQL unless both $properties tables contain the exact equal five-key TPC-H provenance contract; Return a bounded canonical metadata-DB XCom segment-revenue artifact only after schema, source reconciliation, and unchanged postflight property/snapshot checks.
cdc_streaming-online_retail-spark-iceberg	intentionally unscheduled long-running streaming	Streaming Data Engineering	Operator-started Atlas Zeppelin or Jupyter Spark Structured Streaming session; Intentionally unscheduled; an operator starts, monitors, and stops the continuous query	None	Atlas Redpanda and Spark Kafka integration; online_retail_cdc topic and checkpoint ownership policy #85.	The notebook owns an unbounded Redpanda readStream and foreachBatch MERGE with durable checkpoint state; a periodic batch schedule would misrepresent its lifecycle.	Apply ordered CDC micro-batches idempotently to lakehouse.silver.online_retail_cdc through foreachBatch MERGE; Preserve and monitor the owned checkpoint; fail rather than launch a second writer with ambiguous ownership; Use issue #86 manual exact-leaf retention only after reviewed eligibility; automatic and scheduled destructive apply remain disabled pending stronger MinIO capabilities.
data_quality-nyc_taxi-spark-iceberg	existing production DAG	Data Quality Engineering	Production Atlas Airflow and Spark standalone application; Serialized @daily after the matching successful nyc_taxi_etl logical date	spark-apps/nyc-taxi-data-quality/dag.py	#82 execution-mode matrix; #81 verified dataset provenance; #91 durable quality monitoring; nyc_taxi_etl Bronze table.	The reviewed application binds one Bronze snapshot, conserves every row through a null-safe Silver split, persists eight governed Gold facts, and passed matching-ETL, same-date retry, dashboard, Airflow, and Spark live acceptance.	Bind the exact Bronze snapshot and schema, preserve its null-safe row multiset across clean and quarantine, and read back eight deterministic versioned Gold facts; Serialize after the same-logical-date NYC Taxi ETL, converge a same-date rerun without duplicate facts, and expose three fixed bounded Trino dashboard queries.
feature_engineering-movielens-spark-iceberg	existing production DAG	ML Data Engineering	Atlas Airflow and Spark standalone via the movielens-feature-pipeline JAR; @daily with manual trigger and explicit dataset_scale supported; max_active_runs=1 serializes the two-table replacement	spark-apps/movielens-feature-pipeline/dag.py	#82 execution-mode matrix; #81 verified MovieLens generations; #108 feature pipeline implementation and live acceptance.	The reviewed application preserves the notebook aggregations, binds both Gold tables to one verified MovieLens generation, and passed repeatable Airflow and Spark acceptance.	Deterministically replace lakehouse.gold.ml_user_features and lakehouse.gold.ml_movie_features from one verified MovieLens generation; Freeze schemas, duplicate-row aggregation semantics, key behavior, count invariants, five-key provenance, and idempotence with live Airflow and Spark success.
federated_query-nyc_taxi-trino-iceberg	existing production DAG	Analytics Engineering	Atlas Airflow PythonOperator using the bounded consumer-owned Trino HTTP statement client; Daily at 02:00 UTC after the TPC-H BI cadence; max_active_runs=1 and one retry after two minutes	airflow-dags/trino_bi/dag.py	#82 execution-mode matrix; #83 Trino Airflow execution contract; nyc_taxi_etl Bronze table.	The daily read-only Airflow task binds to one unchanged NYC Bronze Iceberg snapshot and accepts a bounded canonical daily-fares XCom artifact after schema and source-count reconciliation.	Bind to one unchanged NYC Bronze Iceberg snapshot and reconcile nonempty daily counts to the independent source count; Return a bounded canonical metadata-DB XCom daily-fares artifact without claiming resolver-generation provenance or writing Iceberg.
incremental_upsert-online_retail-spark-iceberg	intentionally notebook-only	Data Engineering Education	Atlas Zeppelin or Jupyter interactive Spark session; Unscheduled notebook demonstration; reset scenario-owned tables before a clean rerun	None	Atlas Spark and Iceberg runtime; Paired notebook live-reproducibility gate.	The notebook intentionally seeds inline rows and demonstrates repeatable MERGE semantics; its synthetic changes are not a governed production feed.	Demonstrate matched updates and unmatched inserts without duplicating the MERGE result; Keep Scala and PySpark outputs equivalent from the same clean scenario-local state.
join_optimization-tpch-spark-iceberg	intentionally notebook-only	Data Engineering Education	Atlas Zeppelin or Jupyter interactive Spark session; Unscheduled performance experiment with operator-selected dataset_scale	None	#81 verified TPC-H generation; Atlas Spark AQE runtime.	Broadcast and adaptive-query behavior is an interactive experiment whose plan and timing depend on scale and cluster configuration, not a stable scheduled data product.	Show the selected physical join plan and produce the same segment-revenue result in Scala and PySpark; Treat timing as observational evidence rather than a fixed pass threshold.
json_flatten-gh_archive-spark-iceberg	existing production DAG	Data Engineering	Atlas Airflow and Spark standalone via the gh-archive-pipeline JAR; @daily with manual trigger and explicit dataset_scale supported; max_active_runs=1 serializes flatten and sessionization	spark-apps/gh-archive-pipeline/dag.py	#82 execution-mode matrix; #81 verified GH Archive generations; #109 coupled flatten and sessionization pipeline.	The reviewed flatten application preserves canonical duplicate events, rejects conflicting IDs, replaces the typed event table from one verified generation, and passed repeatable Airflow and Spark acceptance.	Independently validate nested source records and deterministically replace lakehouse.silver.gh_events with exact source-row multiplicity; Freeze schema, timestamps, duplicate/conflict behavior, five-key provenance, readback, and terminal Airflow/Spark success before sessionization.
medallion-nyc_taxi-spark-iceberg	existing production DAG	Data Engineering	Atlas Airflow and Spark standalone via the nyc-taxi-medallion JAR; @daily with manual trigger supported after the matching Bronze prerequisite exists	spark-apps/nyc-taxi-medallion/dag.py	#78 operator-owned Spark submission; #81 resolver provenance gate; nyc_taxi_etl Bronze table.	The reviewed medallion application replaces Silver and Gold NYC Taxi tables from Bronze; the former scenario-local DAG was a superseded no-op with a misleading identifier.	Gate the run on one expected-scale immutable NYC Taxi generation and the existing Bronze table; Replace the documented Silver and Gold tables and end with Airflow success plus Spark FINISHED and success=true.
scd2-online_retail-spark-iceberg	intentionally notebook-only	Analytics Engineering Education	Atlas Zeppelin or Jupyter interactive Spark session; Unscheduled notebook demonstration; reset scenario-owned tables before a clean rerun	None	Atlas Spark and Iceberg runtime; Paired notebook live-reproducibility gate.	The notebook uses inline customer changes to teach close-and-insert SCD2 mechanics; it has no governed source history or dimension ownership contract.	Preserve exactly one current version and closed history for the demonstrated customer transition; Keep Scala and PySpark table state equivalent from the same clean starting point.
schema_evolution-gh_archive-spark-iceberg	intentionally notebook-only	Data Engineering Education	Atlas Zeppelin or Jupyter interactive Spark session; Unscheduled operator-run notebook against a disposable scenario-owned table	None	Atlas Spark and Iceberg runtime; Paired notebook live-reproducibility gate.	ADD and RENAME operations intentionally mutate a scenario-local schema for teaching; replaying them on a schedule is neither idempotent nor an owned migration process.	Demonstrate additive and rename evolution with the expected historical null behavior; Run only against the scenario-owned table and preserve Scala/PySpark parity after reset.
sessionization-gh_archive-spark-iceberg	existing production DAG	Analytics Data Engineering	Atlas Airflow and Spark standalone via the gh-archive-pipeline JAR; @daily after matching flatten success with manual trigger and explicit dataset_scale supported; max_active_runs=1 serializes both stages	spark-apps/gh-archive-pipeline/dag.py	#82 execution-mode matrix; #109 coupled flatten and sessionization pipeline; lakehouse.silver.gh_events from the matching flatten stage.	The reviewed sessionization application consumes the matching-provenance event table, preserves row multiplicity, applies the exact 30-minute boundary, and passed repeatable Airflow and Spark acceptance.	Independently validate ordering, gap boundaries, null handling, row conservation, and deterministic replacement of lakehouse.silver.gh_sessions; Refuse mismatched gh_events provenance before reading rows, propagate either stage failure, and prove four REST-confirmed drivers across two reruns.
star_schema-tpch-spark-iceberg	existing production DAG	Analytics Data Engineering	Atlas Airflow and Spark standalone via the tpch-star-schema JAR; @daily with manual trigger and explicit dataset_scale supported before dependent Trino BI work	spark-apps/tpch-star-schema/dag.py	#82 execution-mode matrix; #81 verified TPC-H generations; #107 TPC-H star-schema implementation and live acceptance.	The reviewed production application preserves notebook transform semantics, binds both tables to one verified TPC-H generation, and passed repeatable Airflow and Spark acceptance.	Deterministically replace lakehouse.gold.dim_customer and lakehouse.gold.fct_orders from one verified TPC-H generation; Freeze join, measure, schema, and idempotency contracts and prove live Airflow/Spark success before #83 consumes them.
streaming_ingest-events-spark-iceberg	intentionally unscheduled long-running streaming	Streaming Data Engineering	Operator-started Atlas Zeppelin or Jupyter Spark Structured Streaming session; Intentionally unscheduled; an operator starts, monitors, and stops the continuous query	None	Atlas Redpanda and Spark Kafka integration; events producer and checkpoint ownership policy #85.	The notebook owns an unbounded Redpanda readStream and durable checkpoint; a periodic Airflow task cannot truthfully represent its start, supervision, and stop lifecycle.	Consume valid event records continuously into lakehouse.bronze.events with owned checkpoint recovery; Fail malformed input or ambiguous concurrent ownership and preserve equivalent bounded live-test results in both notebooks; Use issue #86 manual exact-leaf retention only after reviewed eligibility; automatic and scheduled destructive apply remain disabled pending stronger MinIO capabilities.
streaming_ingest-gh_archive-spark-iceberg	intentionally notebook-only	Streaming Data Engineering Education	Atlas Zeppelin or Jupyter interactive Spark Structured Streaming session; Unscheduled notebook demonstration with an operator-bounded run and scenario-owned checkpoint	None	#81 verified GH Archive generation; checkpoint ownership policy #85.	The notebook applies Structured Streaming mechanics to a finite resolver-locked file generation and then waits indefinitely; that is a teaching contrast, not a truthful daily workload.	Process the selected immutable file set once without mixing generations and demonstrate checkpointed file discovery; Bound the live-reproducibility run explicitly and preserve Scala/PySpark result parity; Use issue #86 manual exact-generation leaf retention only after reviewed eligibility and sink-reset evidence; automatic and scheduled destructive apply remain disabled pending stronger MinIO capabilities.
streaming_windows-events-spark-iceberg	intentionally unscheduled long-running streaming	Streaming Data Engineering	Operator-started Atlas Zeppelin or Jupyter Spark Structured Streaming session; Intentionally unscheduled; an operator starts, monitors watermark progress, and stops the continuous query	None	streaming_ingest-events producer contract; Atlas Redpanda and Spark Kafka integration; checkpoint ownership policy #85.	Watermarked windows close according to an unbounded event stream and durable checkpoint; a daily no-op task never represented this continuous lifecycle.	Emit only watermark-closed event windows into lakehouse.gold.event_windows with recoverable checkpoint state; Prove late-event and window semantics in bounded live tests without advertising a batch schedule; Use issue #86 manual exact-leaf retention only after reviewed eligibility; automatic and scheduled destructive apply remain disabled pending stronger MinIO capabilities.
table_maintenance-nyc_taxi-spark-iceberg	intentionally notebook-only	Lakehouse Operations Education	Atlas Zeppelin or Jupyter interactive Spark session; Unscheduled operator-run notebook against the isolated nyc_taxi_tm demonstration table	None	nyc_taxi_etl Bronze table; Atlas Spark and Iceberg maintenance procedures.	The notebook executes rewrite_data_files compaction, expire_snapshots, and remove_orphan_files on an isolated table; these destructive procedures lack the retention, authorization, concurrency, and recovery policy required for scheduling.	Execute rewrite_data_files, expire_snapshots with retain_last 1, and remove_orphan_files only on the scenario-owned nyc_taxi_tm table; Show before/after snapshot and file evidence with equivalent notebook outcomes after reset.
time_travel-nyc_taxi-spark-iceberg	intentionally notebook-only	Lakehouse Operations Education	Atlas Zeppelin or Jupyter interactive Spark session; Unscheduled operator-run notebook against the isolated nyc_taxi_tt demonstration table	None	nyc_taxi_etl Bronze table; Atlas Spark and Iceberg runtime.	The notebook creates snapshots, creates an audit branch, and inspects history metadata; its time-travel query and rollback examples remain comments rather than executed operations.	Execute snapshot creation, CREATE BRANCH, and history metadata inspection only on the isolated nyc_taxi_tt table; The current notebook does not execute a time-travel query or rollback; keep Scala and PySpark observations equivalent from the same reset scenario state.

1. Classification Rules

• existing production DAG: the entrypoint exists under spark-apps/ or airflow-dags/, is mounted into Airflow, and performs reviewed work.
• approved new production DAG: no production entrypoint exists yet; the linked child owns implementation and live acceptance.
• intentionally notebook-only: the scenario is an educational, experimental, or operator-sensitive notebook workflow with no schedule.
• intentionally unscheduled long-running streaming: an operator starts and supervises the continuous notebook query; Airflow does not treat it as a batch task.
• deprecated or superseded: a row uses this only when the scenario itself is retired. Deleted no-op DAG artifacts do not create extra scenario rows.

2. Review Boundary

A child may begin implementation only after this matrix is merged to both protected branches. Productionization must replace the approved row with an existing production entrypoint only after its code, focused tests, documentation, and live Airflow/Spark or Trino acceptance pass.