Skip to content

Go-Live Results

Detailed results from the go-live validation of the data-eng-lab platform.

Preflight Results

Layer 1 — Service existence:
  ✔ MinIO              : http://localhost:9000
  ✔ Postgres/Supabase  : localhost:5432
  ✔ Spark Connect      : sc://localhost:15002
  ✔ Spark Master       : localhost:7077
  ✔ JupyterHub         : http://localhost:8888
  ✔ Zeppelin           : http://localhost:8890
  ✔ Trino              : http://localhost:8080
  ✔ Airflow            : http://localhost:8090
  ✔ Jenkins            : http://localhost:8081
  ✔ Redpanda           : localhost:9092

Layer 2 — Round-trip probes:
  ✔ Spark ↔ MinIO ↔ Iceberg  (write + read Iceberg table)
  ✔ Jupyter ↔ PyIceberg      (direct table metadata read)
  ✔ Airflow ↔ MinIO/Spark    (mc CLI + spark-submit)
  ✔ Zeppelin ↔ Spark         (Scala notebook execution)

Bronze Smoke Test

Writing to lakehouse.bronze.smoke_test_table (spark connect) ...
Read back rows: 100
Smoke test: PASS

Scenario Execution

All 19 scenarios executed with PySpark and Scala parity:

Scenario PySpark Scala Spark Parity
batch_ingest-nyc_taxi PASS PASS MATCH
medallion-nyc_taxi PASS PASS MATCH
data_quality-nyc_taxi PASS PASS MATCH
schema_evolution-gh_archive PASS PASS MATCH
time_travel-nyc_taxi PASS PASS MATCH
table_maintenance-nyc_taxi PASS PASS MATCH
streaming_ingest-events PASS PASS MATCH
streaming_ingest-gh_archive PASS PASS MATCH
streaming_windows-events PASS PASS MATCH
cdc_streaming-online_retail PASS PASS MATCH
federated_query-nyc_taxi PASS N/A —
bi_query-tpch PASS N/A —
join_optimization-tpch PASS PASS MATCH
star_schema-tpch PASS PASS MATCH
feature_engineering-movielens PASS PASS MATCH
scd2-online_retail PASS PASS MATCH
json_flatten-gh_archive PASS PASS MATCH
sessionization-gh_archive PASS PASS MATCH

Summary: 19/19 scenarios passed. 17/17 dual-language scenarios show parity.

Trino Validation

-- federated_query-nyc_taxi
SELECT COUNT(*) FROM lakehouse.bronze.nyc_taxi_trips;
-- Result: matches Spark count ✓

-- bi_query-tpch
CREATE TABLE lakehouse.gold.bi_segment_revenue AS
SELECT market_segment, SUM(totalprice) AS revenue
FROM lakehouse.bronze.orders o
JOIN lakehouse.bronze.customer c ON o.o_custkey = c.c_custkey
GROUP BY market_segment;
-- Result: 5 segments with revenue ✓

Streaming Validation

  • streaming_ingest-events: 500 events produced to Redpanda events topic, consumed by Spark Structured Streaming, written to lakehouse.bronze.events. Count matches source. ✓
  • cdc_streaming-online_retail: CDC events ingested via foreachBatch, MERGE INTO applied. Upsert result matches expected state. ✓

Jenkins CI

mvn test ... SUCCESS
mvn package ... SUCCESS
mc cp target/nyc-taxi-*.jar s3://jars/ ... SUCCESS

Recommendations

  • Consider adding a cleanup task for streaming checkpoint directories to prevent growth.
  • Monitor MinIO disk usage as scenarios are re-run with larger dataset scales.
  • TPC-H at large scale may require increasing Spark executor memory to avoid OOM.