6.3. medallion-nyc_taxi-spark-iceberg¶

Documents the scenario's paired Jupyter (notebook.ipynb) and Zeppelin (notebook.zpln) implementations. Both notebooks implement identical logic in PySpark and Scala.

1. Section map¶

Subsection	Scala (Zeppelin)	PySpark (Jupyter)
2.1 Setup	✓	✓
2.2 Read	✓	✓
2.3 Transform	✓	✓
2.4 Write	✓	✓
2.5 Verify	✓	✓

2. Walkthrough¶

2.1 Setup¶

Scala (Zeppelin):

import spark.implicits._
import org.apache.spark.sql.functions._
// spark pre-bound (Spark Connect + lakehouse catalog)

PySpark (Jupyter):

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.remote("sc://spark-connect:15002").getOrCreate()

2.2 Read¶

Scala (Zeppelin):

val bronze = spark.table("lakehouse.bronze.nyc_taxi_trips")

PySpark (Jupyter):

bronze = spark.table("lakehouse.bronze.nyc_taxi_trips")

2.3 Transform¶

Scala (Zeppelin):

val silver = bronze.dropDuplicates("tpep_pickup_datetime", "tpep_dropoff_datetime")
val gold = silver.groupBy($"trip_date")
  .agg(count("*").as("trips"), avg($"fare_amount").as("avg_fare"))

PySpark (Jupyter):

silver = bronze.dropDuplicates(["tpep_pickup_datetime", "tpep_dropoff_datetime"])
gold = (silver.groupBy("trip_date")
        .agg(F.count("*").alias("trips"), F.avg("fare_amount").alias("avg_fare")))

2.4 Write¶

Scala (Zeppelin):

silver.writeTo("lakehouse.silver.nyc_taxi_trips").using("iceberg").createOrReplace()
gold.writeTo("lakehouse.gold.nyc_taxi_daily").using("iceberg").createOrReplace()

PySpark (Jupyter):

silver.writeTo("lakehouse.silver.nyc_taxi_trips").using("iceberg").createOrReplace()
gold.writeTo("lakehouse.gold.nyc_taxi_daily").using("iceberg").createOrReplace()

2.5 Verify¶

Scala (Zeppelin):

spark.table("lakehouse.gold.nyc_taxi_daily").orderBy($"trip_date").show()

PySpark (Jupyter):

spark.table("lakehouse.gold.nyc_taxi_daily").orderBy("trip_date").show()

3. Scala / PySpark parity¶

Both notebooks share the same numbered sections and produce identical Iceberg tables; only the language and interpreter differ.

4. How to run¶

Open the scenario's zeppelin/notebook.zpln on the Atlas Zeppelin UI or jupyter/notebook.ipynb on JupyterHub, then run all paragraphs/cells top to bottom.