Skip to content

Datasets

data-eng-lab lands a curated set of five standard open datasets into MinIO's landing bucket, driven by a declarative registry. Each dataset comes in multiple scale levels (tiny, small, medium) to support CI, local development, and heavier analytical workloads.

4.1 Registry

datasets/registry.yaml declares each dataset with the following properties:

Property Description
format File format: parquet, csv, or json.gz
license Dataset license
landing_prefix MinIO object key prefix (e.g. nyc_taxi/)
fetch.kind http for direct downloads with optional unzip, tpch for DuckDB-generated TPC-H
fetch.scale_params Per-scale parameters (tiny / small / medium)

The registry is schema-validated by the repo verifier. Adding a dataset requires only a new entry in this file and a corresponding fetch function — no code change to the core pipeline is needed.

4.2 Current Datasets

Dataset Shape Format Fetch Scenarios
nyc_taxi Columnar analytical Parquet http batch_ingest, data_quality, medallion, federated_query, table_maintenance, time_travel, join_optimization, star_schema, bi_query, incremental_upsert, scd2
gh_archive Semi-structured JSON events JSON.gz http json_flatten, schema_evolution, sessionization, streaming_ingest
movielens Rating and join data CSV http (unzip) feature_engineering
online_retail Transactional retail invoices CSV http (unzip) incremental_upsert, scd2, cdc_streaming
tpch Benchmark star-schema Parquet tpch (DuckDB) star_schema, join_optimization, bi_query

4.3 Adding a Dataset

  1. Add an entry to datasets/registry.yaml with format, license, landing_prefix, fetch.kind, and fetch.scale_params.
  2. Add a fetch function in scripts/download_datasets.py if fetch.kind is not http or tpch.
  3. Run make datasets SCALE=tiny to land the dataset in MinIO.
  4. Verify with uv run python scripts/download_datasets.py --dry-run.
  5. Create a scenario folder and write the notebooks.

4.4 Usage

make up                  # boot the Atlas data-eng track (MinIO must be running)
make datasets            # land the 'small' tier
make datasets SCALE=tiny # CI-sized subset
make datasets SCALE=medium # more data; heavier queries
uv run python scripts/download_datasets.py --scale medium --only nyc_taxi
uv run python scripts/download_datasets.py --dry-run    # show what would be landed

The downloader reads MinIO credentials and the published S3 port from infra/.env and is idempotent — existing objects are skipped unless --force is specified.

NYC Taxi (nyc_taxi)

TPC-H (tpch)

MovieLens (movielens)

Online Retail (online_retail)

GitHub Archive (gh_archive)

4.6 See Also