Real-World Graph Dataset Adapters¶

This directory documents the external dataset adapters used by the dataset complexity ladder in compare/datasets.yaml. Most heavy datasets are intentionally generated into corpus/generated/* rather than committed directly. The bounded cyber-threat slice is committed under corpus/cyber_threat_intel/ because it is the next repeatable graph-native rung.

1. STaRK¶

STaRK is the first target because it is built for retrieval over textual and relational knowledge bases. It has three useful domains:

prime: precision medicine;
mag: scholarly papers/authors/concepts/citations;
amazon: product graph retrieval.

Suggested setup:

python3 -m pip install stark-qa

Suggested materialization command shape:

uv run python corpus/adapters/stark_export.py \
  --dataset prime \
  --limit 200 \
  --output corpus/generated/stark_prime
uv run python corpus/adapters/stark_export.py \
  --dataset mag \
  --limit 200 \
  --output corpus/generated/stark_mag

The generated markdown should go under:

corpus/generated/stark_prime/
corpus/generated/stark_mag/

Use the query files:

2. OpenAlex Scholarly Graph¶

OpenAlex is the best real-world scholarly graph source: works, authors, institutions, concepts, venues, references, and abstracts. Use a bounded topic slice so LightRAG extraction stays tractable.

Suggested command shape:

uv run python corpus/adapters/openalex_scholarly.py \
  --search "graph rag knowledge graph retrieval" \
  --limit 150 \
  --output corpus/generated/openalex_scholarly

Use demo/openalex_scholarly_queries.yaml.

3. GDELT Events¶

GDELT is the best event/timeline graph candidate. Use narrow time windows and topic filters, then materialize each event cluster as a relation-heavy dossier with source URLs.

Suggested command shape:

uv run python corpus/adapters/gdelt_events.py \
  --query "artificial intelligence regulation" \
  --start 20240101000000 \
  --end 20240131235959 \
  --limit 150 \
  --output corpus/generated/gdelt_events

Use demo/gdelt_events_queries.yaml.

4. Cyber Threat Intelligence¶

The committed cyber dataset is a bounded MITRE ATT&CK Enterprise export with retained ATT&CK and STIX IDs:

ATT&CK intrusion groups, campaigns, malware, tools, techniques, and mitigations;
explicit uses and mitigates relation lines with human-readable target names.

Suggested command shape:

uv run python corpus/adapters/cyber_threat_intel.py \
  --limit 60 \
  --output corpus/cyber_threat_intel

Use demo/cyber_threat_intel_queries.yaml.

5. Evaluation Loop¶

For each generated dataset:

./scripts/start-all.sh
docker exec -e PYTHONPATH=/app/plugins rag-showcase-backend \
  python /app/ingest/ingest.py /app/<corpus_path-from-compare-datasets.yaml>

MATRIX_QUERIES_FILE=demo/<dataset>_queries.yaml \
MATRIX_RESULTS_FILE=<dataset>_matrix.json \
uv run python compare/run_matrix.py

JUDGE_MATRIX_FILE=<dataset>_matrix.json \
JUDGE_RESULTS_FILE=<dataset>_judgments.json \
uv run python compare/judge.py

uv run python compare/report_datasets.py --output docs/dataset-complexity-report.md

After a successful run, copy the raw result files from compare/results/ into docs/results/, update compare/datasets.yaml from candidate to measured, and regenerate the report.