Skip to content

6.1 Corpus Overview

Corpus

1. Contents

corpus/ provides two committed showcase corpora plus generated raw inputs:

  • raw/ is generated by fetch_corpus.py from a MultiHop-RAG subset plus the hand-picked keyword docs in keyword_docs/ (rare identifiers like WIDGET-ERR-7741 that exercise the exact-keyword contrast). The populated corpus/raw/ directory is gitignored; fetch_corpus.py is the source of truth.
  • graph_native/ is committed source data: 10 concise, relation-dense real-world dossiers with explicit Relations: bullets. It is designed to test whether graph-rag benefits from entity bridges, shared actors, timelines, and relationship-chain prompts.
  • cyber_threat_intel/ is a committed bounded MITRE ATT&CK Enterprise STIX export: 60 dossiers spanning intrusion sets, campaigns, malware, tools, techniques, and mitigations with explicit uses / mitigates relation lines. It is the first heavier graph-native candidate for the expanded flavor run.

The measured baseline_curated rung in compare/datasets.yaml is scored against corpus/subset/ — a frozen, hand-curated 11-document slice (10 MultiHop-RAG articles plus keyword_docs/widget-error-codes.md) picked from corpus/raw/. Like raw/, it is gitignored because it holds third-party MultiHop-RAG text, so it is neither committed nor auto-regenerated. To re-run that rung, either materialize an equivalent slice under corpus/subset/, or point the dataset's corpus_path at corpus/raw/ — the fetched superset that start-all.sh ingests by default (a larger, fully reproducible baseline).

2. Setup

python3 corpus/fetch_corpus.py

This copies the keyword docs into corpus/raw/ and appends a MultiHop-RAG slice. The MultiHop-RAG slice needs the optional datasets package, which is kept out of the uv-managed deps (its transitive closure is heavy) — install it on demand:

python3 -m pip install datasets

If datasets is unavailable, the script degrades gracefully to keyword-docs only.

The graph-native and cyber-threat corpora are already committed and do not need a fetch step. Use demo/graph_native_queries.yaml with the comparison harness for graph-native and demo/cyber_threat_intel_queries.yaml for the cyber-threat slice.

3. Real-world graph dataset ladder

The next evaluation layer is tracked in compare/datasets.yaml and summarized in docs/dataset-complexity-report.md. It orders input datasets by graph complexity so the project can show how approach rankings change as the corpus moves from curated text to real semi-structured knowledge bases.

Candidate heavy datasets are mostly generated, not committed:

  • STaRK-Prime and STaRK-MAG via the optional stark-qa package;
  • OpenAlex scholarly graph slices;
  • GDELT event/news graph slices;
  • MITRE ATT&CK cyber-threat graph slices; the bounded corpus/cyber_threat_intel/ slice is committed for repeatable local runs.

Adapter commands and evaluation steps live in corpus/adapters/README.md.

4. Atlas ingestion profiles

compare/datasets.yaml maps every ladder dataset to an identically named profile in atlas.consumer.yml. Profile corpus paths are relative to the backend's read-only /app/corpus mount, so corpus/graph_native becomes graph_native in the Atlas manifest. Atlas owns discovery, parser fallback, Chonkie chunking, embedding, the RagBase_<profile> write, LightRAG upload, and extraction drain.

The showcase owns the dataset files and questions. After Atlas completes, it also builds RagContextual_<profile> from the exact plain chunks because contextual blurb generation is an approach-specific transform. Warm re-submission is idempotent at the Atlas job level; the measured ladder uses a cold reset per dataset so removed chunks and LightRAG graph state cannot leak between rungs.