Skip to content

11. Typed decisions

nnx.decisions is NNx's inference branch: a provider-neutral way to ask a typed question and get labelled probabilities back, instead of reading positional logits. A question is one of three primitives. A provider declares what it can answer and returns validated results. Three adapters are providers:

  • the fixed-head adapter turns a trained NNx classifier into a provider;
  • the NLI adapter scores candidates supplied at inference with a caller-supplied NLI model;
  • the Jev adapter asks hosted Jev models through the TypeSafe SDK.

Nothing here trains, exports or executes actions. Importing it starts no provider or model backend and needs no hosted-SDK extra.

from nnx.decisions import Choice, FixedHeadProvider, Option

question = Choice(
    "Which animal is in the photo?",
    (Option("sp-fox", "A fox"), Option("sp-cat", "A cat"), Option("sp-dog", "A dog")),
)
provider = FixedHeadProvider(model, option_map={"sp-cat": "cat", "sp-dog": "dog", "sp-fox": "fox"})
results = provider.decide(question, photos)     # one ChoiceResult per row
results[0].distribution                         # (("sp-fox", 0.01), ("sp-cat", 0.94), ("sp-dog", 0.05))
results[0].top                                  # "sp-cat"

examples/decision_fixed_head.py runs this end to end on a deterministic local model.

1. Questions

Primitive Holds Answered by
Choice(prompt, options) 2+ Option(id, description) with unique ids a distribution over the options
Boolean(prompt) the prompt one p_true in [0, 1]
Score(prompt, levels) 2+ ordered Option(level_id, description), lowest first a distribution over the levels

Options may be given as (id, description) pairs; anything else (a bare string, say) raises InvalidDecisionRequest. kind ("choice", "boolean", "score") is the discriminator. state() / question_from_state(...) round-trip a question as plain data. Malformed state raises InvalidDecisionRequest, never a bare KeyError.

Bookkeeping ids vs model-facing text. An option's id is how results are keyed and never reaches a model; its description is what a model reads. question.model_view() returns exactly the model-facing part: the prompt and the option texts in order, no ids. So a text provider can render a question without leaking routing keys such as "team-7f3a".

Digest. question.digest() is a SHA-256 over the kind, the prompt and every option's id and description in order. Reordering the options or rewording a description changes it. Every result records the digest of the question it answers, so a cached or batched result can never be matched to a different question.

2. Results and validation

ChoiceResult and ScoreResult carry a distribution: (id, p) pairs in the question's order. It must hold 2+ unique ids, each p finite in [0, 1], summing to 1 within PROBABILITY_TOLERANCE = 1e-6. BooleanResult carries one p_true (p_false is derived). A provider's own output stays in raw (kept out of equality) and never mixes with the normalized fields.

validate_response(question, response) is the one validator every provider uses. For a Choice or Score, response is the provider's keyed output — a mapping or (id, p) pairs in any order. It is reordered into the question's order. These raise InvalidDecisionResponse:

  • a missing, duplicate, unknown or unlabeled (empty or non-string) id;
  • a probability outside [0, 1] (one too large for a float included);
  • a distribution that does not sum to 1.

A malformed distribution is never renormalized: dividing by the sum would hide a provider bug. For a Boolean, response is p_true or {"true": p, "false": 1 - p}.

Scores are ordinal. ScoreResult.expected_index is sum(i * p_i) over zero-based levels. It is a position between levels, useful for ranking. But the levels' spacing is undefined, so it is not an interval-scale score. A provider that reports its own number (for example a 1–10 rating) passes it as vendor_score, which is kept separate from the distribution.

3. Providers and capabilities

A DecisionProvider has two methods: capabilities() and decide(question, inputs) -> list[result] (one result per input). Capabilities declares:

  • primitives — the question kinds it answers;
  • modalities — the inputs it reads ("tensor", "text", …);
  • dynamic_labels / labels — whether a request may bring any option set, or must map onto a fixed label space;
  • max_batch — the most inputs per call;
  • inference / training / export — what the backend can do. A hosted provider declares inference only; it inherits no training or export capability.

capabilities().check(question, modality=..., batch_size=..., label_ids=None) raises UnsupportedCapability for anything undeclared. Without dynamic labels, that includes a Choice or Score whose ids are not exactly the declared labels. The same holds for label_ids, the provider's own mapping of those ids. With no labels declared, option sets are not checked. Providers call it before any model call or network I/O.

4. The fixed-head adapter

FixedHeadProvider(model, labels=None, option_map=None, ordinal=False, max_batch=None, name="nnx.fixed_head", question=None) adapts a trained NNModel. The head decides what it can answer:

Head Answers
categorical (softmax over C classes) Choice; also Score with ordinal=True, whose levels must follow the head's class order
one Bernoulli logit (BCEWithLogitsLoss, or a one-output multilabel task) the one Boolean it was trained for (question=)
regression nothing — rejected at construction
multi-output multilabel nothing — rejected at construction when the head's width is known (a task's count or a built-in net's output_dim). Otherwise it is refused on the first call, after the model ran.
  • Label space. labels names the head's columns in order. It defaults to the model's TaskSpec labels and must equal them when both are given.
  • Label count. A label count that differs from the head's width is rejected. When the width is known (a task's count or a built-in net's output_dim), this happens at construction. Otherwise it is rejected with InvalidDecisionRequest on the first call.
  • Request ids. A request's option ids must be exactly those labels (in any order), or option_map must map them one-to-one onto the labels. An unseen or missing label raises UnsupportedCapability before the model_calls counter advances.
  • A Boolean head's question. A one-logit head is one probability bound to no prompt, so question=Boolean(...) names the question it was trained for. It is required for such a head and refused for a categorical one (which answers by its labels). ordinal= and option_map= are refused for a one-logit head: a Boolean has no options to order or map.
  • Other Booleans. Any other Boolean — another digest, however close its wording — raises UnsupportedCapability before the model_calls counter advances. So a decision job refuses it and a benchmark records it unsupported, never answered with the head's probability.
  • Columns. Logits come from model.predict_proba (the prediction contract). Probabilities are recomputed from them in float64, so a half-precision head still meets the 1e-6 tolerance. They are reordered into the request's option order and passed through validate_response. Each row's logits (a copy) become raw. An empty batch returns [] without calling the model.
  • Modes. Prediction runs in eval mode and restores every submodule's training mode, on success and on failure.
  • Failures. An exception from the model itself becomes ProviderFailure, with the original error as __cause__.
  • Checking without calling. check(question, inputs) runs every check decide makes before the model call and calls nothing. Those checks cover question, modality, batch size and label space, or a Boolean's digest. A decision job (§7) uses it to refuse a request up front.

5. The NLI baseline adapter

A fixed head answers only the labels it was trained on. NLIProvider is a local label-conditioned baseline. It scores a text (the premise) against candidate descriptions that arrive with each request, each rendered into a hypothesis. Scoring uses a natural-language-inference (NLI) cross-encoder the caller supplies. So no candidate needs to exist when the provider is built, and no output head is resized.

from nnx.decisions import Boolean, Choice, NLIProvider

provider = NLIProvider(
    model, tokenizer,                    # loaded by the caller (a local path, a pinned revision)
    entailment_id="entailment",          # or the class id; names resolve through model.config.label2id
    contradiction_id="contradiction",
    hypothesis_template="This text is about {}.",
    revision="<the revision you loaded>",
)
provider.decide(Choice("Topic?", (("t-sport", "sports"), ("t-econ", "the economy"))), texts)
provider.decide(Boolean("This review is positive."), texts)
  • Choice scoring, explicit. A Choice scores every (text, candidate) pair and softmaxes the entailment logits across the candidates (choice_scoring="entailment_softmax"). Logits log(3) and 0 give (0.75, 0.25), keyed by the request's option ids in its order. Only the descriptions are scored: the Choice's prompt is not part of any hypothesis, so write it into hypothesis_template when it matters.
  • Boolean scoring, explicit. A Boolean scores one pair per text, its prompt rendered by boolean_template. It softmaxes that pair's contradiction and entailment logits alone (boolean_scoring="entailment_vs_contradiction"). 0 and log(4) give p_true = 0.8; texts are never normalized together. Score is unsupported and refused before any model call.
  • Validated settings. The entailment and contradiction ids — given or resolved from names — must differ. When the model's config declares a class count (num_labels), they must fit it and the logits must have exactly that many classes. A label name must be one its config knows. An integer id that contradicts the config's own entailment / contradiction names is refused.
  • Templates and policies. Templates hold exactly one bare {} (no conversion or format spec). Unknown scoring or truncation policies are refused at construction.
  • The provider contract. NNx imports no NLI library and downloads nothing. Importing nnx.decisions or constructing the provider calls no from_pretrained. The model is called as model(**encoded), returning logits or an object with .logits.
  • The tokenizer call. The tokenizer is called HuggingFace-style: tokenizer(premises, hypotheses, truncation=..., max_length=..., padding=True, return_tensors="pt"). It also gets unpadded measurement calls with truncation=False, padding=False. Their attention_mask sums or input_ids row lengths give each pair's token count.
  • Execution. Pairs go in chunks of pair_batch_size (the last may be shorter) to the model's device. The model runs in eval mode under no-grad. Every submodule's training flag is restored on success and failure. A model error becomes ProviderFailure.
  • Truncation, reported. Pairs are measured untruncated first. truncation="only_first" (default) cuts the premise to max_length and marks which pairs were cut in raw["truncated"]. A hypothesis that leaves no room for the premise (measured with an empty premise) is refused (InvalidDecisionRequest) before any model call. truncation="error" refuses an over-long request before any model call.
  • Records, never calibrated. Each result's raw holds the pair logits, the truncation report and provider.record(), with "calibrated": False. The record holds the templates, label ids, scoring methods, truncation policy and model revision.
  • Transfer is empirical. These are zero-shot scores from a model trained for another task, not calibrated probabilities. How well they transfer to a decision task stays empirical. Measure it on labelled records, as examples/decision_nli.py does. Report accuracy, macro-F1, categorical NLL and Brier, and record the split, revision and settings.

It does not extend nnx.embeddings.embed_texts or the FAISS export, whose signatures are unchanged. Those embed texts with a bi-encoder you trained; this scores pairs with a cross-encoder you supply.

6. The Jev adapter

JevProvider (the jev extra: pip install "thekaveh-nnx[jev]") answers Choice, Boolean and Score questions about text on Jev models. It goes through the TypeSafe Python SDK's system_one call; NNx adds no transport stack of its own. The adapter is tested against typesafe-sdk>=0.7,<0.8 (nnx.decisions.jev.SDK_RANGE). Importing nnx or nnx.decisions never imports the SDK. When the SDK is missing, building a provider without an injected client raises an ImportError naming thekaveh-nnx[jev].

from nnx.decisions import Boolean, Choice, JevProvider, Score

with JevProvider(model="jev-1.13.0") as provider:          # TYPESAFE_API_KEY from the environment
    topic, urgent, severity = provider.decide_many(
        [Choice("Topic?", (("t-bill", "billing"), ("t-ship", "shipping"))),
         Boolean("Does this need an answer today?"),
         Score("How severe?", (("s0", "minor"), ("s1", "serious"), ("s2", "critical")))],
        texts,                                              # one system_one request per text
    )
topic[0].distribution          # (("t-bill", 0.7), ("t-ship", 0.3)), in the question's order
topic[0].raw                    # {"provider": "jev", "model": "jev-1.13.0", "requested_model": "jev-1.13.0",
                                #  "request_id": ..., "usage": {...}, "provider_confidence": 0.7}
  • Translation. The prompt is the instructions, and each question becomes a Jev type:
    • a Choice becomes a choice keyed by its options' descriptions — the model-facing text; option ids are never sent;
    • a Score becomes a score whose criteria are its levels' descriptions, lowest first;
    • a Boolean becomes a noul.
  • Batching. Every question of one decide_many shares the request for each text (max_questions is unbounded), so a decision job batches them into it (§7).
  • Answers. Answers are realigned to each question's own option order and checked by validate_response (§2). A Choice whose options repeat a description is refused before any request. A Score's expected score stays in vendor_score.
  • Confidence. A Choice's or Score's confidence is Jev's own certainty in its selection, recorded as raw["provider_confidence"]. It is not a probability that the answer is correct — use the distribution, and measure correctness on labelled records (§8). A Boolean has none.

  • Metadata. raw is plain JSON with these fields:

    • provider;
    • the model the service resolved (even when an alias was requested);
    • requested_model when one was set;
    • request_id and the reported usage token counts.

    A field the service did not report is absent. Clients, headers and credentials are never recorded, and record() (the benchmark identity) holds none either.

  • Retries. The SDK's RetryPolicy owns retries. Pass retry= when the provider builds its client, or configure the client you inject. The adapter sends each request once and adds no retry loop, nor does a decision job or the benchmark.

  • Failures. What still fails is typed and keeps the SDK error as __cause__. When the service answered, it also keeps its request_id (a timeout or a lost connection has none). All are ProviderFailures:

    • JevTimeout (also a TimeoutError);
    • JevAuthenticationError (401 / 403);
    • JevRateLimited (429, with retry_after_ms);
    • JevMalformedResponse (also an InvalidDecisionResponse): an unparseable body, a missing answer or a distribution that does not fit;
    • JevError for the rest.

    Crossing a decision job or the benchmark keeps the type and request id (JobFailed.__cause__, a record's reason).

  • Cost. Every text is its own request, so one provider call over n texts sends n requests. A benchmark's Budget.max_calls and a job's Limits.max_requests count provider calls, not requests. Set max_batch to bound the requests per call. A failure on one text stops the call; answers already received for earlier texts are not returned.

  • Injected clients. A client / async_client you inject stays yours. The provider never closes it and never builds a second client beside it.
    • With only a sync client injected, the provider is sync-only. adecide / adecide_many are None, so DecisionJob.arun runs it in its worker thread, one call at a time.
    • With only an async client injected, the sync methods are refused.
  • Built clients. With none injected, the provider builds its own, lazily, at the first call. It uses api_key / base_url / timeout / retry or the SDK's environment variables; a missing key fails as a JevError. close() (with) closes a built sync client, and aclose() (async with) closes both — after an error or a cancellation too.
  • Event loops. A built async client belongs to one event loop. A call or aclose() from another loop drops it with a ResourceWarning, so close it with async with inside each asyncio.run.
  • Async. adecide / adecide_many use the async client (built, or injected); DecisionJob.arun awaits them directly.

Ordinary tests need no credentials or network. tests/test_decision_jev.py drives a recording fake of system_one that returns real SDK responses. examples/decision_jev.py runs real SDK clients over an offline mock transport, sync and async.

The opt-in live smoke, scripts/smoke_jev_live.py, sends one request to the pinned jev-1.13.0 (never an alias). It prints the resolved model, SDK version, request id, token usage and wall time. It spends real quota, so CI never runs it.

7. Decision jobs: batching and chaining

A DecisionJob is an immutable, deferred description of decision work. Building one calls no provider, runs no callback and draws no RNG; run and arun are the only effect boundaries.

from nnx.decisions import DecisionJob as Job, Follow, Limits

flags = Job.collect({                                   # independent questions, keyed
    "goal": Job.ask(Boolean("Mentions goal"), id="goal"),
    "bank": Job.ask(Boolean("Mentions bank"), id="bank"),
})
routed = Job.ask(topic, id="topic").then(               # a dependent question
    lambda answers: Follow(Job.ask(keeper, id="keeper"), state=sports_texts(answers))
)
result = Job.collect({"flags": flags, "routed": routed}).run(provider, state=texts, limits=Limits(max_questions=8))
result.value["flags"]["goal"]                          # one result per text
result.outcomes["keeper"].kind                         # "answered"
result.calls                                           # provider calls made
Builder Value
Job.ask(question, id=..., policy=None, model_id=None) the provider's results, one per input row; with an abstention policy (concepts §19), the selective decisions
Job.collect({key: job, ...}) a mapping of each key to its job's value, in the given key order
job.map(fn) fn(value): a pure function, no provider call
job.then(fn) fn(value) returns Follow(next_job, state=...); the value is the next job's
  • Batching. Independent questions over the same state are sent in the fewest calls: ceil(count / cap) per state object. Continuations that return the same object share it; equal but distinct objects do not. Questions go in the order they were collected, keeping their ids and the collect key order.
  • The batch cap. cap is the provider's. A provider with decide_many(questions, inputs) declares max_questions (None: any number per call). A provider with only decide, such as FixedHeadProvider, answers one question per call. Limits.max_questions lowers it.

  • Refused before any call. These raise InvalidJob with no call made:

    • duplicate question ids;
    • a question the provider cannot serve (its check(question, inputs) when it has one, else capabilities().check(...));
    • a Limits.max_tokens cap the provider cannot enforce (it needs count_tokens(questions, inputs));
    • an abstention policy whose labels or model_id do not fit its question (checked by ask).

    A continuation's questions are checked when the continuation runs, before they are sent.

  • Dependent questions. A then continuation runs once, after its prerequisite succeeded. It maps the answer explicitly into the follow-up state; it never feeds a sibling question. Limits.max_depth bounds nesting and Limits.max_requests the run's calls (JobLimitExceeded).

  • Answers are checked. A call's answers must hold one result list per question and one result per input row. Each must be a result of that question (its digest). Answers in the wrong order, short rows or a flat list are an InvalidDecisionResponse failure, never filed under the wrong id.
  • Fail-fast. Nothing more is scheduled when a call raises, or when a provider hook fails. A hook fails when its check, capabilities() or count_tokens raises anything but a declared refusal.
  • What JobFailed carries. JobFailed carries four fields: outcomes (every outcome so far, in scheduling order) and completed (the answered ones). Also failed (the questions of that call) and skipped (known questions never sent).
    • Continuations waiting on a failed answer never run, and nothing completed is re-run.
    • A job refused before any call has no outcomes and nothing skipped.
    • The error pickles across a process boundary. A provider error that does not survive the round trip is replaced by a ProviderFailure naming it.
    • The job never retries: retries belong to the provider.
    • A partial result is only what the error carries: JobFailed has no value.
  • Async. await job.arun(provider, state=..., limits=..., cancel=None) runs up to Limits.max_concurrency calls at once through the provider's adecide_many / adecide. arun still needs the provider's synchronous decide (the decision protocol); adecide / adecide_many are used when present.
  • Synchronous providers. A provider with only synchronous methods is called in a worker thread, never alongside another call. So its methods need not be thread-safe. A running thread cannot be interrupted, so the run waits for it before returning. A cancellation that arrives meanwhile is delivered once it is done.
  • The cancel event. Setting the cancel event stops scheduling; no continuation runs after it. The run returns a JobResult with status="cancelled" and no value.
    • The event is an asyncio.Event of the running loop. One bound to another loop is an InvalidJob, never a cancellation.
    • The event counts if set at any point, even if cleared again.
    • A cancel set in the same tick as the last answer still cancels the run: its result has no value.
  • Other cancellations. A cancellation the provider raises itself is a provider failure, not a cancelled job. Cancelling the task cancels only the job's own tasks and re-raises.
  • Timeouts and limits. Limits.timeout raises JobTimeout. A request or depth limit lets the calls already in flight finish, then raises JobLimitExceeded. A request whose call began is reported with sent=True, never as rolled back: whatever the provider did with it stays done.
  • Hooks in arun. The provider's hooks (check, capabilities(), count_tokens) are synchronous and run on the event loop in arun. Keep them local and fast (no network round trip). The timeout is checked between hook calls, so a slow hook overruns it by at most one call.
  • Outcomes. result.outcomes[id].kind is one of four kinds, and the four never blur:
    • "answered" (rows may still be "abstained" under a policy);
    • "failed" (with error);
    • "skipped" (never sent: a failure, a limit or the timeout stopped scheduling);
    • "cancelled" (with sent): the run was cancelled, a sibling call failed first, or the timeout elapsed while the call was in flight (arun).
  • The provider is borrowed. run and arun never close it, and it stays usable after a failure or a cancellation.
  • Serialisation. A job of ask and collect only pickles as plain data (job.state()). A job holding a runtime function (map, then) refuses to pickle: rebuild it where it runs.

Limits of value equivalence. Two equalities hold when the provider is deterministic and answers a question the same way whatever it is batched with. job.map(lambda x: x) and job have the same value, and job.map(f).map(g) equals job.map(lambda x: g(f(x))). A sampling provider, or one whose answer depends on the other questions in its call, keeps the job's call structure but not that equality.

Batching is computational, not statistical. Answers asked together are separate marginals, and the job never multiplies them into a joint probability.

What it is not. It is not LogitsChainBuilder. That is a mutable builder of LM-decoding processors that build() sorts into a canonical order, while a job is immutable and describes provider requests. It is not an ExperimentPlan (nnx.plans). A plan compiles a training run, while a job only asks questions of an already-trained or hosted provider.

examples/decision_jobs.py runs batching, a continuation, fail-fast and the async path end to end.

8. Benchmarking providers

nnx.decisions.benchmark scores decision providers on identical samples, offline first. A live collection runs once. Every report after that is a replay of saved records: no provider call, no credentials, no network, nothing fitted.

from nnx.decisions.benchmark import Budget, Sample, collect, evaluate, read_records, write_records

samples = [Sample("pet-0", question, row, "sp-cat", family="pets"), ...]
collection = collect(provider, samples, provider_id="fixed-head-v1", budget=Budget(max_calls=20))
write_records("records.jsonl", collection.records)           # once, live
report = evaluate(samples, read_records("records.jsonl"), split="animals-v1")   # any time, offline
print(report.text())
  • Samples carry the join key (id) and the typed question (its digest is the schema identity). They also carry the provider input, the true label and the task family (heldout=True families report apart). Finally, they carry the grouping unit (group) and the perturbation that produced them. A sample's input is a text, a tensor or array, or a tuple of them (several inputs per sample, batched part by part).
  • Perturbations. Perturbed samples keep their original's grouping unit. These helpers derive them:
    • permute_options;
    • redescribe (new label descriptions);
    • add_distractors and add_none_of_the_above;
    • add_context (long irrelevant context);
    • rewrite_input (a multilingual or adversarial rewrite of the text input).
  • Replay format. One Record per provider output, one JSON object per line (nnx.decision-record/1, strict JSON). Each record holds:
    • sample id, question digest, provider and status;
    • the answer (distribution or p_true), or the reason there is none;
    • model revision;
    • prompt_identity (by default a digest of the provider's record());
    • execution metadata (batch, size, partial_batch, seconds, attempt).

  • Coverage statuses. Records join samples by sample id and question digest, never by row position. Each sample has one status:

    • eligible (one answered record);
    • missing or duplicate;
    • mismatched (a record for another question digest);
    • invalid (an answer that does not fit its question);
    • unsupported or failed (each with its reasons).

    Records for no sample are the report's extra count. The CSV carries every count, so they add up to each slice's samples.

  • Capabilities. A provider may declare that it cannot serve a request. For instance, FixedHeadProvider declares so outside its label space (or its option_map), or when asked a Boolean other than its own. It declares through its own check(question, inputs) or, without one, its declared capabilities().

    • Such a request gives unsupported records with the provider's own reason, before any call and without spending budget.
    • A check that itself raises gives failed records, also without a call.
    • These records are counted in the slice's coverage, never in a metric's denominator, and never scored as wrong.
  • Batch size. Batches never exceed the provider's declared max_batch.
  • Budgets. collect needs an explicit provider, provider id and Budget(max_calls, max_samples=None). It attempts each batch once (retries are the provider's own). It stops when the budget is spent: the rest is missing, and Collection.stopped says why. It marks a batch the sample budget cut short as partial_batch.
  • Answer checks. Each answer is checked against its sample's question (digest) and put into the question's option order, one by one. A malformed answer fails its own sample only. A provider error or inputs that cannot form one batch fail the batch. Neither ends the collection.

  • Metrics, per slice (in_family, heldout, family:<name>, perturbation:<name>):

    • accuracy and macro-F1;
    • NLL, exact by default: +inf when a true label has probability 0; epsilon= floors it;
    • Brier, and ECE with reliability bins;
    • given an nnx.abstention policy, applied as given, selective coverage and risk. The policy applies to Choice and Score rows only. Boolean rows and rows the policy does not fit are left out of the coverage denominator, and its note says how many.

    Each is a MetricValue with its denominator, or unavailable with the reason; a note qualifies an available value. NLL and Brier follow the definitions of the named nll / brier metrics. One difference: the replay NLL is exact by default, where the named nll floors probabilities at 1e-12. A Boolean is the options ("true", "false").

  • Resources (Resources) say how cost was obtained: warmup, hardware, timing boundary, concurrency, batch count and seconds. They also say whether the numbers were measured (hardware required) or supplied. Anything not declared stays null.

  • Intervals. bootstrap_interval resamples grouping units (whole groups, never single rows of one) with a recorded seed. It runs over the same provider's records evaluate scores. It flags a degenerate interval: fewer than two units, no variation, bounds that coincide, or a non-finite bound such as an exact NLL of +inf. The reason says which.
  • Variant ids. Perturbation helpers give each variant a distinct id, or the id= you pass. The id is the original id, the kind and a digest of the change. The digest is the same in every process for text, numeric arrays and tensors, tuples of them and JSON-able values. An input with no stable digest needs id=.
  • Exports. to_json(), to_csv() and text() agree on units, eligible, failure and extra counts, unavailable states and notes (the CSV's note column). A non-finite value is "Infinity" in the JSON and the CSV.
  • Comparison. compare_reports gives b - a per slice and metric only for reports that share all of these:
    • the same split;
    • the same metric identity (metric set, epsilon, n_bins, policy);
    • the same sample set (a digest of the samples' ids, questions, labels and slicing), when both reports record it.
  • Duplicates. collect refuses duplicate sample ids before any call.

It is not a leaderboard. No paid remote run is a default, and it never tunes a threshold or a prompt on test outcomes. examples/decision_benchmark_offline.py collects once and replays with sockets disabled.

9. Offline teacher distributions

A decision provider's answers are distributions over a question's options. Stored, they can teach a student offline: nnx.paradigms.offline_distillation trains on them without calling the provider again:

from nnx.paradigms.offline_distillation import TeacherDataset, TeacherRecord, write_teacher_records

records = [
    TeacherRecord(
        sample_id=sample.id, question_id="pet-kind",
        candidates=tuple(option_id for option_id, _ in result.distribution),
        probabilities=tuple(p for _, p in result.distribution),
        teacher="fixed-head-v1", revision="r42", schema_digest=question.digest(),
        semantics="predictive",
        provenance={"source": "records.jsonl, 2026-09 collection", "training_rights": "declared: internal use"},
        label=sample.label,
    )
    for sample, result in answered
]
write_teacher_records("teacher.jsonl", records)
data = TeacherDataset(records, inputs, candidates=("sp-fox", "sp-cat", "sp-dog"), schema_digest=question.digest())
student.train(params=..., objective=data.objective(alpha=0.7))
  • What a record states. The sample and question ids, and the ordered candidate ids (the question's option ids) with their probabilities. Also the teacher and its revision, the schema digest (the question's digest()), the probability semantics and the provenance.
  • Record validation. A record without one of these fails validation. read_teacher_records names the path:line of a malformed line in its TeacherRecordError, a number too large for a float included.
  • Provenance is declared, not verified. provenance names the record's source and its training_rights, as the exporter declares them. The rights state how the record may train a student. NNx checks that the declaration is there; it cannot check that it is true. Whether a provider's terms allow training on its outputs is yours to establish.
  • Probabilities, not logits. Live distillation (kd_train_step_factory, kd_objective) softens a running teacher's logits at a temperature and scales the KL by T². A stored distribution has no logits to soften. So the offline objective is KL(teacher ‖ student) at temperature 1, plus an optional hard cross-entropy on the label — nothing invented, no T².
  • Agreement is not quality. The reports keep teacher agreement and imitation loss (over every record) apart from labelled quality (over the labelled records). Each has its own denominator. A student that imitates its teacher perfectly is wrong wherever the teacher is.

See Concepts §10.1 and examples/offline_teacher_distillation.py.

10. Errors

All are nnx.decisions.DecisionErrors, and their names are stable:

Error Raised for
InvalidDecisionRequest (also a ValueError) malformed questions: fewer than 2 options, duplicate or empty ids, empty text; misconfigured adapters
InvalidDecisionResponse (also a ValueError) responses that do not fit their question (see §2)
UnsupportedCapability undeclared primitives, modalities, batch sizes or label spaces — before any model call
ProviderFailure (also a RuntimeError) the backend failing on a valid, supported request
JevError and its JevTimeout, JevAuthenticationError, JevRateLimited, JevMalformedResponse a Jev request failing (§6): __cause__ is the SDK error, request_id the service's
JobError the base of the decision-job errors (§7); outcomes holds every outcome so far, completed the answered ones
InvalidJob (also a ValueError) a job that cannot run as described — before any call or, for a continuation's questions or return value, when the continuation runs
JobFailed a provider call failing inside a job (fail-fast): failed, skipped, __cause__
JobLimitExceeded a job's max_depth or max_requests stopping it before the next call
JobTimeout (also a TimeoutError) a job's timeout elapsing

10.1. Errors as values: nnx.result

Every API above raises. A caller may rather branch on a value — when validating a batch of user-supplied requests, say. For that, the opt-in nnx.result module offers Ok / Err and boundary wrappers; nothing else changes. The same flow both ways:

# Exceptions
try:
    question = question_from_state({"kind": "choice", "prompt": prompt, "options": options})
except InvalidDecisionRequest as error:
    report("invalid_decision_request", error)
    raise
try:
    results = provider.decide(question, texts)  # a provider's own InvalidDecisionRequest propagates
except UnsupportedCapability as error:
    report("unsupported", error)
except InvalidDecisionResponse as error:  # before ProviderFailure: a malformed Jev answer is both
    report("invalid_decision_response", error)
except ProviderFailure as error:
    report("provider_failure", error)

# Result
from nnx.result import decide_result, validate_decision_request_result

outcome = validate_decision_request_result("choice", prompt, options).bind(
    lambda question: decide_result(provider, question, texts)
)
# Ok(results) — or Err(BoundaryError(code="invalid_decision_request" | "invalid_decision_response"
#                                     | "provider_failure" | "unsupported", ...))
  • validate_decision_request_result(kind, prompt, options) builds the question from plain data inside its boundary. Any InvalidDecisionRequest becomes Err(code="invalid_decision_request"), and the provider is never called. That covers duplicate or empty ids, too few options, an empty prompt and an unknown kind.

  • decide_result(provider, question, inputs, policy=None, model_id=None) maps three error types to codes:

    • any InvalidDecisionResponse — an answer that does not fit the question, such as a JevMalformedResponse — becomes "invalid_decision_response";
    • any other ProviderFailure (an outage, a timeout, a refused credential, a rate limit) becomes "provider_failure";
    • UnsupportedCapability becomes "unsupported".

    The first two keep the error's request_id in the context. A policy that does not fit raises before the provider is called, and is not returned as an Err. That is a policy that is not an AbstentionPolicy, or one for a Boolean question, other labels or another model_id.

  • Codes follow the type. The codes follow the exception type, not whether a retry would help. A refused credential is a "provider_failure" too, so branch on the cause when that matters. A JevProvider composes as is: each typed Jev error is its cause.

  • Abstention. With an abstention policy, abstaining is a success: the Ok holds each row's SelectiveDecision, probabilities and reason included.
  • Each Err holds a BoundaryError(code, where, context, cause); any other exception propagates.
  • Combinators and unwrapping.
    • Combinators (map, bind, map_error, recover) never catch a callback's exception.
    • A bind / recover callback that returns a non-Result raises TypeError.
    • unwrap() on an Err raises UnwrapError with the typed error.
    • A Result has no truth value (Ok(0) is a success). Branch with isinstance(outcome, Ok) or match, which also narrow the type for a checker.

See Concepts §27 and examples/result_boundaries.py.

11. Consumers

Decision features share this digest, the kind discriminators and validate_response rather than defining their own. The schema and provider core of nnx.decisions depends on none of them, although the package re-exports the jobs, NLI and JEV features. The consumers are:

  • decision jobs (§7);
  • the provider benchmark (§8);
  • offline teacher-distribution datasets (§9);
  • the local label-conditioned baseline, NLIProvider (§5);
  • the Jev SDK adapter, JevProvider (§6);
  • the optional Result at fallible boundaries, nnx.result (§10.1).

12. What this does not do

  • It does not claim every classifier is a universal decision-maker: the fixed-head adapter answers only what its head justifies.
  • It does not execute actions based on a decision.
  • It does not assume that separately asked questions describe independent events; each question is answered on its own.
  • It does not call hosted models except through an adapter you build. JevProvider (§6) is the only one. It needs the jev extra and your credentials, and NNx never falls back to it silently. It does not fine-tune Jev models or ship credentials, and vendor numbers are not benchmarks — measure them (§8).
  • It does not download or train models: NLIProvider uses the NLI model the caller supplies, as is, and its scores are not calibrated. Training a student on stored answers is nnx.paradigms.offline_distillation (§9).