myxce/.kiro/specs/scheduler-sim-decision-trace/brief.md

# Brief: scheduler-sim-decision-trace

## Problem

People debugging scheduler behavior in the simulator (`xce_scheduler --sim -i`) can see *what* the scheduler decided: dispatch or pend, the jobid, and at best a coarse `xce_pendreason_t`. They cannot see *why*. Nothing in the REPL answers these questions:

- Why was this job picked at this point in the session?
- Which resreq (COMPOUND primary or ALTER alternative) was tried, and which one won?
- Which plans did the planner generate and try, and in what order (seqjob single plan; parjob k = 1..maxTasks with balanced buckets)?
- Which hosts were considered for each plan and bucket, which were rejected, and why? This covers select mismatch, per-resource shortfall with numbers, shared-resource shortfall, and a limit hit.
- What did the final allocation hold (per-host/per-resource amounts, shared allocs, the limit vres debited)?

Most of this information is thrown away today:
- `host_iter.c` rejects hosts with a bare `continue`, so no reason is recorded.
- `unsatisfy_` computes shortfalls into `ctx->unmeet` but only TRACE-logs them.
- ALTER keeps only the last alternative's reason (`scheduler.c:427` FIXME).
- A limit violation is indistinguishable from `PREASON_NO_RESOURCE`.
- Plan and bucket attempts leave no record.

The only workaround is printf debugging or `-vvv` tlog output. That output has no categories, is written to a log file outside the REPL, and cannot be filtered to one job.

## Current State

- The sim loop runs the **same** picker and allocator threads and session protocol as live (`sim_loop.c:1386`, `scheduler-driver-equivalence.md`). Decisions flow allocator → `core->decisions` → the sim drain loop (`sim_loop.c:~1607`), where `sim_print_decision` (`sim_loop.c:479`) echoes only the op and jobid.
- REPL: `info session|config|summary|jobs|queue|worker|resource|limit|break`, decision breakpoints (`repl_break.c`), `set` params (`repl_params.c`), and the continue-mode verbosity/rollup (`scheduler-sim-repl-verbosity`). `info job` shows no pend reason.
- Core isolation: `core*.c`/`sim*.c` may not do I/O (`test_core_isolation.c`, `test_sim_isolation.c`); core→driver communication goes through NULL-checked callbacks (`cache_clear_cb`, `unsatisfiable_*_cb`, `on_decision` in `core.h`).
- Decision seams (from codebase map, 2026-09-28):
  - Picker: `builtin_fcfs_next_job` (FCFS keys: queue priority, then job priority, then queue_time); `sched_election.c:133`.
  - Allocator entry, `plugin_job_election` (return ignored), and the final decision: `sched_election.c:226/236/262/273`.
  - Resreq branch: `sched_make_allocation` `scheduler.c:441`, COMPOUND per primary `:351`, ALTER per alternative `:421`, and the primary chokepoint `sched_make_primary_alloc_` `:235`.
  - Plans, buckets and the shared check: the two `my_alloc()` copies in `seqjob_allocator.c` / `parjob_allocator.c`. **These are being replaced by `sched_allocator_run()` under `scheduler-allocator-runner`.**
  - Host verdicts: `host_iter.c:136` (select), `:158` (resource satisfy, `ctx->unmeet`), and `sched_try_alloc_on_host_` `resource.c:497-573` (prepare fan-out, `limit_prepare_allocation`).
  - Allocation result: `sched_make_allocs` `scheduler.c:1920`, `JobAlloc` (capnp `:441`).
- The unsatisfiable-resreq cache is wired in **live mode only**, so sim always runs the full allocator and a trace never has to explain a cache hit.

## Desired Outcome

In the simulator REPL, an operator can turn decision tracing on at runtime, for every job or for a chosen set of jobids. As they `step`/`continue`, each traced job's decision is printed in the REPL console with a readable, nested explanation:

1. **Job selection**: why this job came up now (picker, queue, sort keys and position), and the election result.
2. **Resreq selection**: each resreq tried (primary / COMPOUND member / ALTER alternative), in order, with its outcome and why it was accepted or rejected.
3. **Planner**: each plan the planner generated and tried (host count, bucket sizes, slots), in order, and which one was committed or why all failed.
4. **Host selection within a plan**: for each bucket, the hosts considered in order and the verdict for each. For rejections: select-expression mismatch, per-resource shortfall (`total / inuse / uncommitted / wanted`), shared-resource shortfall, limit hit (naming the limit and its scope instance), host already claimed by an earlier bucket, or rolled back.
5. **Allocation details**: the final per-host/per-resource amounts, shared allocations, and the limits applied or debited. For a pend, the final reason set.

`info trace <jobid>` re-prints a job's most recent trace. Optionally, `trace ... file <path>` mirrors the same text to a file. When trace is off, REPL output is byte-identical to today. The live scheduler never records a trace.

## Approach

**Trace buffer per decision, rendered by the sim driver (the user's choice, 2026-09-28).**

- A new core module (e.g. `src/scheduler/sched_trace.c` + `include/sched/trace.h`) defines structured trace records and a per-allocator-thread buffer. The allocator thread is single-threaded per decision, so no locking is needed.
- Hooks at the seams above append records only when trace is enabled for the job being allocated. When disabled, the gate is one NULL-pointer / flag check, following the existing `unsatisfiable_*_cb` and `recording` patterns. Only the sim driver ever enables it, so the live scheduler pays a single branch per hook and nothing more.
- The picker-side "why selected" facts are captured when the job is handed to the allocator (via `jbuf`) or recomputed from the sort keys at render time, so the picker thread never writes to the allocator's buffer.
- The finished buffer is attached to `xce_decision_t` and travels through `core->decisions`. The sim drain loop, which already owns console output, renders it in the REPL next to the decision echo, stores the latest trace per job for `info trace`, and optionally mirrors it to a file. The core does no I/O.
- REPL surface (names to be settled in requirements): `trace on all | job <ids> [file <path>]`, `trace off`, `trace status`, `info trace <jobid>`, and `help` entries. Traced decisions are always echoed in full, even in `continue` mode where routine output is collapsed.

**Rejected alternatives:**
- Streaming records to a callback as they happen: I/O would run on the allocator thread, output from the picker and allocator threads would interleave, and there would be no per-job recall for `info trace`.
- A tlog trace level: tlog has no categories and one global level, it writes to a log file rather than the REPL, and the extra log calls would sit in the live path.
- A live-mode trace option (CLI/config/`schedAdmin`): rejected by the user. It would put a performance cost on the production scheduler, and the simulator is the investigation tool.
- A headless `--sim-trace` flag: rejected by the user. Tracing is an interactive REPL feature only.

## Scope

- **In**:
  - Trace record model and per-decision buffer.
  - Hook points across job pick, election, resreq branch, planner (plan/bucket), host iterator verdicts, per-host prepare/limit checks, shared-resource check, and final allocation or pend reason.
  - Attaching the trace to the decision.
  - Rendering in the sim REPL; per-job last-trace store; `info trace`.
  - Runtime on/off with all-jobs or job-set filter; optional file mirror.
  - A cap on trace size per decision, with a visible truncation marker.
  - Tests proving zero behavioral change: identical decisions and identical `--sim-verify` results with trace on vs off, and identical REPL output with trace off.
- **Out**:
  - Any live (non-sim) enablement.
  - Headless-sim CLI flag.
  - Changing pend-reason content published to the controller or compared by `job_preason_eq` / `--sim-verify`.
  - New scheduling, picker, or allocator behavior.
  - Fixing the defects the trace will expose. These are logged as separate bugs, e.g. the ALTER reason leak/overwrite, and the shared-check loop bound (`alloc_req.shared.count` vs `unmeet.count`).

## Boundary Candidates

- **Trace model + buffer + gating** (core, no I/O): record types, per-thread buffer, enable predicate, size cap, attach to `xce_decision_t`.
- **Instrumentation hooks**: picker/election (`sched_election.c`, `builtin_fcfs.c`), resreq branch (`scheduler.c` `sched_make_*`), planner/buckets (inside `sched_allocator_run()` once `scheduler-allocator-runner` lands), host verdicts (`host_iter.c`, `resource.c` including `unsatisfy_` shortfalls), limits (`plugins/limit.c` prepare path, to name the limit behind a vres shortfall).
- **Sim/REPL surface**: `trace` command + `info trace` + help; rendering (nested, human-readable units reusing `scheduler-sim-repl-verbosity` Req 6 formatting); integration with step/continue echo and the rollup; optional file mirror.

## Out of Boundary

- Live scheduler observability of any kind.
- Controller-visible pend reasons (`JobPendReason`), recording/replay file format (`ReplayDecision`), and `--sim-verify` equality semantics.
- `info limit` and limit-usage display (owned by `scheduler-sim-diagnostics`).
- The allocator loop restructuring itself (owned by `scheduler-allocator-runner`).

## Upstream / Downstream

- **Upstream**:
  - `scheduler-allocator-runner`: **hard sequencing dependency (user decision)**. Plan/bucket hooks go into the single `sched_allocator_run()`, not the two `my_alloc()` copies.
  - `scheduler-core`: REPL, sim loop, decision pipeline.
  - `scheduler-sim-repl-verbosity` / `scheduler-sim-step-verbosity`: echo gate, rollup, value formatting.
  - `scheduler-allocation`: seqjob/parjob plan semantics.
- **Downstream**:
  - `scheduler-preemption` (adds PREEMPT decisions; a trace of victim selection is a natural later extension).
  - Future `scheduler-sim-diagnostics` capabilities.
  - Bug triage that uses traces as evidence.

## Existing Spec Touchpoints

- **Extends**: none directly. This spec **takes over** "Seed capability 2 — Decision-reasoning trace" from `scheduler-sim-diagnostics/brief.md`; that brief should point here.
- **Adjacent**:
  - `scheduler-allocator-runner` (hook placement; its "no pend-reason content change" rule also binds this spec).
  - `scheduler-preemption` (also extends `decision.h`; coordinate field additions).
  - `scheduler-sim-discrete-clock` (touches `sim_loop.c`).
  - `libxce-alloc-preason-equality` (owns `job_preason_eq`; the trace must not alter compared reason content).
  - `scheduler-sim-diagnostics` (REPL `info` family).

## Constraints

- **Performance on live**: the user's explicit reason for sim-only. With trace off, which is always the case in live, each hook costs at most one predictable branch: no allocation, no formatting, no argument evaluation. Record formatting happens only in the sim driver.
- **Determinism**: enabling trace must not change any decision, alloc, pend reason, accounting state, or `--sim-verify` outcome. Must hold in both `step` and `continue` modes.
- **Core isolation**: no I/O in `core*.c`/`sched_*` code; rendering and file writes live in the sim/REPL layer (`test_core_isolation.c`, `test_sim_isolation.c`, `test_no_bare_printf.c`).
- **Threading**: the picker and allocator are separate threads with `jbuf` capacity 1 as a determinism invariant. Trace must not add cross-thread synchronization or change hand-off order.
- **File size**: `scheduler.c`, `builtin_fcfs.c`, `repl_info.c` already exceed the 800-line cap; new code goes in new files (`sched_trace.c`, `repl_trace.c`).
- **Memory**: per-decision buffer with a configurable cap and a truncation marker. Large parjob × many-host sessions must not explode memory.
- **TDD**: testsuite tests before C changes (`testing.md`).