myxce/.kiro/specs/scheduler-sim-decision-trace/brief.md

Brief: scheduler-sim-decision-trace

Problem

People debugging scheduler behavior in the simulator (xce_scheduler --sim -i) can see what the scheduler decided: dispatch or pend, the jobid, and at best a coarse xce_pendreason_t. They cannot see why. Nothing in the REPL answers these questions:

• Why was this job picked at this point in the session?
• Which resreq (COMPOUND primary or ALTER alternative) was tried, and which one won?
• Which plans did the planner generate and try, and in what order (seqjob single plan; parjob k = 1..maxTasks with balanced buckets)?
• Which hosts were considered for each plan and bucket, which were rejected, and why? This covers select mismatch, per-resource shortfall with numbers, shared-resource shortfall, and a limit hit.
• What did the final allocation hold (per-host/per-resource amounts, shared allocs, the limit vres debited)?

Most of this information is thrown away today:

• host_iter.c rejects hosts with a bare continue, so no reason is recorded.
• unsatisfy_ computes shortfalls into ctx->unmeet but only TRACE-logs them.
• ALTER keeps only the last alternative's reason (scheduler.c:427 FIXME).
• A limit violation is indistinguishable from PREASON_NO_RESOURCE.
• Plan and bucket attempts leave no record.

The only workaround is printf debugging or -vvv tlog output. That output has no categories, is written to a log file outside the REPL, and cannot be filtered to one job.

Current State

• The sim loop runs the same picker and allocator threads and session protocol as live (sim_loop.c:1386, scheduler-driver-equivalence.md). Decisions flow allocator → core->decisions → the sim drain loop (sim_loop.c:~1607), where sim_print_decision (sim_loop.c:479) echoes only the op and jobid.
• REPL: info session|config|summary|jobs|queue|worker|resource|limit|break, decision breakpoints (repl_break.c), set params (repl_params.c), and the continue-mode verbosity/rollup (scheduler-sim-repl-verbosity). info job shows no pend reason.
• Core isolation: core*.c/sim*.c may not do I/O (test_core_isolation.c, test_sim_isolation.c); core→driver communication goes through NULL-checked callbacks (cache_clear_cb, unsatisfiable_*_cb, on_decision in core.h).
• Decision seams (from codebase map, 2026-09-28):
  • Picker: builtin_fcfs_next_job (FCFS keys: queue priority, then job priority, then queue_time); sched_election.c:133.
  • Allocator entry, plugin_job_election (return ignored), and the final decision: sched_election.c:226/236/262/273.
  • Resreq branch: sched_make_allocation scheduler.c:441, COMPOUND per primary :351, ALTER per alternative :421, and the primary chokepoint sched_make_primary_alloc_ :235.
  • Plans, buckets and the shared check: the two my_alloc() copies in seqjob_allocator.c / parjob_allocator.c. These are being replaced by sched_allocator_run() under scheduler-allocator-runner.
  • Host verdicts: host_iter.c:136 (select), :158 (resource satisfy, ctx->unmeet), and sched_try_alloc_on_host_ resource.c:497-573 (prepare fan-out, limit_prepare_allocation).
  • Allocation result: sched_make_allocs scheduler.c:1920, JobAlloc (capnp :441).
• The unsatisfiable-resreq cache is wired in live mode only, so sim always runs the full allocator and a trace never has to explain a cache hit.

Desired Outcome

In the simulator REPL, an operator can turn decision tracing on at runtime, for every job or for a chosen set of jobids. As they step/continue, each traced job's decision is printed in the REPL console with a readable, nested explanation:

1. Job selection: why this job came up now (picker, queue, sort keys and position), and the election result.
2. Resreq selection: each resreq tried (primary / COMPOUND member / ALTER alternative), in order, with its outcome and why it was accepted or rejected.
3. Planner: each plan the planner generated and tried (host count, bucket sizes, slots), in order, and which one was committed or why all failed.
4. Host selection within a plan: for each bucket, the hosts considered in order and the verdict for each. For rejections: select-expression mismatch, per-resource shortfall (total / inuse / uncommitted / wanted), shared-resource shortfall, limit hit (naming the limit and its scope instance), host already claimed by an earlier bucket, or rolled back.
5. Allocation details: the final per-host/per-resource amounts, shared allocations, and the limits applied or debited. For a pend, the final reason set.

info trace <jobid> re-prints a job's most recent trace. Optionally, trace ... file <path> mirrors the same text to a file. When trace is off, REPL output is byte-identical to today. The live scheduler never records a trace.

Approach

Trace buffer per decision, rendered by the sim driver (the user's choice, 2026-09-28).

• A new core module (e.g. src/scheduler/sched_trace.c + include/sched/trace.h) defines structured trace records and a per-allocator-thread buffer. The allocator thread is single-threaded per decision, so no locking is needed.
• Hooks at the seams above append records only when trace is enabled for the job being allocated. When disabled, the gate is one NULL-pointer / flag check, following the existing unsatisfiable_*_cb and recording patterns. Only the sim driver ever enables it, so the live scheduler pays a single branch per hook and nothing more.
• The picker-side "why selected" facts are captured when the job is handed to the allocator (via jbuf) or recomputed from the sort keys at render time, so the picker thread never writes to the allocator's buffer.
• The finished buffer is attached to xce_decision_t and travels through core->decisions. The sim drain loop, which already owns console output, renders it in the REPL next to the decision echo, stores the latest trace per job for info trace, and optionally mirrors it to a file. The core does no I/O.
• REPL surface (names to be settled in requirements): trace on all | job <ids> [file <path>], trace off, trace status, info trace <jobid>, and help entries. Traced decisions are always echoed in full, even in continue mode where routine output is collapsed.

Rejected alternatives:

• Streaming records to a callback as they happen: I/O would run on the allocator thread, output from the picker and allocator threads would interleave, and there would be no per-job recall for info trace.
• A tlog trace level: tlog has no categories and one global level, it writes to a log file rather than the REPL, and the extra log calls would sit in the live path.
• A live-mode trace option (CLI/config/schedAdmin): rejected by the user. It would put a performance cost on the production scheduler, and the simulator is the investigation tool.
• A headless --sim-trace flag: rejected by the user. Tracing is an interactive REPL feature only.

Scope

• In:
  • Trace record model and per-decision buffer.
  • Hook points across job pick, election, resreq branch, planner (plan/bucket), host iterator verdicts, per-host prepare/limit checks, shared-resource check, and final allocation or pend reason.
  • Attaching the trace to the decision.
  • Rendering in the sim REPL; per-job last-trace store; info trace.
  • Runtime on/off with all-jobs or job-set filter; optional file mirror.
  • A cap on trace size per decision, with a visible truncation marker.
  • Tests proving zero behavioral change: identical decisions and identical --sim-verify results with trace on vs off, and identical REPL output with trace off.
• Out:
  • Any live (non-sim) enablement.
  • Headless-sim CLI flag.
  • Changing pend-reason content published to the controller or compared by job_preason_eq / --sim-verify.
  • New scheduling, picker, or allocator behavior.
  • Fixing the defects the trace will expose. These are logged as separate bugs, e.g. the ALTER reason leak/overwrite, and the shared-check loop bound (alloc_req.shared.count vs unmeet.count).

Boundary Candidates

• Trace model + buffer + gating (core, no I/O): record types, per-thread buffer, enable predicate, size cap, attach to xce_decision_t.
• Instrumentation hooks: picker/election (sched_election.c, builtin_fcfs.c), resreq branch (scheduler.c sched_make_*), planner/buckets (inside sched_allocator_run() once scheduler-allocator-runner lands), host verdicts (host_iter.c, resource.c including unsatisfy_ shortfalls), limits (plugins/limit.c prepare path, to name the limit behind a vres shortfall).
• Sim/REPL surface: trace command + info trace + help; rendering (nested, human-readable units reusing scheduler-sim-repl-verbosity Req 6 formatting); integration with step/continue echo and the rollup; optional file mirror.

Out of Boundary

• Live scheduler observability of any kind.
• Controller-visible pend reasons (JobPendReason), recording/replay file format (ReplayDecision), and --sim-verify equality semantics.
• info limit and limit-usage display (owned by scheduler-sim-diagnostics).
• The allocator loop restructuring itself (owned by scheduler-allocator-runner).

Upstream / Downstream

• Upstream:
  • scheduler-allocator-runner: hard sequencing dependency (user decision). Plan/bucket hooks go into the single sched_allocator_run(), not the two my_alloc() copies.
  • scheduler-core: REPL, sim loop, decision pipeline.
  • scheduler-sim-repl-verbosity / scheduler-sim-step-verbosity: echo gate, rollup, value formatting.
  • scheduler-allocation: seqjob/parjob plan semantics.
• Downstream:
  • scheduler-preemption (adds PREEMPT decisions; a trace of victim selection is a natural later extension).
  • Future scheduler-sim-diagnostics capabilities.
  • Bug triage that uses traces as evidence.

Existing Spec Touchpoints

• Extends: none directly. This spec takes over "Seed capability 2 — Decision-reasoning trace" from scheduler-sim-diagnostics/brief.md; that brief should point here.
• Adjacent:
  • scheduler-allocator-runner (hook placement; its "no pend-reason content change" rule also binds this spec).
  • scheduler-preemption (also extends decision.h; coordinate field additions).
  • scheduler-sim-discrete-clock (touches sim_loop.c).
  • libxce-alloc-preason-equality (owns job_preason_eq; the trace must not alter compared reason content).
  • scheduler-sim-diagnostics (REPL info family).

Constraints

• Performance on live: the user's explicit reason for sim-only. With trace off, which is always the case in live, each hook costs at most one predictable branch: no allocation, no formatting, no argument evaluation. Record formatting happens only in the sim driver.
• Determinism: enabling trace must not change any decision, alloc, pend reason, accounting state, or --sim-verify outcome. Must hold in both step and continue modes.
• Core isolation: no I/O in core*.c/sched_* code; rendering and file writes live in the sim/REPL layer (test_core_isolation.c, test_sim_isolation.c, test_no_bare_printf.c).
• Threading: the picker and allocator are separate threads with jbuf capacity 1 as a determinism invariant. Trace must not add cross-thread synchronization or change hand-off order.
• File size: scheduler.c, builtin_fcfs.c, repl_info.c already exceed the 800-line cap; new code goes in new files (sched_trace.c, repl_trace.c).
• Memory: per-decision buffer with a configurable cap and a truncation marker. Large parjob × many-host sessions must not explode memory.
• TDD: testsuite tests before C changes (testing.md).