docs(spec): scheduler-allocator-runner and scheduler-sim-decision-trace requirements

scheduler-allocator-runner: brief and requirements for moving the duplicated
seqjob/parjob my_alloc() body into one framework-owned allocator runner.

scheduler-sim-decision-trace: brief and requirements for a per-job decision
trace in the simulator REPL (converged after 9 review rounds). The
decision-reasoning seed capability in scheduler-sim-diagnostics now points to
this spec.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.kiro/specs/scheduler-allocator-runner/brief.md

@@ -0,0 +1,129 @@
+# Brief: scheduler-allocator-runner
+
+## Problem
+Scheduler plugin authors and maintainers. `seqjob_allocator.c` (`my_alloc`,
+line 213, ~215 lines) and `parjob_allocator.c` (`my_alloc`, line 545, ~220
+lines) each own the *entire* allocation algorithm, although only plan
+generation is plugin-specific. ~100+ lines are duplicated verbatim (lazy
+pend-reason, `sched_ctx_get_uncomit_shared`, `sched_check_shared_satisfy` +
+`PREASON_NO_RESOURCE` fill, add-uncommitted-shared, plan-iterator loop
+skeleton, commit tail with `sched_make_allocs` + `-ENOMEM` cleanup chain).
+Every bug fix (bug-1882, bug-1767, bug-1768) has to land twice, and a new
+allocator plugin must copy the whole `my_alloc()`.
+Driver: `issues/new_requirements.md` item 4.
+
+## Current State
+- Framework dispatch `sched_make_primary_alloc_()` (`scheduler.c:235`) picks
+  `core->seqjob_allocator` (`maxTasks == 1`) or `core->parjob_allocator`
+  and calls `plugin->job_alloc_fn(plugin, ctx)`.
+- Plan generation is already abstracted behind `xce_alloc_plan_iter_t`
+  (`include/sched/alloc_iter.h`); host scan behind `xce_host_iter_t`.
+- The picker side already uses a framework-owned iface pattern
+  (`xce_job_picker_iface_t`, `plugin->picker_iface`, `plugin.h:88`).
+- `alloc_iter.h` `init`'s third parameter is `xce_scheduler_t*`, unused by
+  both implementations (`(void)sched`); sim mode has no real scheduler.
+- parjob has `parjob_try_plan_` / `parjob_rollback_claims_` /
+  `parjob_snapshot_host_req_` claim bookkeeping; seqjob keeps a single `w`.
+
+## Desired Outcome
+- Neither plugin defines `my_alloc()`; both register
+  `.job_alloc_fn = sched_allocator_run` and supply only an
+  `xce_sched_allocator_iface_t` (optional `accepts` guard + plan iterator)
+  plus their existing `prepare_allocation`.
+- One framework function owns shared-resource prep, the bucket -> host-claim
+  loop with rollback, and the commit tail.
+- Allocation decisions are bit-for-bit identical to today for every existing
+  test and sim-parity fixture.
+- `new_requirements.md` item 4 closed.
+
+## Approach
+**Framework runner, lean** (chosen 2026-09-28). The design proposal of
+2026-09-23 (§5), with the optional `commit` hook left out because nothing
+uses it yet:
+- `xce_sched_allocator_iface_t { accepts; plan_init; plan_next; plan_fini; }`
+  in `plugin.h`; `allocator_iface` pointer on `xce_sched_plugin_t` with the
+  same ownership as `picker_iface`.
+- `sched_allocator_run()` = guards -> `accepts` -> `sched_alloc_prepare_shared_()`
+  -> plan loop -> `sched_try_plan_()` (promoted `parjob_try_plan_`) ->
+  `sched_finalize_job_alloc_()` (one `make_host_alloc` per claim ->
+  `make_job_alloc` -> `allocs_shared` block).
+- Retype `alloc_iter.h` `init`'s third parameter to `xce_sched_core_t*`.
+- Staged migration keeps the suite green: (1) extract shared helpers
+  used by both existing `my_alloc()`s; (2) runner + iface, parjob first;
+  (3) seqjob; (4) retype `plan_init`; (5) close item 4.
+
+Rejected: adding the `commit` override hook (an unused extension point that
+would need its own tests); only extracting helpers (would not meet item 4's
+goal that plugins no longer export `my_alloc()`).
+
+## Scope
+- **In**: new allocator iface + framework runner; shared prep/try-plan/
+  finalize helpers; delete both `my_alloc()`s and parjob's claim helpers;
+  retype `xce_alloc_plan_iter_t.init` to `xce_sched_core_t*`; equivalence
+  tests; update `issues/new_requirements.md` item 4.
+- **Out**: see Out of Boundary.
+
+## Boundary Candidates
+- Plugin ABI (`plugin.h` iface + plugin template wiring)
+- Framework allocation runner (new `plugin_alloc.c` or `scheduler.c`) —
+  shared-resource prep, claim loop/rollback, commit tail
+- Plan-iterator contract (`alloc_iter.h` retype; seqjob/parjob plan iters)
+
+## Out of Boundary
+- Retyping `sched_make_allocs()` to `xce_sched_core_t*` and removing the
+  `(xce_scheduler_t*)plugin->sched` cast (follow-up; callers outside
+  allocators).
+- Collapsing `sched_make_seqjob_allocation` / `sched_make_parjob_allocation`
+  dispatch in `scheduler.c` (optional, only if trivially falls out).
+- The `commit` override hook.
+- The `FIXME: no duration now` (NULL duration to `sched_make_allocs`):
+  carry it over unchanged, don't fix it here.
+- Any change to plan ordering, host ordering, pend-reason content, or
+  preemption.
+
+## Upstream / Downstream
+- **Upstream**: `scheduler-allocation` (plan/host iterators, Req 4
+  combinatorial parjob, Req 9 no-regression); `scheduler-core`
+  (plugin/picker iface pattern, sim mode); `libxce-alloc-preason-equality`
+  (`job_alloc_eq` used by sim verify).
+- **Downstream**: any future allocator plugin (it would need only a plan
+  iterator); `scheduler-preemption`; the follow-up `sched_make_allocs` retype.
+
+## Existing Spec Touchpoints
+- **Extends**: none (scheduler-allocation is implementation-complete; its
+  Req 9 is inherited as a constraint, not reopened).
+- **Adjacent**: `scheduler-allocation`, `scheduler-core`,
+  `scheduler-sim-discrete-clock` (in flight, touches sim_loop — avoid
+  overlapping files), `libxce-alloc-preason-equality`.
+
+## Constraints
+- TDD: tests in `src/scheduler/testsuite/` first; full ASAN build via
+  `/xce-build scheduler`.
+- Behavioral equivalence is mandatory. Preserve exactly: bug-1882 (half-built
+  alloc never left on `ctx->alloc`), bug-1767 (parjob `maxTasks <= 1`
+  returns 0 with no fabricated reason, and before the lazy reason is
+  created), bug-1768; `-ENOMEM` cleanup order of both current tails.
+- Divergences found in discovery that the design MUST resolve explicitly
+  (they are not in the 2026-09-23 proposal):
+  1. parjob does `xvec_idval_fini/init(&ctx->alloc_req.host)` per bucket
+     (`parjob_allocator.c:452`); seqjob never resets it. Decide whether the
+     shared loop always resets, and prove it is a no-op for seqjob.
+  2. seqjob builds its halloc directly from `ctx->alloc_req.host`; under the
+     shared per-claim snapshot it gains a new `-ENOMEM` path. It must go
+     through the same rollback (`sched_ctx_uncommit_host`).
+  3. seqjob does not set `ctx->nslots/nhosts` in its loop. Its values come
+     from `seqjob_prepare_allocation` (1,1). `sched_try_alloc_on_host_()`
+     (`resource.c`) does not read them directly; check which hook does, and
+     that `bucket_slots[0] == 1` for seqjob.
+  4. seqjob has no `maxTasks` guard today. A `seqjob_accepts`
+     (`maxTasks == 1`) changes behaviour for any direct caller with another
+     value (several testsuite files reference `seqjob_allocator`). Either
+     make `accepts` NULL for seqjob or audit those callers.
+  5. Proposal-sketch defects to avoid: `sched_ctx_primary == NULL` typo;
+     `claims[plan->nbuckets]` VLA (heap-allocate or bound by `maxTasks`).
+- Claim `host_req` snapshots are owned by the framework: fini on every exit
+  path (success, bucket failure, OOM).
+- Verify with: `test_seqjob_*`, `test_parjob_*`, `test_seqjob_rewrap_equivalence`,
+  `test_maxtasks_wiring_*`, `test_sim_*_parity`, `test_sim_live_allocation`,
+  `test_snapshot_load_allocator_regression`; `/xce-audit-invariants scheduler`
+  (driver state-equivalence).

.kiro/specs/scheduler-allocator-runner/requirements.md

@@ -0,0 +1,99 @@
+# Requirements Document
+
+## Project Description (Input)
+**Who has the problem:** scheduler plugin authors and maintainers of the XCE scheduler allocation path.
+
+**Current situation:** `seqjob_allocator.c` (`my_alloc`, ~215 lines) and `parjob_allocator.c` (`my_alloc`, ~220 lines) each implement the whole allocation algorithm, although only plan generation is plugin-specific. Over 100 lines are duplicated almost verbatim: lazy pend-reason creation, getting the uncommitted shared resources, the shared-resource satisfaction check with its `PREASON_NO_RESOURCE` fill, adding the uncommitted shared resources, the plan-iterator loop skeleton, and the commit tail (`make_job_alloc` + `sched_make_allocs` + the `-ENOMEM` cleanup chain). Every allocation bug fix (bug-1882, bug-1767, bug-1768) has to land twice, and a new allocator plugin would have to copy the whole `my_alloc()`. Plan generation is already abstracted behind `xce_alloc_plan_iter_t`, whose `init` takes an unused `xce_scheduler_t*` even though sim mode has no real scheduler. Tracked as `issues/new_requirements.md` item 4.
+
+**What should change:** the scheduler framework owns allocation. A new `xce_sched_allocator_iface_t` (an optional `accepts` guard plus `plan_init`/`plan_next`/`plan_fini`), referenced from `xce_sched_plugin_t` in the same way as `picker_iface`, lets one framework function, `sched_allocator_run()`, carry out shared-resource preparation, the bucket → host claim loop with rollback, and the commit tail. Both plugins register `.job_alloc_fn = sched_allocator_run`, supply only their plan iterator and `prepare_allocation`, and no longer define `my_alloc()`. The third parameter of `xce_alloc_plan_iter_t.init` is retyped to `xce_sched_core_t*`. Allocation decisions, pend reasons, and error and `-ENOMEM` behaviour stay identical for every existing test and sim-parity fixture, including bug-1882, bug-1767 and bug-1768. There is no per-plugin `commit` hook and no `sched_make_allocs()` retype (both left out of scope). The design must settle explicitly the seqjob/parjob divergences listed in `brief.md` → Constraints (the per-bucket `alloc_req.host` reset, seqjob's new OOM snapshot path, where `nslots`/`nhosts` are consumed, seqjob `accepts` for direct callers, and no VLA). Full context: `.kiro/specs/scheduler-allocator-runner/brief.md`.
+
+## Introduction
+The scheduler's two built-in allocator plugins, the sequential-job allocator (seqjob, jobs with exactly one task) and the parallel-job allocator (parjob, jobs with more than one task), each carry a private copy of the whole allocation algorithm. This feature moves that algorithm into the scheduler plugin framework. It becomes one framework-owned allocation routine, and an allocator plugin contributes only what is genuinely its own: the candidate-plan generator, an optional "is this job mine" guard, and its existing per-allocation preparation hook.
+
+This is a behaviour-preserving refactor. For every job, the scheduler must reach the same allocation decision, choose the same hosts, debit the same resources, record the same pend reasons, and return the same result codes as it does today, in normal (live) mode and in simulation mode. The only intentional behaviour changes are the two hardening points in Requirement 5 (AC 1 and AC 5), which make formerly undefined sequential-allocator behaviour defined.
+
+Terms:
+- **Allocation attempt**: one call from the scheduler's allocation dispatch to an allocator for one primary resource request.
+- **Plan**: one candidate way of splitting a job's tasks into buckets, each bucket placed on one host.
+- **Claim**: one bucket satisfied on one host within the plan being tried.
+- **Pending signal**: the allocator reports "not satisfiable now" (result 0) without an error.
+
+## Boundary Context
+- **In scope**: a framework-owned allocation routine used by both built-in allocators; the allocator plugin contract (plan generator, optional acceptance guard, and the existing preparation hook); removal of the per-plugin allocation routines; changing the plan-generator initialisation to take the scheduling core instead of the scheduler object; keeping every existing allocation, pend-reason, error and simulation-parity behaviour; closing `issues/new_requirements.md` item 4.
+- **Out of scope**: a per-plugin override hook for assembling results; retyping the allocation-record constructor (`sched_make_allocs`) or removing the plugin's scheduler back-pointer; merging the two dispatch paths (sequential vs parallel) in the scheduler; fixing the missing allocation duration (the existing "no duration now" FIXME is kept as is); any change to plan order, host order, pend-reason vocabulary, the unsatisfiable-request cache, preemption, or the job picker.
+- **Adjacent expectations**: the scheduler's allocation dispatch keeps sending `maxTasks == 1` primaries to seqjob and every other primary to parjob, and keeps treating result `> 0` as satisfied, `0` as pending and `< 0` as error, with an allocation published only on `> 0`. COMPOUND/ALTER handling (bug-1766: a merged allocation is published only if every primary succeeds) and snapshot-restored schedobj construction (bug-1767) stay owned by their existing code. The plan and host iterator contracts from `scheduler-allocation` keep their current ordering semantics.
+
+## Requirements
+
+### Requirement 1: Framework-owned allocation
+**Objective:** As a scheduler plugin author, I want the framework to run the allocation algorithm, so that an allocator plugin only has to describe how it generates candidate plans.
+
+#### Acceptance Criteria
+1. The Scheduler Plugin Framework shall provide one allocation routine that both built-in allocator plugins (seqjob and parjob) register as their job-allocation entry point.
+2. The Scheduler Plugin Framework shall let an allocator plugin supply a candidate-plan generator (initialise, next plan, finish), an optional acceptance guard, and its existing per-allocation preparation hook, and nothing else, to take part in allocation.
+3. Where an allocator plugin supplies no acceptance guard, the Scheduler Plugin Framework shall treat every primary resource request passed to that plugin as accepted.
+4. The seqjob and parjob allocator plugins shall contain no private allocation routine that duplicates the shared-resource check, the plan/host search, or the result assembly.
+5. The Scheduler Plugin Framework shall keep the allocator plugin contract as data the plugin registry does not free, with the same lifetime as the existing job-picker contract: it lives as long as the registered plugin, and the plugin does not release it separately.
+
+### Requirement 2: Shared-resource precheck
+**Objective:** As a scheduler operator, I want jobs whose cluster-wide (shared) resources are unavailable to pend with the same reasons as today, so that pend diagnostics do not change.
+
+#### Acceptance Criteria
+1. When an allocation attempt starts and the attempt has no pend-reason record yet, the Scheduler Plugin Framework shall create one before checking shared resources.
+2. When the job's shared-resource request cannot be met, the Scheduler Plugin Framework shall add one "no resource" pend reason, with no host and naming the resource, for each unmet shared resource. It shall then return the pending signal without trying any host.
+3. When the job's shared-resource request can be met, the Scheduler Plugin Framework shall count the requested shared amounts as uncommitted for the attempt before trying any plan. It shall count them once per attempt, whatever the number of plans or buckets tried.
+4. If creating the pend-reason record, reading the uncommitted shared state, checking shared satisfaction, or counting shared amounts as uncommitted fails, the Scheduler Plugin Framework shall return that failure's negative error code, with no allocation on the attempt.
+
+### Requirement 3: Plan and host search
+**Objective:** As a scheduler operator, I want jobs placed on exactly the hosts they are placed on today, so that the refactor cannot change scheduling outcomes.
+
+#### Acceptance Criteria
+1. The Scheduler Plugin Framework shall try candidate plans in the order the allocator plugin's plan generator produces them, and shall stop at the first plan whose every bucket is satisfied.
+2. While trying a plan, the Scheduler Plugin Framework shall try, for each bucket in order, the candidate hosts in the shared host-iteration order. It shall claim the first candidate host that satisfies the bucket and has not already been claimed by an earlier bucket of the same plan.
+3. If any bucket of a plan cannot be satisfied, the Scheduler Plugin Framework shall release the per-host resource debits of every claim already made for that plan before trying the next plan.
+4. When the plan generator cannot be initialised or yields no plan whose buckets are all satisfied, the Scheduler Plugin Framework shall return the pending signal with no allocation on the attempt.
+5. The Scheduler Plugin Framework shall give each bucket the task count and single-host shape that parjob uses today when checking a candidate host, and shall leave seqjob's per-host checks with the one-task, one-host shape they have today.
+6. While seqjob runs an attempt for a primary that shares its attempt context with an earlier primary of the same COMPOUND or ALTER job, the Scheduler Plugin Framework shall start seqjob's host search from that primary's own per-host request only, as it does today: the job-level preparation pass resets the per-host request for every primary (bug-1883, fixed in `c10f32cc`), so no earlier primary's per-host request reaches seqjob's host pre-filtering or pend reasons.
+7. While parjob runs an attempt, the Scheduler Plugin Framework shall start every bucket's host search from an empty per-host request, as parjob does today.
+8. The Scheduler Plugin Framework shall support plans with any number of buckets up to the job's task count, with no fixed upper limit on the number of buckets that a large task count could exceed.
+
+### Requirement 4: Result assembly
+**Objective:** As a scheduler maintainer, I want the successful allocation result built the same way for both allocators, so that dispatch, commit, and snapshot consumers see identical allocation records.
+
+#### Acceptance Criteria
+1. When every bucket of a plan is satisfied, the Scheduler Plugin Framework shall build one host allocation per claim, in claim order. Each is made from the claimed host's name and that host's resource request as it stood when the claim was made.
+2. When every bucket of a plan is satisfied and the job requests shared resources, the Scheduler Plugin Framework shall attach one shared allocation record per requested shared resource. When the job requests none, it shall attach no shared records.
+3. The Scheduler Plugin Framework shall produce allocation results with the same job identifier, empty reservation name, and absent duration that seqjob and parjob produce today.
+4. When a plan is satisfied, the Scheduler Plugin Framework shall set the attempt's allocation to the newly built result and return a positive result.
+5. When seqjob or parjob is given an input that reaches it today, the Scheduler Plugin Framework shall produce an allocation result identical to that allocator's current result: same host count, host names, per-host resource entries and values, shared records, and return code.
+
+### Requirement 5: Guards and error handling
+**Objective:** As a scheduler maintainer, I want every error and out-of-contract input handled exactly as the fixed bugs require, so that no past fix regresses.
+
+#### Acceptance Criteria
+1. If the scheduling core, the plugin's scheduler back-reference, or the attempt's primary resource request is missing, the Scheduler Plugin Framework shall return an invalid-argument error without creating a pend-reason record, for both allocators. (Hardening for seqjob, whose current behaviour is undefined.)
+2. If parjob receives a primary resource request whose task count is 1 or less, the Scheduler Plugin Framework shall return the pending signal without creating a pend-reason record or checking resources (bug-1767).
+3. The Scheduler Plugin Framework shall let seqjob accept every primary resource request that reaches it, whatever its task count, as today.
+4. If building any part of the allocation result fails for lack of memory, the Scheduler Plugin Framework shall return an out-of-memory error, leave no allocation on the attempt, and free every partly built piece (bug-1882).
+5. If a failure for lack of memory happens after one or more claims were made, the Scheduler Plugin Framework shall release the per-host resource debits of every claim not yet handed over to a built result, for both allocators. (Hardening for seqjob, which today leaves its single debit in place until the attempt's context is discarded.)
+6. The Scheduler Plugin Framework shall set the attempt's allocation only when it returns a positive result, so that callers merging COMPOUND/ALTER results keep publishing only fully satisfied jobs (bug-1766).
+7. When an attempt that initialised the plan generator exits (success, pending, or error), the Scheduler Plugin Framework shall finish the plan generator and release every per-claim resource-request copy it holds.
+8. The Scheduler Plugin Framework shall size each primary's allocation from that primary's own resource request, including primaries nested in COMPOUND and ALTER jobs (bug-1768).
+
+### Requirement 6: Plan-generator initialisation contract
+**Objective:** As a scheduler maintainer, I want plan generators to receive the scheduling core, which exists in both modes, rather than a scheduler object that simulation mode lacks, so that the contract is honest in every mode.
+
+#### Acceptance Criteria
+1. The Scheduler Plugin Framework shall initialise an allocator plugin's plan generator with the primary resource request and the scheduling core.
+2. The Scheduler Plugin Framework shall produce identical plan sequences, for seqjob and for parjob, before and after the change to the initialisation parameter.
+3. While running in simulation mode, the Scheduler Plugin Framework shall run allocation through the same framework routine and plan generators as normal mode, with no scheduler object passed to any plan generator.
+
+### Requirement 7: Equivalence evidence and closure
+**Objective:** As a project maintainer, I want test evidence that nothing changed and the tracked item closed, so that the refactor can be accepted with confidence.
+
+#### Acceptance Criteria
+1. The Scheduler Plugin Framework shall pass, unchanged in their expected outcomes, the existing seqjob allocator, seqjob rewrap-equivalence, parjob combinatorial, parjob end-to-end multi-slot, task-count wiring, unsatisfiable-cache dispatch, simulation parity, simulation live-allocation, simulation limit fan-out, election simulation fixture, and snapshot-load allocator regression tests. Test changes are limited to how the allocation routine is reached and the parameter type of plan-generator initialisation.
+2. The Scheduler Plugin Framework shall have tests showing that, for the same inputs, the framework routine gives the same result code, allocation result, pend reasons, and per-host debits as the removed seqjob and parjob routines. These tests cover success, pending on shared resources, pending on hosts, mid-plan rollback, out-of-memory at each result-building step, the parjob out-of-contract task count, and missing inputs.
+3. The Scheduler Plugin Framework shall keep the existing check that pend-reason records are created only through the shared pend-reason factory, and that check shall cover the new framework allocation code.
+4. When this feature is complete, `issues/new_requirements.md` item 4 shall be marked resolved, with a reference to this spec.
+5. When this feature is complete, the scheduler state-equivalence invariant audit shall report no new violations in the allocation path.

.kiro/specs/scheduler-sim-decision-trace/brief.md

@@ -0,0 +1,124 @@
+# Brief: scheduler-sim-decision-trace
+
+## Problem
+
+People debugging scheduler behavior in the simulator (`xce_scheduler --sim -i`) can see *what* the scheduler decided: dispatch or pend, the jobid, and at best a coarse `xce_pendreason_t`. They cannot see *why*. Nothing in the REPL answers these questions:
+
+- Why was this job picked at this point in the session?
+- Which resreq (COMPOUND primary or ALTER alternative) was tried, and which one won?
+- Which plans did the planner generate and try, and in what order (seqjob single plan; parjob k = 1..maxTasks with balanced buckets)?
+- Which hosts were considered for each plan and bucket, which were rejected, and why? This covers select mismatch, per-resource shortfall with numbers, shared-resource shortfall, and a limit hit.
+- What did the final allocation hold (per-host/per-resource amounts, shared allocs, the limit vres debited)?
+
+Most of this information is thrown away today:
+- `host_iter.c` rejects hosts with a bare `continue`, so no reason is recorded.
+- `unsatisfy_` computes shortfalls into `ctx->unmeet` but only TRACE-logs them.
+- ALTER keeps only the last alternative's reason (`scheduler.c:427` FIXME).
+- A limit violation is indistinguishable from `PREASON_NO_RESOURCE`.
+- Plan and bucket attempts leave no record.
+
+The only workaround is printf debugging or `-vvv` tlog output. That output has no categories, is written to a log file outside the REPL, and cannot be filtered to one job.
+
+## Current State
+
+- The sim loop runs the **same** picker and allocator threads and session protocol as live (`sim_loop.c:1386`, `scheduler-driver-equivalence.md`). Decisions flow allocator → `core->decisions` → the sim drain loop (`sim_loop.c:~1607`), where `sim_print_decision` (`sim_loop.c:479`) echoes only the op and jobid.
+- REPL: `info session|config|summary|jobs|queue|worker|resource|limit|break`, decision breakpoints (`repl_break.c`), `set` params (`repl_params.c`), and the continue-mode verbosity/rollup (`scheduler-sim-repl-verbosity`). `info job` shows no pend reason.
+- Core isolation: `core*.c`/`sim*.c` may not do I/O (`test_core_isolation.c`, `test_sim_isolation.c`); core→driver communication goes through NULL-checked callbacks (`cache_clear_cb`, `unsatisfiable_*_cb`, `on_decision` in `core.h`).
+- Decision seams (from codebase map, 2026-09-28):
+  - Picker: `builtin_fcfs_next_job` (FCFS keys: queue priority, then job priority, then queue_time); `sched_election.c:133`.
+  - Allocator entry, `plugin_job_election` (return ignored), and the final decision: `sched_election.c:226/236/262/273`.
+  - Resreq branch: `sched_make_allocation` `scheduler.c:441`, COMPOUND per primary `:351`, ALTER per alternative `:421`, and the primary chokepoint `sched_make_primary_alloc_` `:235`.
+  - Plans, buckets and the shared check: the two `my_alloc()` copies in `seqjob_allocator.c` / `parjob_allocator.c`. **These are being replaced by `sched_allocator_run()` under `scheduler-allocator-runner`.**
+  - Host verdicts: `host_iter.c:136` (select), `:158` (resource satisfy, `ctx->unmeet`), and `sched_try_alloc_on_host_` `resource.c:497-573` (prepare fan-out, `limit_prepare_allocation`).
+  - Allocation result: `sched_make_allocs` `scheduler.c:1920`, `JobAlloc` (capnp `:441`).
+- The unsatisfiable-resreq cache is wired in **live mode only**, so sim always runs the full allocator and a trace never has to explain a cache hit.
+
+## Desired Outcome
+
+In the simulator REPL, an operator can turn decision tracing on at runtime, for every job or for a chosen set of jobids. As they `step`/`continue`, each traced job's decision is printed in the REPL console with a readable, nested explanation:
+
+1. **Job selection**: why this job came up now (picker, queue, sort keys and position), and the election result.
+2. **Resreq selection**: each resreq tried (primary / COMPOUND member / ALTER alternative), in order, with its outcome and why it was accepted or rejected.
+3. **Planner**: each plan the planner generated and tried (host count, bucket sizes, slots), in order, and which one was committed or why all failed.
+4. **Host selection within a plan**: for each bucket, the hosts considered in order and the verdict for each. For rejections: select-expression mismatch, per-resource shortfall (`total / inuse / uncommitted / wanted`), shared-resource shortfall, limit hit (naming the limit and its scope instance), host already claimed by an earlier bucket, or rolled back.
+5. **Allocation details**: the final per-host/per-resource amounts, shared allocations, and the limits applied or debited. For a pend, the final reason set.
+
+`info trace <jobid>` re-prints a job's most recent trace. Optionally, `trace ... file <path>` mirrors the same text to a file. When trace is off, REPL output is byte-identical to today. The live scheduler never records a trace.
+
+## Approach
+
+**Trace buffer per decision, rendered by the sim driver (the user's choice, 2026-09-28).**
+
+- A new core module (e.g. `src/scheduler/sched_trace.c` + `include/sched/trace.h`) defines structured trace records and a per-allocator-thread buffer. The allocator thread is single-threaded per decision, so no locking is needed.
+- Hooks at the seams above append records only when trace is enabled for the job being allocated. When disabled, the gate is one NULL-pointer / flag check, following the existing `unsatisfiable_*_cb` and `recording` patterns. Only the sim driver ever enables it, so the live scheduler pays a single branch per hook and nothing more.
+- The picker-side "why selected" facts are captured when the job is handed to the allocator (via `jbuf`) or recomputed from the sort keys at render time, so the picker thread never writes to the allocator's buffer.
+- The finished buffer is attached to `xce_decision_t` and travels through `core->decisions`. The sim drain loop, which already owns console output, renders it in the REPL next to the decision echo, stores the latest trace per job for `info trace`, and optionally mirrors it to a file. The core does no I/O.
+- REPL surface (names to be settled in requirements): `trace on all | job <ids> [file <path>]`, `trace off`, `trace status`, `info trace <jobid>`, and `help` entries. Traced decisions are always echoed in full, even in `continue` mode where routine output is collapsed.
+
+**Rejected alternatives:**
+- Streaming records to a callback as they happen: I/O would run on the allocator thread, output from the picker and allocator threads would interleave, and there would be no per-job recall for `info trace`.
+- A tlog trace level: tlog has no categories and one global level, it writes to a log file rather than the REPL, and the extra log calls would sit in the live path.
+- A live-mode trace option (CLI/config/`schedAdmin`): rejected by the user. It would put a performance cost on the production scheduler, and the simulator is the investigation tool.
+- A headless `--sim-trace` flag: rejected by the user. Tracing is an interactive REPL feature only.
+
+## Scope
+
+- **In**:
+  - Trace record model and per-decision buffer.
+  - Hook points across job pick, election, resreq branch, planner (plan/bucket), host iterator verdicts, per-host prepare/limit checks, shared-resource check, and final allocation or pend reason.
+  - Attaching the trace to the decision.
+  - Rendering in the sim REPL; per-job last-trace store; `info trace`.
+  - Runtime on/off with all-jobs or job-set filter; optional file mirror.
+  - A cap on trace size per decision, with a visible truncation marker.
+  - Tests proving zero behavioral change: identical decisions and identical `--sim-verify` results with trace on vs off, and identical REPL output with trace off.
+- **Out**:
+  - Any live (non-sim) enablement.
+  - Headless-sim CLI flag.
+  - Changing pend-reason content published to the controller or compared by `job_preason_eq` / `--sim-verify`.
+  - New scheduling, picker, or allocator behavior.
+  - Fixing the defects the trace will expose. These are logged as separate bugs, e.g. the ALTER reason leak/overwrite, and the shared-check loop bound (`alloc_req.shared.count` vs `unmeet.count`).
+
+## Boundary Candidates
+
+- **Trace model + buffer + gating** (core, no I/O): record types, per-thread buffer, enable predicate, size cap, attach to `xce_decision_t`.
+- **Instrumentation hooks**: picker/election (`sched_election.c`, `builtin_fcfs.c`), resreq branch (`scheduler.c` `sched_make_*`), planner/buckets (inside `sched_allocator_run()` once `scheduler-allocator-runner` lands), host verdicts (`host_iter.c`, `resource.c` including `unsatisfy_` shortfalls), limits (`plugins/limit.c` prepare path, to name the limit behind a vres shortfall).
+- **Sim/REPL surface**: `trace` command + `info trace` + help; rendering (nested, human-readable units reusing `scheduler-sim-repl-verbosity` Req 6 formatting); integration with step/continue echo and the rollup; optional file mirror.
+
+## Out of Boundary
+
+- Live scheduler observability of any kind.
+- Controller-visible pend reasons (`JobPendReason`), recording/replay file format (`ReplayDecision`), and `--sim-verify` equality semantics.
+- `info limit` and limit-usage display (owned by `scheduler-sim-diagnostics`).
+- The allocator loop restructuring itself (owned by `scheduler-allocator-runner`).
+
+## Upstream / Downstream
+
+- **Upstream**:
+  - `scheduler-allocator-runner`: **hard sequencing dependency (user decision)**. Plan/bucket hooks go into the single `sched_allocator_run()`, not the two `my_alloc()` copies.
+  - `scheduler-core`: REPL, sim loop, decision pipeline.
+  - `scheduler-sim-repl-verbosity` / `scheduler-sim-step-verbosity`: echo gate, rollup, value formatting.
+  - `scheduler-allocation`: seqjob/parjob plan semantics.
+- **Downstream**:
+  - `scheduler-preemption` (adds PREEMPT decisions; a trace of victim selection is a natural later extension).
+  - Future `scheduler-sim-diagnostics` capabilities.
+  - Bug triage that uses traces as evidence.
+
+## Existing Spec Touchpoints
+
+- **Extends**: none directly. This spec **takes over** "Seed capability 2 — Decision-reasoning trace" from `scheduler-sim-diagnostics/brief.md`; that brief should point here.
+- **Adjacent**:
+  - `scheduler-allocator-runner` (hook placement; its "no pend-reason content change" rule also binds this spec).
+  - `scheduler-preemption` (also extends `decision.h`; coordinate field additions).
+  - `scheduler-sim-discrete-clock` (touches `sim_loop.c`).
+  - `libxce-alloc-preason-equality` (owns `job_preason_eq`; the trace must not alter compared reason content).
+  - `scheduler-sim-diagnostics` (REPL `info` family).
+
+## Constraints
+
+- **Performance on live**: the user's explicit reason for sim-only. With trace off, which is always the case in live, each hook costs at most one predictable branch: no allocation, no formatting, no argument evaluation. Record formatting happens only in the sim driver.
+- **Determinism**: enabling trace must not change any decision, alloc, pend reason, accounting state, or `--sim-verify` outcome. Must hold in both `step` and `continue` modes.
+- **Core isolation**: no I/O in `core*.c`/`sched_*` code; rendering and file writes live in the sim/REPL layer (`test_core_isolation.c`, `test_sim_isolation.c`, `test_no_bare_printf.c`).
+- **Threading**: the picker and allocator are separate threads with `jbuf` capacity 1 as a determinism invariant. Trace must not add cross-thread synchronization or change hand-off order.
+- **File size**: `scheduler.c`, `builtin_fcfs.c`, `repl_info.c` already exceed the 800-line cap; new code goes in new files (`sched_trace.c`, `repl_trace.c`).
+- **Memory**: per-decision buffer with a configurable cap and a truncation marker. Large parjob × many-host sessions must not explode memory.
+- **TDD**: testsuite tests before C changes (`testing.md`).

.kiro/specs/scheduler-sim-decision-trace/requirements.md

@@ -0,0 +1,200 @@
+# Requirements Document
+
+## Project Description (Input)
+**Who has the problem:** People investigating scheduler behavior in the xce_scheduler simulator REPL (`--sim -i`): scheduler developers and operators debugging why jobs were placed or pended.
+
+**Current situation:** The REPL shows only *what* was decided (dispatch/pend, jobid, at best a coarse `xce_pendreason_t`). It never shows *why*. The reasoning the scheduler computes is discarded:
+- Hosts rejected in `host_iter.c` leave no record.
+- Per-resource shortfalls (`unsatisfy_`) are only TRACE-logged.
+- ALTER keeps only the last alternative's reason.
+- A limit hit is indistinguishable from `NO_RESOURCE`.
+- Plan and bucket attempts are never recorded.
+
+The only workaround is printf debugging or `-vvv` tlog, which is written to a log file rather than the REPL and cannot be filtered to one job.
+
+**What should change:** Add a simulator-only decision trace, enabled at runtime from the REPL for every job or for a chosen set of jobids. For each traced job, the REPL console prints a readable, nested explanation of the full decision:
+1. Why the job was selected (picker, queue, sort keys, position) and the election result.
+2. Each resreq tried (primary / COMPOUND member / ALTER alternative), in order, with its outcome.
+3. Each plan the planner generated and tried (host count, buckets, slots) and which one was committed.
+4. For each bucket, the hosts considered in order and why each could or could not meet the resreq: select mismatch, per-resource shortfall with numbers, shared-resource shortfall, the named limit hit, host already claimed, or rolled back.
+5. The final allocation details (per-host/per-resource amounts, shared allocs, limits applied) or the final pend reason.
+
+`info trace <jobid>` re-prints a job's latest trace, and an optional file mirror is available. Records are collected in a per-decision buffer on the allocator thread, attached to the decision, and rendered by the sim driver, so the core does no I/O.
+
+**Constraints:**
+- The live scheduler never enables the trace. Because tracing has a performance cost, when it is off each hook costs at most one branch.
+- Enabling the trace must not change any decision, allocation, pend reason, accounting state, or `--sim-verify` result.
+- With the trace off, REPL output is unchanged.
+- No new cross-thread synchronization is added.
+- The trace is capped per decision.
+- Trace code goes in new files.
+- Sequenced after `scheduler-allocator-runner`, so the plan/bucket hooks go into `sched_allocator_run()`.
+
+Full context: `brief.md` in this directory.
+
+## Introduction
+
+The xce_scheduler simulator (`--sim` with the interactive REPL, `-i`) replays recorded scheduling sessions so developers and operators can investigate how the scheduler behaves. Today the REPL reports only the outcome of each scheduling decision (dispatch or pend, plus at best a coarse pend reason). The reasoning behind that outcome is not shown:
+- why the job was selected when it was
+- which resource requirement was tried
+- which placement plans were attempted
+- which hosts were considered and why each could or could not satisfy the job
+- which resources and limits were finally allocated
+
+In this document a *round* (the REPL's term) and a *scheduling session* are the same unit: one pass of the scheduler over the pending jobs.
+
+This feature adds a **decision trace** to the simulator REPL. The operator turns it on at runtime for all jobs or for chosen jobs, at a chosen level of detail. The explanation is printed inline as the replay proceeds and can be recalled per job afterwards. The trace is purely an investigation aid: it exists only in the simulator, so the production scheduler pays no cost for it, and enabling it never changes what the scheduler decides.
+
+## Boundary Context
+- **In scope**:
+  - REPL commands to enable, disable, scope and inspect decision tracing.
+  - The content of a decision trace for each traced job's dispatch or pend decision: job selection, resource-requirement selection, placement plans, per-host verdicts, final allocation or pend reason.
+  - Inline display of traces during `step` and `continue`; recall of a job's latest trace.
+  - An optional file copy of trace output.
+  - Bounds on per-decision trace size and on stored-trace memory.
+  - The guarantee that tracing does not alter scheduling results.
+- **Out of scope**:
+  - Tracing in the live (controller-connected) scheduler, by any means (CLI flag, configuration, admin command).
+  - Tracing in a non-interactive simulation (`--sim` without `-i`).
+  - Preemption decisions.
+  - Any change to scheduling, job-picking, or allocation behavior.
+  - Any change to the pend reasons published to the controller, written to recordings, or compared by `--sim-verify`.
+  - Fixing scheduler defects the trace makes visible (for example, an alternative-requirement job reporting only its last alternative's pend reasons); those are tracked as separate bugs.
+  - Limit configuration/usage inspection (`info limit`, owned by `scheduler-sim-diagnostics`).
+- **Adjacent expectations**:
+  - This feature explains decisions made by the existing job picker and allocators (`scheduler-allocation`, as restructured by `scheduler-allocator-runner`) and does not redefine them.
+  - It extends the REPL command set and output conventions shipped by `scheduler-core`, `scheduler-sim-repl-verbosity` and `scheduler-sim-step-verbosity`, including the `set units` display mode, job-identifier range compression, and the sectioned `help` layout.
+  - It amends `scheduler-sim-repl-verbosity` Requirements 2.1 and 3, only while tracing is enabled: a continue-mode round that contains a traced decision also prints that decision's echo line and trace (Requirement 7.3 below), and never joins a collapsed range (Requirement 7.4 below). With tracing off, all REPL behavior defined by those specs is unchanged.
+  - It relies on `scheduler-sim-discrete-clock`: the simulation clock must not advance with the wall-clock time spent collecting traces, so that the time recorded for each decision is the same with tracing on or off.
+  - It takes over the decision-reasoning trace that `scheduler-sim-diagnostics` had deferred.
+
+## Requirements
+
+### Requirement 1: Trace Control
+**Objective:** As an operator investigating a replay, I want to turn decision tracing on and off at runtime and choose which jobs are traced and at what detail, so that I get explanations for exactly the jobs I care about without restarting the simulation.
+
+#### Acceptance Criteria
+1. When the operator issues a trace-enable command naming "all jobs", the Simulator REPL shall trace every dispatch and pend decision made in the scheduling sessions that start after the command, until tracing is disabled.
+2. When the operator issues a trace-enable command naming one or more job identifiers, the Simulator REPL shall trace only decisions for those jobs, where a job-array identifier selects all of its elements and a job-array element identifier selects only that element.
+3. When a trace-enable command names a job identifier that does not yet exist in the replay, the Simulator REPL shall accept it and trace that job's decisions once the job appears.
+4. When the operator issues a trace-enable command while tracing is already enabled, the Simulator REPL shall replace the previous job selection, detail level, size bound and file setting with the new command's settings.
+5. When the operator issues the trace-disable command, the Simulator REPL shall stop tracing decisions in the scheduling sessions that start after the command and shall keep already-captured traces available for recall.
+6. If a trace-enable or trace-disable command is issued while a scheduling session is in progress (for example at a decision breakpoint), the Simulator REPL shall apply the new settings from the next scheduling session, leaving the in-progress session traced according to the settings it started with.
+7. When the operator issues the trace-status command, the Simulator REPL shall display the settings in effect for the scheduling session in progress, or, when none is in progress, the settings the next scheduling session will use, and separately any settings that will replace them at the next scheduling session, each showing whether tracing is enabled, the traced job selection, the detail level, the per-decision size bound, the file copy destination if any, the stored-trace memory bound and current stored-trace memory use.
+8. The Simulator REPL shall support two detail levels, `summary` and `full`, selectable in the trace-enable command, and shall use `summary` when no level is given.
+9. If a trace command has an invalid argument (unknown option, malformed job identifier, invalid level, or a size bound that is not a whole number from 50 to 100000), the Simulator REPL shall print an error naming the invalid argument and leave the current trace settings unchanged.
+10. While tracing is enabled, the Simulator REPL shall keep the trace settings in effect across scheduling sessions until the operator changes or disables them.
+11. The Simulator REPL shall list the trace commands and `info trace` in its `help` output, with the trace commands in their own labeled section and `info trace` among the other `info` subcommands.
+
+### Requirement 2: Job Selection Explanation
+**Objective:** As an operator, I want each traced decision to explain why the job was considered when it was, so that I can understand job ordering and why a job was or was not reached earlier.
+
+#### Acceptance Criteria
+1. When a traced job is selected for allocation in a scheduling session, the Simulator REPL shall show in its trace the session number (numbered as in the REPL's event summary for that session) and the job's 1-based position among the jobs selected for allocation in that session.
+2. When a traced job is selected for allocation, the Simulator REPL shall show the job-ordering policy in use and the job's ordering keys under that policy with the values in effect when the job was selected (for the default first-come-first-served policy: the job's queue, the queue's priority, the job's priority and the job's queue time).
+3. When a traced job is selected for allocation, the Simulator REPL shall show the cluster-wide limits matched to the job before placement, and shall state that host-scoped limits are reported with each host's verdict.
+4. If a traced job is selected for allocation but no scheduling decision is produced for it, the Simulator REPL shall show a trace stating which of these occurred: the job could not be found by the allocator, the allocation attempt ended in an internal error, or the job was pended without any pend reason; when the job could not be found, the trace shall show the session, position and job identifier and state that the ordering keys and limits are unavailable.
+5. When a traced job is dispatched by the simulator's reconstruction of a recorded allocation, rather than by the scheduler's own placement, the Simulator REPL shall state this in place of the selection, requirement, plan and host sections of the trace.
+
+### Requirement 3: Resource Requirement Selection Explanation
+**Objective:** As an operator, I want to see each resource requirement the scheduler tried for a job and the result of each, so that I can understand which requirement was satisfied and why others were not.
+
+#### Acceptance Criteria
+1. When the scheduler evaluates a traced job, the Simulator REPL shall show in the trace every resource requirement tried, in evaluation order, each labelled as a single requirement, the i-th member of n of a compound requirement, or the i-th of n alternatives, with the full nesting path shown when requirements are nested.
+2. The Simulator REPL shall show for each tried resource requirement its minimum and maximum task counts, the requested resources, and whether the requirement was satisfied.
+3. When a member of a compound requirement cannot be satisfied, the Simulator REPL shall show that member as the point of failure and list the remaining members as not tried.
+4. When an alternative requirement is satisfied, the Simulator REPL shall mark it as the chosen alternative and list the remaining alternatives as not tried.
+5. When every alternative requirement fails, the Simulator REPL shall show the failure reasons of each alternative, including an internal error, not only those of the last one.
+
+### Requirement 4: Shared Resource and Placement Plan Explanation
+**Objective:** As an operator, I want to see the cluster-wide checks and the placement plans tried for each resource requirement, so that I understand how the scheduler decided how many hosts and how many tasks per host to use.
+
+#### Acceptance Criteria
+1. When the scheduler checks cluster-wide shared resources for a traced job's resource requirement, the Simulator REPL shall show each shared resource checked, other than cluster-wide limits (which are shown as limit checks with the fields of criterion 2 and whether they passed), with its total, the amount in use, the amount provisionally reserved by earlier parts of this job's evaluation, the amount requested, and whether the check passed.
+2. When the cluster-wide check fails on a cluster-wide limit, the Simulator REPL shall show the failure as a limit rejection naming the limit, the scope instance it applies to (for example the user, queue or project), the limit value, the current usage, the amount provisionally reserved by earlier parts of this job's evaluation and the amount requested, and shall not also show it as a resource shortfall.
+3. When the planner tries a placement plan for a traced job, the Simulator REPL shall show the plan in the order tried, with the number of hosts and the number of tasks assigned to each host group of the plan.
+4. The Simulator REPL shall show for each tried plan whether it succeeded or, if it failed, which host group of the plan could not be placed.
+5. When a plan succeeds, the Simulator REPL shall mark it as the committed plan and show that no further plans were tried for that resource requirement.
+6. If every tried plan for a resource requirement fails, the Simulator REPL shall state that no plan could be placed for that requirement.
+7. If no placement plan could be generated for a resource requirement, the Simulator REPL shall state that no plan was generated and the reason.
+
+### Requirement 5: Host Verdict Explanation
+**Objective:** As an operator, I want to know for each host considered why it could or could not meet the job's resource requirement, so that I can pinpoint the resource, selection expression, or limit that blocked a placement.
+
+#### Acceptance Criteria
+1. While the detail level is `full`, when a host is considered for a group of a traced job's plan, the Simulator REPL shall show that host, in the order considered, with its verdict: accepted, or rejected with every reason found by the checks the scheduler performed for that host; checks the scheduler did not perform for that host shall be shown as not evaluated, and the Simulator REPL shall not perform additional checks to find further reasons.
+2. While the detail level is `full`, when a host is rejected, the Simulator REPL shall indicate whether the host was rejected by the pre-filter check or by the per-host check, and for a pre-filter rejection shall state that the compared request was carried over from an earlier candidate host.
+3. While the detail level is `full`, when a host does not match the job's host selection expression, the Simulator REPL shall show a selection-mismatch verdict for that host.
+4. While the detail level is `full`, when a host is short of one or more resources, the Simulator REPL shall show each resource short on that host with the host's total, the amount already in use, the amount provisionally reserved on that host by earlier parts of this job's evaluation, and the requested amount the check actually compared against.
+5. While the detail level is `full`, when a host-scoped limit would be exceeded on a host, the Simulator REPL shall show a limit rejection naming the limit, the scope instance it applies to, the limit value, the current usage, the amount provisionally reserved by earlier parts of this job's evaluation and the requested amount the check actually compared against, and shall not also show that limit as a resource shortfall.
+6. While the detail level is `full`, when a host is skipped because an earlier group of the same plan already claimed it, the Simulator REPL shall show the host as already claimed.
+7. While the detail level is `full`, when a host could not be evaluated because of an internal error, the Simulator REPL shall show the host as not evaluated with the error.
+8. When a plan fails after some of its groups were placed, the Simulator REPL shall show that the hosts claimed by the earlier groups were released.
+9. While the detail level is `summary`, the Simulator REPL shall show, for each plan group, the hosts accepted and the number of hosts rejected for each verdict kind, counting a host with several rejection reasons once under each reason kind as in Requirement 9.3, instead of one line per host.
+
+### Requirement 6: Decision Outcome Explanation
+**Objective:** As an operator, I want each trace to end with the decision's concrete result, so that I can see exactly what was allocated or why the job stays pending.
+
+#### Acceptance Criteria
+1. When a traced job is dispatched, the Simulator REPL shall show in the trace, for each allocated host, each resource allocated with its amount, and, separately and labelled as limit charges, each host-scoped limit charged on that host with the limit name, its scope instance and the amount charged.
+2. When a traced job is dispatched, the Simulator REPL shall show once for the job the shared resources allocated with their amounts and, separately and labelled as limit charges, each cluster-wide limit charged with the limit name, its scope instance and the amount charged.
+3. The Simulator REPL shall show each limit charge only as a limit charge, never among a host's allocated resources or the job's shared resources.
+4. When a traced job is pended, the Simulator REPL shall show the pend reason set of the decision, with the same content as the pend decision the simulator echoes and records (`--sim-output`) and compares (`--sim-verify`), subject to the host-entry limit of Requirement 9.4.
+
+### Requirement 7: Trace Display and Recall
+**Objective:** As an operator, I want traces printed inline as the replay runs and recallable afterwards, so that I can read explanations as decisions happen and revisit a job later.
+
+#### Acceptance Criteria
+1. While the REPL is in step mode, when a traced decision is echoed, the Simulator REPL shall print that decision's trace immediately after the decision's echo line.
+2. While the REPL is in step mode, when a traced job's selection produces no decision (Requirement 2.4), the Simulator REPL shall print its trace at the point where the job's decision would have been echoed.
+3. While the REPL is in continue mode, when a traced decision is reported or a traced job's selection produces no decision, the Simulator REPL shall at that point end and report any active collapsed range and print the traced decision's echo line followed by its trace (or the no-decision trace), so that a round's traces appear in the order its decisions are reported, with reconstructed dispatches (Requirement 2.5) first, followed by the round's summary line when the round completes.
+4. While the REPL is in continue mode, the Simulator REPL shall not include a round that contains a traced decision or traced no-decision selection in a collapsed range.
+5. When execution stops at a breakpoint, the Simulator REPL shall have printed the traces of the round's traced decisions and no-decision selections reported up to and including the one at which execution stopped, and no later trace from the current round shall be available to `info trace` at the breakpoint; those printed traces shall be available to `info trace` unless discarded under the memory bound (Requirement 9.5), in which case Requirement 9.6 applies.
+6. The Simulator REPL shall present each trace as an indented hierarchy: job selection, then resource requirements, then shared-resource checks and plans, then host groups and hosts, then the outcome.
+7. The Simulator REPL shall show resource amounts in traces using the operator's current `set units` display mode and job identifiers using the REPL's existing job-identifier formatting.
+8. When the operator issues `info trace <jobid>`, the Simulator REPL shall print the most recent stored trace for that job, at the detail level at which it was captured, including for a job that has since finished or been removed from the replay.
+9. When the operator issues `info trace` with a job-array identifier, the Simulator REPL shall print the most recent stored trace of each element of that array that has one, mark each element whose trace was discarded under the memory bound, and, if no element has a stored trace, report that and list the elements' current states.
+10. When a new trace is captured for a job that already has a stored trace, the Simulator REPL shall replace the stored trace with the new one.
+11. If the operator issues `info trace <jobid>` for a job with no stored trace, the Simulator REPL shall report that no trace is available and show the job's current state, or state that the job is unknown.
+
+### Requirement 8: Trace File Copy
+**Objective:** As an operator, I want an optional copy of trace output written to a file, so that I can search or share long traces after the session.
+
+#### Acceptance Criteria
+1. Where the trace-enable command names a file, the Simulator REPL shall append to that file every trace printed inline under Requirement 7 (including no-decision traces and reconstructed-dispatch notices), with the same content as shown on the console, and shall not append `info trace` output.
+2. If the named file cannot be opened for writing, the Simulator REPL shall print an error naming the file and the cause, and shall leave the current trace settings unchanged.
+3. If the named path refers to the same file (same device and inode) as the simulation output file currently in effect (from `--sim-output` or `set output`) or the scheduler's log file, the Simulator REPL shall reject the command with an error and leave the current trace settings unchanged.
+4. If the operator issues `set output` naming the same file (same device and inode) as any trace file still open for writing, including one kept open under criterion 5 until its scheduling session ends, the Simulator REPL shall reject the command with an error and leave the simulation output file unchanged.
+5. When tracing is disabled or the trace settings are replaced, the Simulator REPL shall keep writing traces of decisions captured under the previous settings to the previous file, and shall close that file once the scheduling session in progress when the command was issued has ended, or immediately if no scheduling session was in progress.
+6. When the operator quits the REPL, REPL input ends, or the replay completes, the Simulator REPL shall finish writing every trace already printed inline to the trace file and close it before the simulator exits.
+7. The Simulator REPL shall not write trace content to the scheduler's regular log as a consequence of tracing.
+
+### Requirement 9: Trace Size and Memory Bounds
+**Objective:** As an operator tracing large jobs on large clusters or all jobs of a long replay, I want trace size and memory bounded, so that one decision cannot flood the console and tracing cannot exhaust simulator memory.
+
+#### Acceptance Criteria
+1. The Simulator REPL shall limit the requirement, plan and host sections of each decision's trace to a maximum number of lines, which the operator can set in the trace-enable command to a whole number from 50 to 100000, with a default of 1000 lines when none is given.
+2. When a decision's requirement, plan and host sections would exceed the size bound, the Simulator REPL shall stop capturing lines for them and show a truncation marker stating how many lines were omitted.
+3. When a trace is truncated, the Simulator REPL shall still show the job selection section and, in a number of lines that does not depend on the number of requirements, plans or hosts, the exact number of plans tried and the exact decision-wide number of host evaluations for each verdict kind, where a host evaluation is one host checked for one group of one plan and an evaluation with several rejection reasons is counted once under each reason kind.
+4. The Simulator REPL shall always show the decision outcome, listing at most 50 host entries followed, when more exist, by exact counts per pend-reason kind (for a pend) or the total amount of each resource allocated across the omitted hosts, excluding limit charges, which are totalled per limit (for a dispatch), and a marker stating how many hosts were omitted.
+5. The Simulator REPL shall bound the total memory used by stored traces to 64 MiB and, when the bound is reached, shall discard the oldest stored traces first, never discarding the most recently captured trace.
+6. If the operator issues `info trace <jobid>` for a job whose stored trace was discarded under the memory bound, the Simulator REPL shall report that the trace was discarded.
+
+### Requirement 10: No Interference with Scheduling
+**Objective:** As a developer relying on simulation for faithful reproduction, I want tracing to be purely observational, so that enabling it never changes the behavior I am investigating.
+
+#### Acceptance Criteria
+1. The xce_scheduler simulator shall produce the same sequence of scheduling decisions, allocations and pend reasons for a replay whether tracing is enabled or disabled, at either detail level and for any job selection.
+2. The xce_scheduler simulator shall produce the same `--sim-verify` results, and the same `--sim-output` recording and REPL `snapshot` content apart from wall-clock metadata fields, whether tracing is enabled or disabled.
+3. The xce_scheduler simulator shall produce the same final accounting and job state for a replay whether tracing is enabled or disabled.
+4. While tracing is disabled and no scheduling session that started with tracing enabled is still in progress, the Simulator REPL shall produce output identical to the output it produces without this feature, except for the `help` listing, the output of the new trace commands and `info trace` themselves, and collapsed-range boundaries at the first round after the last traced round.
+5. The Simulator REPL shall not change the behavior or output of any existing REPL command, except that `help` lists the new commands, `set output` rejects any trace file still open for writing (Requirement 8.4), and, while tracing is enabled, `step` and `continue` output includes traces as defined in Requirement 7.
+
+### Requirement 11: Simulator-Only Availability
+**Objective:** As an operator of a production cluster, I want decision tracing to be impossible to enable in the live scheduler, so that production scheduling never pays the cost of trace collection.
+
+#### Acceptance Criteria
+1. The xce_scheduler shall provide no command-line option, configuration setting or administrative request that enables decision tracing in the live (controller-connected) scheduler.
+2. While running as the live scheduler, the xce_scheduler shall not collect, store or output decision-trace information, and shall do no trace-related work beyond checking that tracing is disabled.
+3. While running a simulation without the interactive REPL, the xce_scheduler shall not collect or output decision-trace information.
+4. The xce_scheduler shall make the same scheduling decisions in the live scheduler as it did before this feature.

.kiro/specs/scheduler-sim-diagnostics/brief.md

@@ -23,7 +23,7 @@ Extend the existing REPL/simulation-driver surface (not a new component). Each n
 
 **Seed capability 1 — Limit inspection**: expose limit config (name, per_user/queue/project/host scope, value/percent) and live usage (per-vres-instance current value vs limit) via a new `info limit <name|all>` REPL command. Requires plumbing a read path from the `limit` plugin's private kvmgr tables to the REPL (core has no existing accessor for plugin-private state).
 
-**Seed capability 2 — Decision-reasoning trace**: capture *why* each scheduling decision was made — not just the outcome. For PEND, surface the existing `xce_pendreason_t` (currently captured but not printed). For DISPATCH, there is currently no trail at all of which limit/resource checks were evaluated and passed; this needs new instrumentation in the allocation path (`sched_try_alloc_on_host_`, `limit_prepare_allocation`) to record a per-attempt reasoning trace, then a REPL command to print it for a given job/decision.
+**Seed capability 2 — Decision-reasoning trace** *(MOVED 2026-09-28 to its own spec `scheduler-sim-decision-trace` — see `.kiro/specs/scheduler-sim-decision-trace/brief.md`; text below kept for history)*: capture *why* each scheduling decision was made — not just the outcome. For PEND, surface the existing `xce_pendreason_t` (currently captured but not printed). For DISPATCH, there is currently no trail at all of which limit/resource checks were evaluated and passed; this needs new instrumentation in the allocation path (`sched_try_alloc_on_host_`, `limit_prepare_allocation`) to record a per-attempt reasoning trace, then a REPL command to print it for a given job/decision.
 
 ## Scope
 

.kiro/specs/scheduler-sim-diagnostics/requirements.md

@@ -5,7 +5,7 @@ Operators investigating scheduler behavior in the `--interactive` simulation REP
 
 ## Boundary Context
 - **In scope**: REPL-visible display of a limit's configured setting (the resource it constrains, its configured value or percentage, and which scope dimensions it effectively varies by, including any host-scoping implied by a percentage value on a resource that is not shared cluster-wide) and its live accounted usage (per distinct scope instance, including explicit exclusion-based scope restrictions), in simulation mode only.
-- **Out of scope**: any command that mutates scheduler state; live, controller-connected (non-simulation) inspection; explaining *why* a specific job did or did not trigger a limit (decision-reasoning trace — expected to be added later as a separate requirement of this same open-ended spec, not covered here); any change to scheduling, allocation, or limit-matching behavior.
+- **Out of scope**: any command that mutates scheduler state; live, controller-connected (non-simulation) inspection; explaining *why* a specific job did or did not trigger a limit (decision-reasoning trace — moved 2026-09-28 to its own spec `scheduler-sim-decision-trace`, not covered here); any change to scheduling, allocation, or limit-matching behavior.
 - **Adjacent expectations**: this capability extends the REPL's existing `info` command family (shipped by `scheduler-core`, implementation-complete) without altering any existing subcommand's behavior or output; it must not perturb the guarantee that a simulation replay reproduces identical scheduling decisions and identical accounting state regardless of what inspection commands are issued during it. Requirement 6 (added 2026-09-19, following operator feedback that a limit's raw byte amounts are hard to read) relies on the REPL's existing byte-unit display convention already used by other `info` subcommands for byte-valued resource amounts, including an operator's existing ability to select how those amounts are displayed — it extends that existing convention to limit displays rather than introducing a new one.
 
 ## Requirements