myxce/.kiro/specs/scheduler-sim-decision-trace/research.md

# Gap Analysis: scheduler-sim-decision-trace

_Date: 2026-10-07. Codebase: branch `scheduler-framework` @ `33437a39`, after `scheduler-allocator-runner`, `scheduler-sim-discrete-clock` and `scheduler-sim-repl-verbosity` reached implementation-complete. Paths are relative to `src/scheduler/` unless they start with `src/` (repo root) or `.kiro/`._

## 1. Summary

- All upstream dependencies have landed. Plan and bucket hooks have a single home: `sched_allocator_run()` in `src/alloc_runner.c` (490 lines). There is no `alloc_iter.c`; the interface is `include/sched/alloc_iter.h`.
- Almost every fact the trace needs is computed today and then thrown away at a known site (§3). Only two things are not available where the trace needs them: the per-resource `total/inuse/uncommitted/wanted` numbers (they exist only inside `unsatisfy_`) and a "selected but no decision" channel.
- **Three requirements no longer match the code and should be revised before design** (§4): Req 5.2 (pre-filter "carried over" request), Req 8.3 (the scheduler log path is not known to the sim), and the token limits that affect Req 1/8 argument parsing.
- Most of the cost is integration rather than algorithms. About 10 instrumentation sites, a new decision op or side channel, refactoring the REPL formatting helpers, and updates to about 26 white-box tests that `#include` REPL sources.
- Recommended approach: **Option C (hybrid)**. New `sched_trace.c` and `repl_trace.c` modules hold the logic. Hooks in existing files are one line each, behind a NULL-pointer gate. Two small prerequisite refactors come first.

## 2. Corrections to brief.md

| Brief says | Code today |
|---|---|
| Plan/bucket hooks go in "`sched_allocator_run()` once it lands" | It has landed: `src/alloc_runner.c:457` (`sched_allocator_run`), `:414` (`sched_alloc_search_`), `:270` (`sched_try_plan_`), `:208` (`sched_claim_bucket_`) |
| Shared-check loop bound bug (`alloc_req.shared.count` vs `unmeet.count`) is out of scope | Already fixed (bug-1893), `alloc_runner.c:65,89` |
| ALTER FIXME at `scheduler.c:427` | `scheduler.c:433` |
| `sim_print_decision` at `sim_loop.c:479`, drain loop at ~1607 | `sim_loop.c:529`, drain loop `:1675-1710` |
| "`core*.c`/`sim*.c` may not do I/O" is enforced by tests | Not enforced. `test_core_isolation.c` and `test_sim_isolation.c` forbid controller/network headers and symbols, and wall-clock calls in core; they do not forbid stdio. Keeping `sched_trace.c` I/O-free is a design rule we choose, not one a test checks. |
| `scheduler.c` is 1382 lines (`src/scheduler/CLAUDE.md`) | 2002 lines. CLAUDE.md is stale. |

## 3. Requirement-to-asset map

Tags: **Exists**: the data is available at the seam. **Discarded**: computed, then dropped. **Missing**: not computed anywhere. **Constraint**: an architectural limit.

### Req 1: Trace control
| Need | Asset | Gap |
|---|---|---|
| `trace` command | `repl_dispatch` strcmp chain `repl.c:408` (unknown fallback `:479`) | Missing. Add a branch that routes to `repl_trace.c`. |
| Help section | `repl_print_help` `repl.c:269`. `test_repl_help.c:569-670` checks section order, with a lower bound only on the last section. | Missing. A trailing "Trace" section is safe. |
| Job/array selection | `parse_jobsel`/`jobsel_t` (`src/include/utils.h:231-278`), `array_jobid`/`array_idx` (`src/lib/utils.c:660/671`) | Exists |
| Settings take effect at the next session (1.5/1.6) | `core->session_active` (`include/core.h:260`), set/cleared in `core_session.c:420/479` | Exists. Caveat: at a BEGIN_ROUND breakpoint `session_count` has already been incremented but `session_active==0`. Design must define "next session" there. |
| Pass settings to the allocator thread with no new sync | Window between `xce_core_begin_session` and `xce_core_signal_go` (`sim_loop.c:1621-1673`). The allocator only reads core state after the mutex chain wbuf → picker → jbuf. | Exists. Snapshot settings into a core field in that window. |
| Argument parsing | `repl_split_tokens` (`repl.c:116`): at most **16 tokens of 63 characters** | **Constraint**. Paths longer than 63 characters and long jobid lists do not fit (see §4). |
| `trace status` memory use | `xce_sched_repl_state_s` (`repl_internal.h:171-304`) | Missing. Needs a new state block. |

### Req 2: Job selection
| Need | Asset | Gap |
|---|---|---|
| Session number | `rs->session_count` (`repl_internal.h:178`) is the current session during drain | Exists (render side) |
| 1-based position in session | None. jbuf carries a bare `uint64_t` (`core.c:167`, pinned by `test_core_jbuf_capacity_pin.c:227`) | Missing. Count non-zero, non-sentinel pops in `sched_allocator` (`sched_election.c:197-231`), reset on 0. jbuf is unchanged. |
| Policy and ordering keys | `job->queue->{name,priority}`, `job->priority`, `job->queue_time` (`include/sched/types.h:101-146`); cmp functions `builtin_fcfs.c:183,206` | Exists. Read on the allocator thread after `sched_get_job`. Sim only ever runs `builtin_fcfs` (`sim_picker.c:56-70`). |
| Cluster-wide limits matched | `obj->glimits` filled by `limit_job_election` (`plugins/limit.c:318`) from `plugin_job_election` (`sched_election.c:246`). It is cached per `configver` (`limit.c:336`). | Exists. Read `glimits` after election; do not instrument the match loop. |
| No-decision cases (2.4) | `sched_election.c:226-231` (not found), `:250-254` (`alloc_rc<0`, silent), `:265-269` (pend with NULL reason), `:286` (calloc failure) | **Missing channel**. Nothing reaches the drain loop. See §5 Decision D2. |
| Reconstructed dispatch (2.5) | `sim_reconcile_unaccounted_jobs` (`sim_loop.c:1026`, block `:1309-1329`), runs before `signal_go` | Exists on the render side. The decision does not mark it as reconstructed, but the call site knows. |

### Req 3: Resource requirement selection
| Need | Asset | Gap |
|---|---|---|
| Nesting path | Fixed shape: PRIMARY, COMPOUND (list of primaries), ALTER (list of compounds) (`scheduler.c:306/403/409`, bug-1769) | Exists. Path depth is at most 3. |
| Compound member index and point of failure | `sched_make_compound_alloc_` loop `scheduler.c:339-356`; breaks on first `rc<=0` | **Discarded** (index not kept) |
| Per-alternative outcome and reasons (3.5) | ALTER loop `:409-436`. Each alternative frees and replaces `obj->reason` (`:394-395`). NULL `resreq` is skipped silently (`:423`). | **Discarded**. The trace must snapshot (deep-copy) each alternative's reasons before they are freed. It must not change `obj->reason` itself; that defect stays out of scope. |
| min/max tasks and requested resources | `xce_primary_resreq_t{minTasks,maxTasks,select,rusages}`; staged amounts in `ctx->alloc_req.{shared,host}` | Exists. Note: `minTasks` is never used by allocation, so the trace shows it but nothing depends on it. |
| `accepts()` rejection | `alloc_runner.c:474-476` returns 0 with **no reason**. In PRIMARY the stale `obj->reason` survives (`scheduler.c:324`). | Discarded. The trace should say "allocator did not accept resreq". The stale-reason behaviour is a candidate bug to log separately. |

### Req 4: Shared checks and plans
| Need | Asset | Gap |
|---|---|---|
| Shared check per resource | `sched_alloc_prepare_shared_` (`alloc_runner.c:68-111`) → `sched_check_shared_satisfy` (`:82`) | total/inuse/uncommitted/wanted exist only inside `unsatisfy_` (`resource.c:147-189`, TRACE at `:159,179`). `unmeet` keeps only the shortfall. **Discarded.** Needs a recorder hook inside `unsatisfy_`. |
| Cluster-wide limit vs resource | A limit charge is a vres in `.shared` (`sched_fill_limit_requests`, `resource.c:1062-1099`), with flag `XCE_RESF_LIMIT` (`src/lib/xcelimit.c:526`) and name `limit=<n>[/u=..][/q=..][/p=..]`. Structured `xce_vres_t{vres,res,limit}` (`src/include/xce/types.h:831-844`). | Exists. Classify by `XCE_RESF_LIMIT`, and map idx to `xce_vres_t` through `obj->glimits` / `obj->hlimits[w->idx]`. Pend reasons still say `NO_RESOURCE` (unchanged, out of scope). |
| Plans in order, shape, outcome | `sched_alloc_search_` `:414-455`; `plan->bucket_slots[]`; `sched_try_plan_` knows the failing bucket `b` (`:283`) | **Discarded** |
| No plan generated (4.7) | `plan_init` failure `:427-429` (no reason); malformed plan skipped `:432` | Discarded |
| "Host groups" | Buckets are slot partitions only (seqjob: `[1]`; parjob k=1..maxTasks balanced, `parjob_allocator.c:245-288`) | Terminology: "host group" in the requirements means "bucket". |

### Req 5: Host verdicts
| Need | Asset | Gap |
|---|---|---|
| Select mismatch | `host_iter.c:136-140`, silent `continue` | **Discarded**. Hook in the iterator. |
| Pre-filter vs per-host | Pre-filter is **select-only**. The runner clears `.host` before each `next()` (`alloc_runner.c:216,253`), and `sched_check_host_satisfy` returns 1 on an empty request (`resource.c:419`). Its satisfy branch (`host_iter.c:159-165`) is effectively dead under the runner. | **Req 5.2 is stale** (§4) |
| Per-resource shortfall numbers | `sched_try_alloc_on_host_` `resource.c:530-608` → satisfy `:575` → `unsatisfy_` | Discarded (same recorder hook as Req 4) |
| Host-scoped limit | `obj->hlimits[w->idx]` from `limit_prepare_allocation` (`limit.c:562-607`) | Exists (classify by flag) |
| Already claimed (5.6) | `sched_alloc_is_claimed_` `alloc_runner.c:229`, silent `continue` | Discarded |
| Internal error (5.7) | `sched_try_alloc_on_host_` `<0` (`resource.c:577-579`, uncommitted fetch `:559`); iterator uncommitted error `host_iter.c:149-156`. Runner merges `<0` with rejection (`alloc_runner.c:236`). | Discarded. Visible at the call site. |
| Unreachable `PREASON_MATCH_FAIL` | `resource.c:543-557` | Note only. Select mismatches never reach it. |
| Rollback (5.8) | `sched_rollback_claims_` `alloc_runner.c:187-196` | Discarded |

### Req 6: Outcome
| Need | Asset | Gap |
|---|---|---|
| Per-host allocs, shared allocs | `sched_finalize_job_alloc_` `alloc_runner.c:347-406`; `job->allocs` moved into the decision (`sched_election.c:280`) | Exists. Limit charges are mixed in (host vres in host allocs, global vres in `allocs_shared`). Separable by `XCE_RESF_LIMIT`, but at render time the resource index→flag table is needed. |
| Pend reason set | `make_pend_decision` stores a **pointer** to `job->schedobj->reason` (`decision.c:6-19`) | Constraint. A stored trace for `info trace` must hold its own copy, rendered text or a deep copy, because `schedobj->reason` is replaced in later sessions. |

### Req 7: Display and recall
| Need | Asset | Gap |
|---|---|---|
| Inline after echo (step) | Drain loop order: verify → record → `sim_print_decision_gate` (`:1705`) → `xce_repl_check_decision` (`:1706`) → `free` (`:1709`). Reconstructed: `:1309-1329`. | Exists. Insert render between gate and breakpoint check. This gives 7.5 for free (stop after the trace prints). |
| Continue mode: no collapse for traced rounds | Rollup `repl_rollup.c:285` compares dispatch/preempt counts plus the pend-jobid set from `core->jobs`; flush at `:233` | Missing. Needs a "round had traced output" flag that forces a flush, and must print the echo line that continue mode normally suppresses (`sim_loop.c:587`). |
| Units / jobid formatting | `translate_units_mode`, `repl_format_numeric_resource_value`, `repl_format_consumable_value_and_used`, `repl_current_units_mode` are **static in `repl_info.c`** (1893 lines, `:385-487`). `repl_fmt_jobid_range` (`repl_fmt.c:223`), `fmt_jobid`. | **Prerequisite refactor**: move these to `repl_fmt.c` |
| `info trace` | `repl_cmd_info` `repl_info.c:1837` | Missing. Add a one-line route to `repl_trace.c`. |
| Job state for "no trace" (7.11) | `sched_get_job` / existing `info job` helpers | Exists |

### Req 8: File copy
| Need | Asset | Gap |
|---|---|---|
| dev/inode vs sim output | `sim->record_file` (FILE*, `fstat`able); `rs->output_path` only set by `set output` | Exists (use `fstat(fileno())`) |
| dev/inode vs scheduler log | Log path is a local in `main.c:228/280`, passed only to `mylog_init` | **Missing**. Needs a new field threaded into the sim args, or drop it from the requirement (§4). |
| `set output` rejects open trace file (8.4) | `repl_params.c:241/197` | Missing. Small check. |
| Close at quit/EOF/complete | `repl_cmd_quit` (`repl.c:330`, `exit(0)`), `xce_repl_on_replay_complete` (`repl.c:609`), batch close `sim_loop.c:1785-1798`. `repl_state` is never freed. | Missing. Add trace-file close in quit and complete. |

### Req 9: Bounds
Missing. Pure new logic in `sched_trace.c` / `repl_trace.c`. Note: the reason lists accumulate **across plans without dedup** (`add_*_preason`), so a parjob pend reason can be very large. Req 9.4's cap of 50 host entries is needed in practice, not just in theory.

### Req 10/11: Non-interference and sim-only
| Need | Asset | Gap |
|---|---|---|
| One-branch gate in live | Precedent: NULL-checked core callbacks `unsatisfiable_*_cb`, `cache_clear_cb` (`include/core.h:247,258-259`) | Exists. A `core->trace` pointer, NULL in live and in non-interactive sim. |
| Unsatisfiable cache in sim | `unsatisfiable_lookup_cb` NULL in sim (`scheduler.c:254`) | Exists. A trace never explains a cache hit. |
| Clock unaffected | `xce_core_now` with `speed_num=0` in sim (`sim.c:332`, `core_clock.c:80-121`) | Exists |
| Equivalence testing | `test_repl_equivalence.c` pattern; `--sim-verify`; `scheduler-driver-equivalence.md` | Exists. Add trace-on vs trace-off comparisons. |
| Must not touch `--sim-verify` positional compare | `sim_verify` `di++` per decision | **Constraint** for D2: a trace-only decision must be skipped by verify, record, gate and the REPL tally. |

## 4. Requirements that need revision (recommended before approval)

1. **Req 5.2** assumes the pre-filter compares "a request carried over from an earlier candidate host". Since `scheduler-allocator-runner` (Req 3.6, `alloc_runner.c:137-138,253`) the pre-filter is select-only. Suggested rewording: "...indicate whether the host was rejected by the selection pre-filter or by the per-host resource and limit check." Delete the carried-over clause.
2. **Req 8.3** requires a same-inode check against the scheduler log file, but the sim never sees that path. Either keep the requirement and accept a small `main.c` → sim-args plumbing change, or narrow it to the simulation output file. Keeping it is cheap; flag it as a decision for the user.
3. **Req 1 / Req 8 token limits.** `repl_split_tokens` caps arguments at 16 tokens of 63 characters. Long file paths and long jobid lists (`trace on job 1 2 3 … file /long/path`) will fail. Decide between: (a) raise the limits in the tokenizer (affects every command and its tests), (b) a trace-local re-parse of the raw line, or (c) document the limit in the requirement. Option (b) is the least invasive.
4. **Req 3.2** "minimum task count": it can be shown, but allocation never uses `minTasks`. No change needed, but the design should not suggest it influences plans.
5. Terminology: requirements say "host group"; the code says "bucket" (a slot partition). The design should state the mapping once.

_Applied 2026-10-07 (user-approved):_ item 1 (Req 5.2 rewritten), item 2 (Req 8.3 kept, "if one is in use" added; the log path must be passed through to the sim), item 3 (new Req 1.12: any number of job identifiers and paths up to 4095 bytes; REPL input is libedit `readline()` with no line limit, so `trace` can re-parse the raw line), item 5 ("host group" replaced by a defined *plan position* = one bucket = exactly one host). Review fixes in the same pass: Req 1.6 defines "in progress" (BEGIN_ROUND breakpoint, before the session starts selecting jobs, counts as the next session); new Req 3.6 (skipped member or alternative); Req 4.7 covers an allocator declining a requirement; Req 6.4 no longer claims the echo line shows pend reasons; Req 7.8 renders `info trace` in the current `set units` mode (implies structured storage, D3).

_Review loop 2026-10-07 (7 passes; passes 2-7 by independent reviewers checking against code; pass 7 clean)._ Further changes: glossary in the Introduction defines *in progress* / *ended* / *reported*; Req 2.4 adds the no-scheduling-object case (bug-1897 path, `sched_election.c:236`) and decision-build failure; Req 3.2/3.3 internal-error outcomes; Req 3.6 limited to alternatives (compound members are never skipped, `scheduler.c:339-356` vs `:423`); Req 4.4 covers internal errors when trying a plan, when a plan cannot be tried, and when the host scan fails (a scan failure leaves the position unfilled and later plans are still tried, `alloc_runner.c:222-226`); Req 4.5 counts a plan as committed only after its allocation is built; Req 4.8 (cluster-wide check failed, so no plans tried); Req 4.9 (discarded malformed plans); Req 5.10 lists the verdict kinds ("not performed" checks are not a verdict); Req 5.9 names the limits hit in summary mode; Req 6.1/9.4 host cap semantics; Req 7.11 vs 9.6; Req 10.5 exceptions aligned with 10.4 and 1.6.

## 5. Implementation approach options

### Option A: Extend existing files in place
Put hooks with inline formatting into `alloc_runner.c`, `host_iter.c`, `resource.c` and `scheduler.c`, and put rendering in `sim_loop.c` and `repl_info.c`.
- ✅ Fewest new files.
- ❌ `sim_loop.c` (1819), `repl_info.c` (1893), `scheduler.c` (2002) and `driver.c` (1104) are already over the 800-line cap. This breaks the project convention and the brief's constraint.
- ❌ It mixes rendering into hot-path files and makes the "one branch when off" property hard to audit.
- **Not viable.**

### Option B: New modules only
`include/sched/trace.h` + `src/sched_trace.c` (record model, buffer, bounds, I/O-free) and `src/repl_trace.c` (commands, settings, rendering, store, file). Existing files get only one-line gated hook calls.
- ✅ Clean boundary, easy to unit-test the buffer and renderer in isolation, hook cost is easy to audit.
- ❌ Needs ~10 hook sites with a stable argument contract, and the renderer needs formatting helpers that are currently `static` in `repl_info.c`.

### Option C: Hybrid, Option B plus enabling refactors (recommended)
Option B, plus these preparatory steps, each separately testable:
1. Move the units and value formatting helpers from `repl_info.c` to `repl_fmt.c`, with no behaviour change.
2. Add a recorder seam in `unsatisfy_` (or a sibling that returns `total/inuse/uncommitted/wanted` per unmet idx) used only when tracing.
3. Add the no-decision channel (D2) and a "reconstructed" marker at the reconcile call site.
4. Register `sched_trace.c` in `sched_core_isolation_lib` and `repl_trace.c` in the `test_sim_isolation.c` scans and library. Add `#include "../src/repl_trace.c"` to the ~26 white-box tests.
- ✅ Keeps every new concern in a file under the cap, makes prerequisites reviewable on their own, and keeps the live path to a NULL check.
- ❌ More tasks and a larger diff on test files.

### Key design decisions to carry forward
- **D1: Where the buffer lives.** Choices: (a) `core->trace` (settings snapshot plus current buffer), NULL when off, with one allocator thread per core; or (b) a pointer on the stack-allocated `xce_sched_ctx_t`. (a) reaches `sched_election.c` and `host_iter.c` without changing signatures, and matches the `unsatisfiable_*_cb` precedent. (b) has narrower scope, but ctx does not exist at the election and no-decision sites. **Lean: (a)**, with settings snapshotted between `begin_session` and `signal_go`.
- **D2: No-decision trace channel.** Choices: (a) push a decision with a new op such as `XCE_DECISION_TRACE` and payload `trace`, which every consumer (verify, record, gate, REPL tally, live driver) skips; (b) a side queue drained in the decision loop, which loses positional order relative to decisions. Req 7.2/7.3 need ordering, so (a) is effectively required. Append the op **after** `PREEMPT` so existing values are not renumbered, and coordinate with `scheduler-preemption`, whose draft already disagrees with `decision.h` on PREEMPT placement.
- **D3: Captured representation.** Store structured records (with copied numbers, names and reason snapshots) and render on the sim side, or render to text on the allocator thread. The brief's constraint (no formatting on the allocator thread) favours structured records. For `info trace` storage, rendered text is simplest for the 64 MiB accounting, but it must be re-rendered if units change. Req 7.8 says "at the level captured", not at the units captured, so this is open.
- **D4: Per-resource numbers.** Instrument `unsatisfy_` with a NULL-gated record callback rather than widening `xvec_idval_t unmeet`. Widening would touch every caller of the satisfy API.
- **D5: Limit classification.** Use `XCE_RESF_LIMIT` plus `xce_vres_t` lookup through `obj->glimits` / `obj->hlimits[w->idx]`, captured on the allocator thread. These lists are rebuilt only when the configuration version changes, so they are stable within a session.
- **D6: ALTER per-alternative reasons.** Deep-copy `ctx.reason` into the trace before `scheduler.c:394-395` frees it. Do not change `obj->reason` semantics (out of scope; separate bug).

## 6. Effort and risk

- **Effort: L (1–2 weeks).** About 10 hook sites across 6 hot-path files, a new decision op through 3 consumers, two new modules (model/buffer/bounds and REPL/render/store/file), prerequisite refactors, about 26 test-include updates, and equivalence tests. All of it follows existing patterns.
- **Risk: Medium.** No new technology, and the gating precedent exists. The risks are concentrated in: (1) a new decision op leaking into `--sim-verify`, recording or live driver consumers and breaking determinism; (2) hook arguments being evaluated on the live path (the one-branch rule needs macro discipline); (3) continue-mode rollup and breakpoint ordering interactions.

## 7. Research needed for the design phase

1. Whether a decision-pointer `free` in three places (`driver.c:1093`, `sim_loop.c:1709,1327`) is enough to own a trace buffer, or whether the buffer should be pooled per session to avoid allocator-thread malloc churn.
2. A hook macro shape (`SCHED_TRACE(core, kind, ...)`) that does not evaluate arguments when `core->trace == NULL`, and a test that proves it, for example a counting side-effect argument.
3. A resource idx → `XCE_RESF_LIMIT` lookup available at render time for the outcome section, or captured at finalize.
4. Ordering-key drift between pick and pop (an `on_job_update` landing in between). It is believed benign in sim because events are pushed before the session starts; confirm.
5. Exact interaction with the BEGIN_ROUND breakpoint: `session_count` has already incremented, `session_active==0`, so which session counts as "next"?
6. Line-count budget for `repl_trace.c`: commands, settings, store, file and renderer will likely exceed 800 lines, so plan a split such as `repl_trace.c` (commands/settings/file/store) and `repl_trace_render.c`.
7. Candidate bugs to log separately (not fixed by this spec): stale `obj->reason` after an `accepts()` rejection (`scheduler.c:324`, `alloc_runner.c:474`); a plan `init` failure pending with no reason (`alloc_runner.c:427`); pend reasons duplicated across parjob plans; `PREASON_MATCH_FAIL` branch unreachable (`resource.c:543`); the unsatisfiable cache populating `main_reason`, which the runner never sets (`scheduler.c:282-289`); `src/scheduler/CLAUDE.md` line counts stale.

---

# Design Phase Research (2026-10-07)

## Summary
- **Feature**: `scheduler-sim-decision-trace`
- **Discovery Scope**: Extension. Light discovery, integration-focused, on top of the gap analysis above. No new external dependencies.
- **Key Findings**:
  1. Every allocator-side fact the trace needs can be read **on the allocator thread** at the point it is computed. Reading the same data later on the sim thread would race. The sim drain loop runs concurrently with the allocator working on the next job, and `obj->hlimits` / `worker->resvals` are mutated by `limit_prepare_allocation` during allocation. So the outcome (allocation and pend reasons) is also captured on the allocator thread, before the decision is pushed.
  2. `unsatisfy_` (`resource.c:147`) is the single place where `total/inuse/uncommitted/wanted` exist for both the cluster-wide check and the per-host check. One recorder hook there serves Req 4.1, 4.2, 5.4 and 5.5. Limit rows are told apart by the resource's `XCE_RESF_LIMIT` flag (`xce_resource_t.flags`, via `vget(core->resmgr, idx)`). The limit name and scope come from the `xce_vres_t` found in `obj->glimits` or `obj->hlimits[w->idx]` whose `vres->idx` matches.
  3. `repl_dispatch(sim, line)` (`repl.c:408`) receives the raw input line, and input comes from libedit `readline()` with no length limit. `trace` can therefore re-parse the raw line itself (Req 1.12) without touching `repl_split_tokens`.
  4. The continue-mode rollup (`repl_round_rollup_process`, `repl_rollup.c:285`) is called once per round from `xce_repl_post_session_check`. A per-round "traced" flag checked at its top is enough for Req 7.3/7.4: flush, emit this round alone, and do not start a range.
  5. `main.c` always has a log path (default `/tmp/xce_scheduler.log`, `main.c:228`), so Req 8.3's log-file check always applies once the path is plumbed through `sched_sim_args_t`.
  6. White-box REPL tests `#include` REPL/sim sources (`test_repl_equivalence.c:132-155`) and link `scheduler_test_deps`. Any new file that `repl.c`/`sim_loop.c` calls must be added to those include lists (about 26 tests) or to a library they link. `sched_sim_isolation_lib` (`meson.build:163`) must gain the new `repl_trace*.c` files.

## Research Log

### Thread ownership of trace data
- **Context**: Req 10 (no interference) and the "no new cross-thread synchronization" constraint.
- **Findings**:
  - The allocator thread runs `sched_allocator` (`sched_election.c:162`). The sim thread runs `sched_sim_driver_run` (`sim_loop.c`).
  - The sim thread writes core state only between `xce_core_begin_session` and `xce_core_signal_go`, and after the drain loop has seen the session-end NULL. The allocator is idle (blocked on `jbuf`) in both windows.
  - Values handed sim → allocator are ordered by the wbuf/jbuf cbuf mutexes. Values handed allocator → sim travel through the `core->decisions` cbuf.
- **Implications**: The settings snapshot (`core->trace_ctl`) is written only in those windows. The record under construction (`core->trace_rec`) is touched only by the allocator thread. A finished record changes owner by riding on the decision. No locks are needed.

### Decision transport for no-decision selections
- **Context**: Req 2.4, 7.2, 7.3 require the trace to appear in the decision order.
- **Findings**: The drain loop ends on a NULL decision. `--sim-verify` compares positionally (`di++`, `sim_loop.c:1690`). Recording and the breakpoint tally process every decision.
- **Implications**: Append a new op `XCE_DECISION_TRACE` after `XCE_DECISION_PREEMPT` (no renumbering). The drain loop handles it first and `continue`s before verify, record, gate and breakpoint run. `driver.c` frees and skips it defensively; live never produces one.

### Shortfall numbers
- **Context**: Req 4.1, 4.2, 5.4, 5.5.
- **Findings**: `unsatisfy_(core, nres, idxs, total, inuse, uncomitted, wanted, ...)` is reached from `sched_check_shared_satisfy_` (`resource.c:~290`) and from the host satisfy path (`resource.c:~384`). Only TRACE logging uses the values today.
- **Implications**: One hook at the top of `unsatisfy_` passes all arrays to the current record. The record keeps every row inside a cluster-wide-check scope (4.1 shows passing rows) and only unmet rows inside a host scope (5.4).

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Per-decision record + render in sim driver (selected; brief's choice) | Allocator appends typed entries to an arena-backed record; record rides on the decision; REPL renders, stores and copies to file | No I/O on core thread; per-job recall; deterministic ordering | Needs a new decision op and allocator-thread outcome capture | Matches the existing `unsatisfiable_*_cb` NULL-gate precedent |
| Streaming callback | Hooks call a driver callback per event | No buffer | I/O on allocator thread; picker/allocator output interleaving; no recall | Rejected in brief |
| Render text on allocator thread | Hooks format strings | Simple store | Formatting cost on allocator; units fixed at capture (violates 7.8) | Rejected |

## Design Decisions

### Decision: Two gate pointers on the core
- **Context**: Req 11.2 allows at most one branch per hook when off.
- **Alternatives Considered**:
  1. A single `trace_ctl` pointer, with every hook reading `ctl->rec`: two loads.
  2. `trace_ctl` (session settings) plus `trace_rec` (current record): hooks test only `trace_rec`.
- **Selected Approach**: Option 2. `SCHED_TRACE(core, fn(core->trace_rec, ...))` evaluates nothing unless `core->trace_rec != NULL`. Only the allocator's per-job prologue tests `trace_ctl`.
- **Rationale**: One predictable branch per hook. The live core never sets either pointer.
- **Trade-offs**: Two fields on `xce_sched_core_t` instead of one.

### Decision: Entry budget equals line budget
- **Context**: Req 9.1–9.3 bound lines, but the record is structured.
- **Selected Approach**: Each entry in the requirement, check, plan and host sections renders as exactly one line. The capture-side budget counts entries, and the renderer contract guarantees one line per entry. Counters (plans tried and discarded, verdict counts, limit tallies) keep updating after the budget is exhausted.
- **Rationale**: Truncation happens at capture, so memory is bounded too (9.5), and the omitted-line count is exact.

### Decision: Outcome captured on the allocator thread, capped at capture
- **Context**: Req 6, 9.4; the stored trace must outlive `job->allocs` and `schedobj->reason` (7.8).
- **Selected Approach**: After the decision is built, the allocator copies the first 50 hosts (allocation rows, or pend-reason entries) plus all host-less entries, and aggregates the rest (per-code counts, or per-resource and per-limit totals). Reconstructed dispatches (Req 2.5) are captured the same way on the sim thread, inside the pre-`signal_go` window.
- **Rationale**: This avoids racing the allocator and gives a bounded record size.

### Decision: Structured store, render on demand
- **Context**: Req 7.7 / 7.8 (units mode at render time), 8.1 (file copy has the same content as the console).
- **Selected Approach**: Records hold raw floats plus the resource name and flags. The renderer formats with the current `set units` mode. The inline print renders once to the console and the file (same bytes); `info trace` renders again later.

### Decision: Settings lifecycle pending → active → retired
- **Context**: Req 1.4–1.7, 8.5.
- **Selected Approach**: REPL commands write *pending*. The session-begin hook promotes pending to *active* and publishes `core->trace_ctl`. A replaced or disabled active setting whose file is still open becomes *retired* until the session-end hook closes it.

### Generalization / Build-vs-Adopt / Simplification
- **Generalization**: One recorder path handles the cluster-wide and per-host checks (scope decides which rows to keep). One renderer serves inline, file and `info trace`.
- **Build vs adopt**: No suitable library exists (structured in-process trace, C, sim-only). We reuse libxce `htab` for the jobid→trace index, `parse_jobsel`/`array_jobid` for selection, and the `repl_fmt` helpers for units.
- **Simplification**: No callback indirection. The sim driver is the only consumer, so hooks call `sched_trace_*` functions directly behind the NULL gate. No per-session pooling (records are arena-allocated per traced job; untraced jobs allocate nothing).

## Risks & Mitigations
- New decision op leaks into verify, recording or the live driver → handle the op first in each drain loop; equivalence tests with trace on vs off (`--sim-verify` and `--sim-output` byte compare).
- Hook arguments evaluated in live → statement macro that evaluates its call only inside `if (core->trace_rec != NULL)`; a unit test with a side-effect argument proves it.
- Trace changes scheduling via shared state → capture functions take `const` inputs and never call mutating scheduler APIs; the trace-on/off equivalence tests use a resource-contention fixture (per `scheduler-driver-equivalence.md`).
- ~26 white-box test include lists → update them in one task right after the new files exist; the build fails loudly if one is missed.
- File-size cap → three REPL-side files plus two core-side files, each planned under 800 lines.

## References
- `.kiro/steering/scheduler-driver-equivalence.md`: state-equivalence invariant and contention-fixture rule.
- `.claude/skills/xce-audit-invariants/SKILL.md`: scheduler audit dimensions 1–4.
- `.kiro/specs/scheduler-allocator-runner/`: runner contract (`alloc_iter.h`).

## Design Review Gate (2026-10-07)
- Pass 1 (self): settings lifecycle reduced to reference-counted `next`/`cur` (no separate "retired" state); `pos_scan_error` API added; scope-begin calls take `obj`/`w`; per-host 0 with no rows means selection mismatch (`resource.c` MATCH_FAIL branch); `sim_print_decision` exposed for the continue-mode echo.
- Pass 2 (independent, checked against the code): the link impact on about 20 white-box tests that include hooked sources (D1); `search_end` entry outside the budget plus a `NONE_GENERATED` reason (D2); `round_traced` reset at session begin (D3); `info trace` job-state read placed in the same class as `info job` (D4); resource pre-filter invariant added as a revalidation trigger (D5); host-scope `h=<host>` and name-based limit resolution for reconstructed records (D6); rollback hook only when n > 0 in `sched_try_plan_` (D7); path/no-plan/pos-end enums and bound constants defined (D8); `utils.h` dependency and the isolation-scan target named (D9); reconstructed-block call order spelled out (D10). Threading, drain-loop order, breakpoint ordering and the Req 10/11.2 claims were confirmed.

## Design Review Loop (2026-10-07, after design generation)
Eight independent review passes, each checked against the code; pass 8 found no issues.
- Pass 1 (1 major, 7 minor): compound-container hooks and merge-failure marking; `search_end` counted in the budget, with decision-wide counters; `round_traced` set in every mode; limit units via the limited resource; missing API declarations; `repl.h` externs; `d->trace` ownership; anchor fix.
- Pass 2 (1 high, 1 medium, 2 low): path push/pop with balanced begin/end (non-scope `member_failed` / `alt_result`); schedobj passed for limit-charge resolution; rollup rule after the pend query; session-begin hook after the `rc < 0` check.
- Pass 3 (2 medium, 4 low): no-decision internal-error records render the full hierarchy; renderer example made line-exact; rollup idle formula; `NOT_FOUND` has no selection; `trace_rec` cleared on every hand-off; name-based limit-charge matching; entry-type list and traceability corrected.
- Pass 4 (1 medium, 2 low): failure-reasons line depth; `nmembers` on `resreq_begin`; "unavailable" lines for 2.4.
- Pass 5 (1 medium, 2 low): error-path tests use the real allocator; `search_end`/`rollback` take no data the runner lacks; Modified Files rollup line aligned.
- Pass 6 (2 medium, 2 low): source-include test with macro and link-wrap seams; 4.9 condition `d == g`; `commit` takes no count; search-end line defined for every rc.
- Pass 7 (1 low): `xvec_dup` wrap attributed to the right test.
- Pass 8: NO ISSUES (one citation refinement applied).