myxce/.kiro/specs/scheduler-sim-decision-trace/research.md

Gap Analysis: scheduler-sim-decision-trace

Date: 2026-10-07. Codebase: branch scheduler-framework @ 33437a39, after scheduler-allocator-runner, scheduler-sim-discrete-clock and scheduler-sim-repl-verbosity reached implementation-complete. Paths are relative to src/scheduler/ unless they start with src/ (repo root) or .kiro/.

1. Summary

• All upstream dependencies have landed. Plan and bucket hooks have a single home: sched_allocator_run() in src/alloc_runner.c (490 lines). There is no alloc_iter.c; the interface is include/sched/alloc_iter.h.
• Almost every fact the trace needs is computed today and then thrown away at a known site (§3). Only two things are not available where the trace needs them: the per-resource total/inuse/uncommitted/wanted numbers (they exist only inside unsatisfy_) and a "selected but no decision" channel.
• Three requirements no longer match the code and should be revised before design (§4): Req 5.2 (pre-filter "carried over" request), Req 8.3 (the scheduler log path is not known to the sim), and the token limits that affect Req 1/8 argument parsing.
• Most of the cost is integration rather than algorithms. About 10 instrumentation sites, a new decision op or side channel, refactoring the REPL formatting helpers, and updates to about 26 white-box tests that #include REPL sources.
• Recommended approach: Option C (hybrid). New sched_trace.c and repl_trace.c modules hold the logic. Hooks in existing files are one line each, behind a NULL-pointer gate. Two small prerequisite refactors come first.

2. Corrections to brief.md

Brief says	Code today
Plan/bucket hooks go in "sched_allocator_run() once it lands"	It has landed: src/alloc_runner.c:457 (sched_allocator_run), :414 (sched_alloc_search_), :270 (sched_try_plan_), :208 (sched_claim_bucket_)
Shared-check loop bound bug (alloc_req.shared.count vs unmeet.count) is out of scope	Already fixed (bug-1893), alloc_runner.c:65,89
ALTER FIXME at scheduler.c:427	scheduler.c:433
sim_print_decision at sim_loop.c:479, drain loop at ~1607	sim_loop.c:529, drain loop :1675-1710
"core*.c/sim*.c may not do I/O" is enforced by tests	Not enforced. test_core_isolation.c and test_sim_isolation.c forbid controller/network headers and symbols, and wall-clock calls in core; they do not forbid stdio. Keeping sched_trace.c I/O-free is a design rule we choose, not one a test checks.
scheduler.c is 1382 lines (src/scheduler/CLAUDE.md)	2002 lines. CLAUDE.md is stale.

3. Requirement-to-asset map

Tags: Exists: the data is available at the seam. Discarded: computed, then dropped. Missing: not computed anywhere. Constraint: an architectural limit.

Req 1: Trace control

Need	Asset	Gap
trace command	repl_dispatch strcmp chain repl.c:408 (unknown fallback :479)	Missing. Add a branch that routes to repl_trace.c.
Help section	repl_print_help repl.c:269. test_repl_help.c:569-670 checks section order, with a lower bound only on the last section.	Missing. A trailing "Trace" section is safe.
Job/array selection	parse_jobsel/jobsel_t (src/include/utils.h:231-278), array_jobid/array_idx (src/lib/utils.c:660/671)	Exists
Settings take effect at the next session (1.5/1.6)	core->session_active (include/core.h:260), set/cleared in core_session.c:420/479	Exists. Caveat: at a BEGIN_ROUND breakpoint session_count has already been incremented but session_active==0. Design must define "next session" there.
Pass settings to the allocator thread with no new sync	Window between xce_core_begin_session and xce_core_signal_go (sim_loop.c:1621-1673). The allocator only reads core state after the mutex chain wbuf → picker → jbuf.	Exists. Snapshot settings into a core field in that window.
Argument parsing	repl_split_tokens (repl.c:116): at most 16 tokens of 63 characters	Constraint. Paths longer than 63 characters and long jobid lists do not fit (see §4).
trace status memory use	xce_sched_repl_state_s (repl_internal.h:171-304)	Missing. Needs a new state block.

Req 2: Job selection

Need	Asset	Gap
Session number	rs->session_count (repl_internal.h:178) is the current session during drain	Exists (render side)
1-based position in session	None. jbuf carries a bare uint64_t (core.c:167, pinned by test_core_jbuf_capacity_pin.c:227)	Missing. Count non-zero, non-sentinel pops in sched_allocator (sched_election.c:197-231), reset on 0. jbuf is unchanged.
Policy and ordering keys	job->queue->{name,priority}, job->priority, job->queue_time (include/sched/types.h:101-146); cmp functions builtin_fcfs.c:183,206	Exists. Read on the allocator thread after sched_get_job. Sim only ever runs builtin_fcfs (sim_picker.c:56-70).
Cluster-wide limits matched	obj->glimits filled by limit_job_election (plugins/limit.c:318) from plugin_job_election (sched_election.c:246). It is cached per configver (limit.c:336).	Exists. Read glimits after election; do not instrument the match loop.
No-decision cases (2.4)	sched_election.c:226-231 (not found), :250-254 (alloc_rc<0, silent), :265-269 (pend with NULL reason), :286 (calloc failure)	Missing channel. Nothing reaches the drain loop. See §5 Decision D2.
Reconstructed dispatch (2.5)	sim_reconcile_unaccounted_jobs (sim_loop.c:1026, block :1309-1329), runs before signal_go	Exists on the render side. The decision does not mark it as reconstructed, but the call site knows.

Req 3: Resource requirement selection

Need	Asset	Gap
Nesting path	Fixed shape: PRIMARY, COMPOUND (list of primaries), ALTER (list of compounds) (scheduler.c:306/403/409, bug-1769)	Exists. Path depth is at most 3.
Compound member index and point of failure	sched_make_compound_alloc_ loop scheduler.c:339-356; breaks on first rc<=0	Discarded (index not kept)
Per-alternative outcome and reasons (3.5)	ALTER loop :409-436. Each alternative frees and replaces obj->reason (:394-395). NULL resreq is skipped silently (:423).	Discarded. The trace must snapshot (deep-copy) each alternative's reasons before they are freed. It must not change obj->reason itself; that defect stays out of scope.
min/max tasks and requested resources	xce_primary_resreq_t{minTasks,maxTasks,select,rusages}; staged amounts in ctx->alloc_req.{shared,host}	Exists. Note: minTasks is never used by allocation, so the trace shows it but nothing depends on it.
accepts() rejection	alloc_runner.c:474-476 returns 0 with no reason. In PRIMARY the stale obj->reason survives (scheduler.c:324).	Discarded. The trace should say "allocator did not accept resreq". The stale-reason behaviour is a candidate bug to log separately.

Req 4: Shared checks and plans

Need	Asset	Gap
Shared check per resource	sched_alloc_prepare_shared_ (alloc_runner.c:68-111) → sched_check_shared_satisfy (:82)	total/inuse/uncommitted/wanted exist only inside unsatisfy_ (resource.c:147-189, TRACE at :159,179). unmeet keeps only the shortfall. Discarded. Needs a recorder hook inside unsatisfy_.
Cluster-wide limit vs resource	A limit charge is a vres in .shared (sched_fill_limit_requests, resource.c:1062-1099), with flag XCE_RESF_LIMIT (src/lib/xcelimit.c:526) and name limit=<n>[/u=..][/q=..][/p=..]. Structured xce_vres_t{vres,res,limit} (src/include/xce/types.h:831-844).	Exists. Classify by XCE_RESF_LIMIT, and map idx to xce_vres_t through obj->glimits / obj->hlimits[w->idx]. Pend reasons still say NO_RESOURCE (unchanged, out of scope).
Plans in order, shape, outcome	sched_alloc_search_ :414-455; plan->bucket_slots[]; sched_try_plan_ knows the failing bucket b (:283)	Discarded
No plan generated (4.7)	plan_init failure :427-429 (no reason); malformed plan skipped :432	Discarded
"Host groups"	Buckets are slot partitions only (seqjob: [1]; parjob k=1..maxTasks balanced, parjob_allocator.c:245-288)	Terminology: "host group" in the requirements means "bucket".

Req 5: Host verdicts

Need	Asset	Gap
Select mismatch	host_iter.c:136-140, silent continue	Discarded. Hook in the iterator.
Pre-filter vs per-host	Pre-filter is select-only. The runner clears .host before each next() (alloc_runner.c:216,253), and sched_check_host_satisfy returns 1 on an empty request (resource.c:419). Its satisfy branch (host_iter.c:159-165) is effectively dead under the runner.	Req 5.2 is stale (§4)
Per-resource shortfall numbers	sched_try_alloc_on_host_ resource.c:530-608 → satisfy :575 → unsatisfy_	Discarded (same recorder hook as Req 4)
Host-scoped limit	obj->hlimits[w->idx] from limit_prepare_allocation (limit.c:562-607)	Exists (classify by flag)
Already claimed (5.6)	sched_alloc_is_claimed_ alloc_runner.c:229, silent continue	Discarded
Internal error (5.7)	sched_try_alloc_on_host_ <0 (resource.c:577-579, uncommitted fetch :559); iterator uncommitted error host_iter.c:149-156. Runner merges <0 with rejection (alloc_runner.c:236).	Discarded. Visible at the call site.
Unreachable PREASON_MATCH_FAIL	resource.c:543-557	Note only. Select mismatches never reach it.
Rollback (5.8)	sched_rollback_claims_ alloc_runner.c:187-196	Discarded

Req 6: Outcome

Need	Asset	Gap
Per-host allocs, shared allocs	sched_finalize_job_alloc_ alloc_runner.c:347-406; job->allocs moved into the decision (sched_election.c:280)	Exists. Limit charges are mixed in (host vres in host allocs, global vres in allocs_shared). Separable by XCE_RESF_LIMIT, but at render time the resource index→flag table is needed.
Pend reason set	make_pend_decision stores a pointer to job->schedobj->reason (decision.c:6-19)	Constraint. A stored trace for info trace must hold its own copy, rendered text or a deep copy, because schedobj->reason is replaced in later sessions.

Req 7: Display and recall

Need	Asset	Gap
Inline after echo (step)	Drain loop order: verify → record → sim_print_decision_gate (:1705) → xce_repl_check_decision (:1706) → free (:1709). Reconstructed: :1309-1329.	Exists. Insert render between gate and breakpoint check. This gives 7.5 for free (stop after the trace prints).
Continue mode: no collapse for traced rounds	Rollup repl_rollup.c:285 compares dispatch/preempt counts plus the pend-jobid set from core->jobs; flush at :233	Missing. Needs a "round had traced output" flag that forces a flush, and must print the echo line that continue mode normally suppresses (sim_loop.c:587).
Units / jobid formatting	translate_units_mode, repl_format_numeric_resource_value, repl_format_consumable_value_and_used, repl_current_units_mode are static in repl_info.c (1893 lines, :385-487). repl_fmt_jobid_range (repl_fmt.c:223), fmt_jobid.	Prerequisite refactor: move these to repl_fmt.c
info trace	repl_cmd_info repl_info.c:1837	Missing. Add a one-line route to repl_trace.c.
Job state for "no trace" (7.11)	sched_get_job / existing info job helpers	Exists

Req 8: File copy

Need	Asset	Gap
dev/inode vs sim output	sim->record_file (FILE*, fstatable); rs->output_path only set by set output	Exists (use fstat(fileno()))
dev/inode vs scheduler log	Log path is a local in main.c:228/280, passed only to mylog_init	Missing. Needs a new field threaded into the sim args, or drop it from the requirement (§4).
set output rejects open trace file (8.4)	repl_params.c:241/197	Missing. Small check.
Close at quit/EOF/complete	repl_cmd_quit (repl.c:330, exit(0)), xce_repl_on_replay_complete (repl.c:609), batch close sim_loop.c:1785-1798. repl_state is never freed.	Missing. Add trace-file close in quit and complete.

Req 9: Bounds

Missing. Pure new logic in sched_trace.c / repl_trace.c. Note: the reason lists accumulate across plans without dedup (add_*_preason), so a parjob pend reason can be very large. Req 9.4's cap of 50 host entries is needed in practice, not just in theory.

Req 10/11: Non-interference and sim-only

Need	Asset	Gap
One-branch gate in live	Precedent: NULL-checked core callbacks unsatisfiable_*_cb, cache_clear_cb (include/core.h:247,258-259)	Exists. A core->trace pointer, NULL in live and in non-interactive sim.
Unsatisfiable cache in sim	unsatisfiable_lookup_cb NULL in sim (scheduler.c:254)	Exists. A trace never explains a cache hit.
Clock unaffected	xce_core_now with speed_num=0 in sim (sim.c:332, core_clock.c:80-121)	Exists
Equivalence testing	test_repl_equivalence.c pattern; --sim-verify; scheduler-driver-equivalence.md	Exists. Add trace-on vs trace-off comparisons.
Must not touch --sim-verify positional compare	sim_verify di++ per decision	Constraint for D2: a trace-only decision must be skipped by verify, record, gate and the REPL tally.

4. Requirements that need revision (recommended before approval)

1. Req 5.2 assumes the pre-filter compares "a request carried over from an earlier candidate host". Since scheduler-allocator-runner (Req 3.6, alloc_runner.c:137-138,253) the pre-filter is select-only. Suggested rewording: "...indicate whether the host was rejected by the selection pre-filter or by the per-host resource and limit check." Delete the carried-over clause.
2. Req 8.3 requires a same-inode check against the scheduler log file, but the sim never sees that path. Either keep the requirement and accept a small main.c → sim-args plumbing change, or narrow it to the simulation output file. Keeping it is cheap; flag it as a decision for the user.
3. Req 1 / Req 8 token limits. repl_split_tokens caps arguments at 16 tokens of 63 characters. Long file paths and long jobid lists (trace on job 1 2 3 … file /long/path) will fail. Decide between: (a) raise the limits in the tokenizer (affects every command and its tests), (b) a trace-local re-parse of the raw line, or (c) document the limit in the requirement. Option (b) is the least invasive.
4. Req 3.2 "minimum task count": it can be shown, but allocation never uses minTasks. No change needed, but the design should not suggest it influences plans.
5. Terminology: requirements say "host group"; the code says "bucket" (a slot partition). The design should state the mapping once.

Applied 2026-10-07 (user-approved): item 1 (Req 5.2 rewritten), item 2 (Req 8.3 kept, "if one is in use" added; the log path must be passed through to the sim), item 3 (new Req 1.12: any number of job identifiers and paths up to 4095 bytes; REPL input is libedit readline() with no line limit, so trace can re-parse the raw line), item 5 ("host group" replaced by a defined plan position = one bucket = exactly one host). Review fixes in the same pass: Req 1.6 defines "in progress" (BEGIN_ROUND breakpoint, before the session starts selecting jobs, counts as the next session); new Req 3.6 (skipped member or alternative); Req 4.7 covers an allocator declining a requirement; Req 6.4 no longer claims the echo line shows pend reasons; Req 7.8 renders info trace in the current set units mode (implies structured storage, D3).

Review loop 2026-10-07 (7 passes; passes 2-7 by independent reviewers checking against code; pass 7 clean). Further changes: glossary in the Introduction defines in progress / ended / reported; Req 2.4 adds the no-scheduling-object case (bug-1897 path, sched_election.c:236) and decision-build failure; Req 3.2/3.3 internal-error outcomes; Req 3.6 limited to alternatives (compound members are never skipped, scheduler.c:339-356 vs :423); Req 4.4 covers internal errors when trying a plan, when a plan cannot be tried, and when the host scan fails (a scan failure leaves the position unfilled and later plans are still tried, alloc_runner.c:222-226); Req 4.5 counts a plan as committed only after its allocation is built; Req 4.8 (cluster-wide check failed, so no plans tried); Req 4.9 (discarded malformed plans); Req 5.10 lists the verdict kinds ("not performed" checks are not a verdict); Req 5.9 names the limits hit in summary mode; Req 6.1/9.4 host cap semantics; Req 7.11 vs 9.6; Req 10.5 exceptions aligned with 10.4 and 1.6.

5. Implementation approach options

Option A: Extend existing files in place

Put hooks with inline formatting into alloc_runner.c, host_iter.c, resource.c and scheduler.c, and put rendering in sim_loop.c and repl_info.c.

• ✅ Fewest new files.
• ❌ sim_loop.c (1819), repl_info.c (1893), scheduler.c (2002) and driver.c (1104) are already over the 800-line cap. This breaks the project convention and the brief's constraint.
• ❌ It mixes rendering into hot-path files and makes the "one branch when off" property hard to audit.
• Not viable.

Option B: New modules only

include/sched/trace.h + src/sched_trace.c (record model, buffer, bounds, I/O-free) and src/repl_trace.c (commands, settings, rendering, store, file). Existing files get only one-line gated hook calls.

• ✅ Clean boundary, easy to unit-test the buffer and renderer in isolation, hook cost is easy to audit.
• ❌ Needs ~10 hook sites with a stable argument contract, and the renderer needs formatting helpers that are currently static in repl_info.c.

Option C: Hybrid, Option B plus enabling refactors (recommended)

Option B, plus these preparatory steps, each separately testable:

1. Move the units and value formatting helpers from repl_info.c to repl_fmt.c, with no behaviour change.
2. Add a recorder seam in unsatisfy_ (or a sibling that returns total/inuse/uncommitted/wanted per unmet idx) used only when tracing.
3. Add the no-decision channel (D2) and a "reconstructed" marker at the reconcile call site.
4. Register sched_trace.c in sched_core_isolation_lib and repl_trace.c in the test_sim_isolation.c scans and library. Add #include "../src/repl_trace.c" to the ~26 white-box tests.

• ✅ Keeps every new concern in a file under the cap, makes prerequisites reviewable on their own, and keeps the live path to a NULL check.
• ❌ More tasks and a larger diff on test files.

Key design decisions to carry forward

• D1: Where the buffer lives. Choices: (a) core->trace (settings snapshot plus current buffer), NULL when off, with one allocator thread per core; or (b) a pointer on the stack-allocated xce_sched_ctx_t. (a) reaches sched_election.c and host_iter.c without changing signatures, and matches the unsatisfiable_*_cb precedent. (b) has narrower scope, but ctx does not exist at the election and no-decision sites. Lean: (a), with settings snapshotted between begin_session and signal_go.
• D2: No-decision trace channel. Choices: (a) push a decision with a new op such as XCE_DECISION_TRACE and payload trace, which every consumer (verify, record, gate, REPL tally, live driver) skips; (b) a side queue drained in the decision loop, which loses positional order relative to decisions. Req 7.2/7.3 need ordering, so (a) is effectively required. Append the op after PREEMPT so existing values are not renumbered, and coordinate with scheduler-preemption, whose draft already disagrees with decision.h on PREEMPT placement.
• D3: Captured representation. Store structured records (with copied numbers, names and reason snapshots) and render on the sim side, or render to text on the allocator thread. The brief's constraint (no formatting on the allocator thread) favours structured records. For info trace storage, rendered text is simplest for the 64 MiB accounting, but it must be re-rendered if units change. Req 7.8 says "at the level captured", not at the units captured, so this is open.
• D4: Per-resource numbers. Instrument unsatisfy_ with a NULL-gated record callback rather than widening xvec_idval_t unmeet. Widening would touch every caller of the satisfy API.
• D5: Limit classification. Use XCE_RESF_LIMIT plus xce_vres_t lookup through obj->glimits / obj->hlimits[w->idx], captured on the allocator thread. These lists are rebuilt only when the configuration version changes, so they are stable within a session.
• D6: ALTER per-alternative reasons. Deep-copy ctx.reason into the trace before scheduler.c:394-395 frees it. Do not change obj->reason semantics (out of scope; separate bug).

6. Effort and risk

• Effort: L (1–2 weeks). About 10 hook sites across 6 hot-path files, a new decision op through 3 consumers, two new modules (model/buffer/bounds and REPL/render/store/file), prerequisite refactors, about 26 test-include updates, and equivalence tests. All of it follows existing patterns.
• Risk: Medium. No new technology, and the gating precedent exists. The risks are concentrated in: (1) a new decision op leaking into --sim-verify, recording or live driver consumers and breaking determinism; (2) hook arguments being evaluated on the live path (the one-branch rule needs macro discipline); (3) continue-mode rollup and breakpoint ordering interactions.

7. Research needed for the design phase

1. Whether a decision-pointer free in three places (driver.c:1093, sim_loop.c:1709,1327) is enough to own a trace buffer, or whether the buffer should be pooled per session to avoid allocator-thread malloc churn.
2. A hook macro shape (SCHED_TRACE(core, kind, ...)) that does not evaluate arguments when core->trace == NULL, and a test that proves it, for example a counting side-effect argument.
3. A resource idx → XCE_RESF_LIMIT lookup available at render time for the outcome section, or captured at finalize.
4. Ordering-key drift between pick and pop (an on_job_update landing in between). It is believed benign in sim because events are pushed before the session starts; confirm.
5. Exact interaction with the BEGIN_ROUND breakpoint: session_count has already incremented, session_active==0, so which session counts as "next"?
6. Line-count budget for repl_trace.c: commands, settings, store, file and renderer will likely exceed 800 lines, so plan a split such as repl_trace.c (commands/settings/file/store) and repl_trace_render.c.
7. Candidate bugs to log separately (not fixed by this spec): stale obj->reason after an accepts() rejection (scheduler.c:324, alloc_runner.c:474); a plan init failure pending with no reason (alloc_runner.c:427); pend reasons duplicated across parjob plans; PREASON_MATCH_FAIL branch unreachable (resource.c:543); the unsatisfiable cache populating main_reason, which the runner never sets (scheduler.c:282-289); src/scheduler/CLAUDE.md line counts stale.

---

Design Phase Research (2026-10-07)

Summary

• Feature: scheduler-sim-decision-trace
• Discovery Scope: Extension. Light discovery, integration-focused, on top of the gap analysis above. No new external dependencies.
• Key Findings:
  1. Every allocator-side fact the trace needs can be read on the allocator thread at the point it is computed. Reading the same data later on the sim thread would race. The sim drain loop runs concurrently with the allocator working on the next job, and obj->hlimits / worker->resvals are mutated by limit_prepare_allocation during allocation. So the outcome (allocation and pend reasons) is also captured on the allocator thread, before the decision is pushed.
  2. unsatisfy_ (resource.c:147) is the single place where total/inuse/uncommitted/wanted exist for both the cluster-wide check and the per-host check. One recorder hook there serves Req 4.1, 4.2, 5.4 and 5.5. Limit rows are told apart by the resource's XCE_RESF_LIMIT flag (xce_resource_t.flags, via vget(core->resmgr, idx)). The limit name and scope come from the xce_vres_t found in obj->glimits or obj->hlimits[w->idx] whose vres->idx matches.
  3. repl_dispatch(sim, line) (repl.c:408) receives the raw input line, and input comes from libedit readline() with no length limit. trace can therefore re-parse the raw line itself (Req 1.12) without touching repl_split_tokens.
  4. The continue-mode rollup (repl_round_rollup_process, repl_rollup.c:285) is called once per round from xce_repl_post_session_check. A per-round "traced" flag checked at its top is enough for Req 7.3/7.4: flush, emit this round alone, and do not start a range.
  5. main.c always has a log path (default /tmp/xce_scheduler.log, main.c:228), so Req 8.3's log-file check always applies once the path is plumbed through sched_sim_args_t.
  6. White-box REPL tests #include REPL/sim sources (test_repl_equivalence.c:132-155) and link scheduler_test_deps. Any new file that repl.c/sim_loop.c calls must be added to those include lists (about 26 tests) or to a library they link. sched_sim_isolation_lib (meson.build:163) must gain the new repl_trace*.c files.

Research Log

Thread ownership of trace data

• Context: Req 10 (no interference) and the "no new cross-thread synchronization" constraint.
• Findings:
  • The allocator thread runs sched_allocator (sched_election.c:162). The sim thread runs sched_sim_driver_run (sim_loop.c).
  • The sim thread writes core state only between xce_core_begin_session and xce_core_signal_go, and after the drain loop has seen the session-end NULL. The allocator is idle (blocked on jbuf) in both windows.
  • Values handed sim → allocator are ordered by the wbuf/jbuf cbuf mutexes. Values handed allocator → sim travel through the core->decisions cbuf.
• Implications: The settings snapshot (core->trace_ctl) is written only in those windows. The record under construction (core->trace_rec) is touched only by the allocator thread. A finished record changes owner by riding on the decision. No locks are needed.

Decision transport for no-decision selections

• Context: Req 2.4, 7.2, 7.3 require the trace to appear in the decision order.
• Findings: The drain loop ends on a NULL decision. --sim-verify compares positionally (di++, sim_loop.c:1690). Recording and the breakpoint tally process every decision.
• Implications: Append a new op XCE_DECISION_TRACE after XCE_DECISION_PREEMPT (no renumbering). The drain loop handles it first and continues before verify, record, gate and breakpoint run. driver.c frees and skips it defensively; live never produces one.

Shortfall numbers

• Context: Req 4.1, 4.2, 5.4, 5.5.
• Findings: unsatisfy_(core, nres, idxs, total, inuse, uncomitted, wanted, ...) is reached from sched_check_shared_satisfy_ (resource.c:~290) and from the host satisfy path (resource.c:~384). Only TRACE logging uses the values today.
• Implications: One hook at the top of unsatisfy_ passes all arrays to the current record. The record keeps every row inside a cluster-wide-check scope (4.1 shows passing rows) and only unmet rows inside a host scope (5.4).

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Per-decision record + render in sim driver (selected; brief's choice)	Allocator appends typed entries to an arena-backed record; record rides on the decision; REPL renders, stores and copies to file	No I/O on core thread; per-job recall; deterministic ordering	Needs a new decision op and allocator-thread outcome capture	Matches the existing unsatisfiable_*_cb NULL-gate precedent
Streaming callback	Hooks call a driver callback per event	No buffer	I/O on allocator thread; picker/allocator output interleaving; no recall	Rejected in brief
Render text on allocator thread	Hooks format strings	Simple store	Formatting cost on allocator; units fixed at capture (violates 7.8)	Rejected

Design Decisions

Decision: Two gate pointers on the core

• Context: Req 11.2 allows at most one branch per hook when off.
• Alternatives Considered:
  1. A single trace_ctl pointer, with every hook reading ctl->rec: two loads.
  2. trace_ctl (session settings) plus trace_rec (current record): hooks test only trace_rec.
• Selected Approach: Option 2. SCHED_TRACE(core, fn(core->trace_rec, ...)) evaluates nothing unless core->trace_rec != NULL. Only the allocator's per-job prologue tests trace_ctl.
• Rationale: One predictable branch per hook. The live core never sets either pointer.
• Trade-offs: Two fields on xce_sched_core_t instead of one.

Decision: Entry budget equals line budget

• Context: Req 9.1–9.3 bound lines, but the record is structured.
• Selected Approach: Each entry in the requirement, check, plan and host sections renders as exactly one line. The capture-side budget counts entries, and the renderer contract guarantees one line per entry. Counters (plans tried and discarded, verdict counts, limit tallies) keep updating after the budget is exhausted.
• Rationale: Truncation happens at capture, so memory is bounded too (9.5), and the omitted-line count is exact.

Decision: Outcome captured on the allocator thread, capped at capture

• Context: Req 6, 9.4; the stored trace must outlive job->allocs and schedobj->reason (7.8).
• Selected Approach: After the decision is built, the allocator copies the first 50 hosts (allocation rows, or pend-reason entries) plus all host-less entries, and aggregates the rest (per-code counts, or per-resource and per-limit totals). Reconstructed dispatches (Req 2.5) are captured the same way on the sim thread, inside the pre-signal_go window.
• Rationale: This avoids racing the allocator and gives a bounded record size.

Decision: Structured store, render on demand

• Context: Req 7.7 / 7.8 (units mode at render time), 8.1 (file copy has the same content as the console).
• Selected Approach: Records hold raw floats plus the resource name and flags. The renderer formats with the current set units mode. The inline print renders once to the console and the file (same bytes); info trace renders again later.

Decision: Settings lifecycle pending → active → retired

• Context: Req 1.4–1.7, 8.5.
• Selected Approach: REPL commands write pending. The session-begin hook promotes pending to active and publishes core->trace_ctl. A replaced or disabled active setting whose file is still open becomes retired until the session-end hook closes it.

Generalization / Build-vs-Adopt / Simplification

• Generalization: One recorder path handles the cluster-wide and per-host checks (scope decides which rows to keep). One renderer serves inline, file and info trace.
• Build vs adopt: No suitable library exists (structured in-process trace, C, sim-only). We reuse libxce htab for the jobid→trace index, parse_jobsel/array_jobid for selection, and the repl_fmt helpers for units.
• Simplification: No callback indirection. The sim driver is the only consumer, so hooks call sched_trace_* functions directly behind the NULL gate. No per-session pooling (records are arena-allocated per traced job; untraced jobs allocate nothing).

Risks & Mitigations

• New decision op leaks into verify, recording or the live driver → handle the op first in each drain loop; equivalence tests with trace on vs off (--sim-verify and --sim-output byte compare).
• Hook arguments evaluated in live → statement macro that evaluates its call only inside if (core->trace_rec != NULL); a unit test with a side-effect argument proves it.
• Trace changes scheduling via shared state → capture functions take const inputs and never call mutating scheduler APIs; the trace-on/off equivalence tests use a resource-contention fixture (per scheduler-driver-equivalence.md).
• ~26 white-box test include lists → update them in one task right after the new files exist; the build fails loudly if one is missed.
• File-size cap → three REPL-side files plus two core-side files, each planned under 800 lines.

References

• .kiro/steering/scheduler-driver-equivalence.md: state-equivalence invariant and contention-fixture rule.
• .claude/skills/xce-audit-invariants/SKILL.md: scheduler audit dimensions 1–4.
• .kiro/specs/scheduler-allocator-runner/: runner contract (alloc_iter.h).

Design Review Gate (2026-10-07)

• Pass 1 (self): settings lifecycle reduced to reference-counted next/cur (no separate "retired" state); pos_scan_error API added; scope-begin calls take obj/w; per-host 0 with no rows means selection mismatch (resource.c MATCH_FAIL branch); sim_print_decision exposed for the continue-mode echo.
• Pass 2 (independent, checked against the code): the link impact on about 20 white-box tests that include hooked sources (D1); search_end entry outside the budget plus a NONE_GENERATED reason (D2); round_traced reset at session begin (D3); info trace job-state read placed in the same class as info job (D4); resource pre-filter invariant added as a revalidation trigger (D5); host-scope h=<host> and name-based limit resolution for reconstructed records (D6); rollback hook only when n > 0 in sched_try_plan_ (D7); path/no-plan/pos-end enums and bound constants defined (D8); utils.h dependency and the isolation-scan target named (D9); reconstructed-block call order spelled out (D10). Threading, drain-loop order, breakpoint ordering and the Req 10/11.2 claims were confirmed.

Design Review Loop (2026-10-07, after design generation)

Eight independent review passes, each checked against the code; pass 8 found no issues.

• Pass 1 (1 major, 7 minor): compound-container hooks and merge-failure marking; search_end counted in the budget, with decision-wide counters; round_traced set in every mode; limit units via the limited resource; missing API declarations; repl.h externs; d->trace ownership; anchor fix.
• Pass 2 (1 high, 1 medium, 2 low): path push/pop with balanced begin/end (non-scope member_failed / alt_result); schedobj passed for limit-charge resolution; rollup rule after the pend query; session-begin hook after the rc < 0 check.
• Pass 3 (2 medium, 4 low): no-decision internal-error records render the full hierarchy; renderer example made line-exact; rollup idle formula; NOT_FOUND has no selection; trace_rec cleared on every hand-off; name-based limit-charge matching; entry-type list and traceability corrected.
• Pass 4 (1 medium, 2 low): failure-reasons line depth; nmembers on resreq_begin; "unavailable" lines for 2.4.
• Pass 5 (1 medium, 2 low): error-path tests use the real allocator; search_end/rollback take no data the runner lacks; Modified Files rollup line aligned.
• Pass 6 (2 medium, 2 low): source-include test with macro and link-wrap seams; 4.9 condition d == g; commit takes no count; search-end line defined for every rc.
• Pass 7 (1 low): xvec_dup wrap attributed to the right test.
• Pass 8: NO ISSUES (one citation refinement applied).