From ed201686b67eb91f2e1ab5ab2b607caae9df3ab1 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Thu, 8 Oct 2026 06:48:57 +0000 Subject: [PATCH] docs(spec): scheduler-sim-decision-trace design, research and tasks Co-Authored-By: Claude Opus 5.5 (1M context) --- .../scheduler-sim-decision-trace/design.md | 844 ++++++++++++++++++ .../requirements.md | 59 +- .../scheduler-sim-decision-trace/research.md | 274 ++++++ .../scheduler-sim-decision-trace/spec.json | 16 +- .../scheduler-sim-decision-trace/tasks.md | 336 +++++++ 5 files changed, 1495 insertions(+), 34 deletions(-) create mode 100644 .kiro/specs/scheduler-sim-decision-trace/design.md create mode 100644 .kiro/specs/scheduler-sim-decision-trace/research.md create mode 100644 .kiro/specs/scheduler-sim-decision-trace/tasks.md diff --git a/.kiro/specs/scheduler-sim-decision-trace/design.md b/.kiro/specs/scheduler-sim-decision-trace/design.md new file mode 100644 index 00000000..c3c63dcd --- /dev/null +++ b/.kiro/specs/scheduler-sim-decision-trace/design.md @@ -0,0 +1,844 @@ +# Design Document: scheduler-sim-decision-trace + +## Overview + +**Purpose**: This feature adds a decision trace to the xce_scheduler simulator REPL. For each traced job it explains why the job was selected, which resource requirements were tried, which cluster-wide checks and placement plans were attempted, why each host was accepted or rejected, and what was finally allocated or why the job pends. + +**Users**: Scheduler developers and operators who replay recorded sessions with `xce_scheduler --sim -i`. They enable tracing at runtime for all jobs or for selected jobs, read traces inline during `step`/`continue`, recall them with `info trace`, and optionally copy them to a file. + +**Impact**: +- The allocator thread gains NULL-gated capture hooks at the decision seams. These seams are election, resource-requirement branching, the allocator runner, the host iterator and the shortfall check. +- `xce_decision_t` gains a trace attachment and a trace-only op. +- The simulation driver and REPL gain a trace control, store, renderer and file copy. +- The live scheduler is unchanged except for one never-taken branch per hook. + +### Goals +- Explain every traced dispatch, pend and no-decision selection with the hierarchy and content defined in Requirements 2–6. +- Keep scheduling bit-for-bit identical with tracing on or off (Requirement 10) and keep the live scheduler free of trace work (Requirement 11). +- Bound trace size per decision and stored-trace memory (Requirement 9). + +### Non-Goals +- Live-mode, headless-sim or recording-file tracing. +- Preemption decisions (`XCE_DECISION_PREEMPT`) are never traced. +- Changing pend-reason content, the `--sim-verify` comparison or the `ReplayDecision` format. +- Fixing defects the trace exposes. Examples are ALTER keeping only the last alternative's `obj->reason`, a stale `obj->reason` after an `accepts()` decline, pend reasons duplicated across parjob plans, and an unreachable `PREASON_MATCH_FAIL`. They are logged as separate bugs. +- `info limit` and limit-usage display (`scheduler-sim-diagnostics`). + +## Boundary Commitments + +### This Spec Owns +- The trace record model, its arena storage, entry budget and counters (`sched_trace.c`), and the allocator-side capture helpers that read scheduler state (`sched_trace_capture.c`). +- Two gate fields on `xce_sched_core_t`: `trace_ctl` (session settings snapshot) and `trace_rec` (record under construction). It also owns the `SCHED_TRACE` hook macro. +- The `trace` field on `xce_decision_t`, the `XCE_DECISION_TRACE` op and `make_trace_decision()`. +- The hook call sites in `sched_election.c`, `scheduler.c`, `alloc_runner.c`, `host_iter.c` and `resource.c`. Each is a single gated statement that never changes control flow or return values. +- The REPL `trace` command, trace settings lifecycle, trace file copy, the `set output` trace-file conflict check, the trace store, `info trace`, trace rendering, the help "Trace" section, and the continue-mode traced-round rule. +- Plumbing the scheduler log path into the simulation driver (`sched_sim_args_t.log_path`, `xce_sched_sim_driver_t.log_path`). + +### Out of Boundary +- Scheduling, picking, allocation and accounting behaviour, and the content of `xce_pendreason_t` / `job->allocs`. +- The `jbuf` hand-off (capacity 1, `uint64_t` items) and the picker thread. No hook runs on the picker thread. +- `--sim-verify` comparison logic (`sim_verify.c`), the recording format (`recording*.c`, `sim_record.c`) and snapshot content. +- The unsatisfiable-resreq cache. It is live-only; sim never consults it, so a trace never explains a cache hit. +- Rollup fingerprint semantics for untraced rounds (`scheduler-sim-repl-verbosity`). + +### Allowed Dependencies +- Core side (`sched_trace*.c`): libxce containers and types (`xvec`, `xlist`, `zarray`, `xce/types.h`), `include/sched/alloc_iter.h` (for `xce_alloc_plan_t`), `src/include/utils.h` (only `array_jobid`/`array_idx` for selector matching), `include/sched/types.h`, `include/core.h`, and `include/preason.h`. These files perform no stdio, file or network I/O and do not include REPL or sim headers. +- REPL side (`repl_trace*.c`): `include/sched/trace.h` (read-only use of records), `repl_internal.h`, `repl_fmt.c` helpers, libxce `htab`, and `parse_jobsel`/`array_jobid`/`fmt_jobid` (`utils.h`). It must not call scheduler mutation APIs. +- Dependency direction: `trace.h` types → `sched_trace.c` (model) → `sched_trace_capture.c` (allocator-side readers) → hook sites. Separately, `trace.h` → `repl_trace_render.c` → `repl_trace_store.c` → `repl_trace.c` → `repl.c`/`repl_info.c`/`repl_params.c`/`repl_rollup.c`/`sim_loop.c` call sites. Core-side files never include REPL-side headers. + +### Revalidation Triggers +- Any change to `xce_decision_t` or `xce_decision_op_t` (shared with `scheduler-preemption`). +- Changes to the allocator runner seams (`alloc_runner.c` plan, position and host loop), `host_iter.c` filters, or `unsatisfy_`'s signature. +- Changes to the sim drain-loop order (verify → record → print gate → breakpoint) or to the rollup call site. +- Changes to the limit plugin's vres naming or to `obj->glimits`/`obj->hlimits` ownership. +- Any change that lets the host iterator's resource-availability pre-filter (`host_iter.c:150-166`) see a non-empty request. Today it is a no-op because the runner empties `alloc_req.host` before every `next()` (`alloc_runner.c:216, 253`), and `sched_check_host_satisfy` returns 1 on an empty request (`wanted->count == 0`, `resource.c:418-420`) before it reaches the `unsatisfy_` hook. The design hooks only the selection and internal-error pre-filter paths; a live resource pre-filter would need its own `host_begin` plus pre-filter SHORTFALL/LIMIT verdicts. + +## Architecture + +### Existing Architecture Analysis +- The core (`core*.c`) is a passive state machine driven by two drivers. The picker and allocator threads communicate through `jbuf` (capacity 1). Decisions flow through the `core->decisions` cbuf to the driver's drain loop. Live (`driver.c`) and sim (`sim_loop.c`) run the identical election, allocation and accounting chain (`scheduler-driver-equivalence.md`). +- The precedent for driver-specific behaviour in the core is NULL-gated core fields set by one driver only (`unsatisfiable_lookup_cb`, `cache_clear_cb`). This design follows it. +- Files already over the 800-line cap are `scheduler.c` (2002), `sim_loop.c` (1819), `repl_info.c` (1893), `repl_break.c` (980) and `driver.c` (1104). They receive only call sites. All logic lives in new files. + +### Architecture Pattern & Boundary Map + +```mermaid +graph TB + subgraph AllocatorThread + Election[sched_election allocator loop] + Branch[scheduler resreq branch] + Runner[alloc_runner plans and positions] + HostIter[host_iter prefilter] + Satisfy[resource unsatisfy] + Capture[sched_trace_capture] + Model[sched_trace record model] + end + subgraph SimThread + Drain[sim_loop drain loop] + ReplTrace[repl_trace control and file] + Store[repl_trace_store and info trace] + Render[repl_trace_render] + Rollup[repl_rollup] + end + Election --> Capture + Branch --> Capture + Runner --> Capture + HostIter --> Capture + Satisfy --> Capture + Capture --> Model + Election -->|decision with trace| Decisions[core decisions cbuf] + Decisions --> Drain + Drain --> ReplTrace + ReplTrace --> Render + ReplTrace --> Store + Store --> Render + ReplTrace --> Rollup + ReplTrace -->|session settings| CoreCtl[core trace_ctl] + CoreCtl --> Election +``` + +**Architecture Integration**: +- **Selected pattern**: per-decision structured record built on the allocator thread, transferred with the decision, then rendered and stored by the sim REPL. +- **Boundaries**: + - The core side builds records and never renders. + - The REPL side renders, stores and writes. Inline emission reads only the record, never live scheduler state. The single exception is `info trace` (7.9, 7.11), which reads `job->state` at the prompt. That is the same read-only access, with the same timing, as the existing `info job` at a decision breakpoint; it adds no new race class. +- **Existing patterns preserved**: + - The NULL-gated core field, the `sim->interactive` inline gates in `repl.h`, and drain-loop ownership of decisions. + - `fprintf(stdout, ...)` for all REPL output (`test_no_bare_printf.c`). + - `jbuf` is unchanged. +- **New components rationale**: the record model, capture, control, store and renderer each have a distinct owner and thread (see Components). +- **Steering compliance**: + - The core does no I/O. + - Driver equivalence: the trace only reads state, and live never enables it. + - TDD (`testing.md`) and the 800-line cap. + +### Technology Stack + +| Layer | Choice / Version | Role in Feature | Notes | +|-------|------------------|-----------------|-------| +| Scheduler core | C (gnu17), existing libxce | Record model, capture hooks | No new libraries | +| Simulator REPL | C, libedit `readline()` (existing) | `trace` / `info trace` commands, rendering, file copy | Raw-line re-parse for long arguments | +| Build / test | Meson, existing Unity-style testsuite | New sources added to `scheduler_src`, isolation libs and test include lists | — | + +## File Structure Plan + +### Directory Structure +``` +src/scheduler/ +├── include/sched/ +│ └── trace.h # Record, entry, settings types; SCHED_TRACE macro; model + capture API +├── src/ +│ ├── sched_trace.c # Record model: arena, entry append, budget, counters, verdict tallies, +│ │ # jobsel predicate, record free/size. No I/O, no scheduler-state reads. +│ ├── sched_trace_capture.c # Allocator-side capture helpers that read scheduler structures +│ │ # (job/queue keys, glimits, resource flags, vres lookup, alloc/pend +│ │ # outcome copy with 50-host cap). No I/O. +│ ├── repl_trace.c # `trace on|off|status` parsing (raw line), settings lifecycle +│ │ # (next/cur references), session begin/end hooks, file open/close, +│ │ # dev/inode conflict checks, inline emission, shutdown +│ ├── repl_trace_store.c # jobid -> latest record index, capture-order eviction at 64 MiB, +│ │ # discarded marks, `info trace` (job, array element, array) +│ └── repl_trace_render.c # Record -> lines on one or two FILE* sinks; units mode; jobid fmt; +│ # hierarchy/indent; summary vs full host section; outcome section +└── testsuite/ + ├── test_trace_model.c # sched_trace.c unit tests + ├── test_trace_capture.c # capture helpers against fixture core/schedobj/workers + ├── test_trace_hooks_gate.c # SCHED_TRACE no-evaluation proof; live/non-interactive never trace + ├── test_trace_render.c # renderer output per section, units modes, truncation, outcome cap + ├── test_trace_store.c # store replace/evict/discarded marks, info trace variants + ├── test_repl_trace_cmd.c # trace command grammar, errors, status, lifecycle, file rules + ├── test_repl_trace_e2e.c # step/continue/breakpoint integration over replay fixtures + └── test_trace_equivalence.c # trace on vs off: decisions, --sim-verify, --sim-output, accounting +``` + +### Modified Files +- `src/scheduler/include/core.h`: add `xce_sched_trace_ctl_t* trace_ctl` and `xce_sched_trace_rec_t* trace_rec` to `struct xce_sched_core_s`, each documented as allocator-thread-read and sim-window-written, and NULL in live. +- `src/scheduler/include/decision.h`, `src/decision.c`: + - append `XCE_DECISION_TRACE` after `XCE_DECISION_PREEMPT`; + - add `struct xce_sched_trace_rec_s* trace;` after the union; + - add `make_trace_decision(rec)`. +- `src/scheduler/src/sched_election.c`: hooks in `sched_allocator`: + - per-job prologue: position counter, traced predicate, record begin with selection facts; + - after `plugin_job_election`: cluster-wide limits; + - each no-decision path: no-decision record pushed as a trace decision; + - after the decision is built: outcome capture and attach; + - session-end marker: reset `trace_rec`. +- `src/scheduler/src/scheduler.c`: hooks in `sched_make_allocation` (PRIMARY and top-level COMPOUND containers), `sched_make_compound_alloc_` (member push/begin/end/pop, merge failure, attempt reason snapshot) and the ALTER loop (alternative push/begin/end/pop, skipped, result). +- `src/scheduler/src/alloc_runner.c`: hooks for: + - the accepts decline; + - cluster-wide check begin/end; + - `plan_init` failure; + - each generated plan (discarded, not tried on reserve failure, tried); + - each position (begin, end: filled, unfilled or scan error); + - each candidate (claimed skip, per-host result `>0/0/<0`); + - rollback; + - finalize failure; + - commit. +- `src/scheduler/src/host_iter.c`: hooks for selection mismatch, pre-filter internal error (`sched_ctx_get_uncomit_host` failure) and scan stop on `iter_next < 0`. +- `src/scheduler/src/resource.c`: one hook at the top of `unsatisfy_` passing all rows. +- `src/scheduler/src/sim_loop.c`: + - `xce_repl_trace_session_begin(sim)` right after `xce_core_begin_session`; + - reconstructed-dispatch trace in `sim_reconcile_unaccounted_jobs`' report block; + - drain loop: `XCE_DECISION_TRACE` handled first, then `xce_repl_check_trace(sim, d)` after `sim_print_decision_gate`; + - `xce_repl_trace_session_end(sim)` after the drain loop. +- `src/scheduler/include/repl.h`: `extern` declarations of the forwarded entry points (`repl_trace_session_begin/end`, `repl_trace_on_decision`, `repl_trace_on_trace_only`), following the existing `xce_repl_on_*` pattern so `repl.h` stays self-contained. It also adds inline gates `xce_repl_trace_session_begin/end`, `xce_repl_check_trace` and `xce_repl_check_trace_only`, all forwarding only when `sim->interactive`. +- `src/scheduler/src/repl_internal.h`: `xce_repl_trace_state_t trace` member in `xce_sched_repl_state_t`; declarations for the `repl_trace*` entry points and moved formatting helpers. +- `src/scheduler/src/repl.c`: dispatch `trace` (passing the raw line); help gains a trailing "Trace" section and an `info trace` line; `repl_cmd_quit` and EOF call `repl_trace_shutdown`. +- `src/scheduler/src/repl_info.c`: `info trace` route (one branch); usage string updated; units helpers moved out. +- `src/scheduler/src/repl_fmt.c`: receives `translate_units_mode`, `repl_format_numeric_resource_value`, `repl_format_consumable_value_and_used` and `repl_current_units_mode` (behaviour unchanged, now non-static). +- `src/scheduler/src/repl_params.c`: `set output` calls `repl_trace_file_conflicts(sim, path)` before switching. +- `src/scheduler/src/repl_rollup.c`: the traced-round rule in `repl_round_rollup_process`, placed after `repl_query_pend_jobids` succeeds (see ReplIntegration). +- `src/scheduler/src/driver.c`: the drain loop frees and skips `XCE_DECISION_TRACE` defensively; it is never produced in live. +- `src/scheduler/include/sim.h`, `src/sim.c`, `src/main.c`: `log_path` plumbing. `sim.h` also declares `sim_print_decision`, which `sim_loop.c` makes non-static (format unchanged) so continue-mode trace emission reuses the exact echo line. +- `src/scheduler/meson.build`: + - add the new sources to the scheduler sources; + - add `sched_trace.c` and `sched_trace_capture.c` to `test_core_isolation.c`'s source-text scan list (forbidden headers and symbols, plus a new no-stdio check for these two files only). They are not added to `sched_core_isolation_lib`, whose `nm` check covers the four core state-machine files; the capture file references scheduler symbols outside that archive's scope; + - add `repl_trace*.c` to `sched_sim_isolation_lib`. +- `src/scheduler/testsuite/meson.build`, `test_core_isolation.c`, `test_sim_isolation.c`, plus every white-box test that `#include`s a hooked or calling source: register the new tests, scan the new files, and add `#include` lines for `sched_trace*.c` / `repl_trace*.c`. `SCHED_TRACE` bodies are compiled even when they never run, so a test that includes a hooked file must also include (or link) `sched_trace.c` and `sched_trace_capture.c`. + - **Hooked sources**: `resource.c`, `host_iter.c`, `alloc_runner.c`, `scheduler.c`, `sched_election.c` and `driver.c`. These are included by `test_alloc_runner*.c`, `test_host_iter.c`, `test_alloc_equivalence_*.c`, `test_parjob_combinatorial.c`, `test_core_events.c`, `test.c` and others (about 20). + - **Calling sources**: `repl.c` and `sim_loop.c`, included by about 26 tests. + - The exact list is produced by `grep -l '#include "../src/\(resource\|host_iter\|alloc_runner\|scheduler\|sched_election\|driver\|repl\|sim_loop\).c"' testsuite/*.c` and updated in one dedicated task right after the new files exist. +- `src/scheduler/CLAUDE.md`: correct the stale line counts and list the new files. + +## System Flows + +### Session lifecycle and capture + +```mermaid +sequenceDiagram + participant Op as Operator + participant Repl as repl_trace + participant Sim as sim_loop + participant Core as core + participant Alloc as allocator thread + Op->>Repl: trace on job 42 level full + Repl->>Repl: set next settings + Sim->>Core: begin session + Sim->>Repl: session begin hook + Repl->>Core: publish trace_ctl and reset position + Sim->>Sim: reconcile and report reconstructed dispatches + Sim->>Core: signal go + Alloc->>Core: pop jobid and count position + Alloc->>Alloc: traced so set trace_rec and capture selection + Alloc->>Alloc: hooks append resreq check plan host entries + Alloc->>Alloc: capture outcome and attach record to decision + Alloc->>Core: push decision and clear trace_rec + Sim->>Core: drain decision + Sim->>Repl: check trace + Repl->>Op: echo line and rendered trace and file copy + Repl->>Repl: store record + Sim->>Repl: session end hook + Repl->>Core: clear trace_ctl and release replaced settings +``` + +Key decisions: +- Settings take effect only in the session-begin hook (Requirement 1.6). Commands issued during a session change only `next`. +- `trace_rec` is non-NULL only between the per-job prologue and the decision push of a traced job. The allocator sets and clears it; nothing else touches it. +- A no-decision selection pushes a decision with op `XCE_DECISION_TRACE` that carries only the record. The drain loop handles it before verify, record, print gate and breakpoint check. + +### Drain-loop emission order (per decision) + +```mermaid +graph TB + Get[cbuf get decision] --> IsTrace{op is TRACE} + IsTrace -->|yes| TraceOnly[check trace only then free and continue] + IsTrace -->|no| Verify[verify] + Verify --> Record[record add] + Record --> Gate[print gate] + Gate --> CheckTrace[check trace] + CheckTrace --> Break[breakpoint check] + Break --> Free[free decision] +``` + +- `xce_repl_check_trace` runs before the breakpoint check. A breakpoint therefore stops after the stopping decision's trace is printed and stored, and before any later trace (Requirement 7.5). +- In continue mode, the first traced item of a round flushes the active collapsed range, prints the suppressed echo line, then prints the trace. The round is then marked traced (Requirement 7.3). + +### Trace settings lifecycle + +```mermaid +stateDiagram-v2 + [*] --> NextOnly: trace on + NextOnly --> InSession: session begin + InSession --> NextOnly: session end and still referenced as next + InSession --> Released: session end and replaced or disabled + NextOnly --> Released: replaced or disabled with no session in progress + Released --> [*] +``` + +A settings object (selection, level, bound, open file) is referenced by `rs->trace.next` (the settings the next session will use) and, while a session runs, by `rs->trace.cur` (the settings the session started with). Commands only replace `next`. The session-begin hook sets `cur = next`, and the session-end hook clears `cur`. An object is released, and its file closed, when neither reference holds it. This gives 8.5 directly: a replaced object stays open until the in-progress session ends, or is closed immediately when no session is in progress. + +## Requirements Traceability + +| Requirement | Summary | Components | Interfaces | Flows | +|-------------|---------|------------|------------|-------| +| 1.1, 1.2, 1.3 | Enable all or by jobs (arrays, future ids) | ReplTraceControl, TraceModel | `repl_trace_cmd`, `sched_trace_ctl_wants` | Lifecycle | +| 1.4, 1.5, 1.6, 1.10 | Replace, disable, next-session application, persistence | ReplTraceControl | `next`/`cur` settings, session hooks | Lifecycle, Session | +| 1.7 | Status: in effect, next, memory | ReplTraceControl, TraceStore | `repl_trace_cmd` status, `repl_trace_store_bytes` | — | +| 1.8 | Levels summary/full (default summary) | ReplTraceControl, TraceModel, TraceRenderer | `xce_sched_trace_level_t` | — | +| 1.9, 1.12 | Argument validation; unbounded jobs, 4095-byte path | ReplTraceControl | raw-line parser | — | +| 1.11 | Help section and `info trace` listing | ReplIntegration | `repl_print_help` | — | +| 2.1 | Session number and position | TraceHooks (election), TraceModel | `sched_trace_ctl_t.position`, `.session_no` | Session | +| 2.2 | Policy and FCFS keys | TraceCapture | `sched_trace_capture_selection` | Session | +| 2.3 | Cluster-wide limits matched | TraceCapture | `sched_trace_capture_glimits` | Session | +| 2.4 | No-decision selections | TraceHooks (election), DecisionTransport | `sched_trace_rec_finish`, `XCE_DECISION_TRACE` | Drain order | +| 2.5 | Reconstructed dispatch notice | SimIntegration, TraceCapture | `sched_trace_rec_reconstructed` | Session | +| 3.1, 3.2 | Resreq labels, path, tasks, resources, outcome | TraceHooks (scheduler.c), TraceCapture | `sched_trace_resreq_begin/end` | — | +| 3.3, 3.4, 3.6 | Failure point, chosen alt, not tried, skipped | TraceModel, TraceHooks (scheduler.c) | `resreq_end(not_tried)`, `sched_trace_member_failed`, `sched_trace_alt_result`, `sched_trace_alt_skipped` | — | +| 3.5 | Per-alternative failure reasons | TraceCapture | attempt reason summary entry | — | +| 4.1, 4.2 | Cluster-wide rows; limit rejection | TraceHooks (unsatisfy_, runner), TraceCapture | `sched_trace_check_rows` | — | +| 4.3, 4.4, 4.5, 4.6 | Plans: shape, outcome, failed position, errors, commit | TraceHooks (runner), TraceModel | `sched_trace_plan_*`, `sched_trace_pos_*`, `sched_trace_pos_scan_error`, `sched_trace_commit`, `sched_trace_finalize_failed`, `sched_trace_search_end` | — | +| 4.7, 4.8, 4.9 | No plan generated or declined; no plans after check fail; discarded | TraceHooks (runner) | `sched_trace_no_plan`, `sched_trace_check_end`, `sched_trace_plan_discarded`, `sched_trace_search_end` | — | +| 5.1–5.7, 5.10 | Host verdicts, phases, kinds, not performed | TraceHooks (host_iter, runner, unsatisfy_), TraceModel | `sched_trace_host_*`, verdict bitmask | — | +| 5.8 | Released hosts on rollback | TraceHooks (runner) | `sched_trace_rollback_host` | — | +| 5.9 | Summary per-position counts and limit names | TraceModel, TraceRenderer | `POS_SUMMARY` entry | — | +| 6.1, 6.2, 6.3 | Dispatch outcome with separate limit charges | TraceCapture, TraceRenderer | `sched_trace_capture_dispatch` | — | +| 6.4 | Pend reason set, same content | TraceCapture, TraceRenderer | `sched_trace_capture_pend` | — | +| 7.1, 7.2 | Step-mode inline placement | SimIntegration, ReplTraceControl | `xce_repl_check_trace`, `..._trace_only` | Drain order | +| 7.3, 7.4 | Continue-mode echo and no collapse | ReplTraceControl, RollupIntegration | `rs->trace.round_traced` | Drain order | +| 7.5 | Breakpoint ordering | SimIntegration | hook before `xce_repl_check_decision` | Drain order | +| 7.6, 7.7 | Hierarchy, units, jobid format | TraceRenderer | `repl_trace_render` | — | +| 7.8, 7.9, 7.10, 7.11 | `info trace` job, array, replace, none | TraceStore | `repl_cmd_info_trace` | — | +| 8.1, 8.2 | File copy content; open errors | ReplTraceControl, TraceRenderer | two-sink render | — | +| 8.3, 8.4 | dev/inode conflicts both directions | ReplTraceControl, ReplIntegration (`set output`) | `repl_trace_file_conflicts` | — | +| 8.5, 8.6 | Replaced file kept until session end; shutdown flush | ReplTraceControl | `cur`/`next` references, `repl_trace_shutdown` | Lifecycle | +| 8.7 | Nothing to scheduler log | ReplTraceControl, TraceModel | no tlog in trace code | — | +| 9.1, 9.2, 9.3 | Line bound, marker, exact counts | TraceModel, TraceRenderer | entry budget, counters | — | +| 9.4 | Outcome 50-host cap and aggregates | TraceCapture, TraceRenderer | capped outcome copy | — | +| 9.5, 9.6 | 64 MiB store, eviction, discarded report | TraceStore | `repl_trace_store_put`, discarded marks | — | +| 10.1, 10.2, 10.3 | No scheduling change | All hooks (read-only), DecisionTransport | equivalence tests | — | +| 10.4, 10.5 | REPL output unchanged when off | ReplTraceControl, RollupIntegration, SimIntegration | gates on `rs->trace` | — | +| 11.1–11.4 | Sim-only; single branch in live | TraceHooks, DecisionTransport | `trace_ctl`/`trace_rec` NULL in live | — | + +## Components and Interfaces + +| Component | Domain / Thread | Intent | Requirements | Key Dependencies | Contracts | +|-----------|-----------------|--------|--------------|------------------|-----------| +| TraceModel | Core / allocator (build), sim (read) | Record types, arena, entry budget, counters, jobsel predicate | 1.2, 1.3, 1.8, 3.3, 3.4, 3.6, 4.x, 5.9, 5.10, 9.1–9.3 | libxce containers (P0) | Service, State | +| TraceCapture | Core / allocator (and sim pre-go window) | Read scheduler state into record entries | 2.2, 2.3, 2.5, 3.1, 3.2, 3.5, 4.1, 4.2, 5.4, 5.5, 6.x, 9.4 | core resmgr, schedobj limits, job allocs (P0) | Service | +| TraceHooks | Core / allocator | Gated call sites at the decision seams | 2.1, 2.4, 3.x, 4.x, 5.x, 11.2 | TraceModel, TraceCapture (P0) | — | +| DecisionTransport | Core | Carry records on decisions; trace-only op | 2.4, 7.2, 10.1, 10.2 | `decision.h` consumers (P0) | State | +| SimIntegration | Sim driver | Session hooks, reconstructed notices, drain-loop hand-off | 1.6, 2.5, 7.1–7.5 | repl.h gates (P0) | — | +| ReplTraceControl | REPL / sim thread | Command, settings lifecycle, inline emission, file copy | 1.x, 7.1–7.4, 8.x, 10.4, 10.5 | TraceRenderer, TraceStore (P0) | Service, State | +| TraceStore | REPL / sim thread | Latest record per job; eviction; `info trace` | 1.7, 7.8–7.11, 9.5, 9.6 | libxce htab (P1) | Service, State | +| TraceRenderer | REPL / sim thread | Record to text lines | 6.x, 7.6, 7.7, 8.1, 9.2–9.4 | repl_fmt helpers (P1) | Service | +| ReplIntegration | REPL | help, dispatch, `info` route, `set output`, rollup rule | 1.11, 7.3, 7.4, 8.4, 10.5 | existing REPL files (P1) | — | + +### Core Side + +#### TraceModel (`include/sched/trace.h`, `src/sched_trace.c`) + +**Responsibilities & Constraints** +- Owns `xce_sched_trace_rec_t`: an append-only sequence of typed entries in a per-record arena, plus fixed-size header, selection, counters and outcome blocks. A record is built by exactly one thread at a time and is immutable once attached to a decision. +- Enforces the entry budget (`line_bound`) over the requirement, cluster-wide check, plan and host sections. When the budget is exhausted it stops appending and counts omitted entries, while counters keep updating (9.2, 9.3). +- Owns verdict classification, the per-position summary tallies (5.9) and the decision-wide host-evaluation counts per verdict kind (9.3). +- No I/O and no tlog. It reads only the arguments passed to it (copying names, task counts, requested resources and plan shapes into the arena). It keeps the scope's `obj`/`w` pointers only until the matching scope ends, for TraceCapture's limit lookup. + +**Contracts**: Service [x] / State [x] + +##### Service Interface +```c +typedef enum { XCE_TRACE_LEVEL_SUMMARY = 0, XCE_TRACE_LEVEL_FULL } xce_sched_trace_level_t; + +typedef enum { /* Requirement 5.10 verdict kinds, bit flags */ + XCE_TRACE_V_ACCEPTED = 1u << 0, + XCE_TRACE_V_SELECT = 1u << 1, /* selection mismatch (pre-filter) */ + XCE_TRACE_V_SHORTFALL = 1u << 2, /* resource shortfall (per-host check) */ + XCE_TRACE_V_LIMIT = 1u << 3, /* limit exceeded (per-host check) */ + XCE_TRACE_V_CLAIMED = 1u << 4, /* already claimed by earlier position */ + XCE_TRACE_V_INTERNAL = 1u << 5 /* not evaluated: internal error */ +} xce_sched_trace_verdict_t; + +typedef enum { /* Requirement 2.4 / 2.5 record kinds */ + XCE_TRACE_REC_DECISION = 0, + XCE_TRACE_REC_NOT_FOUND, + XCE_TRACE_REC_NO_SCHEDOBJ, + XCE_TRACE_REC_INTERNAL_ERROR, /* alloc_rc < 0 or decision build failure */ + XCE_TRACE_REC_PEND_NO_REASON, + XCE_TRACE_REC_RECONSTRUCTED +} xce_sched_trace_rec_kind_t; + +typedef struct xce_sched_trace_sel_s { /* one selector from `trace on job ...` */ + uint64_t key; /* plain jobid, array base id, or element id */ + bool whole_array;/* true: key is an array base, matches every element */ +} xce_sched_trace_sel_t; + +typedef struct xce_sched_trace_ctl_s { /* immutable during a session except position */ + bool all; + int nsel; + xce_sched_trace_sel_t* sel; + xce_sched_trace_level_t level; + int line_bound; /* 50..100000 */ + uint64_t session_no; /* rs->session_count of this session */ + const char* policy_name; /* picker name, owned by the ctl */ + int position; /* allocator-owned, reset at session begin */ +} xce_sched_trace_ctl_t; + +typedef struct xce_sched_trace_rec_s xce_sched_trace_rec_t; /* opaque outside sched_trace*.c / renderer */ + +#define XCE_TRACE_LINE_BOUND_DEFAULT 1000 /* Requirement 9.1 */ +#define XCE_TRACE_LINE_BOUND_MIN 50 +#define XCE_TRACE_LINE_BOUND_MAX 100000 +#define XCE_TRACE_OUTCOME_HOSTS 50 /* Requirement 9.4 */ + +typedef enum { + XCE_TRACE_PATH_SINGLE, /* PRIMARY job: the single requirement */ + XCE_TRACE_PATH_COMPOUND, /* top-level COMPOUND container */ + XCE_TRACE_PATH_ALT, /* alternative i/n of an ALTER (a compound) */ + XCE_TRACE_PATH_MEMBER /* member j/m of the enclosing compound */ +} xce_sched_trace_path_kind_t; +typedef struct { /* kept inside the record, not passed by hooks */ + int depth; /* 0..2: container (COMPOUND|ALT) > MEMBER, or SINGLE */ + struct { xce_sched_trace_path_kind_t kind; int i; int n; } lvl[2]; +} sched_trace_path_t; + +typedef enum { /* Requirement 4.7 reasons */ + XCE_TRACE_NOPLAN_DECLINED, /* allocator accepts() declined */ + XCE_TRACE_NOPLAN_GENERATOR_FAILED, /* plan_init failed */ + XCE_TRACE_NOPLAN_NONE_GENERATED /* generator produced no plan */ +} xce_sched_trace_noplan_t; + +typedef enum { + XCE_TRACE_POS_FILLED, + XCE_TRACE_POS_UNFILLED, + XCE_TRACE_POS_SCAN_ERROR, /* host scan could not start or stopped (4.4) */ + XCE_TRACE_POS_INTERNAL /* host accepted but claim snapshot failed */ +} xce_sched_trace_pos_end_t; + +/* gate: evaluates `call` only when a record is being built */ +#define SCHED_TRACE(core, call) \ + do { if ((core)->trace_rec != NULL) { call; } } while (0) + +bool sched_trace_ctl_wants(const xce_sched_trace_ctl_t* ctl, uint64_t jobid); +xce_sched_trace_rec_t* sched_trace_rec_new(const xce_sched_trace_ctl_t* ctl, uint64_t jobid, + xce_sched_trace_rec_kind_t kind); +void sched_trace_rec_free(xce_sched_trace_rec_t* rec); /* NULL-safe */ +/* finish a record as a no-decision kind (NOT_FOUND, NO_SCHEDOBJ, INTERNAL_ERROR, + * PEND_NO_REASON) before it is pushed via make_trace_decision (2.4) */ +void sched_trace_rec_finish(xce_sched_trace_rec_t* rec, xce_sched_trace_rec_kind_t kind, int err); +size_t sched_trace_rec_bytes(const xce_sched_trace_rec_t* rec); /* arena + blocks */ + +/* structural entries (each = one rendered line in the budgeted sections) */ +void sched_trace_path_push(xce_sched_trace_rec_t*, xce_sched_trace_path_kind_t kind, int i, int n); +void sched_trace_path_pop(xce_sched_trace_rec_t*); +/* begin/end of the requirement named by the current path; exactly one end per begin */ +void sched_trace_resreq_begin(xce_sched_trace_rec_t*, + const xce_primary_resreq_t* primary /* NULL for a container */, + int nmembers /* container: resreq->primaries.count; else 0 */); +void sched_trace_resreq_end(xce_sched_trace_rec_t*, int rc, int not_tried_after); +/* non-scope entries */ +void sched_trace_member_failed(xce_sched_trace_rec_t*, int j, int m, int err); /* merge failure after member j of m */ +void sched_trace_alt_result(xce_sched_trace_rec_t*, int chosen_i /* 0 = none */, int n); +void sched_trace_alt_skipped(xce_sched_trace_rec_t*, int i, int n); +void sched_trace_check_begin(xce_sched_trace_rec_t*, const xce_sched_obj_t* obj); /* cluster-wide scope */ +void sched_trace_check_end(xce_sched_trace_rec_t*, int rc, bool plans_follow); +void sched_trace_no_plan(xce_sched_trace_rec_t*, xce_sched_trace_noplan_t why); +void sched_trace_plan_begin(xce_sched_trace_rec_t*, const xce_alloc_plan_t* plan); +void sched_trace_plan_discarded(xce_sched_trace_rec_t*, const xce_alloc_plan_t* plan); +void sched_trace_plan_not_tried(xce_sched_trace_rec_t*, const xce_alloc_plan_t* plan, int err); +void sched_trace_plan_end(xce_sched_trace_rec_t*, int rc, int failed_pos); +void sched_trace_pos_begin(xce_sched_trace_rec_t*, int pos, int slots); +void sched_trace_pos_end(xce_sched_trace_rec_t*, xce_sched_trace_pos_end_t how, const char* host); +void sched_trace_host_begin(xce_sched_trace_rec_t*, const xce_sched_obj_t* obj, + const xce_sched_worker_t* w); /* per-host scope */ +void sched_trace_pos_scan_error(xce_sched_trace_rec_t*, int err); /* host scan stopped */ +void sched_trace_host_verdict(xce_sched_trace_rec_t*, unsigned verdicts, int err); +void sched_trace_rollback_host(xce_sched_trace_rec_t*, const char* host); /* one call per released claim */ +void sched_trace_commit(xce_sched_trace_rec_t*); /* committed plan = the last plan_begin */ +/* end of sched_alloc_search_: one budgeted line; also updates decision-wide counters */ +void sched_trace_search_end(xce_sched_trace_rec_t*, int rc); /* counts come from the model (see below) */ +void sched_trace_finalize_failed(xce_sched_trace_rec_t*, int err); +``` +- **Preconditions**: `rec != NULL` for every call (guaranteed by `SCHED_TRACE`). Calls nest in the order the allocator executes them. Every `path_push` has exactly one `path_pop`, and every `*_begin` has exactly one `*_end`, on every return path at the call site. Outcome corrections that occur after an `*_end` (a merge failure, an alternative's selection) are separate non-scope entries, never a second `*_end`. +- **Postconditions**: + - Entries beyond `line_bound` are not stored; `omitted` is incremented by the number of lines they would have produced. + - In `XCE_TRACE_LEVEL_SUMMARY`, host entries are not stored. Instead `pos_end` stores one position-summary entry: accepted host, count per verdict kind, and (limit name, scope, count) tallies. + - Counters (`plans_tried`, `plans_discarded`, `host_evals[kind]`) are exact regardless of truncation. +- **Invariants**: a record never references scheduler-owned memory. All strings and numbers are copied into its arena. + +##### State Management +- **Record states**: building (allocator) → attached (on decision) → delivered (REPL) → stored or freed. +- **Concurrency**: no locks. Ownership moves with the decision through the `core->decisions` cbuf. + +**Implementation Notes** +- Integration: the record keeps the current `sched_trace_path_t`. Callers push the container level (SINGLE, COMPOUND 1/1, or ALT i/n) before calling into the allocation chain, and `sched_make_compound_alloc_` pushes MEMBER j/m per member. Each `resreq_begin` stores the full current path, so the renderer prints labels such as `alternative 2/3 > member 1/2` (3.1) without the compound function needing to know whether it serves an ALTER. +- Validation: `test_trace_model.c` covers budget edges (bound-1, bound, bound+1), summary tallies with multi-kind hosts, counters under truncation, and `ctl_wants` for plain, element and array selectors. +- Risk: arena growth uses `malloc` on the allocator thread. This happens only for traced jobs, never in live. + +#### TraceCapture (`src/sched_trace_capture.c`) + +**Responsibilities & Constraints** +- Reads scheduler state on the thread that owns it at the moment of capture, and writes entries through TraceModel: + - `sched_trace_capture_selection(rec, core, job)`: policy name, queue name and priority, job priority, `queue_time` (2.2). + - `sched_trace_capture_glimits(rec, obj)`: each `xce_vres_t` in `obj->glimits` as (limit name, scope instance) (2.3). + - `sched_trace_check_rows(rec, core, nres, idxs, total, inuse, uncomitted, wanted)`: called from `unsatisfy_`. + - For each row it resolves `xce_resource_t` through `vget(core->resmgr, idx)`. + - A row whose flags include `XCE_RESF_LIMIT` becomes a limit row. Its limit name comes from the matching `xce_vres_t` in the current scope's limit list: `obj->glimits` for the cluster-wide scope, `obj->hlimits[w->idx]` for a host scope. The scope instance is the vres name with its `limit=` prefix removed, rendered as `u=..,q=..,p=..`. In a host scope it is followed by `h=name>`, because host-limit vres names do not encode the host. When no scope pointer exists (reconstructed records, Requirement 2.5, captured on the sim thread), the limit name and scope are parsed from the `XCE_RESF_LIMIT` resource's own name (`limit=/...`). The value, usage, reserved and requested amounts are `total`, `inuse`, `uncomitted` and `wanted`. + - Every other row is a resource row. + - In the cluster-wide scope every row is kept (4.1). In a host scope only unmet rows are kept, and they set `SHORTFALL` and/or `LIMIT` on the host's verdict (5.4, 5.5). + - `sched_trace_capture_attempt_reasons(rec, reason)`: per-code counts and the hosts/resources named, from a compound attempt's `ctx.reason`. It is captured before `scheduler.c` replaces `obj->reason` and renders as the failure-reasons line of a failed member or alternative (3.5). + - `sched_trace_capture_dispatch(rec, core, obj, alloc)`: per-host allocation rows and shared rows, with limit charges separated by `XCE_RESF_LIMIT` (6.1–6.3). + - Allocation rows carry only a resource name (`xce_alloc_t.resname`). Each row is classified as a limit charge when its resource, looked up by name in `core->resmgr`, has `XCE_RESF_LIMIT`; when the name is not registered in this replay, the `limit=` name prefix decides. + - A limit charge is resolved to its `xce_vres_t` by matching `resname` against `vres->vres->name`. Shared charges are searched in `obj->glimits`. Host charges are searched in `obj->hlimits[w->idx]`, where `w` is found by the allocation's host name. + - The vres supplies the limit name, scope and limited resource name. The first 50 hosts are stored in detail. The rest are aggregated into per-resource totals (excluding limits) and per-limit totals, plus an omitted-host count (9.4). + - `sched_trace_capture_pend(rec, reason)`: a deep copy of the decision's `xce_pendreason_t`. `main_reason` and host-less entries are always kept. Host entries of the first 50 distinct hosts are kept, and the rest are counted per pend-reason code with an omitted-host count (6.4, 9.4). + - `sched_trace_rec_reconstructed(ctl, core, jobid, obj, allocs)`: builds a `RECONSTRUCTED` record with a dispatch outcome (2.5), resolving limit charges the same way. + - A job never evaluated by this replay has empty `glimits`/`hlimits`, or a NULL `obj`. A limit charge with no matching vres is still listed as a limit charge (by its `XCE_RESF_LIMIT` flag), with its name and scope parsed from the vres resource name. + - Its amount is then shown as a raw number, because the limited resource is unknown. This is the only case where a limit amount is not unit-formatted. +- **Constraint**: inputs are read-only. No function here calls a scheduler API that mutates state (Requirement 10). + +**Contracts**: Service [x] + +```c +/* declared in include/sched/trace.h, implemented in sched_trace_capture.c */ +void sched_trace_capture_selection(xce_sched_trace_rec_t*, const xce_sched_core_t* core, + const xce_sched_job_t* job); +void sched_trace_capture_glimits(xce_sched_trace_rec_t*, const xce_sched_obj_t* obj); +void sched_trace_check_rows(xce_sched_trace_rec_t*, xce_sched_core_t* core, int nres, + const int* idxs, const float* total, const float* inuse, + const xvec_t* uncomitted, const float* wanted); +void sched_trace_capture_attempt_reasons(xce_sched_trace_rec_t*, const xce_pendreason_t* reason); +void sched_trace_capture_dispatch(xce_sched_trace_rec_t*, xce_sched_core_t* core, + const xce_sched_obj_t* obj, const xce_job_alloc_t* alloc); +void sched_trace_capture_pend(xce_sched_trace_rec_t*, const xce_pendreason_t* reason); +xce_sched_trace_rec_t* sched_trace_rec_reconstructed(const xce_sched_trace_ctl_t* ctl, + xce_sched_core_t* core, uint64_t jobid, + const xce_sched_obj_t* obj /* job->schedobj, nullable */, + const xce_job_alloc_t* allocs); +``` +`core` is non-const only because `vget(core->resmgr, ...)` takes a non-const container; the functions never write to it. + +**Implementation Notes** +- Integration: the current scope's schedobj and worker are remembered by the record when `check_begin` / `host_begin` run (the runner and the host iterator pass `ctx->schedobj` and the candidate worker). `unsatisfy_` therefore needs only `core`. +- Per-host verdict when `sched_try_alloc_on_host_` returns 0: it is SHORTFALL and/or LIMIT, from the unmet rows recorded during the call. If no row was recorded, the 0 came from the function's own selection branch (`PREASON_MATCH_FAIL`), so the verdict is SELECT in the per-host phase. +- Validation: `test_trace_capture.c` uses a fixture core with a limit vres on a shared and a host resource to check: + - limit rows are never reported as shortfalls; + - scope strings are correct; + - dispatch limit charges are listed separately; + - the 51-host aggregation; + - the pend copy equals the source under `job_preason_eq` when there are 50 or fewer hosts. +- Risk: the vres lookup is linear in the number of matched limits per job. That number is small (configured limits), and it is paid only for traced jobs. + +#### TraceHooks (call sites) + +Each hook is one `SCHED_TRACE(core, ...)` statement placed after the value it reports has been computed. It does not change control flow, return values or arguments to existing calls. + +| Seam | File:anchor | Hook(s) | Requirements | +|------|-------------|---------|--------------| +| Allocator per-job prologue | `sched_election.c` after `cbuf_get` (jobid ≠ 0, not RECONCILE_WAKE) | `if (core->trace_ctl) { ctl->position++; if (wants) core->trace_rec = rec_new(...) }` (the only non-macro hook; one branch in live) | 2.1, 11.2 | +| Job not found / no schedobj | `sched_election.c:226-241` | `NOT_FOUND`: no selection capture (`job == NULL`). `NO_SCHEDOBJ`: `capture_selection` without limits. Then `rec_finish`, push the trace decision and clear `trace_rec` | 2.4 | +| After election | after `plugin_job_election` | `capture_selection`, `capture_glimits` | 2.2, 2.3 | +| Alloc error / pend without reason / decision build failure | `sched_election.c:250-254, 265-269, 286` | `rec_finish(INTERNAL_ERROR / PEND_NO_REASON, err)`, push the trace decision, and clear `trace_rec` before the `continue`. Every path that hands a record away clears `trace_rec`, so no later job in the session can append to a record the REPL owns. | 2.4 | +| Decision built | before `sched_commit_decision` | `capture_dispatch(core, job->schedobj, alloc)` / `capture_pend`; `d->trace = rec`; clear `trace_rec` | 6.x, 9.4 | +| Session-end marker | jobid 0 | `core->trace_rec = NULL` (defensive) | — | +| PRIMARY job | `scheduler.c` `sched_make_allocation`, PRIMARY case | `path_push(SINGLE,1,1)`; `resreq_begin(primary, 0)`; call; `resreq_end(rc, 0)`; `path_pop` | 3.1, 3.2 | +| Top-level COMPOUND | `sched_make_allocation`, COMPOUND case, around the `sched_make_compound_allocation` call | `path_push(COMPOUND,1,1)`; `resreq_begin(NULL, resreq->primaries.count)`; call; `resreq_end(rc, 0)`; `path_pop` | 3.1, 3.2 | +| ALTER loop | `scheduler.c:409-436`, per alternative i of n | NULL resreq → `alt_skipped(i,n)`. Otherwise `path_push(ALT,i,n)`; `resreq_begin(NULL, alt->resreq->primaries.count)`; call `sched_make_compound_alloc_`; `resreq_end(rc, 0)`; `path_pop`. On the first success, `alt_result(i,n)` renders "alternative i/n chosen; alternatives i+1..n not tried" (3.4). If the loop ends with none chosen, `alt_result(0,n)` renders "all alternatives failed" (3.5). | 3.4–3.6 | +| Compound members | `sched_make_compound_alloc_` per member j of m | `path_push(MEMBER,j,m)`; `resreq_begin(primary, 0)`; call; `resreq_end(rc, rc<=0 ? m-j : 0)`; `path_pop`. Here `m-j` is the number of members after j, with j 1-based. If `merge_job_alloc` then fails (`scheduler.c:339-401`), `member_failed(j, m, mrc)` renders "member j/m: merge failed (internal error); members j+1..m not tried" (3.3). At the end of a failed attempt, `capture_attempt_reasons(ctx.reason)` records the failure reasons before `obj->reason` is replaced (3.5). | 3.1–3.3, 3.5 | +| Runner entry | `alloc_runner.c` accepts decline | `no_plan(DECLINED)` | 4.7 | +| Cluster-wide check | around `sched_alloc_prepare_shared_` | `check_begin` / `check_end(rc, plans_follow=rc>0)` → on failure renders "no plans tried" | 4.1, 4.2, 4.8 | +| Generator start failure | `plan_init != 0` | `no_plan(GENERATOR_FAILED)` | 4.7 | +| Search end | end of `sched_alloc_search_` | `search_end(rc)`. The model counts generated, tried and discarded plans per requirement scope from the `plan_begin` / `plan_discarded` / `plan_not_tried` calls it already receives, so the runner keeps no extra bookkeeping outside the gate (11.2). It always renders as one line, `plans: g generated, t tried, d discarded, x not tried`, extended on the same line by at most one statement: +- `; no plan generated` (`NONE_GENERATED`, 4.7) when g = 0; +- `; no plan tried: all generated plans discarded` (4.9) when g > 0 and d = g; +- `; no plan could be placed` (4.6) when rc = 0, t > 0 and no plan ended in an internal error. + +When rc > 0 or rc < 0 there is no statement; the commit or internal-error entries already say what happened. Like every other entry it counts against the line budget. It also bumps the matching decision-wide counter (requirements with no plan generated / all discarded / none placed), under the same conditions, which appear in the fixed-size totals block after truncation (9.3). | 4.6, 4.7, 4.9 | +| Plan loop | malformed / reserve failure / try / result | `plan_discarded`; `plan_not_tried(err)`; `plan_begin`; `plan_end(rc, failed_pos)` | 4.3, 4.4, 4.6, 4.9 | +| Position | `sched_claim_bucket_` | `pos_begin`; iterator init failure → `pos_end(SCAN_ERROR)`; snapshot failure after acceptance (`alloc_runner.c:237-243`) → `pos_end(INTERNAL)`; else `pos_end(FILLED/UNFILLED/SCAN_ERROR, host)` | 4.4, 5.9 | +| Candidate | claimed skip / per-host result | `host_begin` + `host_verdict(CLAIMED)`; `host_begin`, call, `host_verdict(rc>0 ? ACCEPTED : rc==0 ? from rows : INTERNAL, err)` | 5.1–5.7 | +| Pre-filter | `host_iter.c:136-156` | `host_begin` + `host_verdict(SELECT)`; uncomit error → `host_verdict(INTERNAL)` | 5.2, 5.3, 5.7 | +| Scan stop | `host_iter.c:124-127` (`rc < 0`) | `pos_scan_error(rc)`; the runner's following `pos_end(UNFILLED)` is recorded as SCAN_ERROR | 4.4 | +| Rollback | in `sched_try_plan_` after a failed position, only when `n > 0` (not inside `sched_rollback_claims_`, which is also called with `n == 0` and from finalize's partial rollbacks) | `SCHED_TRACE(core, for (k = 0; k < n; k++) sched_trace_rollback_host(core->trace_rec, claims[k].w->name))`, a single gated statement. The model merges consecutive calls into one `ROLLBACK` entry, "released hosts: n01, n02" (one line). | 5.8 | +| Finalize | `sched_finalize_job_alloc_` result | `commit` or `finalize_failed(err)` | 4.5 | +| Shortfall rows | `resource.c` top of `unsatisfy_` | `check_rows(...)` | 4.1, 4.2, 5.4, 5.5 | + +- The "pre-filter" phase covers verdicts produced inside `host_iter.c`. The "per-host" phase covers verdicts produced by `sched_try_alloc_on_host_` (5.2). +- The "not performed" rendering (5.1) follows from phase order: a pre-filter rejection implies the per-host check was not performed. +- `ALREADY CLAIMED` and `INTERNAL` carry no phase label (5.2 applies only to the selection, shortfall and limit kinds). + +#### DecisionTransport (`include/decision.h`, `src/decision.c`) + +**Contracts**: State [x] +- `xce_decision_op_t` gains `XCE_DECISION_TRACE` after `XCE_DECISION_PREEMPT`. Existing values are unchanged. +- `xce_decision_t` gains `struct xce_sched_trace_rec_s* trace;`. It is NULL for every existing constructor (`calloc`). +- `xce_decision_t* make_trace_decision(xce_sched_trace_rec_t* rec)` returns a decision with `op = XCE_DECISION_TRACE`, a zeroed union and `trace = rec`. +- **Ownership**: + - The drain loop that receives a decision owns `d->trace` and hands it to the REPL (`xce_repl_check_trace*`), which stores it or frees it. + - The REPL takes ownership by setting `d->trace = NULL`. Every sim call site (drain loop, trace-only branch, reconstructed block) then calls `sched_trace_rec_free(d->trace)` right before `free(d)`. This frees the record only if the REPL did not take it, for example when `!sim->interactive`, which cannot happen because `trace_ctl` is only set when interactive. + - `driver.c` frees any non-NULL `d->trace` and skips `XCE_DECISION_TRACE`. Both are unreachable in live; this is defensive. +- **Coordination**: `scheduler-preemption` also extends `decision.h`. Both specs append only, and neither reorders. + +### Simulator Side + +#### SimIntegration (`sim_loop.c`, `repl.h` gates) + +- `xce_repl_trace_session_begin(sim)` runs after `xce_core_begin_session` has succeeded, which means after its `rc < 0` check (`sim_loop.c:1620-1631`), whose failure branch `continue`s without draining. It runs before reconciliation. A failed session begin therefore publishes nothing and holds no settings reference. It resets `rs->trace.round_traced` and `rs->trace.round_flushed` to false in every mode, so a round that ended in step mode (for example continue → breakpoint → `step`) cannot leak the flag into the next continue-mode round. It then sets `cur = next`, then sets `ctl->session_no = rs->session_count` and `ctl->position = 0`, and publishes `sim->core.trace_ctl = cur ? &cur->ctl : NULL`. +- In the reconstructed-dispatch report block (`sim_loop.c:1309-1329`), when `trace_ctl` wants the job, the block builds a `RECONSTRUCTED` record and sets `d->trace = rec` right after `make_dispatch_decision`. The order is then: record add → `sim_print_decision_gate` → `xce_repl_check_trace(sim, d)` → `xce_repl_check_decision(sim, d)` → `sched_trace_rec_free(d->trace)` → `free(d)`. This is the same order as the drain loop, so 7.5 holds for reconstructed dispatches too (2.5). This runs before `signal_go`, so these traces precede the round's live decisions (7.3). +- Drain loop: + - `if (d->op == XCE_DECISION_TRACE) { xce_repl_check_trace_only(sim, d); sched_trace_rec_free(d->trace); free(d); continue; }` comes first. + - `xce_repl_check_trace(sim, d)` sits between `sim_print_decision_gate` and `xce_repl_check_decision`. It is a no-op when `d->trace == NULL`. `sched_trace_rec_free(d->trace)` is called just before the existing `free(d)`. +- `xce_repl_trace_session_end(sim)` runs after the drain loop, before `plugin_end_session`. It sets `sim->core.trace_ctl = NULL`, drops `cur` (closing its file if it is no longer `next`), and clears the in-progress flag. + +#### ReplTraceControl (`src/repl_trace.c`) + +**Responsibilities & Constraints** +- Parses the raw input line for `trace` (1.9, 1.12): + ``` + trace on all [level summary|full] [lines N] [file PATH] + trace on job JOBSEL [JOBSEL ...] [level summary|full] [lines N] [file PATH] + trace off + trace status + ``` + - `JOBSEL` is `parse_jobsel` syntax (job, `job[idx]`, array base). + - `file` must be the last option. `PATH` is the rest of the line, trimmed, with one optional pair of surrounding quotes removed, and must be at most 4095 bytes. + - On any invalid argument, it prints `[sim] trace: invalid : ` and changes nothing. +- Keeps settings in `rs->trace` as reference-counted settings objects (see Trace settings lifecycle): + - `next`: the settings the next session will use, or NULL when tracing is off. + - `cur`: the settings of the session in progress, or NULL. + - `session_in_progress` and `round_traced`. +- File rules: + - The file is opened at command time in append mode, so open errors are reported immediately (8.2). + - Before opening, the path is compared by `stat()` dev/inode against `fstat(fileno(sim->record_file))` when an output file is set, and against `stat(sim->log_path)` (8.3). + - Replacing or disabling `next` drops its reference. If `cur` still holds that object, its file stays open until the session-end hook drops `cur`; otherwise the file is closed immediately (8.5). +- `repl_trace_file_conflicts(sim, path)` returns true when `path` matches, by dev/inode, the file of `next` or `cur`. These are every trace file still open. `set output` rejects such a path (8.4). +- Inline emission (`xce_repl_check_trace`, `xce_repl_check_trace_only`) runs on the sim thread: + 1. In any mode it sets `round_traced = true`, so a round traced in step mode and finished with `continue` is never collapsed (7.4). In continue mode only: if this is the round's first traced item, it calls `repl_round_collapse_flush(sim)`; then, for every traced decision (not trace-only), it prints the echo line that the print gate suppressed, by calling `sim_print_decision(d)`. That function is made non-static and declared in `sim.h`, with its format unchanged (7.3). A per-round `round_flushed` bit, also reset at session begin, records whether the flush has been done. + 2. Renders the record to stdout and, if `cur` (the settings the record was captured under) has a file, to that file, then `fflush`es both (7.1, 7.2, 8.1, 8.6). + 3. Hands the record to the TraceStore (7.10). +- `repl_trace_shutdown(sim)` is called from `repl_cmd_quit`, the EOF path and process exit. It flushes and closes the files of `cur` and `next` (8.6). +- Writes nothing through tlog (8.7). + +**Contracts**: Service [x] / State [x] +```c +bool repl_trace_cmd(xce_sched_sim_driver_t* sim, const char* raw_line); /* returns false: re-prompt */ +void repl_trace_session_begin(xce_sched_sim_driver_t* sim); +void repl_trace_session_end(xce_sched_sim_driver_t* sim); +void repl_trace_on_decision(xce_sched_sim_driver_t* sim, xce_decision_t* d); /* takes d->trace */ +void repl_trace_on_trace_only(xce_sched_sim_driver_t* sim, xce_decision_t* d); +bool repl_trace_file_conflicts(const xce_sched_sim_driver_t* sim, const char* path); +void repl_trace_shutdown(xce_sched_sim_driver_t* sim); +``` + +`trace status` output (1.7) shows a block for `cur` ("in effect", when a session is in progress) or `next` ("next session", otherwise), plus a second "replaces at next session" block when `next` differs from `cur`. Each block has enabled, selection (rendered with `repl_fmt_jobid_range`), level, line bound and file. A final line gives stored-trace memory `used / 64 MiB`. + +#### TraceStore (`src/repl_trace_store.c`) + +**Responsibilities & Constraints** +- An `htab` keyed by jobid maps to `{rec, bytes, seq}`. A capture-order FIFO supports eviction, and a set of discarded jobids (with their seq) supports 9.6. +- `put(rec)` replaces any existing entry for the jobid and frees the old record (7.10). It then evicts the oldest entries until `total_bytes <= 64 MiB`, never evicting the newest; evicted jobids are marked discarded (9.5). A newer capture clears the discarded mark. +- `info trace `: + - **Element or plain job**: renders the stored record at its captured level with the current units mode (7.8). If the trace was discarded it reports `trace discarded (memory bound)` (9.6). Otherwise it reports `no trace available` with the job's current state, or `unknown job` (7.11). + - **Array base**: iterates the stored and discarded entries whose `array_jobid` matches, in index order. If there are none, it lists the elements' current states from `core->jobs` (7.9). +- The store is only read and written on the sim thread. It never reads live scheduler state except job state for 7.9 and 7.11, at the REPL prompt. + +**Contracts**: Service [x] / State [x] +```c +void repl_trace_store_put(xce_repl_trace_state_t* ts, xce_sched_trace_rec_t* rec); +size_t repl_trace_store_bytes(const xce_repl_trace_state_t* ts); +bool repl_cmd_info_trace(xce_sched_sim_driver_t* sim, char tokens[][XCE_REPL_TOKEN_LEN], int ntok); + +/* repl_internal.h: the state block embedded in xce_sched_repl_state_t */ +typedef struct xce_repl_trace_settings_s xce_repl_trace_settings_t; /* refcounted; holds ctl + file */ +typedef struct { + xce_repl_trace_settings_t* next; /* settings the next session will use, NULL = off */ + xce_repl_trace_settings_t* cur; /* settings of the session in progress, NULL = none */ + bool session_in_progress; + bool round_traced; + bool round_flushed; + htab_t by_job; /* jobid -> stored entry */ + htab_t discarded; /* jobid -> eviction seq */ + /* capture-order FIFO head/tail, total_bytes, seq counter */ +} xce_repl_trace_state_t; +``` + +#### TraceRenderer (`src/repl_trace_render.c`) + +**Responsibilities & Constraints** +- `repl_trace_render(const xce_sched_trace_rec_t* rec, xce_repl_units_mode_t units, FILE* out1, FILE* out2 /* nullable */)`. +- **Line contract**: every stored entry in the budgeted sections produces exactly one line. Header, selection and outcome lines are outside the budget. +- **Hierarchy** (7.6), two-space indent per nesting level. The example below shows every stored entry: each line from the first `requirement` line to the last `alternative` line is exactly one entry and counts against the line budget. The header, selection and outcome lines are outside the budget. + ``` + [trace] job 1-0-0-42 session 7 position 3 level full + selection: policy builtin_fcfs queue normal (priority 50) job priority 0 queued 2026-10-07 10:00:01 + cluster-wide limits: lim_user (u=alice); host-scoped limits are reported with each host's verdict + requirement alternative 1/2 (compound, 1 member) + requirement alternative 1/2 > member 1/1 tasks 4..4 requests mem=8 GiB slots=1 license=2 + cluster-wide: license total 10 in-use 9 reserved 0 requested 2 failed + cluster-wide check failed; no placement plans tried + member 1/1 not satisfied + failure reasons: NO_RESOURCE x1 (resources: license) + alternative 1/2 not satisfied + requirement alternative 2/2 (compound, 1 member) + requirement alternative 2/2 > member 1/1 tasks 4..4 requests mem=8 GiB slots=1 license=1 + cluster-wide: license total 10 in-use 9 reserved 0 requested 1 passed + cluster-wide limit lim_user (u=alice) on slots: value 100 usage 96 reserved 0 requested 4 passed + cluster-wide check passed + plan 1 hosts 1 tasks per position [4] + position 1 (4 tasks) + host n01 rejected (per-host check): resource shortfall + mem total 64 GiB in-use 60 GiB reserved 0 requested 8 GiB + host n02 rejected (pre-filter): selection mismatch; per-host check not performed + host n03 accepted + position 1 filled by n03 + plan 1 succeeded + plan 1 committed; no further plans tried + plans: 1 generated, 1 tried, 0 discarded, 0 not tried + member 1/1 satisfied + alternative 2/2 satisfied + alternative 2/2 chosen; no alternatives after it + outcome: DISPATCH + host n03: mem=8 GiB slots=4 limit charges: lim_host (h=n03) slots 4 + shared: license=1 limit charges: lim_user (u=alice) slots 4 + ``` +- **Summary level**: the position line is followed by `accepted n03; rejected: selection mismatch 1, resource shortfall 1, limit exceeded 0 (...)` and no per-host lines (5.9). +- **Truncation** (9.2, 9.3): after the last stored entry, the marker `... trace lines omitted (line bound N)`, then a fixed-size totals block: plans tried, plans discarded, requirements with no plan generated / all plans discarded / no plan placed, and host evaluations per verdict kind. +- **Outcome** (6.x, 9.4): up to 50 hosts with all their entries, then aggregates and `... hosts omitted`. A pend lists `main_reason`, host-less entries, then host entries. +- **Values** use the moved `repl_fmt` helpers with the units mode passed in (7.7). Jobids use `fmt_jobid` and selection lists use `repl_fmt_jobid_range`. +- **No-decision records** (2.4): a header, then: + - `NOT_FOUND`: `selection: ordering keys and limits unavailable (job not found by the allocator)`. + - `NO_SCHEDOBJ`: the selection line, then `cluster-wide limits: unavailable (no scheduling object)`. + - `INTERNAL_ERROR` and `PEND_NO_REASON`: the selection and limit lines, then the full entry hierarchy the hooks captured. Allocator internal errors that end the attempt (a plan not tried, a snapshot failure, a finalize failure, an error in the cluster-wide check, a member merge failure) surface as `alloc_rc < 0` for PRIMARY and top-level COMPOUND jobs. Their entries therefore carry the 3.2 / 4.4 / 4.5 outcomes. Per-host errors (5.7) and host-scan errors (4.4) do not end the attempt; they appear as host and position entries in any record kind. + - Every kind ends with one line in place of the outcome, naming the kind and the error. +- **Reconstructed records** (2.5): a header, one line stating the dispatch was reconstructed from the recorded allocation, and the outcome. + +**Contracts**: Service [x] + +#### ReplIntegration (existing REPL files) + +Summary only; no new boundaries. +- **`repl.c`**: + - `repl_dispatch` routes `trace` to `repl_trace_cmd(sim, line)`. + - `repl_print_help` adds a trailing "Trace" section (`trace on all|job ...`, `trace off`, `trace status`) and an `info trace ` line in State Inspection (1.11). + - `repl_cmd_quit` calls `repl_trace_shutdown`. +- **`repl_info.c`**: `repl_cmd_info` routes `tokens[1] == "trace"` to `repl_cmd_info_trace`. +- **`repl_params.c`**: `set output` returns an error `[sim] set output: is an open trace file` when `repl_trace_file_conflicts` is true (8.4, 10.5). +- **`repl_rollup.c`**: in `repl_round_rollup_process`, the traced-round rule runs right after `repl_query_pend_jobids` has built this round's pend list. When `rs->trace.round_traced` is set: + 1. call `repl_round_collapse_flush(sim)` (idempotent, and covers rounds that were traced only in step mode); + 2. emit this round alone via `repl_round_emit(round_no, round_no, round_ts, round_ts, dispatch_count, preempt_count, pend_jobids, n_pend_jobids, idle)` with `idle = (dispatch_count == 0) && (preempt_count == 0) && (n_pend_jobids == 0)`. This `idle` matches `repl_round_collapse_flush` (`repl_rollup.c:251-253`), not the allocation-failure fallback, so a pend-only traced round is not reported as idle; + 3. free the pend list and return without starting a range (7.3, 7.4). + + If the pend query itself fails, the existing fallback already emits the round alone. `round_traced` is reset at the next session begin. + +## Data Models + +### Domain Model +- **TraceSettings** (REPL-owned, reference-counted): `all`, selectors, level, line bound, file (path + `FILE*` + dev/inode), and the `xce_sched_trace_ctl_t` it publishes. It is referenced by `next` and/or `cur` (see Trace settings lifecycle). +- **TraceControlBlock** (`xce_sched_trace_ctl_t`, REPL-owned, published to the core for one session): an immutable copy of the selection, level and bound, plus `session_no`, `policy_name` and the allocator-owned `position`. +- **TraceRecord** (aggregate root, one per traced selection): + - **Header**: jobid, kind, level, session_no, position, line_bound, the file tag of the settings it was captured under. + - **Selection block**: policy, queue name and priority, job priority, `queue_time`, cluster-wide limits list. + - **Entries**: an ordered list. Each entry has a type, a depth, and a payload of copied strings and floats with resource flags. + - **Counters**: `plans_tried`, `plans_discarded`, `reqs_none_generated`, `reqs_all_discarded`, `reqs_none_placed`, `host_evals[6]`, `omitted_lines`. + - **Outcome**: a dispatch block (≤ 50 hosts in detail plus aggregates) or a pend block (≤ 50 hosts plus per-code counts), or none. +- **Invariants**: + - A record is self-contained (no pointers into scheduler memory). + - `stored entries ≤ line_bound`. + - Counters are exact. + +### Logical Data Model +- **Entry types** (each renders as exactly one line): + - `RESREQ_BEGIN`, `RESREQ_END`, `ALT_SKIPPED`, `ALT_RESULT`, `MEMBER_FAILED`, `ATTEMPT_REASONS` + - `CHECK_ROW` (resource or limit), `CHECK_END` (passed / failed, with "no placement plans tried" on failure), `NO_PLAN` + - `PLAN_BEGIN`, `PLAN_DISCARDED`, `PLAN_NOT_TRIED`, `PLAN_END`, `COMMIT`, `FINALIZE_FAILED`, `SEARCH_END` + - `POS_BEGIN`, `POS_SUMMARY` (summary level), `POS_END` (filled / unfilled / scan error / internal; the scan error is set by `pos_scan_error`) + - `HOST` (verdict bitmask, phase, error), `HOST_ROW` (shortfall or limit row under a host) + - `ROLLBACK` +- **Row payload**: resource name, flags, valtype, total, in-use, reserved, requested, met. Limit rows and limit charges add the limit name, the scope instance and the limited resource's name (`vres->res->name`). When no vres is in scope, the limited resource is unknown: the name and scope are parsed from the `limit=` resource name and the amount is a raw number (TraceCapture's fallback). The renderer formats limit amounts by that limited resource's name, matching the existing `info` limit display (`repl_info.c:1359-1360`), so a memory limit renders in bytes units (7.7). +- **Store**: `htab`, a FIFO by capture sequence, and `htab`. Memory accounting is the sum of `sched_trace_rec_bytes`. The discarded-mark table is not counted; it stays small because it holds one mark per evicted jobid. + +## Error Handling + +### Error Strategy +- **Capture-side allocation failure** (arena growth on the allocator thread): the record sets an `incomplete` flag and stops appending. The renderer prints `... trace incomplete (out of memory)`. Scheduling is unaffected. If allocating the record itself fails, the job is not traced. +- **Trace decision allocation failure** (`make_trace_decision` returns NULL): the record is freed and that no-decision trace is lost. This is the only best-effort gap, and it happens only after an earlier allocation failure. +- **User errors** (1.9, 8.2, 8.3, 8.4): one `[sim] trace: ...` or `[sim] set output: ...` line naming the argument or file and the cause (`strerror(errno)`). Settings are left unchanged. +- **File write errors** during emission: reported once per session as `[sim] trace: write to failed: `. The file stays attached until the settings change, and console output continues. + +### Monitoring +None beyond REPL output. Trace code never writes to tlog (8.7). + +## Testing Strategy + +TDD per `testing.md`: each test below is written before the code it covers. + +### Unit Tests +1. `test_trace_model.c`: + - entry budget at bound−1, bound and bound+1, with the exact omitted count; + - counters stay exact after truncation; + - summary tallies count a host with SHORTFALL+LIMIT once per kind (5.9, 9.3); + - `sched_trace_ctl_wants` for plain, element, whole-array and unknown-yet-future ids (1.2, 1.3). +2. `test_trace_capture.c`: + - limit rows vs resource rows in both cluster-wide and host scope; + - limit never reported as a shortfall (4.2, 5.5); + - dispatch limit charges separated (6.1–6.3); + - 51-host dispatch and pend aggregates (9.4); + - the pend copy equals the source under `job_preason_eq` (6.4). +3. `test_trace_hooks_gate.c`: + - `SCHED_TRACE` does not evaluate a side-effecting argument when `trace_rec == NULL`; + - a full live-style allocation with `trace_ctl == NULL` allocates no record (11.2); + - a non-interactive sim run never publishes `trace_ctl` (11.3). +4. `test_trace_render.c`: + - golden output per section for full and summary levels; + - units modes (7.7); + - truncation marker and totals block (9.2, 9.3); + - outcome cap rendering (9.4); + - no-decision and reconstructed records (2.4, 2.5); + - one line per budgeted entry. +5. `test_trace_store.c`: + - replace (7.10); + - eviction at 64 MiB never drops the newest (9.5); + - discarded vs no-trace vs unknown job (7.11, 9.6); + - array listing with mixed stored and discarded entries (7.9). + +### Integration Tests (sim driver with real picker and allocator threads) +1. `test_repl_trace_cmd.c`: + - grammar and every invalid-argument class (1.9); + - a path longer than 63 characters and 20 job ids (1.12); + - `next` vs `cur` across a decision breakpoint and a round-begin breakpoint (1.6); + - `trace status` blocks (1.7); + - file open failure (8.2); + - dev/inode rejection against `--sim-output`, `set output` and the log path, including a hard link (8.3, 8.4); + - a replaced file is kept open until session end, or closed immediately between sessions (8.5); + - quit flushes (8.6). +2. `test_repl_trace_e2e.c`, over a contention fixture with an ALTER job, a parjob, a shared license, a cluster-wide limit and a host-scoped limit: + - every section of Requirements 2–6 present; + - step-mode placement (7.1, 7.2); + - continue-mode flush and no collapse (7.3, 7.4); + - breakpoint ordering with `info trace` at the stop (7.5); + - the file copy byte-equals the console trace output (8.1); + - `info trace` after the job finishes, with the units mode changed (7.8). +3. Error-path seams use the **real** `sched_allocator`, never a stub `sched_allocator()`, because a stub would bypass the hooks under test. + - The test `#include`s `sched_election.c`, `scheduler.c`, `alloc_runner.c`, `host_iter.c`, `resource.c`, `sched_trace*.c` and the other needed sources directly, instead of linking `scheduler_srcs`. This lets source-level macros apply. + - Faults are injected as follows: + - **Plan reserve failure** (4.4): `#define realloc test_realloc` around the `alloc_runner.c` include, as in `test_alloc_runner_search.c:30-33` (macro) and `:87-95` (`test_realloc`). + - **Snapshot and finalize failures** (4.4, 4.5): the link-level wraps of two existing tests, all three listed in the new test's `link_args`: + - `test_alloc_runner_search.c`: `--wrap=xvec_dup` (`testsuite/meson.build:1780`, `__wrap_xvec_dup` at `test_alloc_runner_search.c:102-114`); + - `test_alloc_runner_result.c`: `--wrap=make_host_alloc` and `--wrap=make_job_alloc` (`meson.build:1793-1794`). + + With every scheduler source included, `xvec_dup` is called from more places, so the existing pattern of arming the failure on a specific call number (`test_alloc_runner_search.c:98-114`) is kept. `test_alloc_runner_result.c`'s `sched_make_allocs` stub is not used, because it would collide with the real `scheduler.c`. + - **Host-scan error** (4.4): a worker iterator whose `iter_next` returns `< 0`. + - **Per-host internal error** (5.7): a failing per-host uncommitted-vector fetch. + - **`alloc_rc < 0` no-decision trace** (2.4): a stub `job_alloc_fn` returning `-EIO`. + - **Job-not-found** (2.4): a direct `cbuf_put` of an unknown jobid on `jbuf`. + +### Equivalence Tests (Requirement 10) +1. `test_trace_equivalence.c`: the same contention replay is run four ways: trace off, `all`/`summary`, `all`/`full`, and a job subset. Each run must match on: + - the decision sequence and pend-reason sets (`job_preason_eq`); + - final `resInUse`/`sharedInUse`/`job->accounts`; + - `--sim-verify` mismatch count; + - `--sim-output` bytes, excluding wall-clock metadata (10.1–10.3). +2. With trace off, the REPL transcript of a scripted session is byte-identical to a baseline captured before the feature, except for the `help` text (10.4, 10.5). +3. The existing `test_repl_equivalence.c`, `test_sim_verify*` and live driver tests stay green unchanged (11.4). + +### Performance +- Live and trace-off cost: `test_trace_hooks_gate.c` asserts no record allocation. Code review checks that each hook is a single `SCHED_TRACE` statement (11.2). +- Large trace: a 1000-host, maxTasks = 64 parjob traced at `full` with the default bound completes. The record stays within the line bound, and the store stays within 64 MiB after 10,000 traced decisions (9.1, 9.5). + +## Security Considerations +- The trace file path is operator-supplied in an interactive local tool. It is opened in append mode with the operator's permissions. The dev/inode checks stop the trace from overwriting the replay output or the scheduler log. There are no new network or privileged surfaces. + +## Invariant Cross-Check (scheduler audit dimensions) +1. **Driver state-equivalence**: hooks only read state. The live driver never sets `trace_ctl`/`trace_rec`. The equivalence tests use a contention fixture, as `scheduler-driver-equivalence.md` requires. **Holds.** +2. **Core I/O isolation**: `sched_trace*.c` do no I/O and are added to `test_core_isolation.c`'s source-text scan, with an additional no-stdio check (not to `sched_core_isolation_lib`; see Modified Files). `core.h` gains two opaque pointers only. **Holds.** +3. **jbuf capacity-1 rendezvous**: unchanged. The position is counted by the allocator from its own pops, and no picker-side hook exists. **Holds.** +4. **Snapshot field-completeness**: not applicable. Trace state is not part of the scheduler snapshot, recording or WAL; `snapshot` content must be unchanged (10.2). No save/restore mechanism is added. Stored traces are deliberately not persisted. diff --git a/.kiro/specs/scheduler-sim-decision-trace/requirements.md b/.kiro/specs/scheduler-sim-decision-trace/requirements.md index f0edd11a..11eb144f 100644 --- a/.kiro/specs/scheduler-sim-decision-trace/requirements.md +++ b/.kiro/specs/scheduler-sim-decision-trace/requirements.md @@ -41,7 +41,9 @@ The xce_scheduler simulator (`--sim` with the interactive REPL, `-i`) replays re - which hosts were considered and why each could or could not satisfy the job - which resources and limits were finally allocated -In this document a *round* (the REPL's term) and a *scheduling session* are the same unit: one pass of the scheduler over the pending jobs. +In this document a *round* (the REPL's term) and a *scheduling session* are the same unit: one pass of the scheduler over the pending jobs. A scheduling session is *in progress* from the moment the scheduler begins it, before its first reconstructed dispatch (Requirement 2.5) or job selection, until the simulator has reported the session's last decision or no-decision selection; it has *ended* after that point. A breakpoint that stops before the session begins (for example a round-begin breakpoint) is not within that session. A decision or no-decision selection is *reported* when the simulator processes it in the order the scheduler produced it, which is the point at which step mode echoes a decision. + +A *placement plan* divides a resource requirement's tasks across a number of hosts. Each *plan position* of a plan has a fixed number of tasks and is filled by exactly one host, and no host fills two positions of the same plan. (In the code a plan position is called a *bucket*.) This feature adds a **decision trace** to the simulator REPL. The operator turns it on at runtime for all jobs or for chosen jobs, at a chosen level of detail. The explanation is printed inline as the replay proceeds and can be recalled per job afterwards. The trace is purely an investigation aid: it exists only in the simulator, so the production scheduler pays no cost for it, and enabling it never changes what the scheduler decides. @@ -81,10 +83,11 @@ This feature adds a **decision trace** to the simulator REPL. The operator turns 5. When the operator issues the trace-disable command, the Simulator REPL shall stop tracing decisions in the scheduling sessions that start after the command and shall keep already-captured traces available for recall. 6. If a trace-enable or trace-disable command is issued while a scheduling session is in progress (for example at a decision breakpoint), the Simulator REPL shall apply the new settings from the next scheduling session, leaving the in-progress session traced according to the settings it started with. 7. When the operator issues the trace-status command, the Simulator REPL shall display the settings in effect for the scheduling session in progress, or, when none is in progress, the settings the next scheduling session will use, and separately any settings that will replace them at the next scheduling session, each showing whether tracing is enabled, the traced job selection, the detail level, the per-decision size bound, the file copy destination if any, the stored-trace memory bound and current stored-trace memory use. -8. The Simulator REPL shall support two detail levels, `summary` and `full`, selectable in the trace-enable command, and shall use `summary` when no level is given. -9. If a trace command has an invalid argument (unknown option, malformed job identifier, invalid level, or a size bound that is not a whole number from 50 to 100000), the Simulator REPL shall print an error naming the invalid argument and leave the current trace settings unchanged. +8. The Simulator REPL shall support two detail levels, `summary` and `full`, selectable in the trace-enable command, and shall use `summary` when no level is given; the two levels differ only in how host verdicts are shown (Requirement 5). +9. If a trace command has an invalid argument (unknown option, malformed job identifier, invalid level, a size bound that is not a whole number from 50 to 100000, or a file path longer than 4095 bytes), the Simulator REPL shall print an error naming the invalid argument and leave the current trace settings unchanged. 10. While tracing is enabled, the Simulator REPL shall keep the trace settings in effect across scheduling sessions until the operator changes or disables them. 11. The Simulator REPL shall list the trace commands and `info trace` in its `help` output, with the trace commands in their own labeled section and `info trace` among the other `info` subcommands. +12. The Simulator REPL shall accept in a trace-enable command any number of job identifiers and a file path of up to 4095 bytes, even where these exceed the argument count or argument length accepted by other REPL commands. ### Requirement 2: Job Selection Explanation **Objective:** As an operator, I want each traced decision to explain why the job was considered when it was, so that I can understand job ordering and why a job was or was not reached earlier. @@ -93,7 +96,7 @@ This feature adds a **decision trace** to the simulator REPL. The operator turns 1. When a traced job is selected for allocation in a scheduling session, the Simulator REPL shall show in its trace the session number (numbered as in the REPL's event summary for that session) and the job's 1-based position among the jobs selected for allocation in that session. 2. When a traced job is selected for allocation, the Simulator REPL shall show the job-ordering policy in use and the job's ordering keys under that policy with the values in effect when the job was selected (for the default first-come-first-served policy: the job's queue, the queue's priority, the job's priority and the job's queue time). 3. When a traced job is selected for allocation, the Simulator REPL shall show the cluster-wide limits matched to the job before placement, and shall state that host-scoped limits are reported with each host's verdict. -4. If a traced job is selected for allocation but no scheduling decision is produced for it, the Simulator REPL shall show a trace stating which of these occurred: the job could not be found by the allocator, the allocation attempt ended in an internal error, or the job was pended without any pend reason; when the job could not be found, the trace shall show the session, position and job identifier and state that the ordering keys and limits are unavailable. +4. If a traced job is selected for allocation but no scheduling decision is produced for it, the Simulator REPL shall show a trace stating which of these occurred: the job could not be found by the allocator; the job has no scheduling object (its resource requirement could not be parsed); the allocation attempt ended in an internal error, including a failure to build the decision; or the job was pended without any pend reason. When the job could not be found, the trace shall show the session, position and job identifier and state that the ordering keys and limits are unavailable; when the job has no scheduling object, the trace shall show the selection section without limits and state that the limits are unavailable. 5. When a traced job is dispatched by the simulator's reconstruction of a recorded allocation, rather than by the scheduler's own placement, the Simulator REPL shall state this in place of the selection, requirement, plan and host sections of the trace. ### Requirement 3: Resource Requirement Selection Explanation @@ -101,10 +104,11 @@ This feature adds a **decision trace** to the simulator REPL. The operator turns #### Acceptance Criteria 1. When the scheduler evaluates a traced job, the Simulator REPL shall show in the trace every resource requirement tried, in evaluation order, each labelled as a single requirement, the i-th member of n of a compound requirement, or the i-th of n alternatives, with the full nesting path shown when requirements are nested. -2. The Simulator REPL shall show for each tried resource requirement its minimum and maximum task counts, the requested resources, and whether the requirement was satisfied. -3. When a member of a compound requirement cannot be satisfied, the Simulator REPL shall show that member as the point of failure and list the remaining members as not tried. +2. The Simulator REPL shall show for each tried single requirement or compound member its minimum and maximum task counts, its requested resources and whether it was satisfied, not satisfied, or ended in an internal error, and for each tried compound requirement or alternative whether it was satisfied, not satisfied, or ended in an internal error. +3. When a member of a compound requirement cannot be satisfied or ends in an internal error, the Simulator REPL shall show that member as the point of failure with that outcome and list the remaining members as not tried. 4. When an alternative requirement is satisfied, the Simulator REPL shall mark it as the chosen alternative and list the remaining alternatives as not tried. 5. When every alternative requirement fails, the Simulator REPL shall show the failure reasons of each alternative, including an internal error, not only those of the last one. +6. When the scheduler skips an alternative without evaluating it because the alternative has no requirement content, the Simulator REPL shall show it as skipped, in its position in the evaluation order. ### Requirement 4: Shared Resource and Placement Plan Explanation **Objective:** As an operator, I want to see the cluster-wide checks and the placement plans tried for each resource requirement, so that I understand how the scheduler decided how many hosts and how many tasks per host to use. @@ -112,34 +116,37 @@ This feature adds a **decision trace** to the simulator REPL. The operator turns #### Acceptance Criteria 1. When the scheduler checks cluster-wide shared resources for a traced job's resource requirement, the Simulator REPL shall show each shared resource checked, other than cluster-wide limits (which are shown as limit checks with the fields of criterion 2 and whether they passed), with its total, the amount in use, the amount provisionally reserved by earlier parts of this job's evaluation, the amount requested, and whether the check passed. 2. When the cluster-wide check fails on a cluster-wide limit, the Simulator REPL shall show the failure as a limit rejection naming the limit, the scope instance it applies to (for example the user, queue or project), the limit value, the current usage, the amount provisionally reserved by earlier parts of this job's evaluation and the amount requested, and shall not also show it as a resource shortfall. -3. When the planner tries a placement plan for a traced job, the Simulator REPL shall show the plan in the order tried, with the number of hosts and the number of tasks assigned to each host group of the plan. -4. The Simulator REPL shall show for each tried plan whether it succeeded or, if it failed, which host group of the plan could not be placed. -5. When a plan succeeds, the Simulator REPL shall mark it as the committed plan and show that no further plans were tried for that resource requirement. +3. When the planner tries a placement plan for a traced job, the Simulator REPL shall show the plan in the order tried, with the number of hosts and the number of tasks assigned to each plan position. +4. The Simulator REPL shall show for each tried plan whether it succeeded or, if it failed, which plan position could not be filled or that the attempt ended in an internal error, in which case the Simulator REPL shall state that no further plans were tried; if a generated plan cannot be tried because of an internal error, the Simulator REPL shall show that plan as not tried with the error and state that no further plans were tried. If a plan position could not be filled because the host scan could not start or stopped on an internal error, the Simulator REPL shall show that position as not filled because of an internal error in the host scan, with the hosts considered before the error shown as Requirement 5 specifies for the current detail level, and shall show the plans tried after that plan. +5. When a plan succeeds and the allocation built from it is completed, the Simulator REPL shall mark it as the committed plan and show that no further plans were tried for that resource requirement; if every position of a plan is filled but building the allocation from it ends in an internal error, the Simulator REPL shall show that plan as ended in an internal error, not as committed, and state that no further plans were tried. 6. If every tried plan for a resource requirement fails, the Simulator REPL shall state that no plan could be placed for that requirement. -7. If no placement plan could be generated for a resource requirement, the Simulator REPL shall state that no plan was generated and the reason. +7. If no placement plan could be generated for a resource requirement, including when the allocator declines the requirement before generating plans, the Simulator REPL shall state that no plan was generated and the reason. +8. When the cluster-wide check for a resource requirement fails, the Simulator REPL shall state that no placement plans were tried for that requirement because of that failure. +9. When the planner generates a plan that the scheduler discards without trying it, the Simulator REPL shall show that plan as discarded, in the order generated; if every generated plan is discarded, the Simulator REPL shall state that no plan was tried because all generated plans were discarded. ### Requirement 5: Host Verdict Explanation **Objective:** As an operator, I want to know for each host considered why it could or could not meet the job's resource requirement, so that I can pinpoint the resource, selection expression, or limit that blocked a placement. #### Acceptance Criteria -1. While the detail level is `full`, when a host is considered for a group of a traced job's plan, the Simulator REPL shall show that host, in the order considered, with its verdict: accepted, or rejected with every reason found by the checks the scheduler performed for that host; checks the scheduler did not perform for that host shall be shown as not evaluated, and the Simulator REPL shall not perform additional checks to find further reasons. -2. While the detail level is `full`, when a host is rejected, the Simulator REPL shall indicate whether the host was rejected by the pre-filter check or by the per-host check, and for a pre-filter rejection shall state that the compared request was carried over from an earlier candidate host. +1. While the detail level is `full`, when a host is considered for a position of a traced job's plan, the Simulator REPL shall show that host, in the order considered, with its verdict: accepted, or rejected with every reason found by the checks the scheduler performed for that host; checks the scheduler did not perform for that host shall be shown as not performed, which is not a verdict kind (Requirement 5.10), and the Simulator REPL shall not perform additional checks to find further reasons. +2. While the detail level is `full`, when a host is rejected with a selection-mismatch, resource-shortfall or limit-exceeded verdict, the Simulator REPL shall indicate whether the host was rejected by the host-selection pre-filter or by the per-host resource and limit check. 3. While the detail level is `full`, when a host does not match the job's host selection expression, the Simulator REPL shall show a selection-mismatch verdict for that host. 4. While the detail level is `full`, when a host is short of one or more resources, the Simulator REPL shall show each resource short on that host with the host's total, the amount already in use, the amount provisionally reserved on that host by earlier parts of this job's evaluation, and the requested amount the check actually compared against. 5. While the detail level is `full`, when a host-scoped limit would be exceeded on a host, the Simulator REPL shall show a limit rejection naming the limit, the scope instance it applies to, the limit value, the current usage, the amount provisionally reserved by earlier parts of this job's evaluation and the requested amount the check actually compared against, and shall not also show that limit as a resource shortfall. -6. While the detail level is `full`, when a host is skipped because an earlier group of the same plan already claimed it, the Simulator REPL shall show the host as already claimed. +6. While the detail level is `full`, when a host is skipped because an earlier position of the same plan already claimed it, the Simulator REPL shall show the host as already claimed. 7. While the detail level is `full`, when a host could not be evaluated because of an internal error, the Simulator REPL shall show the host as not evaluated with the error. -8. When a plan fails after some of its groups were placed, the Simulator REPL shall show that the hosts claimed by the earlier groups were released. -9. While the detail level is `summary`, the Simulator REPL shall show, for each plan group, the hosts accepted and the number of hosts rejected for each verdict kind, counting a host with several rejection reasons once under each reason kind as in Requirement 9.3, instead of one line per host. +8. When a plan fails after some of its positions were filled, the Simulator REPL shall show that the hosts claimed for the earlier positions were released. +9. While the detail level is `summary`, the Simulator REPL shall show, for each plan position, the host accepted (if any) and the number of hosts rejected for each verdict kind, counting a host with several rejection reasons once under each verdict kind it matched as in Requirement 9.3, and, for the limit-exceeded count, the name and scope instance of each limit hit with its count, instead of one line per host. +10. The Simulator REPL shall use these host verdict kinds wherever host verdicts are listed or counted: accepted, selection mismatch, resource shortfall, limit exceeded, already claimed, and not evaluated because of an internal error. ### Requirement 6: Decision Outcome Explanation **Objective:** As an operator, I want each trace to end with the decision's concrete result, so that I can see exactly what was allocated or why the job stays pending. #### Acceptance Criteria -1. When a traced job is dispatched, the Simulator REPL shall show in the trace, for each allocated host, each resource allocated with its amount, and, separately and labelled as limit charges, each host-scoped limit charged on that host with the limit name, its scope instance and the amount charged. +1. When a traced job is dispatched, the Simulator REPL shall show in the trace, for each allocated host, each resource allocated with its amount, and, separately and labelled as limit charges, each host-scoped limit charged on that host with the limit name, its scope instance and the amount charged, subject to the host limit of Requirement 9.4. 2. When a traced job is dispatched, the Simulator REPL shall show once for the job the shared resources allocated with their amounts and, separately and labelled as limit charges, each cluster-wide limit charged with the limit name, its scope instance and the amount charged. 3. The Simulator REPL shall show each limit charge only as a limit charge, never among a host's allocated resources or the job's shared resources. -4. When a traced job is pended, the Simulator REPL shall show the pend reason set of the decision, with the same content as the pend decision the simulator echoes and records (`--sim-output`) and compares (`--sim-verify`), subject to the host-entry limit of Requirement 9.4. +4. When a traced job is pended, the Simulator REPL shall show the pend reason set of the decision, with the same content as the pend decision the simulator records (`--sim-output`) and compares (`--sim-verify`), subject to the host limit of Requirement 9.4. ### Requirement 7: Trace Display and Recall **Objective:** As an operator, I want traces printed inline as the replay runs and recallable afterwards, so that I can read explanations as decisions happen and revisit a job later. @@ -150,12 +157,12 @@ This feature adds a **decision trace** to the simulator REPL. The operator turns 3. While the REPL is in continue mode, when a traced decision is reported or a traced job's selection produces no decision, the Simulator REPL shall at that point end and report any active collapsed range and print the traced decision's echo line followed by its trace (or the no-decision trace), so that a round's traces appear in the order its decisions are reported, with reconstructed dispatches (Requirement 2.5) first, followed by the round's summary line when the round completes. 4. While the REPL is in continue mode, the Simulator REPL shall not include a round that contains a traced decision or traced no-decision selection in a collapsed range. 5. When execution stops at a breakpoint, the Simulator REPL shall have printed the traces of the round's traced decisions and no-decision selections reported up to and including the one at which execution stopped, and no later trace from the current round shall be available to `info trace` at the breakpoint; those printed traces shall be available to `info trace` unless discarded under the memory bound (Requirement 9.5), in which case Requirement 9.6 applies. -6. The Simulator REPL shall present each trace as an indented hierarchy: job selection, then resource requirements, then shared-resource checks and plans, then host groups and hosts, then the outcome. +6. The Simulator REPL shall present each trace as an indented hierarchy: job selection, then resource requirements, then shared-resource checks and plans, then plan positions and hosts, then the outcome. 7. The Simulator REPL shall show resource amounts in traces using the operator's current `set units` display mode and job identifiers using the REPL's existing job-identifier formatting. -8. When the operator issues `info trace `, the Simulator REPL shall print the most recent stored trace for that job, at the detail level at which it was captured, including for a job that has since finished or been removed from the replay. +8. When the operator issues `info trace `, the Simulator REPL shall print the most recent stored trace for that job, at the detail level at which it was captured and with resource amounts shown in the `set units` display mode in effect when `info trace` is issued, including for a job that has since finished or been removed from the replay. 9. When the operator issues `info trace` with a job-array identifier, the Simulator REPL shall print the most recent stored trace of each element of that array that has one, mark each element whose trace was discarded under the memory bound, and, if no element has a stored trace, report that and list the elements' current states. 10. When a new trace is captured for a job that already has a stored trace, the Simulator REPL shall replace the stored trace with the new one. -11. If the operator issues `info trace ` for a job with no stored trace, the Simulator REPL shall report that no trace is available and show the job's current state, or state that the job is unknown. +11. If the operator issues `info trace ` for a job that has no stored trace and none discarded under Requirement 9.5, the Simulator REPL shall report that no trace is available and show the job's current state, or state that the job is unknown. ### Requirement 8: Trace File Copy **Objective:** As an operator, I want an optional copy of trace output written to a file, so that I can search or share long traces after the session. @@ -163,7 +170,7 @@ This feature adds a **decision trace** to the simulator REPL. The operator turns #### Acceptance Criteria 1. Where the trace-enable command names a file, the Simulator REPL shall append to that file every trace printed inline under Requirement 7 (including no-decision traces and reconstructed-dispatch notices), with the same content as shown on the console, and shall not append `info trace` output. 2. If the named file cannot be opened for writing, the Simulator REPL shall print an error naming the file and the cause, and shall leave the current trace settings unchanged. -3. If the named path refers to the same file (same device and inode) as the simulation output file currently in effect (from `--sim-output` or `set output`) or the scheduler's log file, the Simulator REPL shall reject the command with an error and leave the current trace settings unchanged. +3. If the named path refers to the same file (same device and inode) as the simulation output file currently in effect (from `--sim-output` or `set output`) or the scheduler's log file if one is in use, the Simulator REPL shall reject the command with an error and leave the current trace settings unchanged. 4. If the operator issues `set output` naming the same file (same device and inode) as any trace file still open for writing, including one kept open under criterion 5 until its scheduling session ends, the Simulator REPL shall reject the command with an error and leave the simulation output file unchanged. 5. When tracing is disabled or the trace settings are replaced, the Simulator REPL shall keep writing traces of decisions captured under the previous settings to the previous file, and shall close that file once the scheduling session in progress when the command was issued has ended, or immediately if no scheduling session was in progress. 6. When the operator quits the REPL, REPL input ends, or the replay completes, the Simulator REPL shall finish writing every trace already printed inline to the trace file and close it before the simulator exits. @@ -173,10 +180,10 @@ This feature adds a **decision trace** to the simulator REPL. The operator turns **Objective:** As an operator tracing large jobs on large clusters or all jobs of a long replay, I want trace size and memory bounded, so that one decision cannot flood the console and tracing cannot exhaust simulator memory. #### Acceptance Criteria -1. The Simulator REPL shall limit the requirement, plan and host sections of each decision's trace to a maximum number of lines, which the operator can set in the trace-enable command to a whole number from 50 to 100000, with a default of 1000 lines when none is given. -2. When a decision's requirement, plan and host sections would exceed the size bound, the Simulator REPL shall stop capturing lines for them and show a truncation marker stating how many lines were omitted. -3. When a trace is truncated, the Simulator REPL shall still show the job selection section and, in a number of lines that does not depend on the number of requirements, plans or hosts, the exact number of plans tried and the exact decision-wide number of host evaluations for each verdict kind, where a host evaluation is one host checked for one group of one plan and an evaluation with several rejection reasons is counted once under each reason kind. -4. The Simulator REPL shall always show the decision outcome, listing at most 50 host entries followed, when more exist, by exact counts per pend-reason kind (for a pend) or the total amount of each resource allocated across the omitted hosts, excluding limit charges, which are totalled per limit (for a dispatch), and a marker stating how many hosts were omitted. +1. The Simulator REPL shall limit the requirement, cluster-wide check, plan and host sections of each decision's trace to a maximum number of lines, which the operator can set in the trace-enable command to a whole number from 50 to 100000, with a default of 1000 lines when none is given. +2. When a decision's requirement, cluster-wide check, plan and host sections would exceed the size bound, the Simulator REPL shall stop capturing lines for them and show a truncation marker stating how many lines were omitted. +3. When a trace is truncated, the Simulator REPL shall still show the job selection section and, in a number of lines that does not depend on the number of requirements, plans or hosts, the exact decision-wide numbers of plans tried and plans discarded and the exact decision-wide number of host evaluations for each verdict kind, where a host evaluation is one host considered for one position of one plan and an evaluation with several rejection reasons is counted once under each verdict kind it matched (Requirement 5.10). +4. The Simulator REPL shall always show the decision outcome, listing at most 50 hosts, each with all of its entries, and always listing the entries not tied to a host; when more hosts exist, it shall follow the list with exact counts of the omitted hosts' pend-reason entries per pend-reason code (for a pend) or the total amount of each resource allocated across the omitted hosts, excluding limit charges, which are totalled per limit (for a dispatch), and a marker stating how many hosts were omitted. 5. The Simulator REPL shall bound the total memory used by stored traces to 64 MiB and, when the bound is reached, shall discard the oldest stored traces first, never discarding the most recently captured trace. 6. If the operator issues `info trace ` for a job whose stored trace was discarded under the memory bound, the Simulator REPL shall report that the trace was discarded. @@ -188,7 +195,7 @@ This feature adds a **decision trace** to the simulator REPL. The operator turns 2. The xce_scheduler simulator shall produce the same `--sim-verify` results, and the same `--sim-output` recording and REPL `snapshot` content apart from wall-clock metadata fields, whether tracing is enabled or disabled. 3. The xce_scheduler simulator shall produce the same final accounting and job state for a replay whether tracing is enabled or disabled. 4. While tracing is disabled and no scheduling session that started with tracing enabled is still in progress, the Simulator REPL shall produce output identical to the output it produces without this feature, except for the `help` listing, the output of the new trace commands and `info trace` themselves, and collapsed-range boundaries at the first round after the last traced round. -5. The Simulator REPL shall not change the behavior or output of any existing REPL command, except that `help` lists the new commands, `set output` rejects any trace file still open for writing (Requirement 8.4), and, while tracing is enabled, `step` and `continue` output includes traces as defined in Requirement 7. +5. The Simulator REPL shall not change the behavior or output of any existing REPL command, except that `help` lists the new commands, `info` accepts the `trace` subcommand (Requirements 7.8-7.11), `set output` rejects any trace file still open for writing (Requirement 8.4), and, while tracing is enabled or a scheduling session that started with tracing enabled is in progress, `step` and `continue` output includes traces and the collapsed-range changes defined in Requirement 7, with the collapsed-range boundary effect allowed by criterion 4. ### Requirement 11: Simulator-Only Availability **Objective:** As an operator of a production cluster, I want decision tracing to be impossible to enable in the live scheduler, so that production scheduling never pays the cost of trace collection. diff --git a/.kiro/specs/scheduler-sim-decision-trace/research.md b/.kiro/specs/scheduler-sim-decision-trace/research.md new file mode 100644 index 00000000..d8e25109 --- /dev/null +++ b/.kiro/specs/scheduler-sim-decision-trace/research.md @@ -0,0 +1,274 @@ +# Gap Analysis: scheduler-sim-decision-trace + +_Date: 2026-10-07. Codebase: branch `scheduler-framework` @ `33437a39`, after `scheduler-allocator-runner`, `scheduler-sim-discrete-clock` and `scheduler-sim-repl-verbosity` reached implementation-complete. Paths are relative to `src/scheduler/` unless they start with `src/` (repo root) or `.kiro/`._ + +## 1. Summary + +- All upstream dependencies have landed. Plan and bucket hooks have a single home: `sched_allocator_run()` in `src/alloc_runner.c` (490 lines). There is no `alloc_iter.c`; the interface is `include/sched/alloc_iter.h`. +- Almost every fact the trace needs is computed today and then thrown away at a known site (§3). Only two things are not available where the trace needs them: the per-resource `total/inuse/uncommitted/wanted` numbers (they exist only inside `unsatisfy_`) and a "selected but no decision" channel. +- **Three requirements no longer match the code and should be revised before design** (§4): Req 5.2 (pre-filter "carried over" request), Req 8.3 (the scheduler log path is not known to the sim), and the token limits that affect Req 1/8 argument parsing. +- Most of the cost is integration rather than algorithms. About 10 instrumentation sites, a new decision op or side channel, refactoring the REPL formatting helpers, and updates to about 26 white-box tests that `#include` REPL sources. +- Recommended approach: **Option C (hybrid)**. New `sched_trace.c` and `repl_trace.c` modules hold the logic. Hooks in existing files are one line each, behind a NULL-pointer gate. Two small prerequisite refactors come first. + +## 2. Corrections to brief.md + +| Brief says | Code today | +|---|---| +| Plan/bucket hooks go in "`sched_allocator_run()` once it lands" | It has landed: `src/alloc_runner.c:457` (`sched_allocator_run`), `:414` (`sched_alloc_search_`), `:270` (`sched_try_plan_`), `:208` (`sched_claim_bucket_`) | +| Shared-check loop bound bug (`alloc_req.shared.count` vs `unmeet.count`) is out of scope | Already fixed (bug-1893), `alloc_runner.c:65,89` | +| ALTER FIXME at `scheduler.c:427` | `scheduler.c:433` | +| `sim_print_decision` at `sim_loop.c:479`, drain loop at ~1607 | `sim_loop.c:529`, drain loop `:1675-1710` | +| "`core*.c`/`sim*.c` may not do I/O" is enforced by tests | Not enforced. `test_core_isolation.c` and `test_sim_isolation.c` forbid controller/network headers and symbols, and wall-clock calls in core; they do not forbid stdio. Keeping `sched_trace.c` I/O-free is a design rule we choose, not one a test checks. | +| `scheduler.c` is 1382 lines (`src/scheduler/CLAUDE.md`) | 2002 lines. CLAUDE.md is stale. | + +## 3. Requirement-to-asset map + +Tags: **Exists**: the data is available at the seam. **Discarded**: computed, then dropped. **Missing**: not computed anywhere. **Constraint**: an architectural limit. + +### Req 1: Trace control +| Need | Asset | Gap | +|---|---|---| +| `trace` command | `repl_dispatch` strcmp chain `repl.c:408` (unknown fallback `:479`) | Missing. Add a branch that routes to `repl_trace.c`. | +| Help section | `repl_print_help` `repl.c:269`. `test_repl_help.c:569-670` checks section order, with a lower bound only on the last section. | Missing. A trailing "Trace" section is safe. | +| Job/array selection | `parse_jobsel`/`jobsel_t` (`src/include/utils.h:231-278`), `array_jobid`/`array_idx` (`src/lib/utils.c:660/671`) | Exists | +| Settings take effect at the next session (1.5/1.6) | `core->session_active` (`include/core.h:260`), set/cleared in `core_session.c:420/479` | Exists. Caveat: at a BEGIN_ROUND breakpoint `session_count` has already been incremented but `session_active==0`. Design must define "next session" there. | +| Pass settings to the allocator thread with no new sync | Window between `xce_core_begin_session` and `xce_core_signal_go` (`sim_loop.c:1621-1673`). The allocator only reads core state after the mutex chain wbuf → picker → jbuf. | Exists. Snapshot settings into a core field in that window. | +| Argument parsing | `repl_split_tokens` (`repl.c:116`): at most **16 tokens of 63 characters** | **Constraint**. Paths longer than 63 characters and long jobid lists do not fit (see §4). | +| `trace status` memory use | `xce_sched_repl_state_s` (`repl_internal.h:171-304`) | Missing. Needs a new state block. | + +### Req 2: Job selection +| Need | Asset | Gap | +|---|---|---| +| Session number | `rs->session_count` (`repl_internal.h:178`) is the current session during drain | Exists (render side) | +| 1-based position in session | None. jbuf carries a bare `uint64_t` (`core.c:167`, pinned by `test_core_jbuf_capacity_pin.c:227`) | Missing. Count non-zero, non-sentinel pops in `sched_allocator` (`sched_election.c:197-231`), reset on 0. jbuf is unchanged. | +| Policy and ordering keys | `job->queue->{name,priority}`, `job->priority`, `job->queue_time` (`include/sched/types.h:101-146`); cmp functions `builtin_fcfs.c:183,206` | Exists. Read on the allocator thread after `sched_get_job`. Sim only ever runs `builtin_fcfs` (`sim_picker.c:56-70`). | +| Cluster-wide limits matched | `obj->glimits` filled by `limit_job_election` (`plugins/limit.c:318`) from `plugin_job_election` (`sched_election.c:246`). It is cached per `configver` (`limit.c:336`). | Exists. Read `glimits` after election; do not instrument the match loop. | +| No-decision cases (2.4) | `sched_election.c:226-231` (not found), `:250-254` (`alloc_rc<0`, silent), `:265-269` (pend with NULL reason), `:286` (calloc failure) | **Missing channel**. Nothing reaches the drain loop. See §5 Decision D2. | +| Reconstructed dispatch (2.5) | `sim_reconcile_unaccounted_jobs` (`sim_loop.c:1026`, block `:1309-1329`), runs before `signal_go` | Exists on the render side. The decision does not mark it as reconstructed, but the call site knows. | + +### Req 3: Resource requirement selection +| Need | Asset | Gap | +|---|---|---| +| Nesting path | Fixed shape: PRIMARY, COMPOUND (list of primaries), ALTER (list of compounds) (`scheduler.c:306/403/409`, bug-1769) | Exists. Path depth is at most 3. | +| Compound member index and point of failure | `sched_make_compound_alloc_` loop `scheduler.c:339-356`; breaks on first `rc<=0` | **Discarded** (index not kept) | +| Per-alternative outcome and reasons (3.5) | ALTER loop `:409-436`. Each alternative frees and replaces `obj->reason` (`:394-395`). NULL `resreq` is skipped silently (`:423`). | **Discarded**. The trace must snapshot (deep-copy) each alternative's reasons before they are freed. It must not change `obj->reason` itself; that defect stays out of scope. | +| min/max tasks and requested resources | `xce_primary_resreq_t{minTasks,maxTasks,select,rusages}`; staged amounts in `ctx->alloc_req.{shared,host}` | Exists. Note: `minTasks` is never used by allocation, so the trace shows it but nothing depends on it. | +| `accepts()` rejection | `alloc_runner.c:474-476` returns 0 with **no reason**. In PRIMARY the stale `obj->reason` survives (`scheduler.c:324`). | Discarded. The trace should say "allocator did not accept resreq". The stale-reason behaviour is a candidate bug to log separately. | + +### Req 4: Shared checks and plans +| Need | Asset | Gap | +|---|---|---| +| Shared check per resource | `sched_alloc_prepare_shared_` (`alloc_runner.c:68-111`) → `sched_check_shared_satisfy` (`:82`) | total/inuse/uncommitted/wanted exist only inside `unsatisfy_` (`resource.c:147-189`, TRACE at `:159,179`). `unmeet` keeps only the shortfall. **Discarded.** Needs a recorder hook inside `unsatisfy_`. | +| Cluster-wide limit vs resource | A limit charge is a vres in `.shared` (`sched_fill_limit_requests`, `resource.c:1062-1099`), with flag `XCE_RESF_LIMIT` (`src/lib/xcelimit.c:526`) and name `limit=[/u=..][/q=..][/p=..]`. Structured `xce_vres_t{vres,res,limit}` (`src/include/xce/types.h:831-844`). | Exists. Classify by `XCE_RESF_LIMIT`, and map idx to `xce_vres_t` through `obj->glimits` / `obj->hlimits[w->idx]`. Pend reasons still say `NO_RESOURCE` (unchanged, out of scope). | +| Plans in order, shape, outcome | `sched_alloc_search_` `:414-455`; `plan->bucket_slots[]`; `sched_try_plan_` knows the failing bucket `b` (`:283`) | **Discarded** | +| No plan generated (4.7) | `plan_init` failure `:427-429` (no reason); malformed plan skipped `:432` | Discarded | +| "Host groups" | Buckets are slot partitions only (seqjob: `[1]`; parjob k=1..maxTasks balanced, `parjob_allocator.c:245-288`) | Terminology: "host group" in the requirements means "bucket". | + +### Req 5: Host verdicts +| Need | Asset | Gap | +|---|---|---| +| Select mismatch | `host_iter.c:136-140`, silent `continue` | **Discarded**. Hook in the iterator. | +| Pre-filter vs per-host | Pre-filter is **select-only**. The runner clears `.host` before each `next()` (`alloc_runner.c:216,253`), and `sched_check_host_satisfy` returns 1 on an empty request (`resource.c:419`). Its satisfy branch (`host_iter.c:159-165`) is effectively dead under the runner. | **Req 5.2 is stale** (§4) | +| Per-resource shortfall numbers | `sched_try_alloc_on_host_` `resource.c:530-608` → satisfy `:575` → `unsatisfy_` | Discarded (same recorder hook as Req 4) | +| Host-scoped limit | `obj->hlimits[w->idx]` from `limit_prepare_allocation` (`limit.c:562-607`) | Exists (classify by flag) | +| Already claimed (5.6) | `sched_alloc_is_claimed_` `alloc_runner.c:229`, silent `continue` | Discarded | +| Internal error (5.7) | `sched_try_alloc_on_host_` `<0` (`resource.c:577-579`, uncommitted fetch `:559`); iterator uncommitted error `host_iter.c:149-156`. Runner merges `<0` with rejection (`alloc_runner.c:236`). | Discarded. Visible at the call site. | +| Unreachable `PREASON_MATCH_FAIL` | `resource.c:543-557` | Note only. Select mismatches never reach it. | +| Rollback (5.8) | `sched_rollback_claims_` `alloc_runner.c:187-196` | Discarded | + +### Req 6: Outcome +| Need | Asset | Gap | +|---|---|---| +| Per-host allocs, shared allocs | `sched_finalize_job_alloc_` `alloc_runner.c:347-406`; `job->allocs` moved into the decision (`sched_election.c:280`) | Exists. Limit charges are mixed in (host vres in host allocs, global vres in `allocs_shared`). Separable by `XCE_RESF_LIMIT`, but at render time the resource index→flag table is needed. | +| Pend reason set | `make_pend_decision` stores a **pointer** to `job->schedobj->reason` (`decision.c:6-19`) | Constraint. A stored trace for `info trace` must hold its own copy, rendered text or a deep copy, because `schedobj->reason` is replaced in later sessions. | + +### Req 7: Display and recall +| Need | Asset | Gap | +|---|---|---| +| Inline after echo (step) | Drain loop order: verify → record → `sim_print_decision_gate` (`:1705`) → `xce_repl_check_decision` (`:1706`) → `free` (`:1709`). Reconstructed: `:1309-1329`. | Exists. Insert render between gate and breakpoint check. This gives 7.5 for free (stop after the trace prints). | +| Continue mode: no collapse for traced rounds | Rollup `repl_rollup.c:285` compares dispatch/preempt counts plus the pend-jobid set from `core->jobs`; flush at `:233` | Missing. Needs a "round had traced output" flag that forces a flush, and must print the echo line that continue mode normally suppresses (`sim_loop.c:587`). | +| Units / jobid formatting | `translate_units_mode`, `repl_format_numeric_resource_value`, `repl_format_consumable_value_and_used`, `repl_current_units_mode` are **static in `repl_info.c`** (1893 lines, `:385-487`). `repl_fmt_jobid_range` (`repl_fmt.c:223`), `fmt_jobid`. | **Prerequisite refactor**: move these to `repl_fmt.c` | +| `info trace` | `repl_cmd_info` `repl_info.c:1837` | Missing. Add a one-line route to `repl_trace.c`. | +| Job state for "no trace" (7.11) | `sched_get_job` / existing `info job` helpers | Exists | + +### Req 8: File copy +| Need | Asset | Gap | +|---|---|---| +| dev/inode vs sim output | `sim->record_file` (FILE*, `fstat`able); `rs->output_path` only set by `set output` | Exists (use `fstat(fileno())`) | +| dev/inode vs scheduler log | Log path is a local in `main.c:228/280`, passed only to `mylog_init` | **Missing**. Needs a new field threaded into the sim args, or drop it from the requirement (§4). | +| `set output` rejects open trace file (8.4) | `repl_params.c:241/197` | Missing. Small check. | +| Close at quit/EOF/complete | `repl_cmd_quit` (`repl.c:330`, `exit(0)`), `xce_repl_on_replay_complete` (`repl.c:609`), batch close `sim_loop.c:1785-1798`. `repl_state` is never freed. | Missing. Add trace-file close in quit and complete. | + +### Req 9: Bounds +Missing. Pure new logic in `sched_trace.c` / `repl_trace.c`. Note: the reason lists accumulate **across plans without dedup** (`add_*_preason`), so a parjob pend reason can be very large. Req 9.4's cap of 50 host entries is needed in practice, not just in theory. + +### Req 10/11: Non-interference and sim-only +| Need | Asset | Gap | +|---|---|---| +| One-branch gate in live | Precedent: NULL-checked core callbacks `unsatisfiable_*_cb`, `cache_clear_cb` (`include/core.h:247,258-259`) | Exists. A `core->trace` pointer, NULL in live and in non-interactive sim. | +| Unsatisfiable cache in sim | `unsatisfiable_lookup_cb` NULL in sim (`scheduler.c:254`) | Exists. A trace never explains a cache hit. | +| Clock unaffected | `xce_core_now` with `speed_num=0` in sim (`sim.c:332`, `core_clock.c:80-121`) | Exists | +| Equivalence testing | `test_repl_equivalence.c` pattern; `--sim-verify`; `scheduler-driver-equivalence.md` | Exists. Add trace-on vs trace-off comparisons. | +| Must not touch `--sim-verify` positional compare | `sim_verify` `di++` per decision | **Constraint** for D2: a trace-only decision must be skipped by verify, record, gate and the REPL tally. | + +## 4. Requirements that need revision (recommended before approval) + +1. **Req 5.2** assumes the pre-filter compares "a request carried over from an earlier candidate host". Since `scheduler-allocator-runner` (Req 3.6, `alloc_runner.c:137-138,253`) the pre-filter is select-only. Suggested rewording: "...indicate whether the host was rejected by the selection pre-filter or by the per-host resource and limit check." Delete the carried-over clause. +2. **Req 8.3** requires a same-inode check against the scheduler log file, but the sim never sees that path. Either keep the requirement and accept a small `main.c` → sim-args plumbing change, or narrow it to the simulation output file. Keeping it is cheap; flag it as a decision for the user. +3. **Req 1 / Req 8 token limits.** `repl_split_tokens` caps arguments at 16 tokens of 63 characters. Long file paths and long jobid lists (`trace on job 1 2 3 … file /long/path`) will fail. Decide between: (a) raise the limits in the tokenizer (affects every command and its tests), (b) a trace-local re-parse of the raw line, or (c) document the limit in the requirement. Option (b) is the least invasive. +4. **Req 3.2** "minimum task count": it can be shown, but allocation never uses `minTasks`. No change needed, but the design should not suggest it influences plans. +5. Terminology: requirements say "host group"; the code says "bucket" (a slot partition). The design should state the mapping once. + +_Applied 2026-10-07 (user-approved):_ item 1 (Req 5.2 rewritten), item 2 (Req 8.3 kept, "if one is in use" added; the log path must be passed through to the sim), item 3 (new Req 1.12: any number of job identifiers and paths up to 4095 bytes; REPL input is libedit `readline()` with no line limit, so `trace` can re-parse the raw line), item 5 ("host group" replaced by a defined *plan position* = one bucket = exactly one host). Review fixes in the same pass: Req 1.6 defines "in progress" (BEGIN_ROUND breakpoint, before the session starts selecting jobs, counts as the next session); new Req 3.6 (skipped member or alternative); Req 4.7 covers an allocator declining a requirement; Req 6.4 no longer claims the echo line shows pend reasons; Req 7.8 renders `info trace` in the current `set units` mode (implies structured storage, D3). + +_Review loop 2026-10-07 (7 passes; passes 2-7 by independent reviewers checking against code; pass 7 clean)._ Further changes: glossary in the Introduction defines *in progress* / *ended* / *reported*; Req 2.4 adds the no-scheduling-object case (bug-1897 path, `sched_election.c:236`) and decision-build failure; Req 3.2/3.3 internal-error outcomes; Req 3.6 limited to alternatives (compound members are never skipped, `scheduler.c:339-356` vs `:423`); Req 4.4 covers internal errors when trying a plan, when a plan cannot be tried, and when the host scan fails (a scan failure leaves the position unfilled and later plans are still tried, `alloc_runner.c:222-226`); Req 4.5 counts a plan as committed only after its allocation is built; Req 4.8 (cluster-wide check failed, so no plans tried); Req 4.9 (discarded malformed plans); Req 5.10 lists the verdict kinds ("not performed" checks are not a verdict); Req 5.9 names the limits hit in summary mode; Req 6.1/9.4 host cap semantics; Req 7.11 vs 9.6; Req 10.5 exceptions aligned with 10.4 and 1.6. + +## 5. Implementation approach options + +### Option A: Extend existing files in place +Put hooks with inline formatting into `alloc_runner.c`, `host_iter.c`, `resource.c` and `scheduler.c`, and put rendering in `sim_loop.c` and `repl_info.c`. +- ✅ Fewest new files. +- ❌ `sim_loop.c` (1819), `repl_info.c` (1893), `scheduler.c` (2002) and `driver.c` (1104) are already over the 800-line cap. This breaks the project convention and the brief's constraint. +- ❌ It mixes rendering into hot-path files and makes the "one branch when off" property hard to audit. +- **Not viable.** + +### Option B: New modules only +`include/sched/trace.h` + `src/sched_trace.c` (record model, buffer, bounds, I/O-free) and `src/repl_trace.c` (commands, settings, rendering, store, file). Existing files get only one-line gated hook calls. +- ✅ Clean boundary, easy to unit-test the buffer and renderer in isolation, hook cost is easy to audit. +- ❌ Needs ~10 hook sites with a stable argument contract, and the renderer needs formatting helpers that are currently `static` in `repl_info.c`. + +### Option C: Hybrid, Option B plus enabling refactors (recommended) +Option B, plus these preparatory steps, each separately testable: +1. Move the units and value formatting helpers from `repl_info.c` to `repl_fmt.c`, with no behaviour change. +2. Add a recorder seam in `unsatisfy_` (or a sibling that returns `total/inuse/uncommitted/wanted` per unmet idx) used only when tracing. +3. Add the no-decision channel (D2) and a "reconstructed" marker at the reconcile call site. +4. Register `sched_trace.c` in `sched_core_isolation_lib` and `repl_trace.c` in the `test_sim_isolation.c` scans and library. Add `#include "../src/repl_trace.c"` to the ~26 white-box tests. +- ✅ Keeps every new concern in a file under the cap, makes prerequisites reviewable on their own, and keeps the live path to a NULL check. +- ❌ More tasks and a larger diff on test files. + +### Key design decisions to carry forward +- **D1: Where the buffer lives.** Choices: (a) `core->trace` (settings snapshot plus current buffer), NULL when off, with one allocator thread per core; or (b) a pointer on the stack-allocated `xce_sched_ctx_t`. (a) reaches `sched_election.c` and `host_iter.c` without changing signatures, and matches the `unsatisfiable_*_cb` precedent. (b) has narrower scope, but ctx does not exist at the election and no-decision sites. **Lean: (a)**, with settings snapshotted between `begin_session` and `signal_go`. +- **D2: No-decision trace channel.** Choices: (a) push a decision with a new op such as `XCE_DECISION_TRACE` and payload `trace`, which every consumer (verify, record, gate, REPL tally, live driver) skips; (b) a side queue drained in the decision loop, which loses positional order relative to decisions. Req 7.2/7.3 need ordering, so (a) is effectively required. Append the op **after** `PREEMPT` so existing values are not renumbered, and coordinate with `scheduler-preemption`, whose draft already disagrees with `decision.h` on PREEMPT placement. +- **D3: Captured representation.** Store structured records (with copied numbers, names and reason snapshots) and render on the sim side, or render to text on the allocator thread. The brief's constraint (no formatting on the allocator thread) favours structured records. For `info trace` storage, rendered text is simplest for the 64 MiB accounting, but it must be re-rendered if units change. Req 7.8 says "at the level captured", not at the units captured, so this is open. +- **D4: Per-resource numbers.** Instrument `unsatisfy_` with a NULL-gated record callback rather than widening `xvec_idval_t unmeet`. Widening would touch every caller of the satisfy API. +- **D5: Limit classification.** Use `XCE_RESF_LIMIT` plus `xce_vres_t` lookup through `obj->glimits` / `obj->hlimits[w->idx]`, captured on the allocator thread. These lists are rebuilt only when the configuration version changes, so they are stable within a session. +- **D6: ALTER per-alternative reasons.** Deep-copy `ctx.reason` into the trace before `scheduler.c:394-395` frees it. Do not change `obj->reason` semantics (out of scope; separate bug). + +## 6. Effort and risk + +- **Effort: L (1–2 weeks).** About 10 hook sites across 6 hot-path files, a new decision op through 3 consumers, two new modules (model/buffer/bounds and REPL/render/store/file), prerequisite refactors, about 26 test-include updates, and equivalence tests. All of it follows existing patterns. +- **Risk: Medium.** No new technology, and the gating precedent exists. The risks are concentrated in: (1) a new decision op leaking into `--sim-verify`, recording or live driver consumers and breaking determinism; (2) hook arguments being evaluated on the live path (the one-branch rule needs macro discipline); (3) continue-mode rollup and breakpoint ordering interactions. + +## 7. Research needed for the design phase + +1. Whether a decision-pointer `free` in three places (`driver.c:1093`, `sim_loop.c:1709,1327`) is enough to own a trace buffer, or whether the buffer should be pooled per session to avoid allocator-thread malloc churn. +2. A hook macro shape (`SCHED_TRACE(core, kind, ...)`) that does not evaluate arguments when `core->trace == NULL`, and a test that proves it, for example a counting side-effect argument. +3. A resource idx → `XCE_RESF_LIMIT` lookup available at render time for the outcome section, or captured at finalize. +4. Ordering-key drift between pick and pop (an `on_job_update` landing in between). It is believed benign in sim because events are pushed before the session starts; confirm. +5. Exact interaction with the BEGIN_ROUND breakpoint: `session_count` has already incremented, `session_active==0`, so which session counts as "next"? +6. Line-count budget for `repl_trace.c`: commands, settings, store, file and renderer will likely exceed 800 lines, so plan a split such as `repl_trace.c` (commands/settings/file/store) and `repl_trace_render.c`. +7. Candidate bugs to log separately (not fixed by this spec): stale `obj->reason` after an `accepts()` rejection (`scheduler.c:324`, `alloc_runner.c:474`); a plan `init` failure pending with no reason (`alloc_runner.c:427`); pend reasons duplicated across parjob plans; `PREASON_MATCH_FAIL` branch unreachable (`resource.c:543`); the unsatisfiable cache populating `main_reason`, which the runner never sets (`scheduler.c:282-289`); `src/scheduler/CLAUDE.md` line counts stale. + +--- + +# Design Phase Research (2026-10-07) + +## Summary +- **Feature**: `scheduler-sim-decision-trace` +- **Discovery Scope**: Extension. Light discovery, integration-focused, on top of the gap analysis above. No new external dependencies. +- **Key Findings**: + 1. Every allocator-side fact the trace needs can be read **on the allocator thread** at the point it is computed. Reading the same data later on the sim thread would race. The sim drain loop runs concurrently with the allocator working on the next job, and `obj->hlimits` / `worker->resvals` are mutated by `limit_prepare_allocation` during allocation. So the outcome (allocation and pend reasons) is also captured on the allocator thread, before the decision is pushed. + 2. `unsatisfy_` (`resource.c:147`) is the single place where `total/inuse/uncommitted/wanted` exist for both the cluster-wide check and the per-host check. One recorder hook there serves Req 4.1, 4.2, 5.4 and 5.5. Limit rows are told apart by the resource's `XCE_RESF_LIMIT` flag (`xce_resource_t.flags`, via `vget(core->resmgr, idx)`). The limit name and scope come from the `xce_vres_t` found in `obj->glimits` or `obj->hlimits[w->idx]` whose `vres->idx` matches. + 3. `repl_dispatch(sim, line)` (`repl.c:408`) receives the raw input line, and input comes from libedit `readline()` with no length limit. `trace` can therefore re-parse the raw line itself (Req 1.12) without touching `repl_split_tokens`. + 4. The continue-mode rollup (`repl_round_rollup_process`, `repl_rollup.c:285`) is called once per round from `xce_repl_post_session_check`. A per-round "traced" flag checked at its top is enough for Req 7.3/7.4: flush, emit this round alone, and do not start a range. + 5. `main.c` always has a log path (default `/tmp/xce_scheduler.log`, `main.c:228`), so Req 8.3's log-file check always applies once the path is plumbed through `sched_sim_args_t`. + 6. White-box REPL tests `#include` REPL/sim sources (`test_repl_equivalence.c:132-155`) and link `scheduler_test_deps`. Any new file that `repl.c`/`sim_loop.c` calls must be added to those include lists (about 26 tests) or to a library they link. `sched_sim_isolation_lib` (`meson.build:163`) must gain the new `repl_trace*.c` files. + +## Research Log + +### Thread ownership of trace data +- **Context**: Req 10 (no interference) and the "no new cross-thread synchronization" constraint. +- **Findings**: + - The allocator thread runs `sched_allocator` (`sched_election.c:162`). The sim thread runs `sched_sim_driver_run` (`sim_loop.c`). + - The sim thread writes core state only between `xce_core_begin_session` and `xce_core_signal_go`, and after the drain loop has seen the session-end NULL. The allocator is idle (blocked on `jbuf`) in both windows. + - Values handed sim → allocator are ordered by the wbuf/jbuf cbuf mutexes. Values handed allocator → sim travel through the `core->decisions` cbuf. +- **Implications**: The settings snapshot (`core->trace_ctl`) is written only in those windows. The record under construction (`core->trace_rec`) is touched only by the allocator thread. A finished record changes owner by riding on the decision. No locks are needed. + +### Decision transport for no-decision selections +- **Context**: Req 2.4, 7.2, 7.3 require the trace to appear in the decision order. +- **Findings**: The drain loop ends on a NULL decision. `--sim-verify` compares positionally (`di++`, `sim_loop.c:1690`). Recording and the breakpoint tally process every decision. +- **Implications**: Append a new op `XCE_DECISION_TRACE` after `XCE_DECISION_PREEMPT` (no renumbering). The drain loop handles it first and `continue`s before verify, record, gate and breakpoint run. `driver.c` frees and skips it defensively; live never produces one. + +### Shortfall numbers +- **Context**: Req 4.1, 4.2, 5.4, 5.5. +- **Findings**: `unsatisfy_(core, nres, idxs, total, inuse, uncomitted, wanted, ...)` is reached from `sched_check_shared_satisfy_` (`resource.c:~290`) and from the host satisfy path (`resource.c:~384`). Only TRACE logging uses the values today. +- **Implications**: One hook at the top of `unsatisfy_` passes all arrays to the current record. The record keeps every row inside a cluster-wide-check scope (4.1 shows passing rows) and only unmet rows inside a host scope (5.4). + +## Architecture Pattern Evaluation + +| Option | Description | Strengths | Risks / Limitations | Notes | +|--------|-------------|-----------|---------------------|-------| +| Per-decision record + render in sim driver (selected; brief's choice) | Allocator appends typed entries to an arena-backed record; record rides on the decision; REPL renders, stores and copies to file | No I/O on core thread; per-job recall; deterministic ordering | Needs a new decision op and allocator-thread outcome capture | Matches the existing `unsatisfiable_*_cb` NULL-gate precedent | +| Streaming callback | Hooks call a driver callback per event | No buffer | I/O on allocator thread; picker/allocator output interleaving; no recall | Rejected in brief | +| Render text on allocator thread | Hooks format strings | Simple store | Formatting cost on allocator; units fixed at capture (violates 7.8) | Rejected | + +## Design Decisions + +### Decision: Two gate pointers on the core +- **Context**: Req 11.2 allows at most one branch per hook when off. +- **Alternatives Considered**: + 1. A single `trace_ctl` pointer, with every hook reading `ctl->rec`: two loads. + 2. `trace_ctl` (session settings) plus `trace_rec` (current record): hooks test only `trace_rec`. +- **Selected Approach**: Option 2. `SCHED_TRACE(core, fn(core->trace_rec, ...))` evaluates nothing unless `core->trace_rec != NULL`. Only the allocator's per-job prologue tests `trace_ctl`. +- **Rationale**: One predictable branch per hook. The live core never sets either pointer. +- **Trade-offs**: Two fields on `xce_sched_core_t` instead of one. + +### Decision: Entry budget equals line budget +- **Context**: Req 9.1–9.3 bound lines, but the record is structured. +- **Selected Approach**: Each entry in the requirement, check, plan and host sections renders as exactly one line. The capture-side budget counts entries, and the renderer contract guarantees one line per entry. Counters (plans tried and discarded, verdict counts, limit tallies) keep updating after the budget is exhausted. +- **Rationale**: Truncation happens at capture, so memory is bounded too (9.5), and the omitted-line count is exact. + +### Decision: Outcome captured on the allocator thread, capped at capture +- **Context**: Req 6, 9.4; the stored trace must outlive `job->allocs` and `schedobj->reason` (7.8). +- **Selected Approach**: After the decision is built, the allocator copies the first 50 hosts (allocation rows, or pend-reason entries) plus all host-less entries, and aggregates the rest (per-code counts, or per-resource and per-limit totals). Reconstructed dispatches (Req 2.5) are captured the same way on the sim thread, inside the pre-`signal_go` window. +- **Rationale**: This avoids racing the allocator and gives a bounded record size. + +### Decision: Structured store, render on demand +- **Context**: Req 7.7 / 7.8 (units mode at render time), 8.1 (file copy has the same content as the console). +- **Selected Approach**: Records hold raw floats plus the resource name and flags. The renderer formats with the current `set units` mode. The inline print renders once to the console and the file (same bytes); `info trace` renders again later. + +### Decision: Settings lifecycle pending → active → retired +- **Context**: Req 1.4–1.7, 8.5. +- **Selected Approach**: REPL commands write *pending*. The session-begin hook promotes pending to *active* and publishes `core->trace_ctl`. A replaced or disabled active setting whose file is still open becomes *retired* until the session-end hook closes it. + +### Generalization / Build-vs-Adopt / Simplification +- **Generalization**: One recorder path handles the cluster-wide and per-host checks (scope decides which rows to keep). One renderer serves inline, file and `info trace`. +- **Build vs adopt**: No suitable library exists (structured in-process trace, C, sim-only). We reuse libxce `htab` for the jobid→trace index, `parse_jobsel`/`array_jobid` for selection, and the `repl_fmt` helpers for units. +- **Simplification**: No callback indirection. The sim driver is the only consumer, so hooks call `sched_trace_*` functions directly behind the NULL gate. No per-session pooling (records are arena-allocated per traced job; untraced jobs allocate nothing). + +## Risks & Mitigations +- New decision op leaks into verify, recording or the live driver → handle the op first in each drain loop; equivalence tests with trace on vs off (`--sim-verify` and `--sim-output` byte compare). +- Hook arguments evaluated in live → statement macro that evaluates its call only inside `if (core->trace_rec != NULL)`; a unit test with a side-effect argument proves it. +- Trace changes scheduling via shared state → capture functions take `const` inputs and never call mutating scheduler APIs; the trace-on/off equivalence tests use a resource-contention fixture (per `scheduler-driver-equivalence.md`). +- ~26 white-box test include lists → update them in one task right after the new files exist; the build fails loudly if one is missed. +- File-size cap → three REPL-side files plus two core-side files, each planned under 800 lines. + +## References +- `.kiro/steering/scheduler-driver-equivalence.md`: state-equivalence invariant and contention-fixture rule. +- `.claude/skills/xce-audit-invariants/SKILL.md`: scheduler audit dimensions 1–4. +- `.kiro/specs/scheduler-allocator-runner/`: runner contract (`alloc_iter.h`). + +## Design Review Gate (2026-10-07) +- Pass 1 (self): settings lifecycle reduced to reference-counted `next`/`cur` (no separate "retired" state); `pos_scan_error` API added; scope-begin calls take `obj`/`w`; per-host 0 with no rows means selection mismatch (`resource.c` MATCH_FAIL branch); `sim_print_decision` exposed for the continue-mode echo. +- Pass 2 (independent, checked against the code): the link impact on about 20 white-box tests that include hooked sources (D1); `search_end` entry outside the budget plus a `NONE_GENERATED` reason (D2); `round_traced` reset at session begin (D3); `info trace` job-state read placed in the same class as `info job` (D4); resource pre-filter invariant added as a revalidation trigger (D5); host-scope `h=` and name-based limit resolution for reconstructed records (D6); rollback hook only when n > 0 in `sched_try_plan_` (D7); path/no-plan/pos-end enums and bound constants defined (D8); `utils.h` dependency and the isolation-scan target named (D9); reconstructed-block call order spelled out (D10). Threading, drain-loop order, breakpoint ordering and the Req 10/11.2 claims were confirmed. + +## Design Review Loop (2026-10-07, after design generation) +Eight independent review passes, each checked against the code; pass 8 found no issues. +- Pass 1 (1 major, 7 minor): compound-container hooks and merge-failure marking; `search_end` counted in the budget, with decision-wide counters; `round_traced` set in every mode; limit units via the limited resource; missing API declarations; `repl.h` externs; `d->trace` ownership; anchor fix. +- Pass 2 (1 high, 1 medium, 2 low): path push/pop with balanced begin/end (non-scope `member_failed` / `alt_result`); schedobj passed for limit-charge resolution; rollup rule after the pend query; session-begin hook after the `rc < 0` check. +- Pass 3 (2 medium, 4 low): no-decision internal-error records render the full hierarchy; renderer example made line-exact; rollup idle formula; `NOT_FOUND` has no selection; `trace_rec` cleared on every hand-off; name-based limit-charge matching; entry-type list and traceability corrected. +- Pass 4 (1 medium, 2 low): failure-reasons line depth; `nmembers` on `resreq_begin`; "unavailable" lines for 2.4. +- Pass 5 (1 medium, 2 low): error-path tests use the real allocator; `search_end`/`rollback` take no data the runner lacks; Modified Files rollup line aligned. +- Pass 6 (2 medium, 2 low): source-include test with macro and link-wrap seams; 4.9 condition `d == g`; `commit` takes no count; search-end line defined for every rc. +- Pass 7 (1 low): `xvec_dup` wrap attributed to the right test. +- Pass 8: NO ISSUES (one citation refinement applied). diff --git a/.kiro/specs/scheduler-sim-decision-trace/spec.json b/.kiro/specs/scheduler-sim-decision-trace/spec.json index 7151bfb0..306848e4 100644 --- a/.kiro/specs/scheduler-sim-decision-trace/spec.json +++ b/.kiro/specs/scheduler-sim-decision-trace/spec.json @@ -1,24 +1,24 @@ { "feature_name": "scheduler-sim-decision-trace", "created_at": "2026-09-29T04:19:32Z", - "updated_at": "2026-09-29T04:50:20Z", + "updated_at": "2026-10-08T06:43:22Z", "language": "en", - "phase": "requirements-generated", + "phase": "tasks-generated", "approvals": { "requirements": { "generated": true, - "approved": false + "approved": true }, "design": { - "generated": false, - "approved": false + "generated": true, + "approved": true }, "tasks": { - "generated": false, - "approved": false + "generated": true, + "approved": true } }, - "ready_for_implementation": false, + "ready_for_implementation": true, "gitea": { "milestone_id": 68, "issues": { diff --git a/.kiro/specs/scheduler-sim-decision-trace/tasks.md b/.kiro/specs/scheduler-sim-decision-trace/tasks.md new file mode 100644 index 00000000..7657d6a3 --- /dev/null +++ b/.kiro/specs/scheduler-sim-decision-trace/tasks.md @@ -0,0 +1,336 @@ +# Implementation Plan + +> Notes for every task below: +> - Every new test file is registered in the scheduler test build with the source set it needs, and runs in the scheduler suite. Registration only appends entries, so parallel tasks that each register a test do not conflict. +> - Every new trace source file is added to the scheduler's main source list by the task that creates it. +> - "Under ASAN" means the standard debug + ASAN scheduler build. + +- [ ] 1. Foundation: baselines, shared formatting, trace model and transport +- [ ] 1.1 Capture the trace-off REPL transcript baseline before any feature change + - Build a scripted REPL session over a resource-contention replay fixture. The fixture has several jobs competing for a shared license, a cluster-wide limit and a host-scoped limit, plus a parallel job and an alternative-requirement job. + - The session exercises step, continue with collapsed ranges, a decision breakpoint, a round-begin breakpoint, info subcommands, set units and set output. + - Store the captured console transcript, the `--sim-output` recording and the `--sim-verify` result as committed test fixtures. Add a test that replays the script and compares byte for byte, ignoring only wall-clock metadata fields. + - Produce the baseline on the tree before any trace code exists. Later tasks use it to prove that output is unchanged with tracing off. + - Done when the new baseline test passes on the unmodified tree under ASAN and the fixtures are committed. + - _Requirements: 10.4, 10.5_ + +- [ ] 1.2 Move the units and value formatting helpers into the shared REPL formatting module + - Move these helpers out of the info module so a new renderer can reuse them, with no change in behaviour: + - units-mode translation; + - numeric-resource and consumable value formatting; + - the current-units lookup. + - Done when every existing REPL test and the 1.1 baseline test pass unchanged under ASAN, and the helpers can be called from other REPL files. + - _Requirements: 7.7_ + +- [ ] 1.3 Build the trace record model foundation, test first + - Define these types: record, settings snapshot, verdict kinds, record kinds, path, no-plan and position-end. Define these constants: line bound (default 1000, range 50 to 100000) and outcome hosts (50). + - The per-record arena copies every string and number, so a record never points into scheduler memory. The record reports its own byte size, and freeing a NULL record is a no-op. + - Job selection matching covers all jobs, a plain job, an array element, and every element of an array (including elements created later). + - Finishing a record as a no-decision kind records the kind and the error. + - Create the capture source next to the model source, as an empty module for now, and register both in the scheduler build. + - Add both to the core isolation test's source scan, with a new check that rejects stdio and file I/O in those two files. They are not added to the isolated core state-machine library. + - Done when model unit tests pass under ASAN for selection matching, arena copying and size, and record finishing, and the isolation test passes with the new files scanned. + - _Requirements: 1.2, 1.3, 1.8, 2.4, 9.1, 11.2_ + +- [ ] 1.4 Build the trace entry sequence and the line budget, test first + - Path push and pop, with balanced requirement begin and end entries. Each entry stores its full nesting path: single, compound container, alternative i/n, or member j/m. + - Entry kinds, each rendering as exactly one line: + - requirement begin and end; + - a skipped alternative, the alternative result, and a member merge failure; + - attempt reasons; + - cluster-wide check end; + - no plan; + - plan begin, discarded, not tried, and end; + - position begin, end, and scan error; + - host verdict; + - commit; + - finalize failure. + - The line budget stops storing entries once the bound is reached and counts the omitted lines exactly. + - Done when model tests pass under ASAN for nesting and balance, every entry kind, and budget edges (bound-1, bound and bound+1) with exact omitted counts. + - _Requirements: 3.3, 3.4, 3.6, 4.3, 4.4, 4.5, 4.7, 4.8, 5.10, 9.1, 9.2_ + +- [ ] 1.5 Add plan and verdict counters, search-end statements, summary tallies and rollback merging, test first + - The model counts generated, tried, discarded and not-tried plans for each requirement scope. + - The search-end entry's statement follows the design's conditions: none generated, all discarded, or none placed. The decision-wide counters use the same conditions. + - Per-verdict host evaluations and requirement outcomes stay exact after truncation. + - At summary level, host entries are not stored. Instead, a position-summary entry holds the accepted host, a count for each verdict kind (a host with several kinds counts once under each), and a tally of hits per limit. + - Consecutive released-host calls merge into one rollback entry. + - Done when model tests pass under ASAN for: + - every search-end condition, including a plan that is not tried after a discarded one; + - counters after truncation; + - summary tallies for a host with both a shortfall and a limit hit; + - rollback merging. + - _Requirements: 4.6, 4.9, 5.8, 5.9, 5.10, 9.3_ + +- [ ] 1.6 Make every white-box test that includes a hooked scheduler source also compile the trace model and capture sources + - Use the design's include-pattern search to find the 28 tests that include the election, scheduler, allocator-runner, host-iterator, resource or live-driver sources. Give each one the trace model and capture sources, so the live-driver change in 1.7 and the hook tasks after it still link. + - Done when the full scheduler test suite builds and passes under ASAN with the new includes and no hooks in place yet. + - _Requirements: 11.4_ + +- [ ] 1.7 Add the trace attachment and trace-only op to scheduling decisions + - Decisions gain a trace attachment, which every existing constructor leaves empty. + - A new trace-only op, with its own constructor, is appended after the existing ops without renumbering them. + - The live driver's drain loop skips the trace-only op and frees any attached trace defensively. Live never produces either. + - Done when a decision test shows every existing constructor leaves the attachment empty, existing op values are unchanged and the trace-only constructor works, and the existing live-driver and decision tests pass unchanged under ASAN. + - _Requirements: 2.4, 10.1, 11.4_ + +- [ ] 1.8 Add the core trace gate and the hook macro, test first + - The scheduling core gains a session-settings pointer and a record-under-construction pointer. Both stay empty unless the simulator REPL sets them. + - The hook macro evaluates its call only when a record is being built. + - Done when a gate test passes under ASAN showing that: + - a hook argument with a side effect is not evaluated when no record is being built; + - a full allocation with tracing never enabled creates no record. + - _Requirements: 11.1, 11.2, 11.3_ + +- [ ] 2. Allocator-side capture of scheduler state +- [ ] 2.1 Capture job selection facts, cluster-wide limits and attempt failure reasons, test first + - Selection capture: policy name, the job's queue and queue priority, the job's priority and its queue time. + - Cluster-wide limit capture: each matched limit's name and scope instance, read from the job's matched cluster-wide limits. + - Attempt failure reasons: summarized per pend-reason code, with the hosts and resources they name, captured before the scheduler replaces the job's stored reason. + - Done when capture tests over a fixture core and job pass under ASAN, showing exact copied values that do not change when the source structures change later. + - _Requirements: 2.2, 2.3, 3.5_ + +- [ ] 2.2 Capture shortfall and limit rows from the satisfaction check, test first + - Each checked row carries the resource name, flags, value type, total, in-use, provisionally reserved, requested and met. + - A row whose resource is flagged as a limit is a limit row. It is resolved to its limit through the scope's matched limits and carries the limit name, the scope instance (plus the host for host scope) and the limited resource's name. + - The cluster-wide scope keeps every row. A host scope keeps only unmet rows, and these set the host's shortfall and/or limit verdict. + - Done when capture tests pass under ASAN showing that limit rows are never reported as shortfalls, scope strings are correct, and each scope keeps the right rows. + - _Requirements: 4.1, 4.2, 5.4, 5.5_ + +- [ ] 2.3 Capture decision outcomes and reconstructed dispatches, test first + - **Dispatch:** per-host allocation rows and shared rows, with limit charges listed separately. + - A limit charge is identified by the limit flag, or by its name prefix when the resource is unknown. + - It is resolved to its limit by name through the job's matched limits. An unresolved charge falls back to a raw number. + - The first 50 hosts are kept in detail. The rest are aggregated into per-resource totals (excluding limits), per-limit totals and an omitted-host count. + - **Pend:** a copy of the decision's pend-reason set. + - The main reason and host-less entries are always kept. + - The first 50 hosts are kept with all their entries; the rest are counted per pend-reason code. + - **Reconstructed dispatch:** a record holding the reconstruction notice and the dispatch outcome. + - Done when capture tests pass under ASAN for: + - a dispatch with shared and host limit charges; + - 51-host dispatch and pend aggregates; + - a pend copy equal to its source under the existing pend-reason equality, for 50 or fewer hosts; + - the raw-number fallback. + - The full scheduler test suite, including the 28 tests updated in 1.6, must still build and pass under ASAN with the completed capture source. Missing scheduler sources are added to those tests, or stubbed. + - _Requirements: 2.5, 6.1, 6.2, 6.3, 6.4, 9.4_ + +- [ ] 3. Allocator-thread hooks at the decision seams +- [ ] 3.1 Hook the allocator loop: position, record lifecycle, selection and no-decision paths (P) + - **Position:** when session settings are present, every real job pop increments it. + - **Record start:** a wanted job starts a record with its selection facts; its cluster-wide limits are added after election. + - **No-decision paths:** + - not found: no selection; + - no scheduling object: selection without limits; + - allocation error, pend without a reason, or decision-build failure: the record is finished as that kind and pushed as a trace-only decision. + - **Normal decisions:** a built decision gets its outcome captured and the record attached. + - **Pointer clearing:** every path that hands a record away clears the record-under-construction pointer; the session-end marker also clears it. + - Done when allocator-loop tests using the real allocator thread pass under ASAN for each no-decision kind, the position count, outcome attachment, and the pointer being clear after each job. + - _Requirements: 2.1, 2.2, 2.3, 2.4, 6.1, 6.4, 11.2_ + - _Boundary: TraceHooks (allocator loop)_ + - _Depends: 2.1, 2.3_ + +- [ ] 3.2 Hook resource-requirement branching (P) + - Single, top-level compound, each alternative and each member push their path level and record exactly one begin and one end. + - The begin carries task counts and requested resources for single requirements and members, and the member count for containers. + - A member that fails or errors records the number of members not tried. A merge failure after a member is a separate entry. + - A skipped alternative is recorded in its position. Either the chosen alternative is recorded, with the alternatives after it as not tried, or "all failed" is recorded. + - Attempt failure reasons are captured before the job's stored reason is replaced. + - Done when branching tests pass under ASAN for: + - a single requirement; + - a compound with a failing member and a merge failure; + - an alternative job with a skipped, a failed and a chosen alternative; + - per-alternative reasons; + with every scheduler result unchanged. + - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6_ + - _Boundary: TraceHooks (resource-requirement branch)_ + - _Depends: 2.1_ + +- [ ] 3.3 Hook the host pre-filter, the host scan and the satisfaction check (P) + - The pre-filter records a selection mismatch and a pre-filter internal error as host verdicts. + - A host scan that stops on an error marks the current position as a scan error. + - The satisfaction check hands every row to the current record. + - Done when host-iterator and resource tests pass under ASAN for the selection-mismatch verdict, the pre-filter internal error, scan-error marking, and row capture in both scopes, with iterator and check results identical to an untraced run. + - _Requirements: 4.1, 4.2, 4.4, 5.2, 5.3, 5.4, 5.5, 5.7_ + - _Boundary: TraceHooks (host iterator, satisfaction check)_ + - _Depends: 2.2_ + +- [ ] 3.4 Hook the allocator runner: cluster-wide check, plans, positions, candidates and finalize + - The runner records: + - an allocator decline and a generator start failure, as no-plan reasons; + - the cluster-wide check scope and its end, with "no plans tried" on failure; + - each generated plan as tried, discarded or not tried; + - each plan's end, with the failed position or the internal error; + - each position's begin and end (filled, unfilled, scan error, or internal error after acceptance; the unfilled end is shown as a scan error when 3.3 marked the position); + - each claimed-host skip; + - each candidate's per-host result: accepted; rejected, with the verdict taken from the rows 3.3 records, or a selection mismatch when no rows were recorded; or internal error; + - the released hosts, when a failed plan had claims; + - commit or finalize failure; + - search end. + - Done when runner tests pass under ASAN covering every recorded outcome above, with runner return values and pend reasons identical to an untraced run. + - _Requirements: 4.1, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 4.9, 5.1, 5.2, 5.6, 5.7, 5.8_ + - _Depends: 2.2, 3.3_ + +- [ ] 4. Simulator REPL trace surface +- [ ] 4.1 Render trace records to text, test first (P) + - **Hierarchy:** header, selection, requirements, cluster-wide checks and plans, positions and hosts, then the outcome. Every budgeted entry renders as exactly one line, in the order shown in the design's example. + - **Summary level:** renders position summaries with per-limit names. + - **Truncated records:** render a marker with the omitted line count, followed by the fixed-size totals block. + - **Outcome:** at most 50 hosts, then aggregates and an omitted-host marker. Limit charges are listed separately, with limit amounts formatted by the limited resource. + - **No-decision records:** render the "unavailable" lines. Internal-error and pend-without-reason records also render the captured hierarchy. + - **Reconstructed records:** render the reconstruction notice and the outcome. + - **Formatting:** amounts use the units mode passed in, and job identifiers use the existing job-identifier formatting. Output goes to one or two sinks, with identical bytes in each. + - Add the renderer source to the scheduler's main source list. + - The renderer test includes only the model, capture-free fixtures and the shared formatting module. + - Done when golden renderer tests pass under ASAN for full and summary levels, every units mode, truncation, the outcome cap, each no-decision kind and reconstructed records. + - _Requirements: 2.1, 2.4, 2.5, 3.1, 3.2, 4.3, 5.1, 5.9, 6.1, 6.2, 6.3, 6.4, 7.6, 7.7, 9.2, 9.3, 9.4_ + - _Boundary: TraceRenderer_ + +- [ ] 4.2 Store the latest trace per job and serve info trace, test first + - **Storing:** a new trace for a job replaces its earlier one. + - **Memory bound:** accounted from each record's size. + - At 64 MiB, the oldest traces are evicted, never the newest. + - Evicted jobs are marked as discarded; a newer capture clears the mark. + - **`info trace` for a job or element:** renders the stored trace at the level it was captured, using the current units mode. With no stored trace, it reports one of: "discarded", "no trace" with the job's state, or "unknown job". + - **`info trace` for an array:** renders each element's trace in index order and marks discarded elements. When no element has a stored trace, it lists the elements' states. + - Add the store source to the scheduler's main source list. + - The store test includes the model, the renderer and the shared formatting module, with a fixture job table. + - Done when store tests pass under ASAN for replacement, eviction that never drops the newest trace, the three no-trace replies, and an array mixing stored and discarded elements. + - _Requirements: 1.7, 7.8, 7.9, 7.10, 7.11, 9.5, 9.6_ + - _Depends: 4.1_ + +- [ ] 4.3 Parse the trace command and manage trace settings, test first + - `trace` re-parses the raw input line, so it is not limited by the normal tokenizer. It accepts: + - `on all`, or `on job` followed by any number of job selectors; + - an optional level (default summary) and an optional line bound (50 to 100000, default 1000); + - an optional trailing `file` path of up to 4095 bytes, taken from the rest of the line, with optional quotes; + - `off` and `status`. + - An invalid argument is named in an error, and the settings stay unchanged. + - This task only parses and stores the file path. Opening the file, the same-file checks and leaving settings unchanged when the open fails are added in 4.5. + - Add the trace control source to the scheduler's main source list. + - Settings objects are reference-counted and held as "next session" and "current session". Commands change only "next". + - `trace status` shows: + - the settings in effect, or the next settings when no session is running; + - any replacement waiting for the next session; + - stored-trace memory use against its bound. + - Done when command tests pass under ASAN for every grammar form, every invalid-argument class, a path longer than 63 characters, 20 job selectors, and status output before, during and after a session. + - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.7, 1.8, 1.9, 1.10, 1.12_ + - _Depends: 4.2_ + +- [ ] 4.4 Pass the scheduler log path to the simulation driver (P) + - The scheduler's log path goes from the command line through the simulation arguments into the simulation driver, where the trace file checks can read it. Only the simulation driver and the command-line entry point change. + - Done when a simulation start test under ASAN shows the driver holds the configured log path, or the default path when none is given. + - _Requirements: 8.3_ + - _Boundary: SimIntegration (driver arguments)_ + +- [ ] 4.5 Manage the trace file copy and output-file conflicts, test first + - When a command names a file, it is opened for appending at once. An open failure names the file and the cause, and the settings stay unchanged. + - A path that is the same file (device and inode) as the current simulation output file or the scheduler log is rejected. + - Expose a conflict check, so `set output` can reject any trace file that is still open. + - A replaced or disabled file stays open until the session in progress ends; with no session in progress it is closed immediately. + - Shutdown flushes and closes every open trace file. Trace code never writes to the scheduler log. + - Done when file tests pass under ASAN for: + - an open failure; + - each same-file rejection, including through a hard link; + - a replaced file kept open until the session ends; + - an immediate close; + - flushing at shutdown. + - _Requirements: 8.2, 8.3, 8.4, 8.5, 8.6, 8.7_ + - _Depends: 4.3, 4.4_ + +- [ ] 4.6 Emit traces inline and apply settings at session boundaries, test first + - At session begin: + - reset the per-round traced and flushed flags; + - make the next settings current, stamping the session number and resetting the position; + - publish the settings to the scheduling core, or publish nothing when tracing is off. + - At session end: + - withdraw the published settings; + - release the current settings, closing their file if it was replaced; + - clear the in-progress state. + - This task owns making the simulator's decision echo printer non-static and declaring it in the simulator header, with its format unchanged, so continue-mode emission prints exactly the echo line that was suppressed. No other simulation-loop change belongs here. + - Emitting a traced decision or trace-only record: + - marks the round as traced, in any mode; + - in continue mode, flushes any active collapsed range on the round's first traced item, then prints the suppressed echo line for traced decisions; + - renders the trace to the console and to the current file with identical bytes, flushes both, and hands the record to the store. + - The emission test includes the REPL and simulation sources it calls: the collapsed-range flush and the echo printer. + - Done when emission tests pass under ASAN showing: + - trace placement in step mode; + - flush, then echo, then trace in continue mode; + - the flags reset after a round that ended in step mode; + - the file copy byte-equal to the console output. + - _Requirements: 1.6, 7.1, 7.2, 7.3, 8.1, 8.5_ + +- [ ] 5. Wire the trace into the REPL and the simulation driver +- [ ] 5.1 Make every white-box test that includes REPL or simulation sources compile the trace sources, and register the trace REPL sources with the simulation isolation checks + - Add the trace control, store and renderer sources to the simulation isolation library and to the simulation isolation test's scan list. + - Find the tests with the design's full include-pattern search. To the 26 tests that include the REPL or simulation-loop sources, add the trace REPL sources. + - Of those 26, 19 include no hooked source, so they also get the trace model and capture sources. Where the capture source needs a scheduler source those tests do not already include, add it or provide a stub, and list it in the task notes. + - Done when all of the following pass under ASAN: a full scheduler build including the scheduler binary, the full scheduler test suite, and the simulation isolation test. + - _Requirements: 10.5, 11.3_ + +- [ ] 5.2 Wire the trace commands into the REPL + - Route `trace` with the raw line, and route `info trace`. + - Help gains a trailing Trace section, and the info subcommand list and info usage line include `info trace`. + - `set output` rejects an open trace file. Quit and end of input shut tracing down. + - The continue-mode rollup emits a traced round on its own, after its pend list is built, using the same idle rule as range flushes, and never starts a collapsed range from it. + - Add the trace entry points and their interactive-only gates to the REPL header. + - Update the 1.1 baseline fixture for exactly the changed help and info-usage lines and nothing else. The test's diff against the old fixture shows only those lines. + - Done when REPL tests pass under ASAN for help sections and alignment, command routing, the `set output` rejection and the traced-round rollup, and the updated baseline test passes. + - _Requirements: 1.11, 7.3, 7.4, 8.4, 8.6, 10.4, 10.5_ + +- [ ] 5.3 Wire tracing into the simulation driver's session and decision flow + - **Session hooks:** session begin runs after a successful begin-session and before reconciliation. Session end runs after the decision stream ends. + - **Drain loop:** + - trace-only decisions are handled first, with no verification, recording, echo or breakpoint check; + - traced decisions go to the trace emitter after the echo gate and before the breakpoint check; + - unclaimed traces are freed before each decision is freed. + - **Reconstructed dispatches:** wanted jobs get a reconstructed record, handled in the same order. + - The echo-printer export belongs to 4.6. + - Done when driver tests pass under ASAN showing: + - trace-only decisions never reach verification or recording; + - a decision breakpoint stops after that decision's trace is printed and stored, with no later trace stored; + - reconstructed traces print before the round's live decisions; + - a non-interactive simulation never publishes settings. + - _Requirements: 1.6, 2.5, 7.1, 7.2, 7.3, 7.5, 10.1, 11.3_ + +- [ ] 6. Validation +- [ ] 6.1 End-to-end trace scenarios over the contention fixture + - Drive the real simulator REPL with tracing on for selected jobs and for all jobs, at both levels. + - Assert that every section of Requirements 2 to 6 appears with the expected content: + - alternative and compound labels; + - cluster-wide rows and limit rejections; + - plans, positions and host verdicts; + - rollback; + - dispatch and pend outcomes. + - Assert ordering in step mode, continue mode and at breakpoints. + - Assert `info trace` at a breakpoint, after the job finishes, for an array, and after changing units mode. + - The trace file must equal the inline traces byte for byte. + - Done when the end-to-end test passes under ASAN. + - _Requirements: 1.1, 1.2, 1.6, 2.1, 2.2, 2.3, 3.1, 3.4, 4.1, 4.2, 4.3, 4.5, 5.1, 5.3, 5.4, 5.5, 5.6, 5.8, 5.9, 6.1, 6.2, 6.4, 7.1, 7.3, 7.4, 7.5, 7.8, 7.9, 8.1_ + +- [ ] 6.2 Error-path trace scenarios with the real allocator + - Include the scheduler sources directly, so source-level and link-level fault injection apply. + - Inject each fault and assert its traced outcome: + - a plan-claims growth failure, through the heap-allocator macro; + - a snapshot failure, through the vector-copy wrap; + - finalize failures, through the allocation-constructor wraps; + - a failing host scan; + - a failing per-host uncommitted lookup; + - an allocation function that returns an error; + - an unknown job identifier placed directly on the job hand-off buffer. + - Done when the error-path test passes under ASAN and each injected fault produces its specified trace lines. + - _Requirements: 2.4, 3.2, 4.4, 4.5, 5.7_ + +- [ ] 6.3 Prove tracing never changes scheduling + - Run the contention replay four ways: tracing off, all jobs at summary level, all jobs at full level, and a job subset. + - Every run must produce identical decision sequences, pend-reason sets, final accounting state, `--sim-verify` results and `--sim-output` recordings (apart from wall-clock fields). + - With tracing off, the baseline transcript (as updated in 5.2) must stay unchanged. + - The existing equivalence, verification and live-driver suites must pass unchanged. + - Done when the equivalence test and the full scheduler suite pass under ASAN. + - _Requirements: 10.1, 10.2, 10.3, 10.4, 10.5, 11.4_ + +- [ ] 6.4 Bounds under large traces and the final full build + - Trace a 1000-host parallel job with up to 64 tasks, at full level with the default bound. The stored entries must stay within the bound, with exact counters. + - Run 10,000 traced decisions. Stored-trace memory must never exceed 64 MiB. + - Correct the stale file sizes in the scheduler component guide and list the new files. + - Done when the bounds test passes and a full ASAN build of all components passes with no test failures. + - _Requirements: 9.1, 9.3, 9.5, 11.2_