myxce/.kiro/specs/scheduler-sim-decision-trace/design.md

Design Document: scheduler-sim-decision-trace

Overview

Purpose: This feature adds a decision trace to the xce_scheduler simulator REPL. For each traced job it explains why the job was selected, which resource requirements were tried, which cluster-wide checks and placement plans were attempted, why each host was accepted or rejected, and what was finally allocated or why the job pends.

Users: Scheduler developers and operators who replay recorded sessions with xce_scheduler --sim <file> -i. They enable tracing at runtime for all jobs or for selected jobs, read traces inline during step/continue, recall them with info trace, and optionally copy them to a file.

Impact:

• The allocator thread gains NULL-gated capture hooks at the decision seams. These seams are election, resource-requirement branching, the allocator runner, the host iterator and the shortfall check.
• xce_decision_t gains a trace attachment and a trace-only op.
• The simulation driver and REPL gain a trace control, store, renderer and file copy.
• The live scheduler is unchanged except for one never-taken branch per hook.

Goals

• Explain every traced dispatch, pend and no-decision selection with the hierarchy and content defined in Requirements 2–6.
• Keep scheduling bit-for-bit identical with tracing on or off (Requirement 10) and keep the live scheduler free of trace work (Requirement 11).
• Bound trace size per decision and stored-trace memory (Requirement 9).

Non-Goals

• Live-mode, headless-sim or recording-file tracing.
• Preemption decisions (XCE_DECISION_PREEMPT) are never traced.
• Changing pend-reason content, the --sim-verify comparison or the ReplayDecision format.
• Fixing defects the trace exposes. Examples are ALTER keeping only the last alternative's obj->reason, a stale obj->reason after an accepts() decline, pend reasons duplicated across parjob plans, and an unreachable PREASON_MATCH_FAIL. They are logged as separate bugs.
• info limit and limit-usage display (scheduler-sim-diagnostics).

Boundary Commitments

This Spec Owns

• The trace record model, its arena storage, entry budget and counters (sched_trace.c, sched_trace_scan.c), and the allocator-side capture helpers that read scheduler state (sched_trace_capture.c, sched_trace_outcome.c), with their shared internal header sched_trace_internal.h.
• Two gate fields on xce_sched_core_t: trace_ctl (session settings snapshot) and trace_rec (record under construction). It also owns the SCHED_TRACE hook macro.
• The trace field on xce_decision_t, the XCE_DECISION_TRACE op and make_trace_decision().
• The hook call sites in sched_election.c, scheduler.c, alloc_runner.c, host_iter.c and resource.c. Each is a single gated statement that never changes control flow or return values.
• The REPL trace command, trace settings lifecycle, trace file copy, the set output trace-file conflict check, the trace store, info trace, trace rendering, the help "Trace" section, and the continue-mode traced-round rule.
• Plumbing the scheduler log path into the simulation driver (sched_sim_args_t.log_path, xce_sched_sim_driver_t.log_path).

Out of Boundary

• Scheduling, picking, allocation and accounting behaviour, and the content of xce_pendreason_t / job->allocs.
• The jbuf hand-off (capacity 1, uint64_t items) and the picker thread. No hook runs on the picker thread.
• --sim-verify comparison logic (sim_verify.c), the recording format (recording*.c, sim_record.c) and snapshot content.
• The unsatisfiable-resreq cache. It is live-only; sim never consults it, so a trace never explains a cache hit.
• Rollup fingerprint semantics for untraced rounds (scheduler-sim-repl-verbosity).

Allowed Dependencies

• Core side (sched_trace*.c): libxce containers and types (xvec, xlist, zarray, xce/types.h), include/sched/alloc_iter.h (for xce_alloc_plan_t), src/include/utils.h (only array_jobid/array_idx for selector matching), include/sched/types.h, include/core.h, and include/preason.h. These files perform no stdio, file or network I/O and do not include REPL or sim headers.
• REPL side (repl_trace*.c): include/sched/trace.h (read-only use of records), repl_internal.h, repl_fmt.c helpers, libxce htab, and parse_jobsel/array_jobid/fmt_jobid (utils.h). It must not call scheduler mutation APIs.
• Dependency direction: trace.h types → sched_trace.c/sched_trace_scan.c (model) → sched_trace_capture.c/sched_trace_outcome.c (allocator-side readers) → hook sites. Separately, trace.h → repl_trace_render.c → repl_trace_store.c → repl_trace.c → repl.c/repl_info.c/repl_params.c/repl_rollup.c/sim_loop.c call sites. Core-side files never include REPL-side headers.

Revalidation Triggers

• Any change to xce_decision_t or xce_decision_op_t (shared with scheduler-preemption).
• Changes to the allocator runner seams (alloc_runner.c plan, position and host loop), host_iter.c filters, or unsatisfy_'s signature.
• Changes to the sim drain-loop order (verify → record → print gate → breakpoint) or to the rollup call site.
• Changes to the limit plugin's vres naming or to obj->glimits/obj->hlimits ownership.
• Any change that lets the host iterator's resource-availability pre-filter (host_iter.c:150-166) see a non-empty request. Today it is a no-op because the runner empties alloc_req.host before every next() (alloc_runner.c:216, 253), and sched_check_host_satisfy returns 1 on an empty request (wanted->count == 0, resource.c:418-420) before it reaches the unsatisfy_ hook. The design hooks only the selection and internal-error pre-filter paths; a live resource pre-filter would need its own host_begin plus pre-filter SHORTFALL/LIMIT verdicts.

Architecture

Existing Architecture Analysis

• The core (core*.c) is a passive state machine driven by two drivers. The picker and allocator threads communicate through jbuf (capacity 1). Decisions flow through the core->decisions cbuf to the driver's drain loop. Live (driver.c) and sim (sim_loop.c) run the identical election, allocation and accounting chain (scheduler-driver-equivalence.md).
• The precedent for driver-specific behaviour in the core is NULL-gated core fields set by one driver only (unsatisfiable_lookup_cb, cache_clear_cb). This design follows it.
• Files already over the 800-line cap are scheduler.c (2002), sim_loop.c (1819), repl_info.c (1893), repl_break.c (980) and driver.c (1104). They receive only call sites. All logic lives in new files.

Architecture Pattern & Boundary Map

graph TB
    subgraph AllocatorThread
        Election[sched_election allocator loop]
        Branch[scheduler resreq branch]
        Runner[alloc_runner plans and positions]
        HostIter[host_iter prefilter]
        Satisfy[resource unsatisfy]
        Capture[sched_trace_capture]
        Model[sched_trace record model]
    end
    subgraph SimThread
        Drain[sim_loop drain loop]
        ReplTrace[repl_trace control and file]
        Store[repl_trace_store and info trace]
        Render[repl_trace_render]
        Rollup[repl_rollup]
    end
    Election --> Capture
    Branch --> Capture
    Runner --> Capture
    HostIter --> Capture
    Satisfy --> Capture
    Capture --> Model
    Election -->|decision with trace| Decisions[core decisions cbuf]
    Decisions --> Drain
    Drain --> ReplTrace
    ReplTrace --> Render
    ReplTrace --> Store
    Store --> Render
    ReplTrace --> Rollup
    ReplTrace -->|session settings| CoreCtl[core trace_ctl]
    CoreCtl --> Election

Architecture Integration:

• Selected pattern: per-decision structured record built on the allocator thread, transferred with the decision, then rendered and stored by the sim REPL.
• Boundaries:
  • The core side builds records and never renders.
  • The REPL side renders, stores and writes. Inline emission reads only the record, never live scheduler state. The single exception is info trace (7.9, 7.11), which reads job->state at the prompt. That is the same read-only access, with the same timing, as the existing info job at a decision breakpoint; it adds no new race class.
• Existing patterns preserved:
  • The NULL-gated core field, the sim->interactive inline gates in repl.h, and drain-loop ownership of decisions.
  • fprintf(stdout, ...) for all REPL output (test_no_bare_printf.c).
  • jbuf is unchanged.
• New components rationale: the record model, capture, control, store and renderer each have a distinct owner and thread (see Components).
• Steering compliance:
  • The core does no I/O.
  • Driver equivalence: the trace only reads state, and live never enables it.
  • TDD (testing.md) and the 800-line cap.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Scheduler core	C (gnu17), existing libxce	Record model, capture hooks	No new libraries
Simulator REPL	C, libedit readline() (existing)	trace / info trace commands, rendering, file copy	Raw-line re-parse for long arguments
Build / test	Meson, existing Unity-style testsuite	New sources added to scheduler_src, isolation libs and test include lists	—

File Structure Plan

Directory Structure

src/scheduler/
├── include/sched/
│   └── trace.h                 # Record, entry, settings types; SCHED_TRACE macro; model + capture API
├── src/
│   ├── sched_trace.c           # Record model: arena, entry append, budget, requirement/plan/check
│   │                           #   entries, jobsel predicate, record free/size. No I/O, no
│   │                           #   scheduler-state reads.
│   ├── sched_trace_scan.c      # Record model, part 2: positions, host verdicts and rows, counters,
│   │                           #   verdict tallies, rollback, commit, search end. No I/O.
│   ├── sched_trace_capture.c   # Allocator-side capture helpers that read scheduler structures
│   │                           #   (job/queue keys, glimits, resource flags, vres lookup, check
│   │                           #   rows, attempt reasons). No I/O.
│   ├── sched_trace_outcome.c   # Capture, part 2: alloc/pend outcome copy with 50-host cap,
│   │                           #   reconstructed-dispatch record. No I/O.
│   ├── sched_trace_internal.h  # Helpers shared by the sched_trace*.c files (task 2.4 split to
│   │                           #   stay under the 800-line cap); not installed.
│   ├── repl_trace.c            # `trace on|off|status` parsing (raw line), refcounted settings
│   │                           #   (next/cur references), file open/close, dev/inode conflict
│   │                           #   checks, shutdown
│   ├── repl_trace_emit.c       # Session begin/end hooks, inline emission (live, trace-only and
│   │                           #   reconstructed records), file write-error reporting
│   ├── repl_trace_store.c      # jobid -> latest record index, capture-order eviction at 64 MiB,
│   │                           #   discarded marks, `info trace` (job, array element, array)
│   ├── repl_trace_render.c     # Record -> lines on one or two FILE* sinks; units mode; jobid fmt;
│   │                           #   hierarchy/indent; summary vs full host section
│   ├── repl_trace_render_outcome.c # Renderer, part 2: outcome section (50-host cap), pend reasons,
│   │                           #   no-decision and reconstructed endings (800-line split)
│   └── repl_trace_render.h     # Helpers shared by the two renderer files; not installed
└── testsuite/
    ├── test_trace_model.c          # sched_trace.c + sched_trace_scan.c unit tests
    ├── test_trace_capture.c        # capture helpers against fixture core/schedobj/workers
    ├── test_decision_trace.c       # XCE_DECISION_TRACE, make_trace_decision, decision->trace transport
    ├── test_trace_hooks_gate.c     # SCHED_TRACE no-evaluation proof; live/non-interactive never trace
    ├── test_trace_hooks_alloc_loop.c # sched_allocator hooks: selection, limits, no-decision records
    ├── test_trace_hooks_resreq.c   # requirement-branch hooks (single, compound, alternatives)
    ├── test_trace_host_hooks.c     # host pre-filter, host scan and satisfaction-check row hooks
    ├── test_trace_hooks_runner.c   # alloc_runner hooks: plans, positions, hosts, rollback, commit
    ├── test_trace_render.c         # renderer output per section, units modes, truncation, outcome cap
    ├── test_trace_store.c          # store replace/evict/discarded marks, info trace variants
    ├── test_repl_trace_baseline.c  # pre-feature REPL transcript baseline (tracing off, 10.4)
    ├── test_repl_trace_cmd.c       # trace command grammar, errors, status, lifecycle
    ├── test_repl_trace_file.c      # trace file rules: open errors, conflicts, replace, shutdown
    ├── test_repl_trace_emit.c      # session hooks, emission order per mode, file copy, write errors
    ├── test_repl_trace_wiring.c    # repl.c/repl_info.c/repl_params.c routing, interactive-only gates
    ├── test_sim_trace_driver.c     # sim driver wiring: trace-only path, breakpoints, reconstructed
    │                               #   records, trace off retention and output (1.5, 10.4), 11.3
    ├── test_repl_trace_e2e.c       # step/continue/breakpoint integration over replay fixtures
    │                               #   (+ test_repl_trace_e2e_fixture.c, test_repl_trace_e2e.h)
    ├── test_trace_error_paths.c    # fault injection into the real allocator thread
    ├── test_trace_bounds.c         # line bound and 64 MiB store bound under large traces
    └── test_trace_equivalence.c    # trace on vs off: decisions, --sim-verify, --sim-output, accounting
                                    #   (+ test_trace_equivalence_cmp.c, test_trace_equivalence.h)

Modified Files

• src/scheduler/include/core.h: add xce_sched_trace_ctl_t* trace_ctl and xce_sched_trace_rec_t* trace_rec to struct xce_sched_core_s, each documented as allocator-thread-read and sim-window-written, and NULL in live.
• src/scheduler/include/decision.h, src/decision.c:
  • append XCE_DECISION_TRACE after XCE_DECISION_PREEMPT;
  • add struct xce_sched_trace_rec_s* trace; after the union;
  • add make_trace_decision(rec).
• src/scheduler/src/sched_election.c: hooks in sched_allocator:
  • per-job prologue: position counter, traced predicate, record begin with selection facts;
  • after plugin_job_election: cluster-wide limits;
  • each no-decision path: no-decision record pushed as a trace decision;
  • after the decision is built: outcome capture and attach;
  • session-end marker: reset trace_rec.
• src/scheduler/src/scheduler.c: hooks in sched_make_allocation (PRIMARY and top-level COMPOUND containers), sched_make_compound_alloc_ (member push/begin/end/pop, merge failure, attempt reason snapshot) and the ALTER loop (alternative push/begin/end/pop, skipped, result).
• src/scheduler/src/alloc_runner.c: hooks for:
  • the accepts decline;
  • cluster-wide check begin/end;
  • plan_init failure;
  • each generated plan (discarded, not tried on reserve failure, tried);
  • each position (begin, end: filled, unfilled or scan error);
  • each candidate (claimed skip, per-host result >0/0/<0);
  • rollback;
  • finalize failure;
  • commit.
• src/scheduler/src/host_iter.c: hooks for selection mismatch, pre-filter internal error (sched_ctx_get_uncomit_host failure) and scan stop on iter_next < 0.
• src/scheduler/src/resource.c: one hook at the top of unsatisfy_ passing all rows.
• src/scheduler/src/sim_loop.c:
  • xce_repl_trace_session_begin(sim) right after xce_core_begin_session;
  • reconstructed-dispatch trace in sim_reconcile_unaccounted_jobs' report block;
  • drain loop: XCE_DECISION_TRACE handled first, then xce_repl_check_trace(sim, d) after sim_print_decision_gate;
  • xce_repl_trace_session_end(sim) after the drain loop.
• src/scheduler/include/repl.h: extern declarations of the forwarded entry points (repl_trace_session_begin/end, repl_trace_on_decision, repl_trace_on_trace_only, repl_trace_on_reconstructed), following the existing xce_repl_on_* pattern so repl.h stays self-contained; this is their only declaration site. It also adds inline gates xce_repl_trace_session_begin/end, xce_repl_check_trace, xce_repl_check_trace_only and xce_repl_trace_reconstructed, all forwarding only when sim->interactive.
• src/scheduler/src/repl_internal.h: xce_repl_trace_state_t trace member in xce_sched_repl_state_t; declarations for the other repl_trace* entry points (command, store, renderer, settings file, conflicts, shutdown) and moved formatting helpers.
• src/scheduler/src/repl.c: dispatch trace (passing the raw line); help gains a trailing "Trace" section and an info trace line; repl_cmd_quit and EOF call repl_trace_shutdown.
• src/scheduler/src/repl_info.c: info trace route (one branch); usage string updated; units helpers moved out.
• src/scheduler/src/repl_fmt.c: receives translate_units_mode, repl_format_numeric_resource_value, repl_format_consumable_value_and_used and repl_current_units_mode (behaviour unchanged, now non-static).
• src/scheduler/src/repl_params.c: set output calls repl_trace_file_conflicts(sim, path) before switching.
• src/scheduler/src/repl_rollup.c: the traced-round rule in repl_round_rollup_process, placed after repl_query_pend_jobids succeeds (see ReplIntegration).
• src/scheduler/src/driver.c: the drain loop frees and skips XCE_DECISION_TRACE defensively; it is never produced in live.
• src/scheduler/include/sim.h, src/sim.c, src/main.c: log_path plumbing. sim.h also declares sim_print_decision, which sim_loop.c makes non-static (format unchanged) so continue-mode trace emission reuses the exact echo line.
• src/scheduler/meson.build:
  • add the new sources to the scheduler sources;
  • add the trace sources (sched_trace.c, sched_trace_scan.c, sched_trace_capture.c, sched_trace_outcome.c and sched_trace_internal.h) to test_core_isolation.c's source-text scan list (forbidden headers and symbols, plus a new no-stdio check for these files only). They are not added to sched_core_isolation_lib, whose nm check covers the four core state-machine files; the capture file references scheduler symbols outside that archive's scope;
  • add repl_trace*.c to sched_sim_isolation_lib.
• src/scheduler/testsuite/meson.build, test_core_isolation.c, test_sim_isolation.c, plus every white-box test that #includes a hooked or calling source: register the new tests, scan the new files, and add #include lines for sched_trace*.c / repl_trace*.c. SCHED_TRACE bodies are compiled even when they never run, so a test that includes a hooked file must also include (or link) sched_trace.c, sched_trace_scan.c, sched_trace_capture.c and sched_trace_outcome.c.
  • Hooked sources: resource.c, host_iter.c, alloc_runner.c, scheduler.c, sched_election.c and driver.c. These are included by test_alloc_runner*.c, test_host_iter.c, test_alloc_equivalence_*.c, test_parjob_combinatorial.c, test_core_events.c, test.c and others (about 20).
  • Calling sources: repl.c and sim_loop.c, included by about 26 tests.
  • The exact list is produced by grep -l '#include "../src/\(resource\|host_iter\|alloc_runner\|scheduler\|sched_election\|driver\|repl\|sim_loop\).c"' testsuite/*.c and updated in one dedicated task right after the new files exist.
• src/scheduler/CLAUDE.md: correct the stale line counts and list the new files.

System Flows

Session lifecycle and capture

sequenceDiagram
    participant Op as Operator
    participant Repl as repl_trace
    participant Sim as sim_loop
    participant Core as core
    participant Alloc as allocator thread
    Op->>Repl: trace on job 42 level full
    Repl->>Repl: set next settings
    Sim->>Core: begin session
    Sim->>Repl: session begin hook
    Repl->>Core: publish trace_ctl and reset position
    Sim->>Sim: reconcile and report reconstructed dispatches
    Sim->>Core: signal go
    Alloc->>Core: pop jobid and count position
    Alloc->>Alloc: traced so set trace_rec and capture selection
    Alloc->>Alloc: hooks append resreq check plan host entries
    Alloc->>Alloc: capture outcome and attach record to decision
    Alloc->>Core: push decision and clear trace_rec
    Sim->>Core: drain decision
    Sim->>Repl: check trace
    Repl->>Op: echo line and rendered trace and file copy
    Repl->>Repl: store record
    Sim->>Repl: session end hook
    Repl->>Core: clear trace_ctl and release replaced settings

Key decisions:

• Settings take effect only in the session-begin hook (Requirement 1.6). Commands issued during a session change only next.
• trace_rec is non-NULL only between the per-job prologue and the decision push of a traced job. The allocator sets and clears it; nothing else touches it.
• A no-decision selection pushes a decision with op XCE_DECISION_TRACE that carries only the record. The drain loop handles it before verify, record, print gate and breakpoint check.

Drain-loop emission order (per decision)

graph TB
    Get[cbuf get decision] --> IsTrace{op is TRACE}
    IsTrace -->|yes| TraceOnly[check trace only then free and continue]
    IsTrace -->|no| Verify[verify]
    Verify --> Record[record add]
    Record --> Gate[print gate]
    Gate --> CheckTrace[check trace]
    CheckTrace --> Break[breakpoint check]
    Break --> Free[free decision]

• xce_repl_check_trace runs before the breakpoint check. A breakpoint therefore stops after the stopping decision's trace is printed and stored, and before any later trace (Requirement 7.5).
• In continue mode, the first traced item of a round flushes the active collapsed range, prints the suppressed echo line, then prints the trace. The round is then marked traced (Requirement 7.3).

Trace settings lifecycle

stateDiagram-v2
    [*] --> NextOnly: trace on
    NextOnly --> InSession: session begin
    InSession --> NextOnly: session end and still referenced as next
    InSession --> Released: session end and replaced or disabled
    NextOnly --> Released: replaced or disabled with no session in progress
    Released --> [*]

A settings object (selection, level, bound, open file) is referenced by rs->trace.next (the settings the next session will use) and, while a session runs, by rs->trace.cur (the settings the session started with). Commands only replace next. The session-begin hook sets cur = next, and the session-end hook clears cur. An object is released, and its file closed, when neither reference holds it. This gives 8.5 directly: a replaced object stays open until the in-progress session ends, or is closed immediately when no session is in progress.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1, 1.2, 1.3	Enable all or by jobs (arrays, future ids)	ReplTraceControl, TraceModel	repl_trace_cmd, sched_trace_ctl_wants	Lifecycle
1.4, 1.5, 1.6, 1.10	Replace, disable, next-session application, persistence	ReplTraceControl	next/cur settings, session hooks	Lifecycle, Session
1.7	Status: in effect, next, memory	ReplTraceControl, TraceStore	repl_trace_cmd status, repl_trace_store_bytes	—
1.8	Levels summary/full (default summary)	ReplTraceControl, TraceModel, TraceRenderer	xce_sched_trace_level_t	—
1.9, 1.12	Argument validation; unbounded jobs, 4095-byte path	ReplTraceControl	raw-line parser	—
1.11	Help section and info trace listing	ReplIntegration	repl_print_help	—
2.1	Session number and position	TraceHooks (election), TraceModel	sched_trace_ctl_t.position, .session_no	Session
2.2	Policy and FCFS keys	TraceCapture	sched_trace_capture_selection	Session
2.3	Cluster-wide limits matched	TraceCapture	sched_trace_capture_glimits	Session
2.4	No-decision selections	TraceHooks (election), DecisionTransport	sched_trace_rec_finish, XCE_DECISION_TRACE	Drain order
2.5	Reconstructed dispatch notice	SimIntegration, TraceCapture	sched_trace_rec_reconstructed	Session
3.1, 3.2	Resreq labels, path, tasks, resources, outcome	TraceHooks (scheduler.c), TraceCapture	sched_trace_resreq_begin/end	—
3.3, 3.4, 3.6	Failure point, chosen alt, not tried, skipped	TraceModel, TraceHooks (scheduler.c)	resreq_end(not_tried), sched_trace_member_failed, sched_trace_alt_result, sched_trace_alt_skipped	—
3.5	Per-alternative failure reasons	TraceCapture	attempt reason summary entry	—
4.1, 4.2	Cluster-wide rows; limit rejection	TraceHooks (unsatisfy_, runner), TraceCapture	sched_trace_check_rows	—
4.3, 4.4, 4.5, 4.6	Plans: shape, outcome, failed position, errors, commit	TraceHooks (runner), TraceModel	sched_trace_plan_*, sched_trace_pos_*, sched_trace_pos_scan_error, sched_trace_commit, sched_trace_finalize_failed, sched_trace_search_end	—
4.7, 4.8, 4.9	No plan generated or declined; no plans after check fail; discarded	TraceHooks (runner)	sched_trace_no_plan, sched_trace_check_end, sched_trace_plan_discarded, sched_trace_search_end	—
5.1–5.7, 5.10	Host verdicts, phases, kinds, not performed	TraceHooks (host_iter, runner, unsatisfy_), TraceModel	sched_trace_host_*, verdict bitmask	—
5.8	Released hosts on rollback	TraceHooks (runner)	sched_trace_rollback_host	—
5.9	Summary per-position counts and limit names	TraceModel, TraceRenderer	POS_SUMMARY entry	—
6.1, 6.2, 6.3	Dispatch outcome with separate limit charges	TraceCapture, TraceRenderer	sched_trace_capture_dispatch	—
6.4	Pend reason set, same content	TraceCapture, TraceRenderer	sched_trace_capture_pend	—
7.1, 7.2	Step-mode inline placement	SimIntegration, ReplTraceControl	xce_repl_check_trace, ..._trace_only	Drain order
7.3, 7.4	Continue-mode echo and no collapse	ReplTraceControl, RollupIntegration	rs->trace.round_traced	Drain order
7.5	Breakpoint ordering	SimIntegration	hook before xce_repl_check_decision	Drain order
7.6, 7.7	Hierarchy, units, jobid format	TraceRenderer	repl_trace_render	—
7.8, 7.9, 7.10, 7.11	info trace job, array, replace, none	TraceStore	repl_cmd_info_trace	—
8.1, 8.2	File copy content; open errors	ReplTraceControl, TraceRenderer	two-sink render	—
8.3, 8.4	dev/inode conflicts both directions	ReplTraceControl, ReplIntegration (set output)	repl_trace_file_conflicts	—
8.5, 8.6	Replaced file kept until session end; shutdown flush	ReplTraceControl	cur/next references, repl_trace_shutdown	Lifecycle
8.7	Nothing to scheduler log	ReplTraceControl, TraceModel	no tlog in trace code	—
9.1, 9.2, 9.3	Line bound, marker, exact counts	TraceModel, TraceRenderer	entry budget, counters	—
9.4	Outcome 50-host cap and aggregates	TraceCapture, TraceRenderer	capped outcome copy	—
9.5, 9.6	64 MiB store, eviction, discarded report	TraceStore	repl_trace_store_put, discarded marks	—
10.1, 10.2, 10.3	No scheduling change	All hooks (read-only), DecisionTransport	equivalence tests	—
10.4, 10.5	REPL output unchanged when off	ReplTraceControl, RollupIntegration, SimIntegration	gates on rs->trace	—
11.1–11.4	Sim-only; single branch in live	TraceHooks, DecisionTransport	trace_ctl/trace_rec NULL in live	—

Components and Interfaces

Component	Domain / Thread	Intent	Requirements	Key Dependencies	Contracts
TraceModel	Core / allocator (build), sim (read)	Record types, arena, entry budget, counters, jobsel predicate	1.2, 1.3, 1.8, 3.3, 3.4, 3.6, 4.x, 5.9, 5.10, 9.1–9.3	libxce containers (P0)	Service, State
TraceCapture	Core / allocator (and sim pre-go window)	Read scheduler state into record entries	2.2, 2.3, 2.5, 3.1, 3.2, 3.5, 4.1, 4.2, 5.4, 5.5, 6.x, 9.4	core resmgr, schedobj limits, job allocs (P0)	Service
TraceHooks	Core / allocator	Gated call sites at the decision seams	2.1, 2.4, 3.x, 4.x, 5.x, 11.2	TraceModel, TraceCapture (P0)	—
DecisionTransport	Core	Carry records on decisions; trace-only op	2.4, 7.2, 10.1, 10.2	decision.h consumers (P0)	State
SimIntegration	Sim driver	Session hooks, reconstructed notices, drain-loop hand-off	1.6, 2.5, 7.1–7.5	repl.h gates (P0)	—
ReplTraceControl	REPL / sim thread	Command, settings lifecycle, inline emission, file copy	1.x, 7.1–7.4, 8.x, 10.4, 10.5	TraceRenderer, TraceStore (P0)	Service, State
TraceStore	REPL / sim thread	Latest record per job; eviction; info trace	1.7, 7.8–7.11, 9.5, 9.6	libxce htab (P1)	Service, State
TraceRenderer	REPL / sim thread	Record to text lines	6.x, 7.6, 7.7, 8.1, 9.2–9.4	repl_fmt helpers (P1)	Service
ReplIntegration	REPL	help, dispatch, info route, set output, rollup rule	1.11, 7.3, 7.4, 8.4, 10.5	existing REPL files (P1)	—

Core Side

TraceModel (include/sched/trace.h, src/sched_trace.c, src/sched_trace_scan.c)

Responsibilities & Constraints

• Owns xce_sched_trace_rec_t: an append-only sequence of typed entries in a per-record arena, plus fixed-size header, selection, counters and outcome blocks. A record is built by exactly one thread at a time and is immutable once attached to a decision.
• Enforces the entry budget (line_bound) over the requirement, cluster-wide check, plan and host sections. When the budget is exhausted it stops appending and counts omitted entries, while counters keep updating (9.2, 9.3).
• Owns verdict classification, the per-position summary tallies (5.9) and the decision-wide host-evaluation counts per verdict kind (9.3).
• No I/O and no tlog. It reads only the arguments passed to it (copying names, task counts, requested resources and plan shapes into the arena). It keeps the scope's obj/w pointers only until the matching scope ends, for TraceCapture's limit lookup.

Contracts: Service [x] / State [x]

Service Interface

typedef enum { XCE_TRACE_LEVEL_SUMMARY = 0, XCE_TRACE_LEVEL_FULL } xce_sched_trace_level_t;

typedef enum {               /* Requirement 5.10 verdict kinds, bit flags */
    XCE_TRACE_V_ACCEPTED   = 1u << 0,
    XCE_TRACE_V_SELECT     = 1u << 1,   /* selection mismatch (pre-filter)      */
    XCE_TRACE_V_SHORTFALL  = 1u << 2,   /* resource shortfall (per-host check)  */
    XCE_TRACE_V_LIMIT      = 1u << 3,   /* limit exceeded (per-host check)      */
    XCE_TRACE_V_CLAIMED    = 1u << 4,   /* already claimed by earlier position  */
    XCE_TRACE_V_INTERNAL   = 1u << 5    /* not evaluated: internal error        */
} xce_sched_trace_verdict_t;

typedef enum {                /* Requirement 2.4 / 2.5 record kinds */
    XCE_TRACE_REC_DECISION = 0,
    XCE_TRACE_REC_NOT_FOUND,
    XCE_TRACE_REC_NO_SCHEDOBJ,
    XCE_TRACE_REC_INTERNAL_ERROR,  /* alloc_rc < 0 or decision build failure */
    XCE_TRACE_REC_PEND_NO_REASON,
    XCE_TRACE_REC_RECONSTRUCTED
} xce_sched_trace_rec_kind_t;

typedef struct xce_sched_trace_sel_s {   /* one selector from `trace on job ...` */
    uint64_t key;        /* plain jobid, array base id, or element id */
    bool     whole_array;/* true: key is an array base, matches every element */
} xce_sched_trace_sel_t;

typedef struct xce_sched_trace_ctl_s {   /* immutable during a session except position */
    bool                     all;
    int                      nsel;
    xce_sched_trace_sel_t*   sel;
    xce_sched_trace_level_t  level;
    int                      line_bound;     /* 50..100000 */
    uint64_t                 session_no;     /* rs->session_count of this session */
    const char*              policy_name;    /* picker name, owned by the ctl */
    int                      position;       /* allocator-owned, reset at session begin */
} xce_sched_trace_ctl_t;

typedef struct xce_sched_trace_rec_s xce_sched_trace_rec_t;   /* opaque outside sched_trace*.c / renderer */

#define XCE_TRACE_LINE_BOUND_DEFAULT 1000      /* Requirement 9.1 */
#define XCE_TRACE_LINE_BOUND_MIN     50
#define XCE_TRACE_LINE_BOUND_MAX     100000
#define XCE_TRACE_OUTCOME_HOSTS      50        /* Requirement 9.4 */

typedef enum {
    XCE_TRACE_PATH_SINGLE,                     /* PRIMARY job: the single requirement      */
    XCE_TRACE_PATH_COMPOUND,                   /* top-level COMPOUND container             */
    XCE_TRACE_PATH_ALT,                        /* alternative i/n of an ALTER (a compound) */
    XCE_TRACE_PATH_MEMBER                      /* member j/m of the enclosing compound     */
} xce_sched_trace_path_kind_t;
typedef struct {                               /* kept inside the record, not passed by hooks */
    int depth;                                 /* 0..2: container (COMPOUND|ALT) > MEMBER, or SINGLE */
    struct { xce_sched_trace_path_kind_t kind; int i; int n; } lvl[2];
} sched_trace_path_t;

typedef enum {                                 /* Requirement 4.7 reasons */
    XCE_TRACE_NOPLAN_DECLINED,                 /* allocator accepts() declined */
    XCE_TRACE_NOPLAN_GENERATOR_FAILED,         /* plan_init failed */
    XCE_TRACE_NOPLAN_NONE_GENERATED            /* generator produced no plan */
} xce_sched_trace_noplan_t;

typedef enum {
    XCE_TRACE_POS_FILLED,
    XCE_TRACE_POS_UNFILLED,
    XCE_TRACE_POS_SCAN_ERROR,                  /* host scan could not start or stopped (4.4) */
    XCE_TRACE_POS_INTERNAL                     /* host accepted but claim snapshot failed */
} xce_sched_trace_pos_end_t;

/* gate: evaluates `call` only when a record is being built */
#define SCHED_TRACE(core, call) \
    do { if ((core)->trace_rec != NULL) { call; } } while (0)

bool  sched_trace_ctl_wants(const xce_sched_trace_ctl_t* ctl, uint64_t jobid);
xce_sched_trace_rec_t* sched_trace_rec_new(const xce_sched_trace_ctl_t* ctl, uint64_t jobid,
                                           xce_sched_trace_rec_kind_t kind);
void   sched_trace_rec_free(xce_sched_trace_rec_t* rec);           /* NULL-safe */
/* finish a record as a no-decision kind (NOT_FOUND, NO_SCHEDOBJ, INTERNAL_ERROR,
 * PEND_NO_REASON) before it is pushed via make_trace_decision (2.4) */
void   sched_trace_rec_finish(xce_sched_trace_rec_t* rec, xce_sched_trace_rec_kind_t kind, int err);
size_t sched_trace_rec_bytes(const xce_sched_trace_rec_t* rec);    /* arena + blocks */

/* structural entries (each = one rendered line in the budgeted sections) */
void sched_trace_path_push(xce_sched_trace_rec_t*, xce_sched_trace_path_kind_t kind, int i, int n);
void sched_trace_path_pop(xce_sched_trace_rec_t*);
/* begin/end of the requirement named by the current path; exactly one end per begin */
void sched_trace_resreq_begin(xce_sched_trace_rec_t*,
                              const xce_primary_resreq_t* primary /* NULL for a container */,
                              int nmembers /* container: resreq->primaries.count; else 0 */);
void sched_trace_resreq_end(xce_sched_trace_rec_t*, int rc, int not_tried_after);
/* non-scope entries */
void sched_trace_member_failed(xce_sched_trace_rec_t*, int j, int m, int err); /* merge failure after member j of m */
void sched_trace_alt_result(xce_sched_trace_rec_t*, int chosen_i /* 0 = none */, int n);
void sched_trace_alt_skipped(xce_sched_trace_rec_t*, int i, int n);
void sched_trace_check_begin(xce_sched_trace_rec_t*, const xce_sched_obj_t* obj); /* cluster-wide scope */
void sched_trace_check_end(xce_sched_trace_rec_t*, int rc, bool plans_follow);
void sched_trace_no_plan(xce_sched_trace_rec_t*, xce_sched_trace_noplan_t why);
void sched_trace_plan_begin(xce_sched_trace_rec_t*, const xce_alloc_plan_t* plan);
void sched_trace_plan_discarded(xce_sched_trace_rec_t*, const xce_alloc_plan_t* plan);
void sched_trace_plan_not_tried(xce_sched_trace_rec_t*, const xce_alloc_plan_t* plan, int err);
void sched_trace_plan_end(xce_sched_trace_rec_t*, int rc, int failed_pos);
void sched_trace_pos_begin(xce_sched_trace_rec_t*, int pos, int slots);
void sched_trace_pos_end(xce_sched_trace_rec_t*, xce_sched_trace_pos_end_t how, const char* host);
void sched_trace_host_begin(xce_sched_trace_rec_t*, const xce_sched_obj_t* obj,
                            const xce_sched_worker_t* w);              /* per-host scope */
void sched_trace_pos_scan_error(xce_sched_trace_rec_t*, int err);    /* host scan stopped */
void sched_trace_host_verdict(xce_sched_trace_rec_t*, unsigned verdicts, int err);
/* added in implementation (Implementation Notes 1.5, 2.2): */
void sched_trace_row(xce_sched_trace_rec_t*, const sched_trace_row_t* row);  /* CHECK_ROW or held HOST_ROW */
void sched_trace_host_limit_hit(xce_sched_trace_rec_t*, const char* limit, const char* scope); /* summary tally */
void sched_trace_rollback_host(xce_sched_trace_rec_t*, const char* host);   /* one call per released claim */
void sched_trace_commit(xce_sched_trace_rec_t*);   /* committed plan = the last plan_begin */
/* end of sched_alloc_search_: one budgeted line; also updates decision-wide counters */
void sched_trace_search_end(xce_sched_trace_rec_t*, int rc);   /* counts come from the model (see below) */
void sched_trace_finalize_failed(xce_sched_trace_rec_t*, int err);

• Preconditions: rec != NULL for every call (guaranteed by SCHED_TRACE). Calls nest in the order the allocator executes them. Every path_push has exactly one path_pop, and every *_begin has exactly one *_end, on every return path at the call site. Outcome corrections that occur after an *_end (a merge failure, an alternative's selection) are separate non-scope entries, never a second *_end.
• Postconditions:
  • Entries beyond line_bound are not stored; omitted is incremented by the number of lines they would have produced.
  • In XCE_TRACE_LEVEL_SUMMARY, host entries are not stored. Instead pos_end stores one position-summary entry: accepted host, count per verdict kind, and (limit name, scope, count) tallies.
  • Counters (plans_tried, plans_discarded, host_evals[kind]) are exact regardless of truncation.
• Invariants: a record never references scheduler-owned memory. All strings and numbers are copied into its arena.

State Management

• Record states: building (allocator) → attached (on decision) → delivered (REPL) → stored or freed.
• Concurrency: no locks. Ownership moves with the decision through the core->decisions cbuf.

Implementation Notes

• Integration: the record keeps the current sched_trace_path_t. Callers push the container level (SINGLE, COMPOUND 1/1, or ALT i/n) before calling into the allocation chain, and sched_make_compound_alloc_ pushes MEMBER j/m per member. Each resreq_begin stores the full current path, so the renderer prints labels such as alternative 2/3 > member 1/2 (3.1) without the compound function needing to know whether it serves an ALTER.
• Validation: test_trace_model.c covers budget edges (bound-1, bound, bound+1), summary tallies with multi-kind hosts, counters under truncation, and ctl_wants for plain, element and array selectors.
• Risk: arena growth uses malloc on the allocator thread. This happens only for traced jobs, never in live.

TraceCapture (src/sched_trace_capture.c, src/sched_trace_outcome.c)

Responsibilities & Constraints

• Reads scheduler state on the thread that owns it at the moment of capture, and writes entries through TraceModel:
  • sched_trace_capture_selection(rec, core, job): policy name, queue name and priority, job priority, queue_time (2.2).
  • sched_trace_capture_glimits(rec, obj): each xce_vres_t in obj->glimits as (limit name, scope instance) (2.3).
  • sched_trace_check_rows(rec, core, nres, idxs, total, inuse, uncomitted, wanted): called from unsatisfy_.
    • For each row it resolves xce_resource_t through vget(core->resmgr, idx).
    • A row whose flags include XCE_RESF_LIMIT becomes a limit row. Its limit name comes from the matching xce_vres_t in the current scope's limit list: obj->glimits for the cluster-wide scope, obj->hlimits[w->idx] for a host scope. The scope instance is the vres name with its limit=<name> prefix removed, rendered as u=..,q=..,p=... In a host scope it is followed by h=<w->name>, because host-limit vres names do not encode the host. When no scope pointer exists (reconstructed records, Requirement 2.5, captured on the sim thread), the limit name and scope are parsed from the XCE_RESF_LIMIT resource's own name (limit=<name>/...). The value, usage, reserved and requested amounts are total, inuse, uncomitted and wanted.
    • Every other row is a resource row.
    • In the cluster-wide scope every row is kept (4.1). In a host scope only unmet rows are kept, and they set SHORTFALL and/or LIMIT on the host's verdict (5.4, 5.5).
  • sched_trace_capture_attempt_reasons(rec, reason): per-code counts and the hosts/resources named, from a compound attempt's ctx.reason. It is captured before scheduler.c replaces obj->reason and renders as the failure-reasons line of a failed member or alternative (3.5).
  • sched_trace_capture_dispatch(rec, core, obj, alloc): per-host allocation rows and shared rows, with limit charges separated by XCE_RESF_LIMIT (6.1–6.3).
    • Allocation rows carry only a resource name (xce_alloc_t.resname). Each row is classified as a limit charge when its resource, looked up by name in core->resmgr, has XCE_RESF_LIMIT; when the name is not registered in this replay, the limit= name prefix decides.
    • A limit charge is resolved to its xce_vres_t by matching resname against vres->vres->name. Shared charges are searched in obj->glimits. Host charges are searched in obj->hlimits[w->idx], where w is found by the allocation's host name.
    • The vres supplies the limit name, scope and limited resource name. The first 50 hosts are stored in detail. The rest are aggregated into per-resource totals (excluding limits) and per-limit totals, plus an omitted-host count (9.4).
  • sched_trace_capture_pend(rec, reason): a deep copy of the decision's xce_pendreason_t. main_reason and host-less entries are always kept. Host entries of the first 50 distinct hosts are kept, and the rest are counted per pend-reason code with an omitted-host count (6.4, 9.4).
  • sched_trace_rec_reconstructed(ctl, core, jobid, obj, allocs): builds a RECONSTRUCTED record with a dispatch outcome (2.5), resolving limit charges the same way.
    • A job never evaluated by this replay has empty glimits/hlimits, or a NULL obj. A limit charge with no matching vres is still listed as a limit charge (by its XCE_RESF_LIMIT flag), with its name and scope parsed from the vres resource name.
    • Its amount is then shown as a raw number, because the limited resource is unknown. This is the only case where a limit amount is not unit-formatted.
• Constraint: inputs are read-only. No function here calls a scheduler API that mutates state (Requirement 10).

Contracts: Service [x]

/* declared in include/sched/trace.h; implemented in sched_trace_capture.c
 * (selection, cluster-wide limits, check rows, attempt reasons) and
 * sched_trace_outcome.c (dispatch, pend, reconstructed record) */
void sched_trace_capture_selection(xce_sched_trace_rec_t*, const xce_sched_core_t* core,
                                   const xce_sched_job_t* job);
void sched_trace_capture_glimits(xce_sched_trace_rec_t*, const xce_sched_obj_t* obj);
void sched_trace_check_rows(xce_sched_trace_rec_t*, xce_sched_core_t* core, int nres,
                            const int* idxs, const float* total, const float* inuse,
                            const xvec_t* uncomitted, const float* wanted);
void sched_trace_capture_attempt_reasons(xce_sched_trace_rec_t*, const xce_pendreason_t* reason);
void sched_trace_capture_dispatch(xce_sched_trace_rec_t*, xce_sched_core_t* core,
                                  const xce_sched_obj_t* obj, const xce_job_alloc_t* alloc);
void sched_trace_capture_pend(xce_sched_trace_rec_t*, const xce_pendreason_t* reason);
xce_sched_trace_rec_t* sched_trace_rec_reconstructed(const xce_sched_trace_ctl_t* ctl,
                                                     xce_sched_core_t* core, uint64_t jobid,
                                                     const xce_sched_obj_t* obj /* job->schedobj, nullable */,
                                                     const xce_job_alloc_t* allocs);

core is non-const only because vget(core->resmgr, ...) takes a non-const container; the functions never write to it.

Implementation Notes

• Integration: the current scope's schedobj and worker are remembered by the record when check_begin / host_begin run (the runner and the host iterator pass ctx->schedobj and the candidate worker). unsatisfy_ therefore needs only core.
• Per-host verdict when sched_try_alloc_on_host_ returns 0: it is SHORTFALL and/or LIMIT, from the unmet rows recorded during the call. If no row was recorded, the 0 came from the function's own selection branch (PREASON_MATCH_FAIL), so the verdict is SELECT in the per-host phase.
• Validation: test_trace_capture.c uses a fixture core with a limit vres on a shared and a host resource to check:
  • limit rows are never reported as shortfalls;
  • scope strings are correct;
  • dispatch limit charges are listed separately;
  • the 51-host aggregation;
  • the pend copy equals the source under job_preason_eq when there are 50 or fewer hosts.
• Risk: the vres lookup is linear in the number of matched limits per job. That number is small (configured limits), and it is paid only for traced jobs.

TraceHooks (call sites)

Each hook is one SCHED_TRACE(core, ...) statement placed after the value it reports has been computed. It does not change control flow, return values or arguments to existing calls.

Seam	File:anchor	Hook(s)	Requirements
Allocator per-job prologue	sched_election.c after cbuf_get (jobid ≠ 0, not RECONCILE_WAKE)	if (core->trace_ctl) { ctl->position++; if (wants) core->trace_rec = rec_new(...) } (the only non-macro hook; one branch in live)	2.1, 11.2
Job not found / no schedobj	sched_election.c:226-241	NOT_FOUND: no selection capture (job == NULL). NO_SCHEDOBJ: capture_selection without limits. Then rec_finish, push the trace decision and clear trace_rec	2.4
After election	after plugin_job_election	capture_selection, capture_glimits	2.2, 2.3
Alloc error / pend without reason / decision build failure	sched_election.c:250-254, 265-269, 286	rec_finish(INTERNAL_ERROR / PEND_NO_REASON, err), push the trace decision, and clear trace_rec before the continue. Every path that hands a record away clears trace_rec, so no later job in the session can append to a record the REPL owns.	2.4
Decision built	before sched_commit_decision	capture_dispatch(core, job->schedobj, alloc) / capture_pend; d->trace = rec; clear trace_rec	6.x, 9.4
Session-end marker	jobid 0	core->trace_rec = NULL (defensive)	—
PRIMARY job	scheduler.c sched_make_allocation, PRIMARY case	path_push(SINGLE,1,1); resreq_begin(primary, 0); call; resreq_end(rc, 0); path_pop	3.1, 3.2
Top-level COMPOUND	sched_make_allocation, COMPOUND case, around the sched_make_compound_allocation call	path_push(COMPOUND,1,1); resreq_begin(NULL, resreq->primaries.count); call; resreq_end(rc, 0); path_pop	3.1, 3.2
ALTER loop	scheduler.c:409-436, per alternative i of n	NULL resreq → alt_skipped(i,n). Otherwise path_push(ALT,i,n); resreq_begin(NULL, alt->resreq->primaries.count); call sched_make_compound_alloc_; resreq_end(rc, 0); path_pop. On the first success, alt_result(i,n) renders "alternative i/n chosen; alternatives i+1..n not tried" (3.4). If the loop ends with none chosen, alt_result(0,n) renders "all alternatives failed" (3.5).	3.4–3.6
Compound members	sched_make_compound_alloc_ per member j of m	path_push(MEMBER,j,m); resreq_begin(primary, 0); call; resreq_end(rc, rc<=0 ? m-j : 0); path_pop. Here m-j is the number of members after j, with j 1-based. If merge_job_alloc then fails (scheduler.c:339-401), member_failed(j, m, mrc) renders "member j/m: merge failed (internal error); members j+1..m not tried" (3.3). At the end of a failed attempt, capture_attempt_reasons(ctx.reason) records the failure reasons before obj->reason is replaced (3.5).	3.1–3.3, 3.5
Runner entry	alloc_runner.c accepts decline	no_plan(DECLINED)	4.7
Cluster-wide check	around sched_alloc_prepare_shared_	check_begin / check_end(rc, plans_follow=rc>0) → on failure renders "no plans tried"	4.1, 4.2, 4.8
Generator start failure	plan_init != 0	no_plan(GENERATOR_FAILED)	4.7
Search end	end of sched_alloc_search_	search_end(rc). The model counts generated, tried and discarded plans per requirement scope from the plan_begin / plan_discarded / plan_not_tried calls it already receives, so the runner keeps no extra bookkeeping outside the gate (11.2). It always renders as one line, plans: g generated, t tried, d discarded, x not tried, extended on the same line by at most one statement:

• ; no plan generated (NONE_GENERATED, 4.7) when g = 0;
• ; no plan tried: all generated plans discarded (4.9) when g > 0 and d = g;
• ; no plan could be placed (4.6) when rc = 0, t > 0 and no plan ended in an internal error.

When rc > 0 or rc < 0 there is no statement; the commit or internal-error entries already say what happened. A declined requirement and a generator that cannot start end the runner before the search end: their no_plan entry is the requirement's "no plan generated" line and bumps that counter itself, once; a search end after a no_plan entry in the same requirement makes no statement and counts nothing. Like every other entry it counts against the line budget. It also bumps the matching decision-wide counter (requirements with no plan generated / all discarded / none placed), under the same conditions, which appear in the fixed-size totals block after truncation (9.3). | 4.6, 4.7, 4.9 | | Plan loop | malformed / reserve failure / try / result | plan_discarded; plan_not_tried(err); plan_begin; plan_end(rc, failed_pos) | 4.3, 4.4, 4.6, 4.9 | | Position | sched_claim_bucket_ | pos_begin; iterator init failure → pos_end(SCAN_ERROR); snapshot failure after acceptance (alloc_runner.c:237-243) → pos_end(INTERNAL); else pos_end(FILLED/UNFILLED/SCAN_ERROR, host) | 4.4, 5.9 | | Candidate | claimed skip / per-host result | host_begin + host_verdict(CLAIMED); host_begin, call, host_verdict(rc>0 ? ACCEPTED : rc==0 ? from rows : INTERNAL, err) | 5.1–5.7 | | Pre-filter | host_iter.c:136-156 | host_begin + host_verdict(SELECT); uncomit error → host_verdict(INTERNAL) | 5.2, 5.3, 5.7 | | Scan stop | host_iter.c:124-127 (rc < 0) | pos_scan_error(rc); the runner's following pos_end(UNFILLED) is recorded as SCAN_ERROR | 4.4 | | Rollback | in sched_try_plan_ after a failed position, only when n > 0 (not inside sched_rollback_claims_, which is also called with n == 0 and from finalize's partial rollbacks) | SCHED_TRACE(core, for (k = 0; k < n; k++) sched_trace_rollback_host(core->trace_rec, claims[k].w->name)), a single gated statement. The model merges consecutive calls into one ROLLBACK entry, "released hosts: n01, n02" (one line). | 5.8 | | Finalize | sched_finalize_job_alloc_ result | commit or finalize_failed(err) | 4.5 | | Shortfall rows | resource.c top of unsatisfy_ | check_rows(...) | 4.1, 4.2, 5.4, 5.5 |

• The "pre-filter" phase covers verdicts produced inside host_iter.c. The "per-host" phase covers verdicts produced by sched_try_alloc_on_host_ (5.2).
• The "not performed" rendering (5.1) follows from phase order: a pre-filter rejection implies the per-host check was not performed.
• ALREADY CLAIMED and INTERNAL carry no phase label (5.2 applies only to the selection, shortfall and limit kinds).

DecisionTransport (include/decision.h, src/decision.c)

Contracts: State [x]

• xce_decision_op_t gains XCE_DECISION_TRACE after XCE_DECISION_PREEMPT. Existing values are unchanged.
• xce_decision_t gains struct xce_sched_trace_rec_s* trace;. It is NULL for every existing constructor (calloc).
• xce_decision_t* make_trace_decision(xce_sched_trace_rec_t* rec) returns a decision with op = XCE_DECISION_TRACE, a zeroed union and trace = rec.
• Ownership:
  • The drain loop that receives a decision owns d->trace and hands it to the REPL (xce_repl_check_trace*), which stores it or frees it.
  • The REPL takes ownership by setting d->trace = NULL. Every sim call site (drain loop, trace-only branch, reconstructed block) then calls sched_trace_rec_free(d->trace) right before free(d). This frees the record only if the REPL did not take it, for example when !sim->interactive, which cannot happen because trace_ctl is only set when interactive.
  • driver.c frees any non-NULL d->trace and skips XCE_DECISION_TRACE. Both are unreachable in live; this is defensive.
• Coordination: scheduler-preemption also extends decision.h. Both specs append only, and neither reorders.

Simulator Side

SimIntegration (sim_loop.c, repl.h gates)

• xce_repl_trace_session_begin(sim) runs after xce_core_begin_session has succeeded, which means after its rc < 0 check (sim_loop.c:1620-1631), whose failure branch continues without draining. It runs before reconciliation. A failed session begin therefore publishes nothing and holds no settings reference. It resets rs->trace.round_traced and rs->trace.round_flushed to false in every mode, so a round that ended in step mode (for example continue → breakpoint → step) cannot leak the flag into the next continue-mode round. It then sets cur = next, then sets ctl->session_no = rs->session_count and ctl->position = 0, and publishes sim->core.trace_ctl = cur ? &cur->ctl : NULL.
• In the reconstructed-dispatch report block (sim_loop.c:1309-1329), when trace_ctl wants the job, the block builds a RECONSTRUCTED record and sets d->trace = rec right after make_dispatch_decision, through the gate xce_repl_trace_reconstructed(sim, d, job) (repl_trace_on_reconstructed in repl_trace_emit.c). The order is then: record add → sim_print_decision_gate → xce_repl_check_trace(sim, d) → xce_repl_check_decision(sim, d) → sched_trace_rec_free(d->trace) → free(d). This is the same order as the drain loop, so 7.5 holds for reconstructed dispatches too (2.5). This runs before signal_go, so these traces precede the round's live decisions (7.3).
• Drain loop:
  • if (d->op == XCE_DECISION_TRACE) { xce_repl_check_trace_only(sim, d); sched_trace_rec_free(d->trace); free(d); continue; } comes first.
  • xce_repl_check_trace(sim, d) sits between sim_print_decision_gate and xce_repl_check_decision. It is a no-op when d->trace == NULL. sched_trace_rec_free(d->trace) is called just before the existing free(d).
• xce_repl_trace_session_end(sim) runs after the drain loop, before plugin_end_session. It sets sim->core.trace_ctl = NULL, drops cur (closing its file if it is no longer next), and clears the in-progress flag.

ReplTraceControl (src/repl_trace.c, src/repl_trace_emit.c)

repl_trace.c holds the command, status, settings, file and shutdown code; repl_trace_emit.c holds the session hooks and inline emission.

Responsibilities & Constraints

• Parses the raw input line for trace (1.9, 1.12):

  trace on all [level summary|full] [lines N] [file PATH]
  trace on job JOBSEL [JOBSEL ...] [level summary|full] [lines N] [file PATH]
  trace off
  trace status
  

  • JOBSEL is parse_jobsel syntax (job, job[idx], array base).
  • file must be the last option. PATH is the rest of the line, trimmed, with one optional pair of surrounding quotes removed, and must be at most 4095 bytes.
  • On any invalid argument, it prints [sim] trace: invalid <what>: <arg> and changes nothing.
• Keeps settings in rs->trace as reference-counted settings objects (see Trace settings lifecycle):
  • next: the settings the next session will use, or NULL when tracing is off.
  • cur: the settings of the session in progress, or NULL.
  • session_in_progress and round_traced.
• File rules:
  • The file is opened at command time in append mode, so open errors are reported immediately (8.2).
  • Before opening, the path is compared by stat() dev/inode against fstat(fileno(sim->record_file)) when an output file is set, and against stat(sim->log_path) (8.3).
  • Replacing or disabling next drops its reference. If cur still holds that object, its file stays open until the session-end hook drops cur; otherwise the file is closed immediately (8.5).
• repl_trace_file_conflicts(sim, path) returns true when path matches, by dev/inode, the file of next or cur. These are every trace file still open. set output rejects such a path (8.4).
• Inline emission (xce_repl_check_trace, xce_repl_check_trace_only) runs on the sim thread:
  1. In any mode it sets round_traced = true, so a round traced in step mode and finished with continue is never collapsed (7.4). In continue mode only: if this is the round's first traced item, it calls repl_round_collapse_flush(sim); then, for every traced decision (not trace-only), it prints the echo line that the print gate suppressed, by calling sim_print_decision(d). That function is made non-static and declared in sim.h, with its format unchanged (7.3). A per-round round_flushed bit, also reset at session begin, records whether the flush has been done.
  2. Renders the record to stdout and, if cur (the settings the record was captured under) has a file, to that file, then fflushes both (7.1, 7.2, 8.1, 8.6).
  3. Hands the record to the TraceStore (7.10).
• repl_trace_shutdown(sim) is called from repl_cmd_quit, the EOF path and process exit. It flushes and closes the files of cur and next (8.6).
• Writes nothing through tlog (8.7).

Contracts: Service [x] / State [x]

/* repl_trace.c (declared in repl_internal.h) */
bool  repl_trace_cmd(xce_sched_sim_driver_t* sim, const char* raw_line);   /* returns false: re-prompt */
FILE* repl_trace_settings_file(const xce_repl_trace_settings_t* s);        /* NULL: no file */
bool  repl_trace_file_conflicts(const xce_sched_sim_driver_t* sim, const char* path);
void  repl_trace_shutdown(xce_sched_sim_driver_t* sim);   /* clears trace_ctl, releases cur/next, store_fini */
/* repl_trace_emit.c (declared in repl.h, called through its interactive-only gates) */
void repl_trace_session_begin(xce_sched_sim_driver_t* sim);
void repl_trace_session_end(xce_sched_sim_driver_t* sim);
void repl_trace_on_decision(xce_sched_sim_driver_t* sim, xce_decision_t* d);   /* takes d->trace */
void repl_trace_on_trace_only(xce_sched_sim_driver_t* sim, xce_decision_t* d);
void repl_trace_on_reconstructed(xce_sched_sim_driver_t* sim, xce_decision_t* d,
                                 const xce_sched_job_t* job);   /* attaches a RECONSTRUCTED record (2.5) */

trace status output (1.7) shows a block for cur ("in effect", when a session is in progress) or next ("next session", otherwise), plus a second "replaces at next session" block when next differs from cur. Each block has enabled, selection (rendered with repl_fmt_jobid_range), level, line bound and file. A final line gives stored-trace memory used / 64 MiB.

TraceStore (src/repl_trace_store.c)

Responsibilities & Constraints

• An htab keyed by jobid maps to {rec, bytes, seq}. A capture-order FIFO supports eviction, and a set of discarded jobids (with their seq) supports 9.6.
• put(rec) replaces any existing entry for the jobid and frees the old record (7.10). It then evicts the oldest entries until total_bytes <= 64 MiB, never evicting the newest; evicted jobids are marked discarded (9.5). A newer capture clears the discarded mark.
• info trace <jobsel>:
  • Element or plain job: renders the stored record at its captured level with the current units mode (7.8). If the trace was discarded it reports trace discarded (memory bound) (9.6). Otherwise it reports no trace available with the job's current state, or unknown job (7.11).
  • Array base: iterates the stored and discarded entries whose array_jobid matches, in index order. If there are none, it lists the elements' current states from core->jobs (7.9).
• The store is only read and written on the sim thread. It never reads live scheduler state except job state for 7.9 and 7.11, at the REPL prompt.

Contracts: Service [x] / State [x]

void   repl_trace_store_put(xce_repl_trace_state_t* ts, xce_sched_trace_rec_t* rec);
size_t repl_trace_store_bytes(const xce_repl_trace_state_t* ts);
void   repl_trace_store_fini(xce_repl_trace_state_t* ts);   /* frees all records; called by repl_trace_shutdown */
bool   repl_cmd_info_trace(xce_sched_sim_driver_t* sim, char tokens[][XCE_REPL_TOKEN_LEN], int ntok);

/* repl_internal.h: the state block embedded in xce_sched_repl_state_t */
typedef struct xce_repl_trace_settings_s xce_repl_trace_settings_t;   /* refcounted; holds ctl + file */
typedef struct {
    xce_repl_trace_settings_t* next;         /* settings the next session will use, NULL = off */
    xce_repl_trace_settings_t* cur;          /* settings of the session in progress, NULL = none */
    bool                       session_in_progress;
    bool                       round_traced;
    bool                       round_flushed;
    htab_t                     by_job;       /* jobid -> stored entry */
    htab_t                     discarded;    /* jobid -> eviction seq */
    /* capture-order FIFO head/tail, total_bytes, seq counter */
} xce_repl_trace_state_t;

TraceRenderer (src/repl_trace_render.c, src/repl_trace_render_outcome.c, private src/repl_trace_render.h)

Responsibilities & Constraints

• repl_trace_render(const xce_sched_trace_rec_t* rec, xce_repl_units_mode_t units, FILE* out1, FILE* out2 /* nullable */).
• Line contract: every stored entry in the budgeted sections produces exactly one line. Header, selection and outcome lines are outside the budget.
• Hierarchy (7.6), two-space indent per nesting level. The example below shows every stored entry: each line from the first requirement line to the last alternative line is exactly one entry and counts against the line budget. The header, selection and outcome lines are outside the budget.

  [trace] job 1-0-0-42  session 7  position 3  level full
    selection: policy builtin_fcfs  queue normal (priority 50)  job priority 0  queued 2026-10-07 10:00:01
    cluster-wide limits: lim_user (u=alice); host-scoped limits are reported with each host's verdict
    requirement alternative 1/2 (compound, 1 member)
      requirement alternative 1/2 > member 1/1  tasks 4..4  requests mem=8 GiB slots=1 license=2
        cluster-wide: license total 10 in-use 9 reserved 0 requested 2  failed
        cluster-wide check failed; no placement plans tried
      member 1/1 not satisfied
      failure reasons: NO_RESOURCE x1 (resources: license)
    alternative 1/2 not satisfied
    requirement alternative 2/2 (compound, 1 member)
      requirement alternative 2/2 > member 1/1  tasks 4..4  requests mem=8 GiB slots=1 license=1
        cluster-wide: license total 10 in-use 9 reserved 0 requested 1  passed
        cluster-wide limit lim_user (u=alice) on slots: value 100 usage 96 reserved 0 requested 4  passed
        cluster-wide check passed
        plan 1  hosts 1  tasks per position [4]
          position 1 (4 tasks)
            host n01  rejected (per-host check): resource shortfall
              mem total 64 GiB in-use 60 GiB reserved 0 requested 8 GiB
            host n02  rejected (pre-filter): selection mismatch; per-host check not performed
            host n03  accepted
          position 1 filled by n03
        plan 1 succeeded
        plan 1 committed; no further plans tried
        plans: 1 generated, 1 tried, 0 discarded, 0 not tried
      member 1/1 satisfied
    alternative 2/2 satisfied
    alternative 2/2 chosen; no alternatives after it
    outcome: DISPATCH
      host n03: mem=8 GiB slots=4  limit charges: lim_host (h=n03) slots 4
      shared: license=1  limit charges: lim_user (u=alice) slots 4
  

• Summary level: the position line is followed by accepted n03; rejected: selection mismatch 1, resource shortfall 1, limit exceeded 0 (...) and no per-host lines (5.9).
• Truncation (9.2, 9.3): after the last stored entry, the marker ... <k> trace lines omitted (line bound N), then a fixed-size totals block: plans tried, plans discarded, requirements with no plan generated / all plans discarded / no plan placed, and host evaluations per verdict kind.
• Outcome (6.x, 9.4): up to 50 hosts with all their entries, then aggregates and ... <k> hosts omitted. A pend lists main_reason, host-less entries, then host entries.
• Values use the moved repl_fmt helpers with the units mode passed in (7.7). Jobids use fmt_jobid and selection lists use repl_fmt_jobid_range.
• No-decision records (2.4): a header, then:
  • NOT_FOUND: selection: ordering keys and limits unavailable (job not found by the allocator).
  • NO_SCHEDOBJ: the selection line, then cluster-wide limits: unavailable (no scheduling object).
  • INTERNAL_ERROR and PEND_NO_REASON: the selection and limit lines, then the full entry hierarchy the hooks captured. Allocator internal errors that end the attempt (a plan not tried, a snapshot failure, a finalize failure, an error in the cluster-wide check, a member merge failure) surface as alloc_rc < 0 for PRIMARY and top-level COMPOUND jobs. Their entries therefore carry the 3.2 / 4.4 / 4.5 outcomes. Per-host errors (5.7) and host-scan errors (4.4) do not end the attempt; they appear as host and position entries in any record kind.
  • Every kind ends with one line in place of the outcome, naming the kind and the error.
• Reconstructed records (2.5): a header, one line stating the dispatch was reconstructed from the recorded allocation, and the outcome.

Contracts: Service [x]

ReplIntegration (existing REPL files)

Summary only; no new boundaries.

• repl.c:

  • repl_dispatch routes trace to repl_trace_cmd(sim, line).
  • repl_print_help adds a trailing "Trace" section (trace on all|job ..., trace off, trace status) and an info trace <jobid> line in State Inspection (1.11).
  • repl_cmd_quit calls repl_trace_shutdown.

• repl_info.c: repl_cmd_info routes tokens[1] == "trace" to repl_cmd_info_trace.

• repl_params.c: set output returns an error [sim] set output: <path> is an open trace file when repl_trace_file_conflicts is true (8.4, 10.5).

• repl_rollup.c: in repl_round_rollup_process, the traced-round rule runs right after repl_query_pend_jobids has built this round's pend list. When rs->trace.round_traced is set:

  1. call repl_round_collapse_flush(sim) (idempotent, and covers rounds that were traced only in step mode);
  2. emit this round alone via repl_round_emit(round_no, round_no, round_ts, round_ts, dispatch_count, preempt_count, pend_jobids, n_pend_jobids, idle) with idle = (dispatch_count == 0) && (preempt_count == 0) && (n_pend_jobids == 0). This idle matches repl_round_collapse_flush (repl_rollup.c:251-253), not the allocation-failure fallback, so a pend-only traced round is not reported as idle;
  3. free the pend list and return without starting a range (7.3, 7.4).

  If the pend query itself fails, the existing fallback already emits the round alone. round_traced is reset at the next session begin.

Data Models

Domain Model

• TraceSettings (REPL-owned, reference-counted): all, selectors, level, line bound, file (path + FILE* + dev/inode), and the xce_sched_trace_ctl_t it publishes. It is referenced by next and/or cur (see Trace settings lifecycle).
• TraceControlBlock (xce_sched_trace_ctl_t, REPL-owned, published to the core for one session): an immutable copy of the selection, level and bound, plus session_no, policy_name and the allocator-owned position.
• TraceRecord (aggregate root, one per traced selection):
  • Header: jobid, kind, level, session_no, position, line_bound. (The planned "file tag of the settings it was captured under" was dropped in implementation, Implementation Note 4.2: info trace is console-only (8.1) and emission writes through the session's cur settings, so a record needs no file reference.)
  • Selection block: policy, queue name and priority, job priority, queue_time, cluster-wide limits list.
  • Entries: an ordered list. Each entry has a type, a depth, and a payload of copied strings and floats with resource flags.
  • Counters: plans_tried, plans_discarded, reqs_none_generated, reqs_all_discarded, reqs_none_placed, host_evals[6], omitted_lines.
  • Outcome: a dispatch block (≤ 50 hosts in detail plus aggregates) or a pend block (≤ 50 hosts plus per-code counts), or none.
• Invariants:
  • A record is self-contained (no pointers into scheduler memory).
  • stored entries ≤ line_bound.
  • Counters are exact.

Logical Data Model

• Entry types (each renders as exactly one line):
  • RESREQ_BEGIN, RESREQ_END, ALT_SKIPPED, ALT_RESULT, MEMBER_FAILED, ATTEMPT_REASONS
  • CHECK_ROW (resource or limit), CHECK_END (passed / failed, with "no placement plans tried" on failure), NO_PLAN
  • PLAN_BEGIN, PLAN_DISCARDED, PLAN_NOT_TRIED, PLAN_END, COMMIT, FINALIZE_FAILED, SEARCH_END
  • POS_BEGIN, POS_SUMMARY (summary level), POS_END (filled / unfilled / scan error / internal; the scan error is set by pos_scan_error)
  • HOST (verdict bitmask, phase, error), HOST_ROW (shortfall or limit row under a host)
  • ROLLBACK
• Row payload: resource name, flags, valtype, total, in-use, reserved, requested, met. Limit rows and limit charges add the limit name, the scope instance and the limited resource's name (vres->res->name). When no vres is in scope, the limited resource is unknown: the name and scope are parsed from the limit= resource name and the amount is a raw number (TraceCapture's fallback). The renderer formats limit amounts by that limited resource's name, matching the existing info limit display (repl_info.c:1359-1360), so a memory limit renders in bytes units (7.7).
• Store: htab<jobid → entry>, a FIFO by capture sequence, and htab<jobid → discarded seq>. Memory accounting is the sum of sched_trace_rec_bytes. The discarded-mark table is not counted; it stays small because it holds one mark per evicted jobid.

Error Handling

Error Strategy

• Capture-side allocation failure (arena growth on the allocator thread): the record sets an incomplete flag and stops appending. The renderer prints ... trace incomplete (out of memory). Scheduling is unaffected. If allocating the record itself fails, the job is not traced.
• Trace decision allocation failure (make_trace_decision returns NULL): the record is freed and that no-decision trace is lost. This is the only best-effort gap, and it happens only after an earlier allocation failure.
• User errors (1.9, 8.2, 8.3, 8.4): one [sim] trace: ... or [sim] set output: ... line naming the argument or file and the cause (strerror(errno)). Settings are left unchanged.
• File write errors during emission: reported once per session as [sim] trace: write to <path> failed: <cause>. The file stays attached until the settings change, and console output continues.

Monitoring

None beyond REPL output. Trace code never writes to tlog (8.7).

Testing Strategy

TDD per testing.md: each test below is written before the code it covers.

Unit Tests

1. test_trace_model.c:
   • entry budget at bound−1, bound and bound+1, with the exact omitted count;
   • counters stay exact after truncation;
   • summary tallies count a host with SHORTFALL+LIMIT once per kind (5.9, 9.3);
   • sched_trace_ctl_wants for plain, element, whole-array and unknown-yet-future ids (1.2, 1.3).
2. test_trace_capture.c:
   • limit rows vs resource rows in both cluster-wide and host scope;
   • limit never reported as a shortfall (4.2, 5.5);
   • dispatch limit charges separated (6.1–6.3);
   • 51-host dispatch and pend aggregates (9.4);
   • the pend copy equals the source under job_preason_eq (6.4).
3. test_trace_hooks_gate.c:
   • SCHED_TRACE does not evaluate a side-effecting argument when trace_rec == NULL;
   • a full live-style allocation with trace_ctl == NULL allocates no record (11.2);
   • a non-interactive sim run never publishes trace_ctl (11.3).
4. test_trace_render.c:
   • golden output per section for full and summary levels;
   • units modes (7.7);
   • truncation marker and totals block (9.2, 9.3);
   • outcome cap rendering (9.4);
   • no-decision and reconstructed records (2.4, 2.5);
   • one line per budgeted entry.
5. test_trace_store.c:
   • replace (7.10);
   • eviction at 64 MiB never drops the newest (9.5);
   • discarded vs no-trace vs unknown job (7.11, 9.6);
   • array listing with mixed stored and discarded entries (7.9).

Integration Tests (sim driver with real picker and allocator threads)

1. test_repl_trace_cmd.c:
   • grammar and every invalid-argument class (1.9);
   • a path longer than 63 characters and 20 job ids (1.12);
   • next vs cur across a decision breakpoint and a round-begin breakpoint (1.6);
   • trace status blocks (1.7);
   • file open failure (8.2);
   • dev/inode rejection against --sim-output, set output and the log path, including a hard link (8.3, 8.4);
   • a replaced file is kept open until session end, or closed immediately between sessions (8.5);
   • quit flushes (8.6).
2. test_repl_trace_e2e.c, over a contention fixture with an ALTER job, a parjob, a shared license, a cluster-wide limit and a host-scoped limit:
   • every section of Requirements 2–6 present;
   • step-mode placement (7.1, 7.2);
   • continue-mode flush and no collapse (7.3, 7.4);
   • breakpoint ordering with info trace at the stop (7.5);
   • the file copy byte-equals the console trace output (8.1);
   • info trace after the job finishes, with the units mode changed (7.8).
3. Error-path seams use the real sched_allocator, never a stub sched_allocator(), because a stub would bypass the hooks under test.

   • The test #includes sched_election.c, scheduler.c, alloc_runner.c, host_iter.c, resource.c, sched_trace*.c and the other needed sources directly, instead of linking scheduler_srcs. This lets source-level macros apply.

   • Faults are injected as follows:

   • Plan reserve failure (4.4): #define realloc test_realloc around the alloc_runner.c include, as in test_alloc_runner_search.c:30-33 (macro) and :87-95 (test_realloc).

   • Snapshot and finalize failures (4.4, 4.5): the link-level wraps of two existing tests, all three listed in the new test's link_args:

     • test_alloc_runner_search.c: --wrap=xvec_dup (testsuite/meson.build:1780, __wrap_xvec_dup at test_alloc_runner_search.c:102-114);
     • test_alloc_runner_result.c: --wrap=make_host_alloc and --wrap=make_job_alloc (meson.build:1793-1794).

     With every scheduler source included, xvec_dup is called from more places, so the existing pattern of arming the failure on a specific call number (test_alloc_runner_search.c:98-114) is kept. test_alloc_runner_result.c's sched_make_allocs stub is not used, because it would collide with the real scheduler.c.

   • Host-scan error (4.4): a worker iterator whose iter_next returns < 0.

   • Per-host internal error (5.7): a failing per-host uncommitted-vector fetch.

   • alloc_rc < 0 no-decision trace (2.4): a stub job_alloc_fn returning -EIO.

   • Job-not-found (2.4): a direct cbuf_put of an unknown jobid on jbuf.

Equivalence Tests (Requirement 10)

1. test_trace_equivalence.c: the same contention replay is run four ways: trace off, all/summary, all/full, and a job subset. Each run must match on:
   • the decision sequence and pend-reason sets (job_preason_eq);
   • final resInUse/sharedInUse/job->accounts;
   • --sim-verify mismatch count;
   • --sim-output bytes, excluding wall-clock metadata (10.1–10.3).
2. With trace off, the REPL transcript of a scripted session is byte-identical to a baseline captured before the feature, except for the help text (10.4, 10.5).
3. The existing test_repl_equivalence.c, test_sim_verify* and live driver tests stay green unchanged (11.4).

Performance

• Live and trace-off cost: test_trace_hooks_gate.c asserts no record allocation. Code review checks that each hook is a single SCHED_TRACE statement (11.2).
• Large trace: a 1000-host, maxTasks = 64 parjob traced at full with the default bound completes. The record stays within the line bound, and the store stays within 64 MiB after 10,000 traced decisions (9.1, 9.5).

Security Considerations

• The trace file path is operator-supplied in an interactive local tool. It is opened in append mode with the operator's permissions. The dev/inode checks stop the trace from overwriting the replay output or the scheduler log. There are no new network or privileged surfaces.

Invariant Cross-Check (scheduler audit dimensions)

1. Driver state-equivalence: hooks only read state. The live driver never sets trace_ctl/trace_rec. The equivalence tests use a contention fixture, as scheduler-driver-equivalence.md requires. Holds.
2. Core I/O isolation: sched_trace*.c do no I/O and are added to test_core_isolation.c's source-text scan, with an additional no-stdio check (not to sched_core_isolation_lib; see Modified Files). core.h gains two opaque pointers only. Holds.
3. jbuf capacity-1 rendezvous: unchanged. The position is counted by the allocator from its own pops, and no picker-side hook exists. Holds.
4. Snapshot field-completeness: not applicable. Trace state is not part of the scheduler snapshot, recording or WAL; snapshot content must be unchanged (10.2). No save/restore mechanism is added. Stored traces are deliberately not persisted.