myxce/.kiro/specs/scheduler-sim-diagnostics/brief.md

Brief: scheduler-sim-diagnostics

Problem

Operators/developers investigating scheduler behavior in simulation mode (the --sim/--interactive REPL shipped by scheduler-core) can inspect job/queue/worker/resource state (info job/queue/worker/resource/summary) but cannot answer two common investigation questions:

1. What limits are configured, and what is their current usage? xce_limit_t/xce_vres_t (limit config + live per-user/queue/project/host usage instances) exist only inside the limit plugin's private kvmgr tables (plugin.c's limit_plugin_data_t) — invisible to the REPL and to sched_get_*() accessors.
2. Why did a job pend, or why did a job that was expected to pend actually run? xce_pendreason_t is attached to pend decisions and to xce_sched_obj_t, but the REPL never prints it, and for a job that did dispatch there is no recorded trail of which limit/resource checks passed — only the final DISPATCH outcome.

Current State

• scheduler-core (implementation-complete) shipped the REPL, breakpoints, and state-inspection commands (Req 5 AC 22) — this is the platform this work extends.
• No limit accessor exists outside the limit plugin's internal tables.
• No decision-reasoning trace exists for the dispatch path; only pend decisions carry a reason, and it isn't surfaced.

Desired Outcome

Simulation-mode REPL (and, over time, other diagnostic surfaces added under this spec) lets an investigator answer "what is configured / in use" and "why did the scheduler decide X" without reading source or adding printf debugging. This is an open-ended, continuously-extended spec — new debug/troubleshoot capabilities get added as requirements over time, driven by discussion or by needs surfacing from other in-flight specs. Not all scope is known upfront.

Approach

Extend the existing REPL/simulation-driver surface (not a new component). Each new capability is its own Requirement added when its scope is settled — this brief captures only the two seed capabilities identified so far; more will be appended.

Seed capability 1 — Limit inspection: expose limit config (name, per_user/queue/project/host scope, value/percent) and live usage (per-vres-instance current value vs limit) via a new info limit <name|all> REPL command. Requires plumbing a read path from the limit plugin's private kvmgr tables to the REPL (core has no existing accessor for plugin-private state).

Seed capability 2 — Decision-reasoning trace: capture why each scheduling decision was made — not just the outcome. For PEND, surface the existing xce_pendreason_t (currently captured but not printed). For DISPATCH, there is currently no trail at all of which limit/resource checks were evaluated and passed; this needs new instrumentation in the allocation path (sched_try_alloc_on_host_, limit_prepare_allocation) to record a per-attempt reasoning trace, then a REPL command to print it for a given job/decision.

Scope

• In: Simulation-mode / REPL-facing investigation tooling only. Read-only inspection — no state mutation (consistent with scheduler-core Req 5's existing "no write-side mutations" boundary). Capabilities are added incrementally as separate ACs/Requirements.
• Out: Live production inspection against a controller-connected scheduler (no OPCode_schedAdmin-style wire work) — explicitly deferred; this spec is simulation-mode only per user decision. Any write/mutation commands. New allocation or picker logic — this spec only observes decisions made by existing logic.

Boundary Candidates

• Limit inspection (read path + REPL command) — self-contained, touches limit.c (expose accessor) + repl_info.c (new command).
• Decision-reasoning trace (new instrumentation + REPL command) — larger: touches the allocation hot path (resource.c, plugin.c) to record trace data, plus decision/REPL plumbing to surface it.
• Future capabilities: unscoped — to be added as separate requirements when identified.

Out of Boundary

• Live/production (controller-connected) diagnostics.
• Any REPL command that mutates scheduler state.
• New scheduling policy, allocator, or picker behavior.

Upstream / Downstream

• Upstream: scheduler-core (REPL, simulation driver, info command family this extends). scheduler-allocation and scheduler-preemption (siblings, currently blocked on scheduler-core, now unblocked) are likely future sources of new diagnostic requirements once they land, since this spec's scope is meant to absorb debug needs "according to the requirement of other projects."
• Downstream: none yet — open-ended.

Existing Spec Touchpoints

• Extends: scheduler-core (REPL/simulation driver architecture, Req 5's info/breakpoint/snapshot command family).
• Adjacent: scheduler-allocation, scheduler-preemption — not yet implemented; expect this spec to gain requirements once those introduce new decision types/state worth inspecting.

Constraints

• Must not alter deterministic replay guarantees (scheduler-core Req 5's normal↔sim equivalence invariant) — inspection commands are read-only and must not perturb core state or event ordering.
• Not starting immediately — requirements will be added/refined per-capability over time rather than fully speced upfront.