myxce/.kiro/specs/scheduler-sim-diagnostics/brief.md

# Brief: scheduler-sim-diagnostics

## Problem

Operators/developers investigating scheduler behavior in simulation mode (the `--sim`/`--interactive` REPL shipped by `scheduler-core`) can inspect job/queue/worker/resource state (`info job/queue/worker/resource/summary`) but cannot answer two common investigation questions:

1. **What limits are configured, and what is their current usage?** `xce_limit_t`/`xce_vres_t` (limit config + live per-user/queue/project/host usage instances) exist only inside the `limit` plugin's private `kvmgr` tables (`plugin.c`'s `limit_plugin_data_t`) — invisible to the REPL and to `sched_get_*()` accessors.
2. **Why did a job pend, or why did a job that was expected to pend actually run?** `xce_pendreason_t` is attached to pend decisions and to `xce_sched_obj_t`, but the REPL never prints it, and for a job that *did* dispatch there is no recorded trail of which limit/resource checks passed — only the final DISPATCH outcome.

## Current State

- `scheduler-core` (implementation-complete) shipped the REPL, breakpoints, and state-inspection commands (Req 5 AC 22) — this is the platform this work extends.
- No limit accessor exists outside the `limit` plugin's internal tables.
- No decision-reasoning trace exists for the dispatch path; only pend decisions carry a reason, and it isn't surfaced.

## Desired Outcome

Simulation-mode REPL (and, over time, other diagnostic surfaces added under this spec) lets an investigator answer "what is configured / in use" and "why did the scheduler decide X" without reading source or adding printf debugging. This is an **open-ended, continuously-extended spec** — new debug/troubleshoot capabilities get added as requirements over time, driven by discussion or by needs surfacing from other in-flight specs. Not all scope is known upfront.

## Approach

Extend the existing REPL/simulation-driver surface (not a new component). Each new capability is its own Requirement added when its scope is settled — this brief captures only the two seed capabilities identified so far; more will be appended.

**Seed capability 1 — Limit inspection**: expose limit config (name, per_user/queue/project/host scope, value/percent) and live usage (per-vres-instance current value vs limit) via a new `info limit <name|all>` REPL command. Requires plumbing a read path from the `limit` plugin's private kvmgr tables to the REPL (core has no existing accessor for plugin-private state).

**Seed capability 2 — Decision-reasoning trace** *(MOVED 2026-09-28 to its own spec `scheduler-sim-decision-trace` — see `.kiro/specs/scheduler-sim-decision-trace/brief.md`; text below kept for history)*: capture *why* each scheduling decision was made — not just the outcome. For PEND, surface the existing `xce_pendreason_t` (currently captured but not printed). For DISPATCH, there is currently no trail at all of which limit/resource checks were evaluated and passed; this needs new instrumentation in the allocation path (`sched_try_alloc_on_host_`, `limit_prepare_allocation`) to record a per-attempt reasoning trace, then a REPL command to print it for a given job/decision.

## Scope

- **In**: Simulation-mode / REPL-facing investigation tooling only. Read-only inspection — no state mutation (consistent with `scheduler-core` Req 5's existing "no write-side mutations" boundary). Capabilities are added incrementally as separate ACs/Requirements.
- **Out**: Live production inspection against a controller-connected scheduler (no `OPCode_schedAdmin`-style wire work) — explicitly deferred; this spec is simulation-mode only per user decision. Any write/mutation commands. New allocation or picker logic — this spec only observes decisions made by existing logic.

## Boundary Candidates

- Limit inspection (read path + REPL command) — self-contained, touches `limit.c` (expose accessor) + `repl_info.c` (new command).
- Decision-reasoning trace (new instrumentation + REPL command) — larger: touches the allocation hot path (`resource.c`, `plugin.c`) to record trace data, plus decision/REPL plumbing to surface it.
- Future capabilities: unscoped — to be added as separate requirements when identified.

## Out of Boundary

- Live/production (controller-connected) diagnostics.
- Any REPL command that mutates scheduler state.
- New scheduling policy, allocator, or picker behavior.

## Upstream / Downstream

- **Upstream**: `scheduler-core` (REPL, simulation driver, `info` command family this extends). `scheduler-allocation` and `scheduler-preemption` (siblings, currently blocked on scheduler-core, now unblocked) are likely future sources of new diagnostic requirements once they land, since this spec's scope is meant to absorb debug needs "according to the requirement of other projects."
- **Downstream**: none yet — open-ended.

## Existing Spec Touchpoints

- **Extends**: `scheduler-core` (REPL/simulation driver architecture, Req 5's `info`/breakpoint/snapshot command family).
- **Adjacent**: `scheduler-allocation`, `scheduler-preemption` — not yet implemented; expect this spec to gain requirements once those introduce new decision types/state worth inspecting.

## Constraints

- Must not alter deterministic replay guarantees (`scheduler-core` Req 5's normal↔sim equivalence invariant) — inspection commands are read-only and must not perturb core state or event ordering.
- Not starting immediately — requirements will be added/refined per-capability over time rather than fully speced upfront.