Rongsong Shen
ce0ee78b0f
- config-upgrade-snapshot, raft-adapter-conformance: correct stale
phase metadata (both showed "tasks-generated" though every task was
already complete) and record their validate-impl GO decisions with
known-limitations notes (bug-1471/bug-1472-remainder, bug-1474
angle-b) for future spec ownership.
- component-failover-adoption: catch up design/requirements/research
docs to the Phase 6 fix (bug-1548/bug-1549, commit d479b707) that
landed after this spec was already marked closed.
- Promote 3 roadmap backlog items to specs (2026-08-10 discovery):
component-rejoin-resync (apiserver/scheduler rejoin resync),
worker-rejoin-resync (worker job-fact resync, hard prereq for
raft-apply-decoupling's facts half), scheduler-sim-diagnostics
(open-ended scheduler REPL debug/troubleshoot capability).
- roadmap.md updated to reflect all of the above.
- kiro-steering/kiro-steering-custom SKILL.md: fix stale template path
(templates/steering/ -> .kiro/settings/templates/steering/) plus
minor doc corrections in the other kiro-spec-* skills.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
58 lines
5.2 KiB
Markdown
58 lines
5.2 KiB
Markdown
# Brief: scheduler-sim-diagnostics
|
|
|
|
## Problem
|
|
|
|
Operators/developers investigating scheduler behavior in simulation mode (the `--sim`/`--interactive` REPL shipped by `scheduler-core`) can inspect job/queue/worker/resource state (`info job/queue/worker/resource/summary`) but cannot answer two common investigation questions:
|
|
|
|
1. **What limits are configured, and what is their current usage?** `xce_limit_t`/`xce_vres_t` (limit config + live per-user/queue/project/host usage instances) exist only inside the `limit` plugin's private `kvmgr` tables (`plugin.c`'s `limit_plugin_data_t`) — invisible to the REPL and to `sched_get_*()` accessors.
|
|
2. **Why did a job pend, or why did a job that was expected to pend actually run?** `xce_pendreason_t` is attached to pend decisions and to `xce_sched_obj_t`, but the REPL never prints it, and for a job that *did* dispatch there is no recorded trail of which limit/resource checks passed — only the final DISPATCH outcome.
|
|
|
|
## Current State
|
|
|
|
- `scheduler-core` (implementation-complete) shipped the REPL, breakpoints, and state-inspection commands (Req 5 AC 22) — this is the platform this work extends.
|
|
- No limit accessor exists outside the `limit` plugin's internal tables.
|
|
- No decision-reasoning trace exists for the dispatch path; only pend decisions carry a reason, and it isn't surfaced.
|
|
|
|
## Desired Outcome
|
|
|
|
Simulation-mode REPL (and, over time, other diagnostic surfaces added under this spec) lets an investigator answer "what is configured / in use" and "why did the scheduler decide X" without reading source or adding printf debugging. This is an **open-ended, continuously-extended spec** — new debug/troubleshoot capabilities get added as requirements over time, driven by discussion or by needs surfacing from other in-flight specs. Not all scope is known upfront.
|
|
|
|
## Approach
|
|
|
|
Extend the existing REPL/simulation-driver surface (not a new component). Each new capability is its own Requirement added when its scope is settled — this brief captures only the two seed capabilities identified so far; more will be appended.
|
|
|
|
**Seed capability 1 — Limit inspection**: expose limit config (name, per_user/queue/project/host scope, value/percent) and live usage (per-vres-instance current value vs limit) via a new `info limit <name|all>` REPL command. Requires plumbing a read path from the `limit` plugin's private kvmgr tables to the REPL (core has no existing accessor for plugin-private state).
|
|
|
|
**Seed capability 2 — Decision-reasoning trace**: capture *why* each scheduling decision was made — not just the outcome. For PEND, surface the existing `xce_pendreason_t` (currently captured but not printed). For DISPATCH, there is currently no trail at all of which limit/resource checks were evaluated and passed; this needs new instrumentation in the allocation path (`sched_try_alloc_on_host_`, `limit_prepare_allocation`) to record a per-attempt reasoning trace, then a REPL command to print it for a given job/decision.
|
|
|
|
## Scope
|
|
|
|
- **In**: Simulation-mode / REPL-facing investigation tooling only. Read-only inspection — no state mutation (consistent with `scheduler-core` Req 5's existing "no write-side mutations" boundary). Capabilities are added incrementally as separate ACs/Requirements.
|
|
- **Out**: Live production inspection against a controller-connected scheduler (no `OPCode_schedAdmin`-style wire work) — explicitly deferred; this spec is simulation-mode only per user decision. Any write/mutation commands. New allocation or picker logic — this spec only observes decisions made by existing logic.
|
|
|
|
## Boundary Candidates
|
|
|
|
- Limit inspection (read path + REPL command) — self-contained, touches `limit.c` (expose accessor) + `repl_info.c` (new command).
|
|
- Decision-reasoning trace (new instrumentation + REPL command) — larger: touches the allocation hot path (`resource.c`, `plugin.c`) to record trace data, plus decision/REPL plumbing to surface it.
|
|
- Future capabilities: unscoped — to be added as separate requirements when identified.
|
|
|
|
## Out of Boundary
|
|
|
|
- Live/production (controller-connected) diagnostics.
|
|
- Any REPL command that mutates scheduler state.
|
|
- New scheduling policy, allocator, or picker behavior.
|
|
|
|
## Upstream / Downstream
|
|
|
|
- **Upstream**: `scheduler-core` (REPL, simulation driver, `info` command family this extends). `scheduler-allocation` and `scheduler-preemption` (siblings, currently blocked on scheduler-core, now unblocked) are likely future sources of new diagnostic requirements once they land, since this spec's scope is meant to absorb debug needs "according to the requirement of other projects."
|
|
- **Downstream**: none yet — open-ended.
|
|
|
|
## Existing Spec Touchpoints
|
|
|
|
- **Extends**: `scheduler-core` (REPL/simulation driver architecture, Req 5's `info`/breakpoint/snapshot command family).
|
|
- **Adjacent**: `scheduler-allocation`, `scheduler-preemption` — not yet implemented; expect this spec to gain requirements once those introduce new decision types/state worth inspecting.
|
|
|
|
## Constraints
|
|
|
|
- Must not alter deterministic replay guarantees (`scheduler-core` Req 5's normal↔sim equivalence invariant) — inspection commands are read-only and must not perturb core state or event ordering.
|
|
- Not starting immediately — requirements will be added/refined per-capability over time rather than fully speced upfront.
|