lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(scheduler-sim-diagnostics): finalize requirements, design, and tasks for limit inspection

Browse source 
Requirement set 1 (info limit <name|all>) of the open-ended scheduler-sim-diagnostics
spec: requirements refined for effective vs. configured scope and exclusion-group
membership, design grounded in the actual limit-plugin/REPL source (accessor boundary,
GLOBAL/SHARED-vs-per-host read split, existence-vs-zero-usage distinction), and an
8-subtask implementation plan with full requirements traceability.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-17 08:22:11 +00:00
parent b19b404386
commit 80e096b85e
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
5 changed files with 474 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

303
.kiro/specs/scheduler-sim-diagnostics/design.md Normal file
View file

@ -0,0 +1,303 @@
# Design Document: scheduler-sim-diagnostics (Requirement Set 1 — Limit Inspection)
## Overview
This design covers the first capability of the open-ended `scheduler-sim-diagnostics` spec: a new `info limit <name|all>` command in the scheduler simulation REPL that displays a resource limit's configured setting and its live, per-instance accounted usage.
**Purpose**: Lets an operator answer "what limit is in effect, and how close is it to being hit" from inside a simulation replay, without reading cluster configuration files or adding ad hoc instrumentation.
**Users**: Operators and developers debugging scheduler behavior via `xce_scheduler -i`/`-s <replay>`.
**Impact**: Extends the existing REPL `info` command family (`scheduler-core`, implementation-complete) with one new subcommand. No existing subcommand's behavior changes. No scheduling, allocation, or limit-matching logic changes.
### Goals
- Display a limit's configured setting: constrained resource, value/percent, and effective scope dimensions (Requirements 1.1–1.7).
- Display live accounted usage per distinct scope instance (Requirements 2.1–2.3, 3.1–3.4).
- Fit the existing `info <name|all>` command shape exactly (Requirements 4.1–4.2).
- Guarantee zero effect on scheduling decisions, accounting state, or other `info` subcommands (Requirements 5.1–5.3).
### Non-Goals
- Decision-reasoning trace (seed capability 2 of this same open-ended spec) — separate, future requirement set.
- Any live, controller-connected (non-simulation) inspection.
- Any REPL command that mutates limit configuration or usage accounting.
- Any change to `match_limit()`/`is_host_based_limit()`/allocation logic — this design reads existing state only.
## Boundary Commitments
### This Spec Owns
- The `info limit <name|all>` REPL command and its output format (`repl_info_limit()`, `repl_info.c`).
- Two new read-only accessor functions exposing the limit plugin's private configured-limits and live-usage-instances tables to REPL code (`limit_get_configured_limits()`, `limit_get_usage_instances()`, `plugins/limit.c`).
### Out of Boundary
- `fmt_limit()`, `fmt_consumer()`, `fmt_group()` (`src/lib/xcelimit.c`, `src/lib/group.c`) — reused as-is, not modified.
- `is_host_based_limit()`, `match_limit()`, and the rest of the limit-matching/allocation path — read from, never modified.
- `repl_info_resource()`/`repl_info_resource_one()`/`repl_info_worker_one()` — unmodified; this design follows their established GLOBAL/SHARED-vs-per-host read pattern but does not touch their code.
- The pre-existing fact that a GLOBAL-scoped limit's synthetic capacity resource can already appear in `info resource all`/`info resource <name>` (flagged `LIMIT`) once instantiated — this is existing `scheduler-core` behavior, not introduced or altered here (Requirement 5.3 is satisfied by not touching that code path at all).
- Decision-reasoning trace (seed capability 2) — future requirement set of this same spec.
### Allowed Dependencies
- `core->resmgr`, `core->sharedInst`, `core->sharedInUse` — already read by `repl_info_resource()`/`repl_info_resource_one()` today. `core->workers[].resvals`/`core->workers[].resInUse` — the capacity side (`resvals`) is already read by `repl_info_worker_one()`; the in-use side (`resInUse`) is new to the REPL layer but is the exact pattern `sched_check_host_satisfy_()` (`resource.c`) already uses in production. `core->plugins` (name-keyed plugin registry) — read elsewhere in the scheduler (`plugin.c`'s hook fan-outs) but genuinely new to the REPL `info` layer; this design is the first `info` subcommand to cross into plugin-private state, which is why the Limit Plugin Accessor exists as its own boundary rather than being folded into `repl_info.c` directly.
- The limit plugin's fixed registration name `"xce_sched_limit"` — owned entirely inside `limit.c`; no other file may hardcode it.
### Revalidation Triggers
- A change to `xce_consumer_type_t` (adding/removing a scope dimension) — Requirement 1.1/3.1 enumerate the four dimensions explicitly.
- A change to `is_host_based_limit()`'s percent/global-resource rule — Requirement 1.6/3.2 depend on its exact current semantics.
- A change to how `add_vres()` flags synthetic capacity resources, or to `repl_info_resource_one()`'s GLOBAL/SHARED-vs-host read split — this design's live-usage read path mirrors that split and must be re-checked if it changes.
- Renaming the limit plugin's registration string away from `"xce_sched_limit"`.
## Architecture
### Existing Architecture Analysis
`scheduler-core` already establishes the pattern this design extends: `repl_dispatch()` (`repl.c`) routes to `repl_cmd_info()` (`repl_info.c`), which routes on a second token to one `repl_info_<subcommand>()` per family member. Of the raw handles this pattern already reads, only `core->jobs`/`core->queues`/`core->workers`/`core->resmgr` have existing REPL-layer precedent (`repl_info_job(s)`/`repl_info_queue`/`repl_info_worker`/`repl_info_resource`); `core->plugins` is already a raw, name-keyed handle used elsewhere in the scheduler (`plugin.c`'s hook fan-outs) but has never been read by `repl_info.c`, and plugin-*private* state (behind `plugin->data`) has no REPL precedent at all — crossing that specific boundary is the one genuinely new seam this design adds.
### Architecture Pattern & Boundary Map
```mermaid
graph TB
Dispatch[repl_dispatch] --> CmdInfo[repl_cmd_info]
CmdInfo --> InfoLimit[repl_info_limit new]
InfoLimit --> Accessor1[limit_get_configured_limits new]
InfoLimit --> Accessor2[limit_get_usage_instances new]
InfoLimit --> FmtLimit[fmt_limit existing]
InfoLimit --> HostCheck[is_host_based_limit existing]
FmtLimit --> FmtConsumer[fmt_consumer existing]
FmtConsumer --> FmtGroup[fmt_group existing]
Accessor1 --> PluginRegistry[core.plugins existing]
Accessor2 --> PluginRegistry
InfoLimit --> SharedState[core.sharedInst core.sharedInUse existing]
InfoLimit --> WorkerState[core.workers resvals and resInUse existing]
```
**Architecture Integration**:
- Selected pattern: two-level command handler (`repl_info_limit()` for `all`, an inner per-limit printer for `<name>`), identical shape to `repl_info_resource()`/`repl_info_resource_one()`.
- Domain/feature boundaries: REPL formatting stays in `repl_info.c`; plugin-private data access stays behind the two new `limit.c` accessors; nothing in the REPL layer touches `limit_plugin_data_t` directly.
- Existing patterns preserved: name-keyed plugin lookup (`core->plugins`), raw-handle-plus-shared-iterator access (`iter_make`/`iter_next`/`iter_destroy`), GLOBAL/SHARED-vs-per-host value read split (bug-1593 precedent).
- New components rationale: the two accessors are the only way to cross the plugin-private-state boundary that has blocked this capability since it was first scoped (brief.md).
- Steering compliance: read-only (`ha-async-io.md`'s "never block"/never-mutate concerns are inapplicable — this is in-process, synchronous, simulation-only REPL code, not on any async I/O or Raft path).
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|-------------------|------------------|-------|
| Scheduler REPL (C) | Existing `src/scheduler` toolchain (Meson/GCC, ASAN debug profile per `build-deployment.md`) | New `info limit` command | No new dependency |
| Scheduler plugin (C) | Existing `src/scheduler/src/plugins/limit.c` | New read-only accessors | No new dependency |
| Shared lib (C) | Existing `src/lib/xcelimit.c`, `src/lib/group.c` | Reused formatters (`fmt_limit`, `fmt_consumer`, `fmt_group`) | No new dependency; already built into `libxce` |
No new external dependency is introduced.
## File Structure Plan
### Modified Files
- `src/scheduler/include/plugin.h` — add two `extern` declarations: `limit_get_configured_limits()`, `limit_get_usage_instances()`, placed next to the existing `register_limit()` declaration.
- `src/scheduler/src/plugins/limit.c` — define the two accessor functions. Each does `key_find(core->plugins, "xce_sched_limit", &plugin)`, returns `NULL` if not found or `plugin->data == NULL`, otherwise returns the relevant `limit_plugin_data_t` field (`limits` or `instances`).
- `src/scheduler/src/repl_info.c` — add `repl_info_limit()` (the `all` case: iterate `limit_get_configured_limits()` via `iter_make()`/`iter_next()`, print each) and an inner per-limit printer (the `<name>` case: `key_find()` directly on `limit_get_configured_limits()`'s handle — the plugin's kvmgr is keyed by limit name, confirmed via bug-1685's duplicate-name `-EEXIST` handling, so this is a direct lookup, not a scan; report "no such limit" per Requirement 1.3 if not found, report "no limits configured" per Requirement 1.7 if the accessor returns `NULL` or an empty handle on the `all` path); add one new `else if (strcmp(tokens[1], "limit") == 0)` branch to `repl_cmd_info()`, and extend its usage-string literal to include `limit`.
- `src/scheduler/testsuite/test_limit_plugin.c` — add unit tests for the two new accessors (registered-plugin case, not-yet-initialized case).
- `src/scheduler/testsuite/test_repl_limit_table.c` (new sibling of `test_repl_resource_table.c`, matching its dispatch-capture harness — kept separate rather than added to `test_repl_resource_table.c` itself, since it needs a different white-box `#include` set, see Implementation Notes below) — REPL-level tests for `info limit <name>`/`info limit all`.
No new files. No existing formatter, matcher, or allocation-path file changes.
## System Flows
```mermaid
sequenceDiagram
participant Op as Operator
participant Dispatch as repl_dispatch
participant Info as repl_cmd_info
participant Limit as repl_info_limit
participant Plugin as limit.c accessors
participant Fmt as fmt_limit/fmt_group
participant Core as core state
Op->>Dispatch: info limit ug1_slots
Dispatch->>Info: repl_cmd_info
Info->>Limit: repl_info_limit sim ug1_slots
Limit->>Plugin: limit_get_configured_limits core
Plugin->>Core: key_find core.plugins xce_sched_limit
Plugin-->>Limit: limits handle or NULL
Limit->>Limit: key_find name lookup
Limit->>Fmt: fmt_limit matched limit
Fmt-->>Limit: configured setting text
Limit->>Limit: is_host_based_limit effective scope check
Limit->>Plugin: limit_get_usage_instances core
Plugin-->>Limit: instances handle
Limit->>Limit: filter instances by pointer equality to matched limit
Limit->>Core: read sharedInst/sharedInUse or per-worker resvals/resInUse
Limit-->>Op: configured setting plus per-instance usage
```
Key decisions not obvious from the diagram: the "no such limit"/"no limits configured" cases (Requirements 1.3, 1.7) short-circuit before any formatter or core-state read is attempted, so a missing/empty case never risks printing partial data. The `all` path repeats this exact per-limit sequence once per configured limit rather than a bulk query, matching `repl_info_resource()`'s existing per-resource iteration shape.
## Requirements Traceability
| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|-------------|------------|-------|
| 1.1 | Display name, resource, value, percent flag, effective scope | REPL Limit Info Command | `repl_info_limit()`, `fmt_limit()`, `is_host_based_limit()` | Sequence diagram |
| 1.2 | `all` shows every configured limit | REPL Limit Info Command | `limit_get_configured_limits()` | — |
| 1.3 | Unknown name reports absence, no fabrication | REPL Limit Info Command | `repl_info_limit()` `key_find()` lookup | Sequence diagram (short-circuit) |
| 1.4 | Percent resolves to current absolute amount, possibly per-host-instance | Limit Plugin Accessor + REPL Limit Info Command | `limit_get_usage_instances()`, shared/per-host read split | — |
| 1.5 | Inclusion/exclusion membership shown | REPL Limit Info Command (reused formatter) | `fmt_group()` | — |
| 1.6 | Effective host-scope shown even when not explicitly configured | REPL Limit Info Command | `is_host_based_limit()` | Sequence diagram |
| 1.7 | Empty "all limits" reports explicitly | REPL Limit Info Command | `limit_get_configured_limits()` NULL/empty check | Sequence diagram (short-circuit) |
| 2.1 | Per-instance accounted amount + capacity | Limit Plugin Accessor + REPL Limit Info Command | `limit_get_usage_instances()`, shared/per-host read split | Sequence diagram |
| 2.2 | Not-yet-evaluated limit reports no instance, not fabricated zero | REPL Limit Info Command | instance-filter (no matching `xce_vres_t`) | — |
| 2.3 | Usage reflects state at command issue time | REPL Limit Info Command | direct `core.sharedInst`/`core.workers` read, no caching | — |
| 3.1 | Multi-dimension combinations each own instance | Limit Plugin Accessor + REPL Limit Info Command | `xce_vres_t` per unique `build_vres_name()` user/queue/project combination (existing), fanned out per-worker by the REPL when host-scoped | — |
| 3.2 | Host dimension broken out per host, including implicit case | REPL Limit Info Command | per-worker `resvals`/`resInUse` fan-out over one `xce_vres_t` (host is not part of `build_vres_name()`) | Sequence diagram |
| 3.3 | No scope dimension → single simulation-wide instance | REPL Limit Info Command | `core.sharedInst` read | — |
| 3.4 | Each instance labeled by scope values | REPL Limit Info Command | `build_vres_name()`-derived label (existing naming, parsed/echoed) | — |
| 4.1 | `<name>|all` invocation shape | REPL dispatch | `repl_cmd_info()` new branch | Sequence diagram |
| 4.2 | Accepted anywhere other `info` subcommands are | REPL dispatch | `repl_cmd_info()` new branch | — |
| 5.1 | No mutation of config/accounting/decisions | REPL Limit Info Command | read-only accessors, no write calls | — |
| 5.2 | Identical replay outcome whether or not queried | REPL Limit Info Command | no state written, no hook installed | — |
| 5.3 | Existing `info` subcommands unchanged | REPL dispatch | additive-only `repl_cmd_info()` branch | — |
## Components and Interfaces
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|---------------|--------|---------------|---------------------------|-----------|
| Limit Plugin Accessor | Scheduler Plugin (`limit.c`) | Expose the limit plugin's private configured-limits/usage-instances tables as raw iterable handles | 1.1, 1.2, 1.4, 2.1, 3.1 | `core->plugins` (P0) | Service |
| REPL Limit Info Command | Scheduler REPL (`repl_info.c`) | Format and print `info limit <name|all>` output | 1.1–1.7, 2.1–2.3, 3.1–3.4, 4.1–4.2, 5.1–5.3 | Limit Plugin Accessor (P0), `fmt_limit`/`fmt_group` (P0), `is_host_based_limit` (P0), `core.sharedInst`/`core.workers` (P1) | Service |
### Scheduler Plugin
#### Limit Plugin Accessor
| Field | Detail |
|-------|--------|
| Intent | Cross the plugin-private-state boundary read-only, without exposing `limit_plugin_data_t` or the plugin's registration name outside `limit.c` |
| Requirements | 1.1, 1.2, 1.4, 2.1, 3.1 |
**Responsibilities & Constraints**
- Owns the `"xce_sched_limit"` plugin-name lookup; no other file may hardcode it.
- Returns `NULL` (never a partially-valid handle) when the plugin is not registered or not yet initialized — callers must treat `NULL` as the state "nothing is configured," never as an error to surface differently. This is a semantic signal only: which user-facing message a caller prints for a `NULL` result is the caller's own decision (the REPL Limit Info Command prints a different message depending on whether the operator asked for `all` or a specific name — see its Responsibilities below); this accessor does not prescribe wording.
- Performs no locking beyond what `key_find(core->plugins, ...)` already does internally (matches every other `core->plugins` consumer, e.g. `plugin_job_update()`).
- Never allocates, mutates, or triggers plugin hooks — a pure read of already-existing pointers.
**Dependencies**
- Inbound: REPL Limit Info Command — the only caller (P0)
- Outbound: `core->plugins` (existing name-keyed registry) — P0
**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]
##### Service Interface
```c
/* src/scheduler/include/plugin.h */
/* Returns the plugin's configured-limits handle (iterable via
* iter_make()/iter_next(), elements are xce_limit_t*), or NULL if the
* limit plugin is not registered or not yet initialized. */
extern var limit_get_configured_limits(xce_sched_core_t* core);
/* Returns the plugin's live usage-instances handle (iterable via
* iter_make()/iter_next(), elements are xce_vres_t*), or NULL under the
* same conditions as above. */
extern var limit_get_usage_instances(xce_sched_core_t* core);
```
- Preconditions: `core` non-NULL and `xce_sched_core_init()`-initialized (same precondition every other `core->`-prefixed accessor in this codebase already assumes).
- Postconditions: return value is either `NULL` or a handle safe to pass to `iter_make()`; no side effects.
- Invariants: never mutates `core`, `core->plugins`, or the limit plugin's private state.
**Implementation Notes**
- Integration: declared in `plugin.h` next to `register_limit()`, matching where every other plugin-registration-adjacent declaration already lives.
- Validation: unit tests cover (a) plugin registered with configured limits, (b) plugin registered with zero limits (empty handle, not NULL, per Requirement 1.7's "report explicitly" needing to distinguish "no plugin" from "plugin present, zero limits" only if that distinction is observable — both collapse to the same REPL message either way), (c) plugin not registered (`NULL`).
- Risks: none — pure read accessor, no new failure mode beyond "plugin absent," which is already a normal simulation configuration (a replay with no limits configured never registers/initializes any limit).
### Scheduler REPL
#### REPL Limit Info Command
| Field | Detail |
|-------|--------|
| Intent | Implement `info limit <name|all>`: format configured setting, effective scope, and live per-instance usage |
| Requirements | 1.1–1.7, 2.1–2.3, 3.1–3.4, 4.1–4.2, 5.1–5.3 |
**Responsibilities & Constraints**
- Single entry point `repl_info_limit(xce_sched_sim_driver_t* sim, const char* arg)`, called from `repl_cmd_info()` exactly like `repl_info_resource(sim, tokens[2])`.
- `arg == "all"`: iterate `limit_get_configured_limits()`; if `NULL`/empty, print the Requirement 1.7 "no limits configured" message and return; otherwise print each limit via the per-limit path below.
- `arg != "all"`: if `limit_get_configured_limits()` returns `NULL` (plugin not registered — a `NULL` handle is never passed to `key_find()`), or `key_find()` on the returned handle finds no match, print the Requirement 1.3 "no such limit `<name>`" message and return (no partial output). Requirement 1.7's "no limits configured" message is reserved for the `all` path specifically — its own acceptance criterion scopes it to "when an operator requests information for all limits" — so a `NULL`-handle result on the named-lookup path reports absence of *that name*, not absence of configuration in general.
- Per-limit output: `fmt_limit()` into a stack buffer, printed line-by-line with the existing `[sim] ` prefix convention; then, only if `is_host_based_limit(core->resmgr, limit) > 0` AND the limit's own `consumers[CONSUMER_HOST] == NULL`, print the Requirement 1.6 effective-host-scope note.
- Live usage: iterate `limit_get_usage_instances()`, filter by `vres->limit == matched_limit` (pointer equality). `build_vres_name()` (existing, `xcelimit.c`) never encodes host in the vresname — so there is exactly one `xce_vres_t` per matched (user, queue, project) combination *regardless of host-scoping*; host breakout is NOT "one `xce_vres_t` per host," it is a per-worker fan-out over that single `xce_vres_t`'s shared resource index. Concretely, for each matching `xce_vres_t`:
- If `vres->vres` is `XCE_RESF_GLOBAL`/`XCE_RESF_SHARED`: exactly one displayed instance, reading capacity from `core.sharedInst[vres->vres->idx]` and accounted usage from `core.sharedInUse[vres->vres->idx]` — the exact read `repl_info_resource_one()` already implements for a GLOBAL/SHARED resource (bug-1593).
- Otherwise (host-scoped, explicit or implicit per Requirement 1.6): one displayed instance *per worker that has a populated entry at `vres->vres->idx`* — iterate `core.workers`, and for each worker `w`, include it only if `zarray_get(&(w->resvals), vres->vres->idx)` is non-NULL (a worker `put_resvals()` has never written to, e.g. an unevaluated host, must not produce a fabricated zero-usage line, matching Requirement 2.2's no-fabrication rule at per-host granularity). For an included worker, capacity comes from `w->resvals[vres->vres->idx]` and accounted usage from `w->resInUse[vres->vres->idx]` — the sibling capacity/in-use zarray pair `sched_check_host_satisfy_()` (`resource.c`) already reads the identical way for ordinary consumable resources.
- **Existence-of-instance vs. zero-usage are distinct signals, and must not be conflated.** A matched `xce_vres_t`'s *capacity* entry (`sharedInst`/`resvals`) is populated synchronously with the `xce_vres_t` being created (`instantiate_limit()` and `update_limit_res_value()` run back-to-back in the same call, per `research.md`), so an existing `xce_vres_t`/populated-worker-entry always has a real capacity to show. Its *in-use* entry (`sharedInUse`/`resInUse`) is populated only once something has actually been charged against it, independently of capacity — so a `NULL` read there means genuine zero current usage (display `accounted=0`), never "no instance" (Requirement 2.2's message is reserved for the case where the `xce_vres_t`/worker-entry itself does not exist at all, i.e. the limit or that host has never been evaluated against any job).
- This is the mechanism behind Requirement 3.1's "per-host and per-user together" example: the `xce_vres_t` count only reflects the user dimension; the host dimension is produced entirely by this per-worker fan-out, one displayed instance per (vres, worker) pair.
- Zero included instances for a matched `xce_vres_t` (the GLOBAL/SHARED *capacity* entry is absent, or no worker has a populated *capacity* entry) → print the Requirement 2.2 "no active usage instance yet" message for that limit, not a fabricated zero.
- Every instance line is labeled with its scope values (Requirement 3.4), derived from the same `limit=<name>[/u=<user>][/q=<queue>][/p=<project>]` naming `build_vres_name()` already produces, plus the host name when the instance came from a per-worker read.
- Never calls `instantiate_limit()`, `match_limit()`, `update_limit_res_value()`, or any other mutating/allocating limit-plugin function — read-only per Requirement 5.1.
**Dependencies**
- Inbound: `repl_cmd_info()` — the only caller (P0)
- Outbound: Limit Plugin Accessor (P0), `fmt_limit()`/`fmt_consumer()`/`fmt_group()` (P0), `is_host_based_limit()` (P0), `core.sharedInst`/`core.sharedInUse`/`core.workers[].resvals`/`core.workers[].resInUse` (P1)
**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]
##### Service Interface
```c
/* src/scheduler/src/repl_info.c (static, mirrors repl_info_resource shape) */
static void repl_info_limit(xce_sched_sim_driver_t* sim, const char* arg);
```
- Preconditions: `sim` non-NULL, `sim->core` initialized (same as every other `repl_info_*` function).
- Postconditions: writes to `stdout` only; no field of `sim`, `sim->core`, or any plugin's private state is modified.
- Invariants: output for a given `arg` is a pure function of current simulation state — calling it twice in a row with no intervening `step`/`continue` produces identical output (Requirement 5.2).
**Implementation Notes**
- Integration: one new `else if (strcmp(tokens[1], "limit") == 0) { ... repl_info_limit(sim, tokens[2]); ... }` branch in `repl_cmd_info()` (`repl_info.c`), plus updating that function's own usage-string literal (`"info <break|session|config|job|jobs|queue|worker|resource|summary|limit> [arg]"`) to keep it accurate — the same maintenance step every prior `info` subcommand addition has made. All output must use `fprintf(stdout, ...)`, never bare `printf()` — `src/scheduler/testsuite/test_no_bare_printf.c` regression-tests every file under `src/scheduler/src/` for this (bug-1599 is a known false-positive in that scanner's *comments*, not a license to use bare `printf()` in real code).
- Validation: dispatch-capture tests per Requirement Traceability above, following `test_repl_resource_table.c`'s existing `run_dispatch_for_capture()` harness; at minimum one fixture with a percent-based limit on a per-host resource configured without `per_host` (exercises Requirement 1.6/3.2's implicit-scope case) and one fixture with an exclusion-based group (exercises Requirement 1.5).
- Risks: **Build wiring** — `test_repl_resource_table.c` white-box `#include`s `repl_info.c` but deliberately does NOT include `plugins/limit.c` or `resource.c` today (confirmed by reading its include list; its own comment notes "resource.c, not linked here"). Adding calls from `repl_info_limit()` to the new `limit_get_configured_limits()`/`limit_get_usage_instances()` accessors without including `plugins/limit.c` (and, transitively, either `resource.c` or link-satisfying stubs for `get_resvals()`/`put_resvals()`, the same pattern `test_limit_plugin.c` already uses) produces undefined-reference link failures — the exact failure class bug-1762 hit and remediated for a different scheduler-sim-repl-verbosity command. This is the reason `test_repl_limit_table.c` is a new file rather than an addition to `test_repl_resource_table.c`: it needs its own white-box include set. Because this test needs to assert *real, non-trivial* accounted-usage values (not a stubbed-zero), it should white-box `resource.c` for real (not stub `get_resvals`/`put_resvals` as no-ops the way `test_limit_plugin.c` does for its narrower config-update-only scope) — left as a task-level decision, flagged here so it is made deliberately rather than discovered via a failed build.
## Data Models
### Domain Model
No new persisted or transient domain entities are introduced. This design reads three existing entities already defined in `src/include/xce/types.h`:
```mermaid
graph TB
Limit[xce_limit_t] --> Consumer[xce_consumer_t x4]
Consumer --> Group[xce_group_t]
Limit -.instantiates one per user queue project combo.-> Vres[xce_vres_t]
Vres --> Capacity[xce_resource_t capacity vres field]
Vres --> Underlying[xce_resource_t underlying res field]
Capacity -.read from.-> Shared[core sharedInst sharedInUse when global]
Capacity -.fanned out over.-> Worker[core workers resvals resInUse when host scoped]
```
- `xce_limit_t` — one per configured limit; owns up to four `xce_consumer_t` (one per scope dimension), each referencing an `xce_group_t` (inclusion `members` + exclusion `ex_members`).
- `xce_vres_t` — one per distinct live (user, queue, project) combination of a limit; created lazily on first job match (`instantiate_limit()`, unmodified by this design). Links back to its owning `xce_limit_t` (`.limit`), its synthetic capacity resource (`.vres`), and the underlying constrained resource (`.res`). **Host is not part of this cardinality** — `build_vres_name()` never encodes it, so a host-scoped limit still has exactly one `xce_vres_t` per (user, queue, project); the host dimension is produced entirely at display time by reading `.vres`'s shared resource index across every worker that has a populated entry (see the REPL Limit Info Command's Responsibilities above). A displayed "instance" (Requirement 3's unit of enumeration) is therefore a `(xce_vres_t, worker)` pair when host-scoped, and the bare `xce_vres_t` when GLOBAL/SHARED.
- No new fields, tables, or relationships. This design is a read-only projection over the above.
### Logical Data Model
Not applicable — no schema change, no new storage.
## Error Handling
### Error Strategy
This is a read-only diagnostic command; there is no user-input validation beyond "does a limit by this name exist" (Requirement 1.3) and "are there any limits at all" (Requirement 1.7) — both are reported as plain REPL messages, matching every other `info` subcommand's existing style (e.g. `repl_info_job()`'s "no such job" message), not as process-level errors.
### Error Categories and Responses
- **Unknown limit name** (Requirement 1.3): report "no such limit `<name>`", no partial output — mirrors `repl_info_job()`'s exact pattern for an unknown jobid.
- **No limits configured at all** (Requirement 1.7): report "no limits configured", not an empty listing.
- **Limit configured but never evaluated** (Requirement 2.2): report "no active usage instance yet" for that limit, not a fabricated zero-usage line.
- **Limit plugin not registered/initialized**: a simulation with no limits configured never registers the plugin's private tables, so this is not a distinct error case from the operator's point of view — it collapses into Requirement 1.7's "no limits configured" message on the `all` path, and into Requirement 1.3's "no such limit `<name>`" message on the named-lookup path (matching that requirement's own scoping — see the REPL Limit Info Command's Responsibilities above).
### Monitoring
Not applicable — this is an interactive/replay-local REPL command with no persistent logging requirement beyond what the REPL already does.
## Testing Strategy
### Unit Tests
- `limit_get_configured_limits()`/`limit_get_usage_instances()` return the correct handle when the plugin is registered and initialized with configured limits (`test_limit_plugin.c`).
- Both accessors return `NULL` when the limit plugin has not been registered (`test_limit_plugin.c`).
- `repl_info_limit()`'s `key_find()` lookup reports "no such limit" for an unconfigured name without touching `stdout` beyond that one line (`test_repl_limit_table.c` or equivalent).
### Integration Tests
- `info limit all` against a fixture with zero configured limits reports "no limits configured" (Requirement 1.7).
- `info limit <name>` against a percent-based limit configured on a per-host resource *without* `per_host` set reports the effective host-scope note and breaks out usage per host (Requirements 1.6, 3.2) — the central scope-ambiguity finding from `research.md`.
- `info limit <name>` against a limit scoped with an exclusion group ("all except X") shows both inclusion and exclusion membership via `fmt_group()` (Requirement 1.5).
- `info limit <name>` issued twice in a row with no intervening `step` produces byte-identical output, and a subsequent `step`/`continue` sequence produces identical scheduling decisions whether or not `info limit` was issued in between (Requirement 5.1, 5.2) — extends the existing normal-vs-sim equivalence test discipline (`scheduler-driver-equivalence.md`) to this new read path specifically.
### E2E/REPL Tests
- Full dispatch-capture round trip for `info limit ug1_slots` and `info limit all` against a realistic multi-limit fixture (mirrors `limits.yaml`'s shape referenced in `research.md`), asserting every existing `info` subcommand's output is byte-identical before and after this change (Requirement 5.3).
## Constraints
- **Project Invariant Cross-Check** (`scheduler-driver-equivalence.md`): this design touches none of `resource.c`, `core_events.c`, `driver.c`, `sim_loop.c`, or `core_session.c` — the files that invariant's testing discipline gates — and is read-only by Requirement 5.1/5.2, so it cannot introduce normal-mode-vs-sim-mode accounting divergence. Checked, not applicable beyond the read-only guarantee already covered by the Integration Tests above.
- **`ha-async-io.md`**: not applicable — this code runs entirely inside the standalone simulation REPL process, never on the controller's iomgr/delaymgr/vtask threads.
- **`resource-accounting.md`** (reservation-vs-actual-usage reconciliation for real resources like `mem`): not applicable, checked by flag rather than by analogy — that document scopes itself explicitly to resources flagged both `XCE_RESF_DYNAMIC` and `XCE_RESF_CONSUMABLE`. A limit's synthetic capacity resource (`vres->vres`, `add_vres()` in `xcelimit.c`) is flagged `XCE_RESF_LIMIT | XCE_RESF_CONSUMABLE | XCE_RESF_RELEASE` plus `XCE_RESF_GLOBAL|XCE_RESF_SHARED` or `XCE_RESF_DEFAULT_HOST` — never `XCE_RESF_DYNAMIC`. Its capacity/in-use pair is an exact scheduler-computed accounting number (charged/released by the same dispatch/completion path as any consumable resource), not an estimate reconciled against independently-drifting OS-measured usage the way `mem` is — so this design's plain read of those two values has no reconciliation-window or staleness concern to account for.

12
.kiro/specs/scheduler-sim-diagnostics/requirements.md
View file

@ -4,7 +4,7 @@
Operators investigating scheduler behavior in the `--interactive` simulation REPL (shipped by `scheduler-core`) can already inspect job, queue, worker, and resource state through the REPL's `info` command family, but cannot see what resource limits are configured or how much of each is currently accounted for. This document covers the first capability of the open-ended `scheduler-sim-diagnostics` spec: giving the REPL visibility into limits — both a limit's configured setting and the live, accounted usage tracked against it — so an operator can answer "what limit is in effect, and how close is it to being hit" without reading cluster configuration files or adding ad hoc instrumentation.
## Boundary Context
- **In scope**: REPL-visible display of a limit's configured setting (the resource it constrains, its configured value or percentage, and which scope dimensions it varies by) and its live accounted usage (per distinct scope instance), in simulation mode only.
- **In scope**: REPL-visible display of a limit's configured setting (the resource it constrains, its configured value or percentage, and which scope dimensions it effectively varies by, including any host-scoping implied by a percentage value on a resource that is not shared cluster-wide) and its live accounted usage (per distinct scope instance, including explicit exclusion-based scope restrictions), in simulation mode only.
- **Out of scope**: any command that mutates scheduler state; live, controller-connected (non-simulation) inspection; explaining *why* a specific job did or did not trigger a limit (decision-reasoning trace — expected to be added later as a separate requirement of this same open-ended spec, not covered here); any change to scheduling, allocation, or limit-matching behavior.
- **Adjacent expectations**: this capability extends the REPL's existing `info` command family (shipped by `scheduler-core`, implementation-complete) without altering any existing subcommand's behavior or output; it must not perturb the guarantee that a simulation replay reproduces identical scheduling decisions and identical accounting state regardless of what inspection commands are issued during it.
@ -14,11 +14,13 @@ Operators investigating scheduler behavior in the `--interactive` simulation REP
**Objective:** As an operator investigating scheduler behavior, I want to see each limit's configured setting, so that I can confirm what constraint is actually in effect without reading cluster configuration files.
#### Acceptance Criteria
1. When an operator requests information for a specific limit by name, the REPL shall display that limit's name, the resource it constrains, its configured value, whether that value is an absolute amount or a percentage, and which scope dimensions (user, queue, project, host) it varies by.
1. When an operator requests information for a specific limit by name, the REPL shall display that limit's name, the resource it constrains, its configured value, whether that value is an absolute amount or a percentage, and which scope dimensions (user, queue, project, host) it effectively varies by.
2. When an operator requests information for all limits, the REPL shall display the configuration described in Acceptance Criterion 1 for every limit currently configured in the simulation.
3. If an operator requests information for a limit name that is not configured, then the REPL shall report that no such limit exists and shall not display partial or fabricated information.
4. Where a limit's configured value is a percentage, the REPL shall also display the current absolute amount that percentage resolves to, since that amount can change as the underlying resource's value changes.
5. Where a limit's scope is restricted to a specific set of users, queues, projects, or hosts rather than applying to all of them, the REPL shall display that restricted membership.
4. Where a limit's configured value is a percentage, the REPL shall also display the current absolute amount that percentage resolves to for each instance the limit applies to, since that amount can change as the underlying resource's value changes and may differ between instances when the limit is host-scoped.
5. Where a limit's scope is restricted to a specific set of users, queues, projects, or hosts — whether by explicit inclusion, explicit exclusion, or both — the REPL shall display that restricted membership, distinguishing included members from explicitly excluded ones.
6. Where a limit's configured value is a percentage of a resource that is not shared cluster-wide, the REPL shall include the host dimension in the scope it displays for that limit even when the limit's configuration does not explicitly restrict it by host, since usage for such a limit is tracked separately per host regardless of that configuration.
7. If no limits are configured in the simulation, then the REPL shall report that no limits are configured when an operator requests information for all limits, rather than displaying an empty listing with no explanation.
### Requirement 2: Live Accounted Usage Display
**Objective:** As an operator watching a simulation replay, I want to see how much of each limit is currently accounted for, so that I can tell whether a limit is close to being exhausted or was the reason a job was denied.
@ -33,7 +35,7 @@ Operators investigating scheduler behavior in the `--interactive` simulation REP
#### Acceptance Criteria
1. When a limit varies by more than one scope dimension at once (for example per-user and per-queue together, or per-host and per-user together), the REPL shall display each distinct combination of scoped values as its own instance with its own accounted usage.
2. When a limit varies by host, the REPL shall break out accounted usage separately for each host on which that limit has been evaluated, even when the limit also varies by another scope dimension at the same time.
2. When a limit varies by host — including when the host dimension is implicit per Requirement 1's Acceptance Criterion 6 rather than explicitly configured — the REPL shall break out accounted usage separately for each host on which that limit has been evaluated, even when the limit also varies by another scope dimension at the same time.
3. When a limit does not vary by any scope dimension, the REPL shall display a single instance representing the entire simulation.
4. The REPL shall label each displayed instance with the scope values it represents, so an operator can tell instances apart.

94
.kiro/specs/scheduler-sim-diagnostics/research.md Normal file
View file

@ -0,0 +1,94 @@
# Research & Design Decisions
## Summary
- **Feature**: `scheduler-sim-diagnostics` — Requirement set 1 (`info limit` REPL command)
- **Discovery Scope**: Extension (extends `scheduler-core`'s existing REPL `info` command family and reuses the limit plugin's existing library-level formatters)
- **Key Findings**:
- The REPL `info` command family is a single dispatch function (`repl_cmd_info()`, `repl_info.c`) with one `else if` branch per subcommand — adding `limit` is a mechanical extension of an existing, well-established pattern.
- The limit plugin's private state (`limit_plugin_data_t.limits`/`.instances`) is genuinely inaccessible outside `src/scheduler/src/plugins/limit.c` today — no header declares the type — confirming the brief's stated gap. Two new one-line accessor functions close it.
- Two library-level formatters already exist and already do most of Requirement 1's work: `fmt_limit()` and `fmt_consumer()`/`fmt_group()` (`src/lib/xcelimit.c`, `src/lib/group.c`), including correct inclusion/exclusion membership rendering (`fmt_group()` already prefixes excluded members with `~`). These are adopted directly rather than rebuilt.
- The live-usage read path (GLOBAL/SHARED single value vs. per-host breakout) is architecturally identical to the pattern `repl_info_resource_one()` already implements for ordinary resources (bug-1593) — the limit's synthetic capacity resource (`vres->vres`) is a real, already-registered `xce_resource_t` carrying the same `XCE_RESF_GLOBAL`/`XCE_RESF_SHARED` flags, so the same shared-vs-per-host read logic applies unmodified.
## Research Log
### Limit plugin data model and matching semantics
- **Context**: Requirement 1 AC1/AC5/AC6 need to know exactly what "scope dimensions" and "restricted membership" mean at the code level, since an earlier requirements-review pass surfaced a scope-vs-runtime ambiguity.
- **Sources Consulted**: `src/include/xce/types.h` (`xce_limit_t`, `xce_limit_config_t`, `xce_consumer_t`, `xce_vres_t`), `src/lib/xcelimit.c` (`match_limit`, `build_vres_name`, `is_host_based_limit`, `instantiate_limit`), `src/scheduler/src/plugins/limit.c` (`match_job_host_limit_`, `match_job_global_limit_`, `update_limit_res_value`, `limit_prepare_allocation`), `.wolf/buglog.json` bug-1784 through bug-1790 (the gitea#46 limit-matching bug chain, 2026-09-07).
- **Findings**:
- `xce_consumer_type_t` has exactly four values (`CONSUMER_USER`, `CONSUMER_QUEUE`, `CONSUMER_PROJECT`, `CONSUMER_HOST`) — the four scope dimensions in Requirement 1 AC1 are the complete set, no fifth dimension exists.
- `is_host_based_limit()` forces per-host live-instance tracking whenever the limit's value is a percentage of a resource that is not `XCE_RESF_GLOBAL`/`XCE_RESF_SHARED` — independent of whether `CONSUMER_HOST` was configured. This is real, current behavior (not hypothetical), confirmed via source and via bug-1789/1790's live-verified fix history for the equivalent allocation-side computation.
- `xce_group_t` supports both `members` and `ex_members` (exclusion). Confirmed as real production configuration via bug-1789's own note describing a live-configured limit ("`limit1`... all users except rshen").
- `fmt_group()` (`src/lib/group.c:388`) already renders both lists correctly (`~name` prefix for exclusions, `all`/`[]` shortcuts for the builtin groups) and is already `extern`-declared in `group.h`. `fmt_consumer()`/`fmt_limit()` (`src/lib/xcelimit.c`) already compose `fmt_group()` into a full per-limit configuration dump.
- The bug-1789/1790 chain (global limits under-counting usage against per-host resources) is fully closed as of 2026-09-07 — the underlying accounting this feature displays is trustworthy across all resource/scope combinations by the time this spec is implemented.
- **Implications**: Requirement 1's "configured setting" display can adopt `fmt_limit()` directly instead of re-deriving formatting logic; only the effective-host-scope annotation (Req 1.6) and the live-instance/usage rendering (Req 2, Req 3) are genuinely new code.
### REPL command dispatch and existing `info resource` precedent
- **Context**: Need the concrete integration point for a new `info limit` subcommand and a working precedent for the GLOBAL/SHARED-vs-per-host live-value read split Requirement 2/3 require.
- **Sources Consulted**: `src/scheduler/src/repl.c` (`repl_dispatch`), `src/scheduler/src/repl_info.c` (`repl_cmd_info`, `repl_info_resource`, `repl_info_resource_one`), `src/scheduler/src/plugin.c` (`plugin_register`, `plugin_register_core`), `src/scheduler/include/plugin.h`, bug-1593 (`repl_info_resource_one`'s original GLOBAL/SHARED-vs-host-based display bug and fix).
- **Findings**:
- `repl_cmd_info()` is a flat `strcmp` chain on `tokens[1]`; every existing subcommand (`job`, `queue`, `worker`, `resource`, ...) follows the identical `<name>|all` argument shape Requirement 4 asks for.
- `repl_info_resource_one()` (post bug-1593) reads a GLOBAL/SHARED resource's value from `sim->core.sharedInst`/`sharedInUse` at `r->idx`, and explicitly declines to fabricate a value for a host-based resource, deferring to `info worker <name>` instead. This is the exact pattern Requirement 2/3's per-instance read path reuses.
- Plugins are registered into `core->plugins`, a name-keyed handle (`key_add(core->plugins, plugin->name, plugin)`); the limit plugin registers under the fixed name `"xce_sched_limit"` (`limit.c:557`). No plugin is ever renamed or re-registered under a different name at runtime.
- Synthetic limit-capacity resources (`vres->vres`, created by `add_vres()`) are inserted into the *same* `core.resmgr` table `info resource`/`info worker` already iterate, flagged `XCE_RESF_LIMIT` (already a named flag in `repl_resource_flags_str`'s table). A GLOBAL-scoped limit's synthetic resource can therefore already appear in `info resource all`/`info resource <name>` today, labeled with the `LIMIT` flag — this is pre-existing behavior, unrelated to and unmodified by this spec.
- **Implications**: New command wiring is a single new branch in `repl_cmd_info()` plus one new static function `repl_info_limit()` in `repl_info.c`, following `repl_info_resource()`'s two-level (`_all` iterates, `_one` prints) shape.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|----------------------|-------|
| Direct struct exposure | Move `limit_plugin_data_t` into a shared header, let `repl_info.c` include it directly | Fewer functions | Breaks the plugin's existing data encapsulation; every other plugin's private state stays file-local — this would be the only exception | Rejected |
| Opaque query API (iterator_t out-params, per-limit filtered iteration) | New multi-function query interface returning filtered iterators | Nice call-site ergonomics | New abstraction with only one consumer today; no existing accessor in this codebase does filtered iteration — the rest (`sched_get_job`, `core->plugins`, `core->jobs`) hand back a raw kvmgr/xlist handle and let the caller `iter_make`/`iter_next` | Rejected — over-built for current scope |
| **Selected: two raw-handle accessors** | `limit_get_configured_limits(core)` / `limit_get_usage_instances(core)` return the plugin's existing `var` (kvmgr/xlist) handles, or `NULL` if the plugin isn't registered; caller iterates with the same `iter_make`/`iter_next`/`iter_destroy` idiom used everywhere else in this codebase | Matches every existing accessor pattern (`core->jobs`, `core->queues`, `core->plugins`); minimal new surface (2 functions); no new types | Caller must filter usage instances by `vres->limit == target_limit` itself (pointer comparison) since there is no per-limit index — acceptable, this table is small (configured-limit count × scope-instance count, not job count) | Selected |
## Design Decisions
### Decision: Plugin accessor shape
- **Context**: `limit_plugin_data_t` (`limits`, `instances`) is a file-local type in `limit.c`; Requirement 1/2/3 need read access to both from `repl_info.c`.
- **Alternatives Considered**:
1. Expose the struct itself in a shared header.
2. Build a filtered query API (per-name lookup, per-limit instance iteration) as new plugin-owned functions.
3. Expose the two raw `var` handles as accessor functions, encapsulating the plugin-name lookup (`"xce_sched_limit"`) inside the accessor itself so `repl_info.c` never needs to know it.
- **Selected Approach**: (3). `limit_get_configured_limits(xce_sched_core_t* core)` and `limit_get_usage_instances(xce_sched_core_t* core)`, both declared in `src/scheduler/include/plugin.h` next to `register_limit`, defined in `limit.c`. Each does its own `key_find(core->plugins, "xce_sched_limit", &plugin)` and returns `plugin->data`'s relevant field (or `NULL` if the plugin is not registered or not yet initialized).
- **Rationale**: Matches the codebase's existing accessor idiom exactly (raw handle + shared iterator trio), keeps the plugin-name string fully inside `limit.c` (the one place that already owns it via `register_limit()`), and adds the minimum surface needed — no speculative per-limit-name lookup function, since both AC1's "specific name" and AC2's "all" cases are satisfiable by iterating the same handle and filtering in the caller.
- **Trade-offs**: `repl_info_limit()` finds a limit by name via a direct `key_find()` (`privdata->limits` is a name-keyed kvmgr — confirmed by bug-1685's duplicate-name `-EEXIST` handling, so this is O(1)-ish, not a scan) and a linear scan + pointer-equality filter to find that limit's usage instances in `privdata->instances` (keyed by vresname, not by owning limit — no index avoids this scan; accepted, same scale argument as below).
- **Follow-up**: None — this is a stable, minimal boundary; a future capability under this same open-ended spec (e.g. decision-reasoning trace) does not need to touch this accessor.
### Decision: Reuse `fmt_limit()`/`fmt_group()` for configured-setting display
- **Context**: Requirement 1 AC1/AC5 need to print a limit's name, per-dimension scope, and inclusion/exclusion membership.
- **Alternatives Considered**:
1. Write new formatting code in `repl_info.c` mirroring the other `repl_info_*_one()` functions' inline `fprintf` style.
2. Call the existing `fmt_limit()` (which internally calls `fmt_consumer()`/`fmt_group()`) into a stack buffer and print it, then append the REPL-specific effective-scope note and live-usage lines the library formatter cannot know about.
- **Selected Approach**: (2).
- **Rationale**: `fmt_limit()` already produces correct, exclusion-aware output and is already used/tested at the library level (`src/lib/testsuite/test_limit.c`); duplicating that logic in `repl_info.c` would both waste effort and risk drifting out of sync with the library's own definition of "configured setting." Build-vs-adopt: adopt.
- **Trade-offs**: `fmt_limit()`'s output format (multi-line, `PER_USER = [...]` style) is fixed by the library; the REPL wraps it with a `[sim]` prefix per line rather than reformatting it, consistent with how other multi-line `info` output is already prefixed in this codebase (e.g. `info jobs`).
- **Follow-up**: None.
### Decision: Effective-scope annotation is REPL-layer, not library-layer
- **Context**: Requirement 1 AC6 (implicit host-scoping for a percent-based limit on a non-shared resource) is not something `fmt_limit()` knows about — it only reflects the *configured* `consumers[]` array.
- **Alternatives Considered**:
1. Change `fmt_limit()` itself to call `is_host_based_limit()` and inject a synthetic `PER_HOST` line.
2. Keep `fmt_limit()` unchanged (it is a general-purpose library formatter used elsewhere, e.g. by any future non-REPL consumer that wants the raw configured shape) and have `repl_info_limit()` call `is_host_based_limit()` itself, appending an explanatory note only when `consumers[CONSUMER_HOST] == NULL` and the check returns true.
- **Selected Approach**: (2).
- **Rationale**: `is_host_based_limit()` is already `extern`-declared in `xcelimit.h` and used exactly this way (a yes/no runtime check) elsewhere; the "effective scope" concept is specific to what an *operator inspecting the REPL* needs to know, not a general property of the limit's configured shape — keeping it at the REPL layer avoids changing a shared library formatter's contract for one caller's needs.
- **Trade-offs**: None material.
- **Follow-up**: None.
## Risks & Mitigations
- **Risk**: Linear scan over `privdata->instances` to filter by `vres->limit == target_limit` (this table is keyed by vresname, not by owning limit, so no index avoids the scan) could be surprising if a deployment ever configures an unusually large number of distinct scope instances (e.g. thousands of users × per-user limits). — **Mitigation**: Accepted for this spec; the existing `match_job_host_limit_`/`match_job_global_limit_` matching path already does a `key_find` per job per limit against the same table, so this table's size is already load-bearing elsewhere in the scheduler's hot path — a read-only REPL scan over it is not a new scaling concern this spec introduces.
- **Risk**: Because `is_host_based_limit()` takes the plugin's `resmgr` handle (a `var`), calling it from `repl_info.c` requires reaching `sim->core.resmgr` — already public (used directly by `repl_info_resource()` today), so no new exposure needed. — **Mitigation**: N/A, confirmed no new risk.
- **Risk**: `build_vres_name()` never encodes host in the vresname it builds — an implementer could reasonably assume "one `xce_vres_t` per (user, queue, project, host)" and design storage/iteration around a cardinality that does not exist. — **Mitigation**: `design.md`'s Data Models section states the correct cardinality explicitly (one `xce_vres_t` per (user, queue, project); host is produced by a per-worker fan-out over that single `xce_vres_t`'s shared resource index, not by additional `xce_vres_t` instances) and cites this finding.
- **Risk**: Build wiring for the new white-box test — `test_repl_resource_table.c`'s existing `#include` set omits `plugins/limit.c` and `resource.c`; naively adding a limit-info test there, or to a careless new file, reproduces the exact undefined-reference class bug-1762 already hit once for a sibling REPL feature. — **Mitigation**: `design.md`'s Testing Strategy calls this out explicitly and directs a new sibling test file (`test_repl_limit_table.c`) with its own white-box include set, real (not stubbed) `resource.c` since the test needs non-trivial accounted-usage values.
## Additional Findings (design review pass)
- `xce_sched_worker_t` carries two separate zarrays per resource idx: `.resvals` (capacity) and `.resInUse` (accounted usage) — confirmed via `sched_check_host_satisfy_()` (`resource.c`), which reads both the identical way this design's per-host instance display needs to.
- `instantiate_limit()` and `update_limit_res_value()` run back-to-back inside the same `match_job_host_limit_`/`match_job_global_limit_` call (`plugins/limit.c`) — a matched `xce_vres_t`'s *capacity* entry is therefore always populated by the time the `xce_vres_t` itself is observable via the new accessors; only its *in-use* entry can legitimately be absent (genuine zero usage, not "no instance").
- `resource-accounting.md`'s reservation-vs-actual-usage reconciliation is explicitly scoped to resources flagged both `XCE_RESF_DYNAMIC` and `XCE_RESF_CONSUMABLE`. A limit's synthetic capacity resource (`add_vres()`) is flagged `XCE_RESF_LIMIT | XCE_RESF_CONSUMABLE | XCE_RESF_RELEASE` plus `GLOBAL|SHARED` or `DEFAULT_HOST` — never `XCE_RESF_DYNAMIC` — so that steering document does not apply here; the capacity/in-use pair this design reads is an exact scheduler-computed accounting number, not an OS-reconciled estimate.
- `src/scheduler/testsuite/test_no_bare_printf.c` regression-tests every file under `src/scheduler/src/` for bare `printf()` calls (must use `fprintf(stdout, ...)`); bug-1599 documents a known scanner false-positive on *comment prose*, not an exemption for real code.
## References
- `.wolf/buglog.json` bug-1784–bug-1790 (gitea#46 limit-matching chain, 2026-09-07) — internal, not a URL.
- `.wolf/buglog.json` bug-1593 (`info resource` GLOBAL/SHARED-vs-host-based display fix, 2026-08-19) — internal, not a URL.
- `.wolf/buglog.json` bug-1685 (limit plugin double-free; also the evidence that `privdata->limits` is name-keyed) — internal, not a URL.
- `.wolf/buglog.json` bug-1762 (build-wiring precedent for white-box REPL test files) — internal, not a URL.
- `.wolf/buglog.json` bug-1599 (bare-printf regression scanner false-positive) — internal, not a URL.
- `.kiro/steering/resource-accounting.md` — reservation-vs-actual-usage policy, scoped to `XCE_RESF_DYNAMIC`; confirmed not applicable to this design.

16
.kiro/specs/scheduler-sim-diagnostics/spec.json
View file

@ -1,24 +1,24 @@
{
"feature_name": "scheduler-sim-diagnostics",
"created_at": "2026-08-16T11:02:06Z",
"updated_at": "2026-09-01T21:15:00Z",
"updated_at": "2026-09-17T02:30:00Z",
"language": "en",
"phase": "requirements-generated",
"phase": "tasks-generated",
"description": "Open-ended simulation-mode REPL diagnostics spec. First capability: limit inspection (info limit <name|all>) showing configured setting (scope, value/percent) and live accounted usage per instance. Future capability (seed capability 2, not yet in requirements.md): decision-reasoning trace for pend/dispatch outcomes.",
"approvals": {
"requirements": {
"generated": true,
"approved": false
"approved": true
},
"design": {
"generated": false,
"approved": false
"generated": true,
"approved": true
},
"tasks": {
"generated": false,
"approved": false
"generated": true,
"approved": true
}
},
"ready_for_implementation": false,
"ready_for_implementation": true,
"blocked_on": []
}

62
.kiro/specs/scheduler-sim-diagnostics/tasks.md Normal file
View file

@ -0,0 +1,62 @@
# Implementation Plan: scheduler-sim-diagnostics (Requirement Set 1 — Limit Inspection)
- [ ] 1. Add read-only accessors exposing the limit plugin's configured-limits and live-usage-instances tables
- Two new functions (`limit_get_configured_limits()`, `limit_get_usage_instances()`) returning raw kvmgr handles (or `NULL` when the plugin is absent), declared in `plugin.h` alongside `register_limit()`, defined in `plugins/limit.c`
- Never allocate, mutate, or trigger plugin hooks — a pure read of already-existing pointers
- New/extended unit tests in `test_limit_plugin.c` cover: plugin registered with configured limits (handle returned, iterable), plugin registered with zero limits, plugin not registered (`NULL`)
- Observable: a test that registers the limit plugin with a known limit and confirms the accessor's returned handle, when iterated via `iter_make()`/`iter_next()`, yields that same limit; a sibling test confirms `NULL` when the plugin was never registered
- _Requirements: 1.1, 1.2, 1.4, 2.1, 3.1_
- _Boundary: Limit Plugin Accessor_
- [ ] 2. Core: Limit lookup and configured-setting display
- [ ] 2.1 Implement name/all resolution with correct absence messaging
- New white-box test file `test_repl_limit_table.c` created (sibling of `test_repl_resource_table.c`, own `#include` set covering `plugins/limit.c` and `resource.c` for real — not stubbed — per `design.md`'s build-wiring risk note citing bug-1762's precedent)
- `repl_info_limit(sim, arg)`: `arg == "all"` iterates `limit_get_configured_limits()`; `NULL`/empty prints "no limits configured" (Requirement 1.7); `arg != "all"` does a `key_find()` by name (not a scan) on the same handle, printing "no such limit '<name>'" (Requirement 1.3) for a `NULL` handle or a miss — a `NULL` handle on this path is never treated as "no limits configured," matching that message's own `all`-only scoping
- Wired as a direct call for now (`repl_dispatch()`/`repl_cmd_info()` wiring itself deferred to task 4); tests in this task call `repl_info_limit()` directly, not through `repl_dispatch()`
- Observable: REPL output shows "no such limit 'foo'" for an unconfigured name against a fixture with other limits configured, and "no limits configured" for `all` against a fixture with zero configured limits
- _Requirements: 1.3, 1.7, 4.1, 4.2_
- _Depends: 1_
- [ ] 2.2 Display configured setting including scope membership
- Matched limit's name, constrained resource, value/percent, and per-dimension scope print via the existing `fmt_limit()` library formatter into a stack buffer, line-prefixed with the existing `[sim] ` convention; all output via `fprintf(stdout, ...)`, never bare `printf()` (`test_no_bare_printf.c` regression-gates this)
- Inclusion and exclusion group membership both render (`fmt_group()` already handles this, adopted as-is)
- Observable: a limit configured with an exclusion group ("all except X") shows both the inclusion and the exclusion in its output; `info limit all` against a fixture with multiple configured limits displays every one's setting
- _Requirements: 1.1, 1.2, 1.5_
- [ ] 2.3 Display effective host-scope for implicitly host-scoped limits
- After `fmt_limit()`'s output, call `is_host_based_limit(core->resmgr, limit)`; if it returns `> 0` AND the limit's own per-host consumer is not configured, print an explanatory host-scope note
- Observable: two REPL fixtures — one percent-based limit on a per-host resource without explicit per-host scoping (note appears), one absolute-value or explicitly-global-scoped limit (note does not appear)
- _Requirements: 1.6_
- [ ] 3. Core: Live accounted-usage display
- [ ] 3.1 Display usage for cluster-wide (non-host-scoped) limit instances
- Iterate `limit_get_usage_instances()`, filter by `vres->limit == matched_limit`; for a GLOBAL/SHARED `vres`, read capacity from `core.sharedInst[idx]` and accounted usage from `core.sharedInUse[idx]`
- Existence-of-instance and zero-usage are distinct: zero matching `xce_vres_t` (no capacity entry at all) prints "no active usage instance yet" (Requirement 2.2); a matched `xce_vres_t` whose `sharedInUse[idx]` is `NULL`/absent instead displays a genuine `accounted=0` (it has been evaluated and has a real capacity, just nothing charged against it yet) — existence is gated on the capacity read, never on the in-use read
- Observable: a global limit that has matched at least one job shows an accounted/capacity pair reflecting current state; a configured-but-never-evaluated limit shows the explicit no-instance message instead; a matched-but-never-charged limit shows `accounted=0` rather than the no-instance message
- _Requirements: 1.4, 2.1, 2.2, 2.3, 3.3_
- _Depends: 1_
- [ ] 3.2 Display per-host usage for host-scoped limit instances
- For a host-scoped `vres` (explicit or implicit per 2.3), iterate `core.workers`; include a worker only if it has a populated capacity entry (`w->resvals[idx]` non-`NULL`) — read capacity from `w->resvals[idx]` and accounted usage from `w->resInUse[idx]` (same existence-vs-zero-usage distinction as 3.1, applied per worker); label each instance with its scope values (user/queue/project from the vresname, plus host)
- A limit varying by more than one scope dimension at once (e.g. per-user and per-host together) shows each distinct combination as its own labeled instance
- Observable: a limit matched against jobs on two distinct hosts shows two separate per-host usage lines each labeled with its own scope values, and a third configured-but-untouched host shows no line at all
- _Requirements: 1.4, 2.1, 2.2, 2.3, 3.1, 3.2, 3.4_
- [ ] 4. Register `info limit` as a recognized `info` subcommand
- New `else if (strcmp(tokens[1], "limit") == 0)` branch in `repl_cmd_info()` (`repl_info.c`) routes to `repl_info_limit()`; usage-string literal updated to list `limit` alongside the existing subcommands
- `test_repl_limit_table.c` gains a full dispatch-round-trip test (via `repl_dispatch()`, not a direct call) for both `info limit <name>` and `info limit all`
- Observable: `info limit <name>` and `info limit all` both work when typed as real REPL input (through `repl_dispatch()`), and `help`/usage output lists the new subcommand
- _Requirements: 4.1, 4.2_
- _Depends: 2.1, 3.1, 3.2_
- [ ] 5. Verify no mutation, replay determinism, and existing-subcommand regression
- Test: displaying limit information never alters configuration, accounting state, or scheduling decisions (assert config/accounting untouched before vs. after)
- Test: a replay run twice — once with `info limit` issued mid-session, once without — produces identical scheduling decisions and identical final accounted state
- Full existing `info` subcommand test suite (job/queue/worker/resource/summary) re-run to confirm zero output changes from this addition
- Observable: the two new determinism/no-mutation tests pass, and the full pre-existing info-command test suite shows zero diffs
- _Requirements: 5.1, 5.2, 5.3_
- _Depends: 4_
## Parallel Execution Note
No `(P)` markers are used anywhere in this plan. Tasks 2.1–2.3 and 3.1–3.2 all incrementally edit the same function (`repl_info_limit()`) in the same file (`repl_info.c`) and largely share the same new test file (`test_repl_limit_table.c`) — the parallel-analysis rules' "no conflicting files or shared mutable resources" criterion rules out marking any of them concurrent. Task 1 is the sole Foundation task (rarely `(P)` by rule, and has no peer to run alongside regardless). Tasks 4 and 5 are Integration/Validation work that structurally depends on Core being complete.