myxce/.kiro/specs/scheduler-sim-diagnostics/requirements.md

Requirements Document

Introduction

Operators investigating scheduler behavior in the --interactive simulation REPL (shipped by scheduler-core) can already inspect job, queue, worker, and resource state through the REPL's info command family, but cannot see what resource limits are configured or how much of each is currently accounted for. This document covers the first capability of the open-ended scheduler-sim-diagnostics spec: giving the REPL visibility into limits — both a limit's configured setting and the live, accounted usage tracked against it — so an operator can answer "what limit is in effect, and how close is it to being hit" without reading cluster configuration files or adding ad hoc instrumentation.

Boundary Context

• In scope: REPL-visible display of a limit's configured setting (the resource it constrains, its configured value or percentage, and which scope dimensions it effectively varies by, including any host-scoping implied by a percentage value on a resource that is not shared cluster-wide) and its live accounted usage (per distinct scope instance, including explicit exclusion-based scope restrictions), in simulation mode only.
• Out of scope: any command that mutates scheduler state; live, controller-connected (non-simulation) inspection; explaining why a specific job did or did not trigger a limit (decision-reasoning trace — moved 2026-09-28 to its own spec scheduler-sim-decision-trace, not covered here); any change to scheduling, allocation, or limit-matching behavior.
• Adjacent expectations: this capability extends the REPL's existing info command family (shipped by scheduler-core, implementation-complete) without altering any existing subcommand's behavior or output; it must not perturb the guarantee that a simulation replay reproduces identical scheduling decisions and identical accounting state regardless of what inspection commands are issued during it. Requirement 6 (added 2026-09-19, following operator feedback that a limit's raw byte amounts are hard to read) relies on the REPL's existing byte-unit display convention already used by other info subcommands for byte-valued resource amounts, including an operator's existing ability to select how those amounts are displayed — it extends that existing convention to limit displays rather than introducing a new one.

Requirements

Requirement 1: Limit Configuration Display

Objective: As an operator investigating scheduler behavior, I want to see each limit's configured setting, so that I can confirm what constraint is actually in effect without reading cluster configuration files.

Acceptance Criteria

1. When an operator requests information for a specific limit by name, the REPL shall display that limit's name, the resource it constrains, its configured value, whether that value is an absolute amount or a percentage, and which scope dimensions (user, queue, project, host) it effectively varies by.
2. When an operator requests information for all limits, the REPL shall display the configuration described in Acceptance Criterion 1 for every limit currently configured in the simulation.
3. If an operator requests information for a limit name that is not configured, then the REPL shall report that no such limit exists and shall not display partial or fabricated information.
4. Where a limit's configured value is a percentage, the REPL shall also display the current absolute amount that percentage resolves to for each instance the limit applies to, since that amount can change as the underlying resource's value changes and may differ between instances when the limit is host-scoped.
5. Where a limit's scope is restricted to a specific set of users, queues, projects, or hosts — whether by explicit inclusion, explicit exclusion, or both — the REPL shall display that restricted membership, distinguishing included members from explicitly excluded ones.
6. Where a limit's configured value is a percentage of a resource that is not shared cluster-wide, the REPL shall include the host dimension in the scope it displays for that limit even when the limit's configuration does not explicitly restrict it by host, since usage for such a limit is tracked separately per host regardless of that configuration.
7. If no limits are configured in the simulation, then the REPL shall report that no limits are configured when an operator requests information for all limits, rather than displaying an empty listing with no explanation.

Requirement 2: Live Accounted Usage Display

Objective: As an operator watching a simulation replay, I want to see how much of each limit is currently accounted for, so that I can tell whether a limit is close to being exhausted or was the reason a job was denied.

Acceptance Criteria

1. When an operator requests information for a limit, the REPL shall display, for each of that limit's currently active instances, the amount currently accounted against it alongside its configured capacity.
2. While a limit has not yet been evaluated against any job, the REPL shall report that it has no active usage instance yet, rather than displaying a fabricated zero-usage instance.
3. The REPL shall display accounted usage reflecting the simulation's state at the moment the command is issued.

Requirement 3: Instance Scoping and Enumeration

Objective: As an operator, I want limits that vary by user, queue, project, or host to show every distinct instance currently in effect, so that I don't mistake one instance's usage for the limit's usage as a whole.

Acceptance Criteria

1. When a limit varies by more than one scope dimension at once (for example per-user and per-queue together, or per-host and per-user together), the REPL shall display each distinct combination of scoped values as its own instance with its own accounted usage.
2. When a limit varies by host — including when the host dimension is implicit per Requirement 1's Acceptance Criterion 6 rather than explicitly configured — the REPL shall break out accounted usage separately for each host on which that limit has been evaluated, even when the limit also varies by another scope dimension at the same time.
3. When a limit does not vary by any scope dimension, the REPL shall display a single instance representing the entire simulation.
4. The REPL shall label each displayed instance with the scope values it represents, so an operator can tell instances apart.

Requirement 4: Command Invocation

Objective: As an operator, I want a predictable way to ask for limit information, consistent with how existing REPL info commands work, so that this new capability fits how I already work.

Acceptance Criteria

1. The REPL shall provide a way to request limit information for a single named limit or for all configured limits, consistent with how existing info subcommands accept a name or "all".
2. The REPL shall accept a request for limit information at any point it accepts other info subcommands.

Requirement 5: No Interference with Scheduling or Determinism

Objective: As an operator relying on simulation replay to faithfully reproduce recorded production behavior, I want limit inspection to be strictly read-only, so that asking for limit information never changes what the simulation does.

Acceptance Criteria

1. The REPL shall not alter any limit's configuration, usage accounting, or any scheduling decision as a result of displaying limit information.
2. The REPL shall produce identical scheduling decisions and identical final accounting state for a replay whether or not limit information was requested during that replay.
3. The REPL shall preserve all existing info subcommands' behavior and output unchanged.

Requirement 6: Human-Readable Units for Byte-Valued Limit Amounts

Objective: As an operator inspecting a limit whose constrained resource is measured in bytes (for example memory), I want large byte amounts displayed in a human-readable scale rather than as a raw byte count, so that I can read and compare limit values and usage at a glance instead of counting digits.

Acceptance Criteria

1. When the REPL displays a limit's configured amount or a limit's live accounted or capacity amount for a resource whose canonical unit is bytes, the REPL shall present that amount using the same human-readable unit scaling (kilobytes, megabytes, gigabytes, or terabytes as appropriate) that it already uses when displaying other byte-valued resource amounts.
2. Where an operator has selected how byte-valued resource amounts are displayed elsewhere in the REPL, the REPL shall present a limit's byte-valued amounts consistent with that same selection.
3. The REPL shall not apply byte-unit scaling to a limit's amount when the limit's constrained resource is not byte-valued, including a limit expressed as a percentage of a resource whose own canonical unit is not bytes.
4. Where a limit's configured value is a percentage of a resource whose canonical unit is bytes, the REPL shall display the resolved absolute amount described in Requirement 1's Acceptance Criterion 4 using the same human-readable unit scaling as a non-percentage byte-valued limit.
5. The REPL shall apply this unit scaling consistently to both a limit's configured setting display and its live accounted usage display, so an operator sees the same units convention throughout a single request for limit information.