myxce/.kiro/specs/scheduler-sim-diagnostics/research.md

# Research & Design Decisions

## Summary
- **Feature**: `scheduler-sim-diagnostics` — Requirement set 1 (`info limit` REPL command)
- **Discovery Scope**: Extension (extends `scheduler-core`'s existing REPL `info` command family and reuses the limit plugin's existing library-level formatters)
- **Key Findings**:
  - The REPL `info` command family is a single dispatch function (`repl_cmd_info()`, `repl_info.c`) with one `else if` branch per subcommand — adding `limit` is a mechanical extension of an existing, well-established pattern.
  - The limit plugin's private state (`limit_plugin_data_t.limits`/`.instances`) is genuinely inaccessible outside `src/scheduler/src/plugins/limit.c` today — no header declares the type — confirming the brief's stated gap. Two new one-line accessor functions close it.
  - Two library-level formatters already exist and already do most of Requirement 1's work: `fmt_limit()` and `fmt_consumer()`/`fmt_group()` (`src/lib/xcelimit.c`, `src/lib/group.c`), including correct inclusion/exclusion membership rendering (`fmt_group()` already prefixes excluded members with `~`). These are adopted directly rather than rebuilt.
  - The live-usage read path (GLOBAL/SHARED single value vs. per-host breakout) is architecturally identical to the pattern `repl_info_resource_one()` already implements for ordinary resources (bug-1593) — the limit's synthetic capacity resource (`vres->vres`) is a real, already-registered `xce_resource_t` carrying the same `XCE_RESF_GLOBAL`/`XCE_RESF_SHARED` flags, so the same shared-vs-per-host read logic applies unmodified.

## Research Log

### Limit plugin data model and matching semantics
- **Context**: Requirement 1 AC1/AC5/AC6 need to know exactly what "scope dimensions" and "restricted membership" mean at the code level, since an earlier requirements-review pass surfaced a scope-vs-runtime ambiguity.
- **Sources Consulted**: `src/include/xce/types.h` (`xce_limit_t`, `xce_limit_config_t`, `xce_consumer_t`, `xce_vres_t`), `src/lib/xcelimit.c` (`match_limit`, `build_vres_name`, `is_host_based_limit`, `instantiate_limit`), `src/scheduler/src/plugins/limit.c` (`match_job_host_limit_`, `match_job_global_limit_`, `update_limit_res_value`, `limit_prepare_allocation`), `.wolf/buglog.json` bug-1784 through bug-1790 (the gitea#46 limit-matching bug chain, 2026-09-07).
- **Findings**:
  - `xce_consumer_type_t` has exactly four values (`CONSUMER_USER`, `CONSUMER_QUEUE`, `CONSUMER_PROJECT`, `CONSUMER_HOST`) — the four scope dimensions in Requirement 1 AC1 are the complete set, no fifth dimension exists.
  - `is_host_based_limit()` forces per-host live-instance tracking whenever the limit's value is a percentage of a resource that is not `XCE_RESF_GLOBAL`/`XCE_RESF_SHARED` — independent of whether `CONSUMER_HOST` was configured. This is real, current behavior (not hypothetical), confirmed via source and via bug-1789/1790's live-verified fix history for the equivalent allocation-side computation.
  - `xce_group_t` supports both `members` and `ex_members` (exclusion). Confirmed as real production configuration via bug-1789's own note describing a live-configured limit ("`limit1`... all users except rshen").
  - `fmt_group()` (`src/lib/group.c:388`) already renders both lists correctly (`~name` prefix for exclusions, `all`/`[]` shortcuts for the builtin groups) and is already `extern`-declared in `group.h`. `fmt_consumer()`/`fmt_limit()` (`src/lib/xcelimit.c`) already compose `fmt_group()` into a full per-limit configuration dump.
  - The bug-1789/1790 chain (global limits under-counting usage against per-host resources) is fully closed as of 2026-09-07 — the underlying accounting this feature displays is trustworthy across all resource/scope combinations by the time this spec is implemented.
- **Implications**: Requirement 1's "configured setting" display can adopt `fmt_limit()` directly instead of re-deriving formatting logic; only the effective-host-scope annotation (Req 1.6) and the live-instance/usage rendering (Req 2, Req 3) are genuinely new code.

### REPL command dispatch and existing `info resource` precedent
- **Context**: Need the concrete integration point for a new `info limit` subcommand and a working precedent for the GLOBAL/SHARED-vs-per-host live-value read split Requirement 2/3 require.
- **Sources Consulted**: `src/scheduler/src/repl.c` (`repl_dispatch`), `src/scheduler/src/repl_info.c` (`repl_cmd_info`, `repl_info_resource`, `repl_info_resource_one`), `src/scheduler/src/plugin.c` (`plugin_register`, `plugin_register_core`), `src/scheduler/include/plugin.h`, bug-1593 (`repl_info_resource_one`'s original GLOBAL/SHARED-vs-host-based display bug and fix).
- **Findings**:
  - `repl_cmd_info()` is a flat `strcmp` chain on `tokens[1]`; every existing subcommand (`job`, `queue`, `worker`, `resource`, ...) follows the identical `<name>|all` argument shape Requirement 4 asks for.
  - `repl_info_resource_one()` (post bug-1593) reads a GLOBAL/SHARED resource's value from `sim->core.sharedInst`/`sharedInUse` at `r->idx`, and explicitly declines to fabricate a value for a host-based resource, deferring to `info worker <name>` instead. This is the exact pattern Requirement 2/3's per-instance read path reuses.
  - Plugins are registered into `core->plugins`, a name-keyed handle (`key_add(core->plugins, plugin->name, plugin)`); the limit plugin registers under the fixed name `"xce_sched_limit"` (`limit.c:557`). No plugin is ever renamed or re-registered under a different name at runtime.
  - Synthetic limit-capacity resources (`vres->vres`, created by `add_vres()`) are inserted into the *same* `core.resmgr` table `info resource`/`info worker` already iterate, flagged `XCE_RESF_LIMIT` (already a named flag in `repl_resource_flags_str`'s table). A GLOBAL-scoped limit's synthetic resource can therefore already appear in `info resource all`/`info resource <name>` today, labeled with the `LIMIT` flag — this is pre-existing behavior, unrelated to and unmodified by this spec.
- **Implications**: New command wiring is a single new branch in `repl_cmd_info()` plus one new static function `repl_info_limit()` in `repl_info.c`, following `repl_info_resource()`'s two-level (`_all` iterates, `_one` prints) shape.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|----------------------|-------|
| Direct struct exposure | Move `limit_plugin_data_t` into a shared header, let `repl_info.c` include it directly | Fewer functions | Breaks the plugin's existing data encapsulation; every other plugin's private state stays file-local — this would be the only exception | Rejected |
| Opaque query API (iterator_t out-params, per-limit filtered iteration) | New multi-function query interface returning filtered iterators | Nice call-site ergonomics | New abstraction with only one consumer today; no existing accessor in this codebase does filtered iteration — the rest (`sched_get_job`, `core->plugins`, `core->jobs`) hand back a raw kvmgr/xlist handle and let the caller `iter_make`/`iter_next` | Rejected — over-built for current scope |
| **Selected: two raw-handle accessors** | `limit_get_configured_limits(core)` / `limit_get_usage_instances(core)` return the plugin's existing `var` (kvmgr/xlist) handles, or `NULL` if the plugin isn't registered; caller iterates with the same `iter_make`/`iter_next`/`iter_destroy` idiom used everywhere else in this codebase | Matches every existing accessor pattern (`core->jobs`, `core->queues`, `core->plugins`); minimal new surface (2 functions); no new types | Caller must filter usage instances by `vres->limit == target_limit` itself (pointer comparison) since there is no per-limit index — acceptable, this table is small (configured-limit count × scope-instance count, not job count) | Selected |

## Design Decisions

### Decision: Plugin accessor shape
- **Context**: `limit_plugin_data_t` (`limits`, `instances`) is a file-local type in `limit.c`; Requirement 1/2/3 need read access to both from `repl_info.c`.
- **Alternatives Considered**:
  1. Expose the struct itself in a shared header.
  2. Build a filtered query API (per-name lookup, per-limit instance iteration) as new plugin-owned functions.
  3. Expose the two raw `var` handles as accessor functions, encapsulating the plugin-name lookup (`"xce_sched_limit"`) inside the accessor itself so `repl_info.c` never needs to know it.
- **Selected Approach**: (3). `limit_get_configured_limits(xce_sched_core_t* core)` and `limit_get_usage_instances(xce_sched_core_t* core)`, both declared in `src/scheduler/include/plugin.h` next to `register_limit`, defined in `limit.c`. Each does its own `key_find(core->plugins, "xce_sched_limit", &plugin)` and returns `plugin->data`'s relevant field (or `NULL` if the plugin is not registered or not yet initialized).
- **Rationale**: Matches the codebase's existing accessor idiom exactly (raw handle + shared iterator trio), keeps the plugin-name string fully inside `limit.c` (the one place that already owns it via `register_limit()`), and adds the minimum surface needed — no speculative per-limit-name lookup function, since both AC1's "specific name" and AC2's "all" cases are satisfiable by iterating the same handle and filtering in the caller.
- **Trade-offs**: `repl_info_limit()` finds a limit by name via a direct `key_find()` (`privdata->limits` is a name-keyed kvmgr — confirmed by bug-1685's duplicate-name `-EEXIST` handling, so this is O(1)-ish, not a scan) and a linear scan + pointer-equality filter to find that limit's usage instances in `privdata->instances` (keyed by vresname, not by owning limit — no index avoids this scan; accepted, same scale argument as below).
- **Follow-up**: None — this is a stable, minimal boundary; a future capability under this same open-ended spec (e.g. decision-reasoning trace) does not need to touch this accessor.

### Decision: Reuse `fmt_limit()`/`fmt_group()` for configured-setting display
- **Context**: Requirement 1 AC1/AC5 need to print a limit's name, per-dimension scope, and inclusion/exclusion membership.
- **Alternatives Considered**:
  1. Write new formatting code in `repl_info.c` mirroring the other `repl_info_*_one()` functions' inline `fprintf` style.
  2. Call the existing `fmt_limit()` (which internally calls `fmt_consumer()`/`fmt_group()`) into a stack buffer and print it, then append the REPL-specific effective-scope note and live-usage lines the library formatter cannot know about.
- **Selected Approach**: (2).
- **Rationale**: `fmt_limit()` already produces correct, exclusion-aware output and is already used/tested at the library level (`src/lib/testsuite/test_limit.c`); duplicating that logic in `repl_info.c` would both waste effort and risk drifting out of sync with the library's own definition of "configured setting." Build-vs-adopt: adopt.
- **Trade-offs**: `fmt_limit()`'s output format (multi-line, `PER_USER = [...]` style) is fixed by the library; the REPL wraps it with a `[sim]` prefix per line rather than reformatting it, consistent with how other multi-line `info` output is already prefixed in this codebase (e.g. `info jobs`).
- **Follow-up**: None.

### Decision: Effective-scope annotation is REPL-layer, not library-layer
- **Context**: Requirement 1 AC6 (implicit host-scoping for a percent-based limit on a non-shared resource) is not something `fmt_limit()` knows about — it only reflects the *configured* `consumers[]` array.
- **Alternatives Considered**:
  1. Change `fmt_limit()` itself to call `is_host_based_limit()` and inject a synthetic `PER_HOST` line.
  2. Keep `fmt_limit()` unchanged (it is a general-purpose library formatter used elsewhere, e.g. by any future non-REPL consumer that wants the raw configured shape) and have `repl_info_limit()` call `is_host_based_limit()` itself, appending an explanatory note only when `consumers[CONSUMER_HOST] == NULL` and the check returns true.
- **Selected Approach**: (2).
- **Rationale**: `is_host_based_limit()` is already `extern`-declared in `xcelimit.h` and used exactly this way (a yes/no runtime check) elsewhere; the "effective scope" concept is specific to what an *operator inspecting the REPL* needs to know, not a general property of the limit's configured shape — keeping it at the REPL layer avoids changing a shared library formatter's contract for one caller's needs.
- **Trade-offs**: None material.
- **Follow-up**: None.

## Risks & Mitigations
- **Risk**: Linear scan over `privdata->instances` to filter by `vres->limit == target_limit` (this table is keyed by vresname, not by owning limit, so no index avoids the scan) could be surprising if a deployment ever configures an unusually large number of distinct scope instances (e.g. thousands of users × per-user limits). — **Mitigation**: Accepted for this spec; the existing `match_job_host_limit_`/`match_job_global_limit_` matching path already does a `key_find` per job per limit against the same table, so this table's size is already load-bearing elsewhere in the scheduler's hot path — a read-only REPL scan over it is not a new scaling concern this spec introduces.
- **Risk**: Because `is_host_based_limit()` takes the plugin's `resmgr` handle (a `var`), calling it from `repl_info.c` requires reaching `sim->core.resmgr` — already public (used directly by `repl_info_resource()` today), so no new exposure needed. — **Mitigation**: N/A, confirmed no new risk.
- **Risk**: `build_vres_name()` never encodes host in the vresname it builds — an implementer could reasonably assume "one `xce_vres_t` per (user, queue, project, host)" and design storage/iteration around a cardinality that does not exist. — **Mitigation**: `design.md`'s Data Models section states the correct cardinality explicitly (one `xce_vres_t` per (user, queue, project); host is produced by a per-worker fan-out over that single `xce_vres_t`'s shared resource index, not by additional `xce_vres_t` instances) and cites this finding.
- **Risk**: Build wiring for the new white-box test — `test_repl_resource_table.c`'s existing `#include` set omits `plugins/limit.c` and `resource.c`; naively adding a limit-info test there, or to a careless new file, reproduces the exact undefined-reference class bug-1762 already hit once for a sibling REPL feature. — **Mitigation**: `design.md`'s Testing Strategy calls this out explicitly and directs a new sibling test file (`test_repl_limit_table.c`) with its own white-box include set, real (not stubbed) `resource.c` since the test needs non-trivial accounted-usage values.

## Additional Findings (design review pass)
- `xce_sched_worker_t` carries two separate zarrays per resource idx: `.resvals` (capacity) and `.resInUse` (accounted usage) — confirmed via `sched_check_host_satisfy_()` (`resource.c`), which reads both the identical way this design's per-host instance display needs to.
- `instantiate_limit()` and `update_limit_res_value()` run back-to-back inside the same `match_job_host_limit_`/`match_job_global_limit_` call (`plugins/limit.c`) — a matched `xce_vres_t`'s *capacity* entry is therefore always populated by the time the `xce_vres_t` itself is observable via the new accessors; only its *in-use* entry can legitimately be absent (genuine zero usage, not "no instance").
- `resource-accounting.md`'s reservation-vs-actual-usage reconciliation is explicitly scoped to resources flagged both `XCE_RESF_DYNAMIC` and `XCE_RESF_CONSUMABLE`. A limit's synthetic capacity resource (`add_vres()`) is flagged `XCE_RESF_LIMIT | XCE_RESF_CONSUMABLE | XCE_RESF_RELEASE` plus `GLOBAL|SHARED` or `DEFAULT_HOST` — never `XCE_RESF_DYNAMIC` — so that steering document does not apply here; the capacity/in-use pair this design reads is an exact scheduler-computed accounting number, not an OS-reconciled estimate.
- `src/scheduler/testsuite/test_no_bare_printf.c` regression-tests every file under `src/scheduler/src/` for bare `printf()` calls (must use `fprintf(stdout, ...)`); bug-1599 documents a known scanner false-positive on *comment prose*, not an exemption for real code.

## Requirement 6 Research (added 2026-09-19 — Human-Readable Units for Byte-Valued Limit Amounts)

### Two independent root causes behind one operator-reported symptom
- **Context**: A live REPL transcript showed a limit's configured amount (`mem = 5000000000.000000`) and its usage amount (`accounted=9011200.00 capacity=4635344384.00`) both as raw byte counts, despite the REPL already having a general byte-formatting convention used by `info worker`/`info resource`.
- **Sources Consulted**: `src/scheduler/src/repl_info.c` (`repl_print_limit_usage()`, `repl_print_limit_usage_per_host()`, `repl_format_consumable_value_and_used()`, `repl_format_numeric_resource_value()`, `repl_current_units_mode()`), `src/scheduler/src/repl_fmt.c` (`repl_is_bytes_valued_resource()`, `repl_format_bytes_human()`), `src/lib/xcelimit.c` (`fmt_limit()`), `src/lib/testsuite/test_limit.c`.
- **Findings**:
  - The usage lines (`accounted=`/`capacity=`) already call `repl_format_consumable_value_and_used()`, which already checks `repl_is_bytes_valued_resource()` — but the argument passed is `vres->vres->name`, a synthesized compound name (`build_vres_name()`, e.g. `limits2/u=rshen/q=xce/p=`), never the literal resource name (`"mem"`). The classifier silently always returns false for a limit's usage instance. The correct string (`limit->val.resname`, or `vres->limit->val.resname` in the per-host helper) is already reachable in scope — this is a genuine, isolated argument bug, not a missing feature.
  - The configured-value line is printed entirely by `fmt_limit()` (`src/lib/xcelimit.c`), a shared-library function with a fixed `"%s = %f%s\n"` format and its own direct unit tests (`test_limit.c`, 5 call sites) — it has no REPL units-mode concept at all, and `repl_info.c`'s own existing comment explicitly documents a prior decision NOT to reformat its output.
  - Full-tree grep confirms `fmt_limit()` has exactly one production caller (`repl_info.c`, this spec's own command) and one direct-test file — no other component (controller, worker, apiserver) calls it, despite `libxce` being linked by all of them.
- **Implications**: the usage-line issue is a one-line-per-call-site bug fix. The config-line issue requires a real design decision (discovery phase surfaced 3 approaches; user selected "reformat at the source").

### Reformat-at-the-source approach: how to add a units-mode parameter without breaking `fmt_limit()`'s contract
- **Context**: the user selected reformatting `fmt_limit()`'s output at its source over a REPL-layer annotation, but `fmt_limit()`'s existing signature has no way to receive a display-mode preference, and its 5 existing `test_limit.c` assertions pin its exact current output.
- **Alternatives Considered**:
  1. Add a units-mode parameter directly to `fmt_limit()`'s existing signature — breaks its contract; all 5 `test_limit.c` call sites and both `repl_info.c` call sites need updating, and the meaning of "default" behavior for `fmt_limit()`'s own unit tests becomes ambiguous.
  2. Add a new sibling function `fmt_limit_units()` taking the extra parameter, and retype `fmt_limit()` itself into a one-line wrapper calling it with a fixed RAW mode.
  3. Duplicate `fmt_limit()`'s whole body into a new function rather than sharing it via delegation.
- **Selected Approach**: (2).
- **Rationale**: verified algebraically that `xce_format_units_value(XCE_UNITS_RAW, v, ...)` produces the bare `"%f"` rendering of `v`, identical to what `fmt_limit()` prints today in both its percent and non-percent branches — so `fmt_limit(buf, len, limit) == fmt_limit_units(buf, len, limit, XCE_UNITS_RAW)` for every input, provably byte-identical. This means `test_limit.c`'s 5 existing assertions require **zero changes**, contrary to this spec's own discovery-phase estimate that they would need updating. Delegation (not duplication) avoids the two functions' consumer-dump loops drifting apart later.
- **Trade-offs**: two function names to maintain instead of one, but this matches the codebase's own precedent (`fmt_jobid()`/`fmt_jobid_ex()`, `limit_seed_config()`/`limit_seed_config_impl()`) for "add an extended sibling rather than break an existing signature."
- **Follow-up**: none.

### Where the byte-formatter should live once it needs a non-scheduler caller
- **Context**: `fmt_limit_units()` lives in `src/lib/xcelimit.c` (shared library) but needs the same byte-scaling algorithm `repl_format_bytes_human()` (`src/scheduler/src/repl_fmt.c`, scheduler-only) already implements.
- **Alternatives Considered**:
  1. Duplicate the algorithm into `xcelimit.c` directly.
  2. Promote the algorithm (and its companion classifier/composer) into an existing shared-lib file, with the scheduler's existing functions becoming thin wrappers.
  3. Create a brand-new shared-lib file solely for this.
- **Selected Approach**: (2), landing in `src/lib/utils.c`/`src/include/utils.h` — the existing home of `fmt_jobid()`/`fmt_jobid_ex()`, the closest existing precedent for a small, generic, cross-component display-formatting helper with no domain coupling (confirmed no similar helper already exists elsewhere in `src/lib`; `src/controller/testsuite/test.c`'s own `format_bytes_human()` is test-only and not a production dependency, so not consolidated by this spec).
- **Rationale**: avoids duplicating the same divide-by-1024-while->=1024 algorithm a third time in the codebase (it already exists twice: `repl_format_bytes_human()` production, `test.c`'s `format_bytes_human()` test-only); `utils.c`/`utils.h` is already linked into every component that would ever need this, with zero new build-system surface.
- **Trade-offs**: `xce_repl_units_mode_t` (scheduler-private, `repl_internal.h`) and the new `xce_units_mode_t` (shared, `utils.h`) are two independently-defined enums with the same three semantic values, rather than one shared type — deliberate, since promoting the scheduler's own enum into shared-lib scope would be a layering violation in the other direction (a `src/lib` type must not be defined in terms of a scheduler-internal concept, nor should the scheduler's already-tested, already-working `xce_repl_units_mode_t` plumbing need to change for this). A small `translate_units_mode()` switch at the one new call site (`repl_info.c`) bridges the two; a Revalidation Trigger in `design.md` flags this mapping for a future maintainer.
- **Follow-up**: none.

### Existing test fixtures were asserting on the bug, not a locked contract
- **Context**: `test_repl_limit_table.c` (built under Requirement Set 1) has multiple fixtures using `"mem"` as the limit's constrained resource, with assertions like `strstr(captured, "accounted=4.00")`.
- **Finding**: these assertions currently pass BECAUSE of the resource-name argument bug (the classifier is never actually asked about `"mem"`), not because `"4.00"` is a deliberately-chosen display contract. Fixing the bug will legitimately change this output to a human-formatted form.
- **Implication**: this is expected, not a regression to avoid — `design.md`'s Testing Strategy requires the implementer to prove this via a temporary revert (old assertion fails for the right reason once the bug is fixed) before finalizing the updated assertion, the same RED→GREEN discipline as any other bug fix in this codebase.

## References
- `.wolf/buglog.json` bug-1784–bug-1790 (gitea#46 limit-matching chain, 2026-09-07) — internal, not a URL.
- `.wolf/buglog.json` bug-1593 (`info resource` GLOBAL/SHARED-vs-host-based display fix, 2026-08-19) — internal, not a URL.
- `.wolf/buglog.json` bug-1685 (limit plugin double-free; also the evidence that `privdata->limits` is name-keyed) — internal, not a URL.
- `.wolf/buglog.json` bug-1762 (build-wiring precedent for white-box REPL test files) — internal, not a URL.
- `.wolf/buglog.json` bug-1599 (bare-printf regression scanner false-positive) — internal, not a URL.
- `.kiro/steering/resource-accounting.md` — reservation-vs-actual-usage policy, scoped to `XCE_RESF_DYNAMIC`; confirmed not applicable to this design.