myxce/.kiro/specs/scheduler-sim-diagnostics/tasks.md

Implementation Plan: scheduler-sim-diagnostics (Requirement Set 1 — Limit Inspection; Requirement 6 — Human-Readable Units)

• 1. Add read-only accessors exposing the limit plugin's configured-limits and live-usage-instances tables

  • Two new functions (limit_get_configured_limits(), limit_get_usage_instances()) returning raw kvmgr handles (or NULL when the plugin is absent), declared in plugin.h alongside register_limit(), defined in plugins/limit.c
  • Never allocate, mutate, or trigger plugin hooks — a pure read of already-existing pointers
  • New/extended unit tests in test_limit_plugin.c cover: plugin registered with configured limits (handle returned, iterable), plugin registered with zero limits, plugin not registered (NULL)
  • Observable: a test that registers the limit plugin with a known limit and confirms the accessor's returned handle, when iterated via iter_make()/iter_next(), yields that same limit; a sibling test confirms NULL when the plugin was never registered
  • Requirements: 1.1, 1.2, 1.4, 2.1, 3.1
  • Boundary: Limit Plugin Accessor

• 6. Bootstrap the limit plugin in sim mode so info limit shows real data against a replay (fixes bug-1825)

  • Extract limit_new_plugin() factory out of register_limit() (plugins/limit.c), mirroring builtin_fcfs_new_plugin()/register_builtin_fcfs()'s existing factory-extraction pattern exactly; register_limit() itself must be behaviorally unchanged (pure refactor)
  • New sim_bootstrap_limit_plugin(xce_sched_sim_driver_t* sim) (new file sim_limit.c, mirroring sim_picker.c's structure/header-comment convention): limit_new_plugin() → plugin_register_core(&(sim->core), plugin) → if sim->core.config != NULL, call plugin->hooks.config_hook(plugin, sim->core.config) directly (NOT the type-unsafe plugin_config_update(), which casts its void* arg to xce_scheduler_t* and would misread xce_sched_sim_driver_t's embedded core field)
  • Wire the call into sim_loop.c right alongside the existing sim_bootstrap_picker(sim) call site — by that point sim->core.config already holds the snapshot's borrowed-reference config (set during sched_sim_driver_start()'s snapshot load, which runs first)
  • Observable: a real xce_sched_sim_driver_t driven through sched_sim_driver_start() against a replay whose snapshot carries a real xce_limit_config_t, after the sim driver's own bootstrap sequence runs, has limit_get_configured_limits() return a handle that iterates to that real limit — no test-only registration helper needed
  • Requirements: 1.1–1.7 (makes them exercisable against a real replay, not just via a test-only fixture helper)
  • Boundary: Limit Plugin Accessor (factory extraction), new Sim Limit Bootstrap component
  • Known deliberate non-goal, tracked separately: this task does NOT wire limit matching (limit_job_election()) into the sim/replay decision path — that function is only ever reached via plugin_job_election(), which is normal-mode-only (one caller, scheduler.c) and itself needs a core-safe variant before it could be called from sim_loop.c. Live usage (Requirements 2.x/3.x) will still show "no active usage instance yet" against a real replay even after this task — only the configured-setting display (1.1-1.7) becomes real. Tracked as a separate follow-up bug, not bundled into this task.

• 2. Core: Limit lookup and configured-setting display

• 2.1 Implement name/all resolution with correct absence messaging

  • New white-box test file test_repl_limit_table.c created (sibling of test_repl_resource_table.c, own #include set covering plugins/limit.c and resource.c for real — not stubbed — per design.md's build-wiring risk note citing bug-1762's precedent)
  • repl_info_limit(sim, arg): arg == "all" iterates limit_get_configured_limits(); NULL/empty prints "no limits configured" (Requirement 1.7); arg != "all" does a key_find() by name (not a scan) on the same handle, printing "no such limit ''" (Requirement 1.3) for a NULL handle or a miss — a NULL handle on this path is never treated as "no limits configured," matching that message's own all-only scoping
  • Wired as a direct call for now (repl_dispatch()/repl_cmd_info() wiring itself deferred to task 4); tests in this task call repl_info_limit() directly, not through repl_dispatch()
  • Observable: REPL output shows "no such limit 'foo'" for an unconfigured name against a fixture with other limits configured, and "no limits configured" for all against a fixture with zero configured limits
  • Requirements: 1.3, 1.7, 4.1, 4.2
  • Depends: 1

• 2.2 Display configured setting including scope membership

  • Matched limit's name, constrained resource, value/percent, and per-dimension scope print via the existing fmt_limit() library formatter into a stack buffer, line-prefixed with the existing [sim] convention; all output via fprintf(stdout, ...), never bare printf() (test_no_bare_printf.c regression-gates this)
  • Inclusion and exclusion group membership both render (fmt_group() already handles this, adopted as-is)
  • Observable: a limit configured with an exclusion group ("all except X") shows both the inclusion and the exclusion in its output; info limit all against a fixture with multiple configured limits displays every one's setting
  • Requirements: 1.1, 1.2, 1.5

• 2.3 Display effective host-scope for implicitly host-scoped limits

  • After fmt_limit()'s output, call is_host_based_limit(core->resmgr, limit); if it returns > 0 AND the limit's own per-host consumer is not configured, print an explanatory host-scope note
  • Observable: two REPL fixtures — one percent-based limit on a per-host resource without explicit per-host scoping (note appears), one absolute-value or explicitly-global-scoped limit (note does not appear)
  • Requirements: 1.6

• 3. Core: Live accounted-usage display

• 3.1 Display usage for cluster-wide (non-host-scoped) limit instances

  • Iterate limit_get_usage_instances(), filter by vres->limit == matched_limit; for a GLOBAL/SHARED vres, read capacity from core.sharedInst[idx] and accounted usage from core.sharedInUse[idx]
  • Existence-of-instance and zero-usage are distinct: zero matching xce_vres_t (no capacity entry at all) prints "no active usage instance yet" (Requirement 2.2); a matched xce_vres_t whose sharedInUse[idx] is NULL/absent instead displays a genuine accounted=0 (it has been evaluated and has a real capacity, just nothing charged against it yet) — existence is gated on the capacity read, never on the in-use read
  • Observable: a global limit that has matched at least one job shows an accounted/capacity pair reflecting current state; a configured-but-never-evaluated limit shows the explicit no-instance message instead; a matched-but-never-charged limit shows accounted=0 rather than the no-instance message
  • Requirements: 1.4, 2.1, 2.2, 2.3, 3.3
  • Depends: 1

• 3.2 Display per-host usage for host-scoped limit instances

  • For a host-scoped vres (explicit or implicit per 2.3), iterate core.workers; include a worker only if it has a populated capacity entry (w->resvals[idx] non-NULL) — read capacity from w->resvals[idx] and accounted usage from w->resInUse[idx] (same existence-vs-zero-usage distinction as 3.1, applied per worker); label each instance with its scope values (user/queue/project from the vresname, plus host)
  • A limit varying by more than one scope dimension at once (e.g. per-user and per-host together) shows each distinct combination as its own labeled instance
  • Observable: a limit matched against jobs on two distinct hosts shows two separate per-host usage lines each labeled with its own scope values, and a third configured-but-untouched host shows no line at all
  • Requirements: 1.4, 2.1, 2.2, 2.3, 3.1, 3.2, 3.4

• 4. Register info limit as a recognized info subcommand

  • New else if (strcmp(tokens[1], "limit") == 0) branch in repl_cmd_info() (repl_info.c) routes to repl_info_limit(); usage-string literal updated to list limit alongside the existing subcommands
  • test_repl_limit_table.c gains a full dispatch-round-trip test (via repl_dispatch(), not a direct call) for both info limit <name> and info limit all
  • Observable: info limit <name> and info limit all both work when typed as real REPL input (through repl_dispatch()), and help/usage output lists the new subcommand
  • Requirements: 4.1, 4.2
  • Depends: 2.1, 3.1, 3.2

• 5. Verify no mutation, replay determinism, and existing-subcommand regression

  • Test: displaying limit information never alters configuration, accounting state, or scheduling decisions (assert config/accounting untouched before vs. after)
  • Test: a replay run twice — once with info limit issued mid-session, once without — produces identical scheduling decisions and identical final accounted state
  • Full existing info subcommand test suite (job/queue/worker/resource/summary) re-run to confirm zero output changes from this addition
  • Observable: the two new determinism/no-mutation tests pass, and the full pre-existing info-command test suite shows zero diffs
  • Requirements: 5.1, 5.2, 5.3
  • Depends: 4

• 7. Foundation: Promote human-readable byte-unit formatting into the shared library

  • Move the existing byte-unit scaling algorithm and its byte-valued-resource classification out of the scheduler REPL and into the shared library, so any component linking the shared library can reach it, not only the scheduler's REPL
  • Add a display-mode-aware value composer to the shared library capturing the existing raw / human-readable / both-together rendering shapes the REPL already uses elsewhere for other byte-valued resources
  • The scheduler's own existing byte-formatting functions become thin callers of the newly shared versions — their names, signatures, and observable output stay exactly as they are today
  • Add the small scheduler-side bridge between the REPL's own existing display-mode selection and the shared library's newly-introduced mode concept, and retype the REPL's existing shared numeric-value composer (the one piece of formatting logic every byte-valued resource display in the REPL already funnels through, including — but not limited to — limit display) to delegate to the new shared composer through that bridge, so every consumer of that shared composer benefits from one single implementation from this point forward, not just the limit display this requirement set is adding
  • Observable: every existing scheduler test that exercises byte-unit formatting continues to pass completely unchanged — including the pre-existing, non-limit displays that already share this exact formatting logic — and new shared-library unit tests directly exercise the promoted scaling, classification, and mode-aware composition logic in isolation
  • Requirements: 6.1, 6.3, 6.5, 5.3
  • Boundary: Shared Units Formatter, REPL Limit Info Command (the retyped composer and its thin-wrapper callers live in repl_info.c/repl_fmt.c)

• 8. Core: Human-readable configured-setting display

• 8.1 Add a display-mode-aware way to render a limit's configured setting

  • New entry point produces the same configured-setting output the existing formatter already produces, except a byte-valued, non-percentage resource's amount renders in human-readable form according to a supplied display mode
  • The existing formatter's own output is provably unchanged for every input it can be given — proven by its complete pre-existing test coverage continuing to pass with zero modification
  • Observable: the existing configured-setting formatter's test coverage passes unmodified, and a new test confirms the mode-aware variant renders a byte-valued resource's amount differently across each supported display mode while leaving a percentage or non-byte-valued resource's rendering untouched
  • Requirements: 6.1, 6.3, 6.5
  • Depends: 7
  • Boundary: Shared Units Formatter

• 8.2 Show configured limit amounts in the operator's selected display mode

  • The REPL's limit configuration display now renders a byte-valued limit's configured amount using the operator's already-selected display mode, reusing the scheduler-side bridge task 7 introduced, the same mode every other byte-valued resource display in the REPL already honors
  • Observable: info limit <name> against a byte-valued limit shows its configured amount in human-readable form under the REPL's default display mode, and the rendering changes consistently when the operator switches display modes
  • Requirements: 6.1, 6.2, 6.5
  • Depends: 8.1
  • Boundary: REPL Limit Info Command

• 9. Core: Correct live usage display's resource classification

  • The live usage display currently classifies a limit's amounts using an internal per-instance identifier rather than the limit's own constrained resource, so a byte-valued limit's usage amounts never actually get evaluated for human-readable rendering even though the surrounding display logic (already retyped onto the shared composer by task 7) already supports it
  • Classify a limit's live usage amounts by the limit's own constrained resource instead, for both the cluster-wide and the per-host usage display paths
  • Observable: a byte-valued limit's accounted and capacity amounts in its live usage display render in human-readable form, matching the same format its configured-setting display already uses; a limit expressed as a percentage of a byte-valued resource shows its resolved absolute amount in human-readable form too
  • Requirements: 6.1, 6.3, 6.4, 6.5
  • Depends: 7 — task 7 retypes the exact shared formatting call chain this task's fix flows through (the limit usage display's classification feeds into that same composer); sequencing after 7 avoids two tasks editing overlapping logic in an uncoordinated order, not a data dependency on any new symbol this task itself needs
  • Boundary: REPL Limit Info Command

• 10. Validation: Confirm correctness, consistency, and no regression

• 10.1 Re-verify existing test coverage against the corrected usage display

  • Task 9 found that fixing the resource-classification defect did NOT break any pre-existing assertion outright — instead, several existing assertions on the live usage display's amounts turned out to be loose substring checks against whole-number fixture values, which happen to still match the corrected, human-augmented output by coincidence of formatting (the old expected text is a literal prefix of the new one) rather than by genuinely exercising the fix
  • Identify every such loose, coincidentally-still-passing assertion, and strengthen each one to assert the actual corrected output precisely enough that it would fail if the resource-classification fix were reverted
  • Observable: each strengthened assertion demonstrably fails when checked against the prior (defective) behavior and passes against the corrected behavior — not merely a new assertion added alongside an untouched, still-coincidentally-passing old one
  • Requirements: 6.1, 6.3, 6.4
  • Depends: 9

• 10.2 Confirm units consistency, exclusion rules, and zero regression end to end

  • A byte-valued limit's configured-setting display and its live usage display agree on units under every display mode, within one request for that limit's information
  • A limit whose constrained resource is not byte-valued, or whose own configured value is itself a percentage, is never subject to byte-unit scaling in either display
  • The complete pre-existing info limit test suite, and the complete pre-existing suite for every other info subcommand — including the two pre-existing, non-limit displays that share the same underlying formatting logic this requirement set retyped in task 7 — show zero output differences beyond what this requirement set intends
  • Observable: a single info limit invocation against a byte-valued limit shows the same units convention in its configured-setting line and its usage line(s) in every display mode, a slot-count limit's display is unaffected by display mode in either line, and the full pre-existing scheduler test suite — including the pre-existing non-limit byte-valued-resource displays task 7 touched — passes with no unintended output changes
  • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 5.1, 5.2, 5.3
  • Depends: 8.2, 9, 10.1

Parallel Execution Note

No (P) markers are used anywhere in this plan. Tasks 2.1–2.3 and 3.1–3.2 all incrementally edit the same function (repl_info_limit()) in the same file (repl_info.c) and largely share the same new test file (test_repl_limit_table.c) — the parallel-analysis rules' "no conflicting files or shared mutable resources" criterion rules out marking any of them concurrent. Task 1 is the sole Foundation task (rarely (P) by rule, and has no peer to run alongside regardless). Tasks 4 and 5 are Integration/Validation work that structurally depends on Core being complete.

Tasks 7–10 (Requirement 6) follow the same reasoning: task 9 has no genuine data dependency on task 8 (they touch different functions — the configured-setting formatter vs. the live-usage formatter), but both edit repl_info.c and both build on task 7's shared-library symbols and scheduler-side bridge, and task 10's validation work spans all of them — marking any of 8/9 (P) would only invite the same same-file coordination risk this plan has already declined once. Task 7 is the sole new Foundation task, and now also owns retyping the REPL's pre-existing shared numeric-value composer (used by info worker/info resource as well as info limit) onto the new shared formatter — a task-graph sanity review (2026-09-20) caught this as a design.md-specified change with no owning task in an earlier draft, corrected here rather than left implicit. Task 10 is Validation work that structurally depends on Core being complete.

Implementation Notes

• Task 1: test_limit_plugin.c's pre-existing fixture_init() never populated core->plugins (plugin_register() is stubbed to a no-op in that white-box TU). Closed with two new test-local fixture helpers that hand-build a real core->plugins kvmgr and key_add() the plugin under its real "xce_sched_limit" name — necessary to exercise the accessors' real key_find() path, not a workaround. test_repl_limit_table.c (tasks 2.1+) is a different white-box TU with its own include set (per design.md's build-wiring risk note) — it should link real plugin.c/core.c machinery where register_limit()/xce_sched_core_init() actually populate core->plugins, so it likely does NOT need this same hand-built-kvmgr workaround; confirm this when starting 2.1 rather than assuming the same gap exists there.
• Task 2.1: Making repl_info.c genuinely call limit_get_configured_limits() broke the link for ~24 pre-existing white-box test files that #include "../src/repl_info.c" without also linking plugins/limit.c — this codebase's white-box test convention means any new cross-file call from an included production file ripples to every file that includes the caller but not the callee. Fixed with an identical 5-line link-satisfying stub (return NULL) added to each, matching the established "duplicate, don't share" stub idiom already used throughout this file family; verified behavior-preserving since none of those 24 fixtures register the limit plugin (the real function would also return NULL there). Task 3 (which calls limit_get_usage_instances()) will very likely hit the identical ripple across an overlapping or identical file set — expect it, and apply the same fix pattern rather than treating it as a surprise.
• Task 3.1: found and fixed a real latent bounds-check imprecision (negative/stale idx not safely rejected by a .count-based pre-check) while writing the new GLOBAL/SHARED usage-read code — logged as bug-1824 in .wolf/buglog.json rather than fixed in the pre-existing repl_info_resource_one() (out of this task's boundary); the new code uses the corrected pattern instead. Task 3.2 confirmed it did NOT repeat the same mistake in its own per-host reads.
• Task 3.2: reviewer approved with one non-blocking suggestion — no test directly exercises Requirement 3.1's "per-user and per-host together" combined-dimension case end-to-end (structurally supported by the code, per independent review inspection, just not covered by a dedicated fixture). Worth adding if this area gets touched again; not required before shipping.
• Task 5 — significant follow-up needed (logged as bug-1825): all 5 tasks are correctly implemented, thoroughly tested, and genuinely read-only/deterministic — but info limit is currently inert in real production usage. register_limit() has exactly one call site in the whole codebase (scheduler.c's normal-mode sched_register_plugins()); sim mode's own plugin bootstrap (sim_bootstrap_picker(), sim_picker.c) only registers the picker plugin, never the limit plugin. So a real operator running xce_scheduler -i/-s <replay> will always see "no limits configured"/"no such limit" from info limit, regardless of what the original live recording had configured — every positive-display test in this spec exercises the code via a test-only plugin-registration helper with no production equivalent in sim mode. This is a pre-existing sim-mode driver gap (same class as scheduler-driver-equivalence.md's other documented normal-vs-sim parity gaps), not a defect in this spec's own work, and was out of task 5's test-only boundary to fix. Fixed by task 6.
• Task 6 — a real memory-safety lesson, caught by review before it shipped: the first implementation attempt set plugin->sched = (void*)sim (a xce_sched_sim_driver_t*, disguised) before invoking the limit plugin's config_hook, relying on both xce_scheduler_t and xce_sched_sim_driver_t happening to have xce_sched_core_t core as their first struct member. An independent reviewer rejected this: core.h's own xce_core_cache_clear_cb_t doc comment documents that this EXACT cross-cast pattern already corrupted memory once before in this codebase (cache_clear() reinterpreting xce_sched_sim_driver_t's alt_events_file as a kvtab_t*), and there was no compiler/runtime guard against a future edit to the callee reading past .core. Fixed by adding limit_seed_config(plugin, xce_sched_core_t* core, config) — an honestly-typed entry point taking core as a real field access, no cast — with limit_update_config() (normal mode) and sim_bootstrap_limit_plugin() (sim mode) both delegating to a shared _impl function. Lesson for future work in this codebase: when a plugin hook or helper is invoked from BOTH a normal-mode (xce_scheduler_t) and a sim-mode (xce_sched_sim_driver_t) context, never derive one struct type from the other via a first-member cast, even when it "happens to work" today — take xce_sched_core_t* as an honest parameter instead. Two independent reviewers (one per round) verified this by reading the actual struct definitions and the full call chain, not by trusting the report.
• Task 7 — a genuine, narrow behavioral corner case surfaced by the promotion, deliberately left as-is: the pre-existing repl_format_numeric_resource_value() had an asymmetry for "swp" (whose wire unit is megabytes, not bytes): RAW and BOTH modes printed the un-converted raw value, while only HUMAN mode used the MB→bytes-converted value for scaling. The new shared xce_format_units_value(mode, bytes_value, ...) interface — a single value argument, matching design.md's specified shape exactly — cannot preserve that asymmetry: swp under RAW/BOTH mode in info worker/info resource now renders the bytes-converted number instead of the original raw MB number. Confirmed by both the implementer and an independent reviewer: no test in the codebase exercises swp under RAW or BOTH mode (all existing swp tests use HUMAN only), and swp is not in Requirement 6's byte-valued-resource list (mem/tmp/maxmem/maxtmp/maxswap), so this is untested and out of this requirement's scope — not a regression caught by any existing coverage, but technically in tension with Requirement 5.3's literal "preserve all existing behavior unchanged." Left as-is rather than special-cased, since fixing it would mean giving the shared composer an swp-specific parameter it has no other reason to carry. If swp's RAW/BOTH display is ever reported as user-visibly wrong, start here.
• Task 8.2 — implementer's own stated regression-safety rationale was inaccurate, though its conclusion held: the report claimed the 28 pre-existing test_repl_limit_table.c tests stay green because they use "slots-based (non-byte-valued)" fixtures unaffected by units-mode. An independent reviewer found this false — every configured-limit fixture in that file hardcodes resource name "mem" (a byte-valued resource); slots-looking names in that file (ug1_slots, etc.) are limit names, not resource names. The real reason those 28 tests stay green is that none of them assert on the formatted "<resname> = <value>" line's exact text at all — they assert only on limit names, group membership/exclusion text, host-scope-note presence, and separate usage/accounting lines. Lesson: when a report explains why an existing test suite is unaffected by a change, verify the stated mechanism against the actual fixtures, not just the pass/fail outcome — a correct outcome can still carry an incorrect explanation, which would mislead the next person who touches this test file expecting a slots fixture to exist.
• Task 9 — design.md/tasks.md's own prediction about which pre-existing assertions would break was wrong, and the fix landed with ZERO pre-existing test failures (not the expected handful): the original design/task-10.1 framing assumed fixing the resname-classification bug would break test_repl_limit_table.c's existing "accounted=4.00"-style assertions (lines ~1355, 1359, 1455, 1537, 1540, 1599). It didn't — every one of those fixtures uses a whole-number double value (4.0, 10.0, 0.0, 2.0, 3.0), and xce_format_units_value(XCE_UNITS_BOTH, ...)'s "%f (%s)" composition renders any whole number's %f expansion as "N.000000", whose leading "N.00" substring still satisfies the old assertions' bare strstr() checks even though the actual output materially changed (now "accounted=4.000000 (4.00 B)" instead of the old "accounted=4.00"). Independently confirmed by task 9's reviewer, tracing the exact format chain and fixture values, with no counterexample among the 6 citations. Consequence for task 10.1: there is no RED→GREEN "fix a broken assertion" work to do — instead, task 10.1 should STRENGTHEN these 6 loose, coincidentally-passing assertions (e.g. assert the full corrected string, or assert a "B)"/unit-suffix is present) so they actually prove the fix rather than passing by accident of whole-number formatting. General lesson: a substring-only (strstr) assertion with no upper bound or exact-match check is blind to a superset-string regression — the old string being a prefix of the new one is enough to mask a real, materially-different output change.