myxce/.kiro/specs/scheduler-sim-diagnostics/research.md

Research & Design Decisions

Summary

• Feature: scheduler-sim-diagnostics — Requirement set 1 (info limit REPL command)
• Discovery Scope: Extension (extends scheduler-core's existing REPL info command family and reuses the limit plugin's existing library-level formatters)
• Key Findings:
  • The REPL info command family is a single dispatch function (repl_cmd_info(), repl_info.c) with one else if branch per subcommand — adding limit is a mechanical extension of an existing, well-established pattern.
  • The limit plugin's private state (limit_plugin_data_t.limits/.instances) is genuinely inaccessible outside src/scheduler/src/plugins/limit.c today — no header declares the type — confirming the brief's stated gap. Two new one-line accessor functions close it.
  • Two library-level formatters already exist and already do most of Requirement 1's work: fmt_limit() and fmt_consumer()/fmt_group() (src/lib/xcelimit.c, src/lib/group.c), including correct inclusion/exclusion membership rendering (fmt_group() already prefixes excluded members with ~). These are adopted directly rather than rebuilt.
  • The live-usage read path (GLOBAL/SHARED single value vs. per-host breakout) is architecturally identical to the pattern repl_info_resource_one() already implements for ordinary resources (bug-1593) — the limit's synthetic capacity resource (vres->vres) is a real, already-registered xce_resource_t carrying the same XCE_RESF_GLOBAL/XCE_RESF_SHARED flags, so the same shared-vs-per-host read logic applies unmodified.

Research Log

Limit plugin data model and matching semantics

• Context: Requirement 1 AC1/AC5/AC6 need to know exactly what "scope dimensions" and "restricted membership" mean at the code level, since an earlier requirements-review pass surfaced a scope-vs-runtime ambiguity.
• Sources Consulted: src/include/xce/types.h (xce_limit_t, xce_limit_config_t, xce_consumer_t, xce_vres_t), src/lib/xcelimit.c (match_limit, build_vres_name, is_host_based_limit, instantiate_limit), src/scheduler/src/plugins/limit.c (match_job_host_limit_, match_job_global_limit_, update_limit_res_value, limit_prepare_allocation), .wolf/buglog.json bug-1784 through bug-1790 (the gitea#46 limit-matching bug chain, 2026-09-07).
• Findings:
  • xce_consumer_type_t has exactly four values (CONSUMER_USER, CONSUMER_QUEUE, CONSUMER_PROJECT, CONSUMER_HOST) — the four scope dimensions in Requirement 1 AC1 are the complete set, no fifth dimension exists.
  • is_host_based_limit() forces per-host live-instance tracking whenever the limit's value is a percentage of a resource that is not XCE_RESF_GLOBAL/XCE_RESF_SHARED — independent of whether CONSUMER_HOST was configured. This is real, current behavior (not hypothetical), confirmed via source and via bug-1789/1790's live-verified fix history for the equivalent allocation-side computation.
  • xce_group_t supports both members and ex_members (exclusion). Confirmed as real production configuration via bug-1789's own note describing a live-configured limit ("limit1... all users except rshen").
  • fmt_group() (src/lib/group.c:388) already renders both lists correctly (~name prefix for exclusions, all/[] shortcuts for the builtin groups) and is already extern-declared in group.h. fmt_consumer()/fmt_limit() (src/lib/xcelimit.c) already compose fmt_group() into a full per-limit configuration dump.
  • The bug-1789/1790 chain (global limits under-counting usage against per-host resources) is fully closed as of 2026-09-07 — the underlying accounting this feature displays is trustworthy across all resource/scope combinations by the time this spec is implemented.
• Implications: Requirement 1's "configured setting" display can adopt fmt_limit() directly instead of re-deriving formatting logic; only the effective-host-scope annotation (Req 1.6) and the live-instance/usage rendering (Req 2, Req 3) are genuinely new code.

REPL command dispatch and existing info resource precedent

• Context: Need the concrete integration point for a new info limit subcommand and a working precedent for the GLOBAL/SHARED-vs-per-host live-value read split Requirement 2/3 require.
• Sources Consulted: src/scheduler/src/repl.c (repl_dispatch), src/scheduler/src/repl_info.c (repl_cmd_info, repl_info_resource, repl_info_resource_one), src/scheduler/src/plugin.c (plugin_register, plugin_register_core), src/scheduler/include/plugin.h, bug-1593 (repl_info_resource_one's original GLOBAL/SHARED-vs-host-based display bug and fix).
• Findings:
  • repl_cmd_info() is a flat strcmp chain on tokens[1]; every existing subcommand (job, queue, worker, resource, ...) follows the identical <name>|all argument shape Requirement 4 asks for.
  • repl_info_resource_one() (post bug-1593) reads a GLOBAL/SHARED resource's value from sim->core.sharedInst/sharedInUse at r->idx, and explicitly declines to fabricate a value for a host-based resource, deferring to info worker <name> instead. This is the exact pattern Requirement 2/3's per-instance read path reuses.
  • Plugins are registered into core->plugins, a name-keyed handle (key_add(core->plugins, plugin->name, plugin)); the limit plugin registers under the fixed name "xce_sched_limit" (limit.c:557). No plugin is ever renamed or re-registered under a different name at runtime.
  • Synthetic limit-capacity resources (vres->vres, created by add_vres()) are inserted into the same core.resmgr table info resource/info worker already iterate, flagged XCE_RESF_LIMIT (already a named flag in repl_resource_flags_str's table). A GLOBAL-scoped limit's synthetic resource can therefore already appear in info resource all/info resource <name> today, labeled with the LIMIT flag — this is pre-existing behavior, unrelated to and unmodified by this spec.
• Implications: New command wiring is a single new branch in repl_cmd_info() plus one new static function repl_info_limit() in repl_info.c, following repl_info_resource()'s two-level (_all iterates, _one prints) shape.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Direct struct exposure	Move limit_plugin_data_t into a shared header, let repl_info.c include it directly	Fewer functions	Breaks the plugin's existing data encapsulation; every other plugin's private state stays file-local — this would be the only exception	Rejected
Opaque query API (iterator_t out-params, per-limit filtered iteration)	New multi-function query interface returning filtered iterators	Nice call-site ergonomics	New abstraction with only one consumer today; no existing accessor in this codebase does filtered iteration — the rest (sched_get_job, core->plugins, core->jobs) hand back a raw kvmgr/xlist handle and let the caller iter_make/iter_next	Rejected — over-built for current scope
Selected: two raw-handle accessors	limit_get_configured_limits(core) / limit_get_usage_instances(core) return the plugin's existing var (kvmgr/xlist) handles, or NULL if the plugin isn't registered; caller iterates with the same iter_make/iter_next/iter_destroy idiom used everywhere else in this codebase	Matches every existing accessor pattern (core->jobs, core->queues, core->plugins); minimal new surface (2 functions); no new types	Caller must filter usage instances by vres->limit == target_limit itself (pointer comparison) since there is no per-limit index — acceptable, this table is small (configured-limit count × scope-instance count, not job count)	Selected

Design Decisions

Decision: Plugin accessor shape

• Context: limit_plugin_data_t (limits, instances) is a file-local type in limit.c; Requirement 1/2/3 need read access to both from repl_info.c.
• Alternatives Considered:
  1. Expose the struct itself in a shared header.
  2. Build a filtered query API (per-name lookup, per-limit instance iteration) as new plugin-owned functions.
  3. Expose the two raw var handles as accessor functions, encapsulating the plugin-name lookup ("xce_sched_limit") inside the accessor itself so repl_info.c never needs to know it.
• Selected Approach: (3). limit_get_configured_limits(xce_sched_core_t* core) and limit_get_usage_instances(xce_sched_core_t* core), both declared in src/scheduler/include/plugin.h next to register_limit, defined in limit.c. Each does its own key_find(core->plugins, "xce_sched_limit", &plugin) and returns plugin->data's relevant field (or NULL if the plugin is not registered or not yet initialized).
• Rationale: Matches the codebase's existing accessor idiom exactly (raw handle + shared iterator trio), keeps the plugin-name string fully inside limit.c (the one place that already owns it via register_limit()), and adds the minimum surface needed — no speculative per-limit-name lookup function, since both AC1's "specific name" and AC2's "all" cases are satisfiable by iterating the same handle and filtering in the caller.
• Trade-offs: repl_info_limit() finds a limit by name via a direct key_find() (privdata->limits is a name-keyed kvmgr — confirmed by bug-1685's duplicate-name -EEXIST handling, so this is O(1)-ish, not a scan) and a linear scan + pointer-equality filter to find that limit's usage instances in privdata->instances (keyed by vresname, not by owning limit — no index avoids this scan; accepted, same scale argument as below).
• Follow-up: None — this is a stable, minimal boundary; a future capability under this same open-ended spec (e.g. decision-reasoning trace) does not need to touch this accessor.

Decision: Reuse fmt_limit()/fmt_group() for configured-setting display

• Context: Requirement 1 AC1/AC5 need to print a limit's name, per-dimension scope, and inclusion/exclusion membership.
• Alternatives Considered:
  1. Write new formatting code in repl_info.c mirroring the other repl_info_*_one() functions' inline fprintf style.
  2. Call the existing fmt_limit() (which internally calls fmt_consumer()/fmt_group()) into a stack buffer and print it, then append the REPL-specific effective-scope note and live-usage lines the library formatter cannot know about.
• Selected Approach: (2).
• Rationale: fmt_limit() already produces correct, exclusion-aware output and is already used/tested at the library level (src/lib/testsuite/test_limit.c); duplicating that logic in repl_info.c would both waste effort and risk drifting out of sync with the library's own definition of "configured setting." Build-vs-adopt: adopt.
• Trade-offs: fmt_limit()'s output format (multi-line, PER_USER = [...] style) is fixed by the library; the REPL wraps it with a [sim] prefix per line rather than reformatting it, consistent with how other multi-line info output is already prefixed in this codebase (e.g. info jobs).
• Follow-up: None.

Decision: Effective-scope annotation is REPL-layer, not library-layer

• Context: Requirement 1 AC6 (implicit host-scoping for a percent-based limit on a non-shared resource) is not something fmt_limit() knows about — it only reflects the configured consumers[] array.
• Alternatives Considered:
  1. Change fmt_limit() itself to call is_host_based_limit() and inject a synthetic PER_HOST line.
  2. Keep fmt_limit() unchanged (it is a general-purpose library formatter used elsewhere, e.g. by any future non-REPL consumer that wants the raw configured shape) and have repl_info_limit() call is_host_based_limit() itself, appending an explanatory note only when consumers[CONSUMER_HOST] == NULL and the check returns true.
• Selected Approach: (2).
• Rationale: is_host_based_limit() is already extern-declared in xcelimit.h and used exactly this way (a yes/no runtime check) elsewhere; the "effective scope" concept is specific to what an operator inspecting the REPL needs to know, not a general property of the limit's configured shape — keeping it at the REPL layer avoids changing a shared library formatter's contract for one caller's needs.
• Trade-offs: None material.
• Follow-up: None.

Risks & Mitigations

• Risk: Linear scan over privdata->instances to filter by vres->limit == target_limit (this table is keyed by vresname, not by owning limit, so no index avoids the scan) could be surprising if a deployment ever configures an unusually large number of distinct scope instances (e.g. thousands of users × per-user limits). — Mitigation: Accepted for this spec; the existing match_job_host_limit_/match_job_global_limit_ matching path already does a key_find per job per limit against the same table, so this table's size is already load-bearing elsewhere in the scheduler's hot path — a read-only REPL scan over it is not a new scaling concern this spec introduces.
• Risk: Because is_host_based_limit() takes the plugin's resmgr handle (a var), calling it from repl_info.c requires reaching sim->core.resmgr — already public (used directly by repl_info_resource() today), so no new exposure needed. — Mitigation: N/A, confirmed no new risk.
• Risk: build_vres_name() never encodes host in the vresname it builds — an implementer could reasonably assume "one xce_vres_t per (user, queue, project, host)" and design storage/iteration around a cardinality that does not exist. — Mitigation: design.md's Data Models section states the correct cardinality explicitly (one xce_vres_t per (user, queue, project); host is produced by a per-worker fan-out over that single xce_vres_t's shared resource index, not by additional xce_vres_t instances) and cites this finding.
• Risk: Build wiring for the new white-box test — test_repl_resource_table.c's existing #include set omits plugins/limit.c and resource.c; naively adding a limit-info test there, or to a careless new file, reproduces the exact undefined-reference class bug-1762 already hit once for a sibling REPL feature. — Mitigation: design.md's Testing Strategy calls this out explicitly and directs a new sibling test file (test_repl_limit_table.c) with its own white-box include set, real (not stubbed) resource.c since the test needs non-trivial accounted-usage values.

Additional Findings (design review pass)

• xce_sched_worker_t carries two separate zarrays per resource idx: .resvals (capacity) and .resInUse (accounted usage) — confirmed via sched_check_host_satisfy_() (resource.c), which reads both the identical way this design's per-host instance display needs to.
• instantiate_limit() and update_limit_res_value() run back-to-back inside the same match_job_host_limit_/match_job_global_limit_ call (plugins/limit.c) — a matched xce_vres_t's capacity entry is therefore always populated by the time the xce_vres_t itself is observable via the new accessors; only its in-use entry can legitimately be absent (genuine zero usage, not "no instance").
• resource-accounting.md's reservation-vs-actual-usage reconciliation is explicitly scoped to resources flagged both XCE_RESF_DYNAMIC and XCE_RESF_CONSUMABLE. A limit's synthetic capacity resource (add_vres()) is flagged XCE_RESF_LIMIT | XCE_RESF_CONSUMABLE | XCE_RESF_RELEASE plus GLOBAL|SHARED or DEFAULT_HOST — never XCE_RESF_DYNAMIC — so that steering document does not apply here; the capacity/in-use pair this design reads is an exact scheduler-computed accounting number, not an OS-reconciled estimate.
• src/scheduler/testsuite/test_no_bare_printf.c regression-tests every file under src/scheduler/src/ for bare printf() calls (must use fprintf(stdout, ...)); bug-1599 documents a known scanner false-positive on comment prose, not an exemption for real code.

Requirement 6 Research (added 2026-09-19 — Human-Readable Units for Byte-Valued Limit Amounts)

Two independent root causes behind one operator-reported symptom

• Context: A live REPL transcript showed a limit's configured amount (mem = 5000000000.000000) and its usage amount (accounted=9011200.00 capacity=4635344384.00) both as raw byte counts, despite the REPL already having a general byte-formatting convention used by info worker/info resource.
• Sources Consulted: src/scheduler/src/repl_info.c (repl_print_limit_usage(), repl_print_limit_usage_per_host(), repl_format_consumable_value_and_used(), repl_format_numeric_resource_value(), repl_current_units_mode()), src/scheduler/src/repl_fmt.c (repl_is_bytes_valued_resource(), repl_format_bytes_human()), src/lib/xcelimit.c (fmt_limit()), src/lib/testsuite/test_limit.c.
• Findings:
  • The usage lines (accounted=/capacity=) already call repl_format_consumable_value_and_used(), which already checks repl_is_bytes_valued_resource() — but the argument passed is vres->vres->name, a synthesized compound name (build_vres_name(), e.g. limits2/u=rshen/q=xce/p=), never the literal resource name ("mem"). The classifier silently always returns false for a limit's usage instance. The correct string (limit->val.resname, or vres->limit->val.resname in the per-host helper) is already reachable in scope — this is a genuine, isolated argument bug, not a missing feature.
  • The configured-value line is printed entirely by fmt_limit() (src/lib/xcelimit.c), a shared-library function with a fixed "%s = %f%s\n" format and its own direct unit tests (test_limit.c, 5 call sites) — it has no REPL units-mode concept at all, and repl_info.c's own existing comment explicitly documents a prior decision NOT to reformat its output.
  • Full-tree grep confirms fmt_limit() has exactly one production caller (repl_info.c, this spec's own command) and one direct-test file — no other component (controller, worker, apiserver) calls it, despite libxce being linked by all of them.
• Implications: the usage-line issue is a one-line-per-call-site bug fix. The config-line issue requires a real design decision (discovery phase surfaced 3 approaches; user selected "reformat at the source").

Reformat-at-the-source approach: how to add a units-mode parameter without breaking fmt_limit()'s contract

• Context: the user selected reformatting fmt_limit()'s output at its source over a REPL-layer annotation, but fmt_limit()'s existing signature has no way to receive a display-mode preference, and its 5 existing test_limit.c assertions pin its exact current output.
• Alternatives Considered:
  1. Add a units-mode parameter directly to fmt_limit()'s existing signature — breaks its contract; all 5 test_limit.c call sites and both repl_info.c call sites need updating, and the meaning of "default" behavior for fmt_limit()'s own unit tests becomes ambiguous.
  2. Add a new sibling function fmt_limit_units() taking the extra parameter, and retype fmt_limit() itself into a one-line wrapper calling it with a fixed RAW mode.
  3. Duplicate fmt_limit()'s whole body into a new function rather than sharing it via delegation.
• Selected Approach: (2).
• Rationale: verified algebraically that xce_format_units_value(XCE_UNITS_RAW, v, ...) produces the bare "%f" rendering of v, identical to what fmt_limit() prints today in both its percent and non-percent branches — so fmt_limit(buf, len, limit) == fmt_limit_units(buf, len, limit, XCE_UNITS_RAW) for every input, provably byte-identical. This means test_limit.c's 5 existing assertions require zero changes, contrary to this spec's own discovery-phase estimate that they would need updating. Delegation (not duplication) avoids the two functions' consumer-dump loops drifting apart later.
• Trade-offs: two function names to maintain instead of one, but this matches the codebase's own precedent (fmt_jobid()/fmt_jobid_ex(), limit_seed_config()/limit_seed_config_impl()) for "add an extended sibling rather than break an existing signature."
• Follow-up: none.

Where the byte-formatter should live once it needs a non-scheduler caller

• Context: fmt_limit_units() lives in src/lib/xcelimit.c (shared library) but needs the same byte-scaling algorithm repl_format_bytes_human() (src/scheduler/src/repl_fmt.c, scheduler-only) already implements.
• Alternatives Considered:
  1. Duplicate the algorithm into xcelimit.c directly.
  2. Promote the algorithm (and its companion classifier/composer) into an existing shared-lib file, with the scheduler's existing functions becoming thin wrappers.
  3. Create a brand-new shared-lib file solely for this.
• Selected Approach: (2), landing in src/lib/utils.c/src/include/utils.h — the existing home of fmt_jobid()/fmt_jobid_ex(), the closest existing precedent for a small, generic, cross-component display-formatting helper with no domain coupling (confirmed no similar helper already exists elsewhere in src/lib; src/controller/testsuite/test.c's own format_bytes_human() is test-only and not a production dependency, so not consolidated by this spec).
• Rationale: avoids duplicating the same divide-by-1024-while->=1024 algorithm a third time in the codebase (it already exists twice: repl_format_bytes_human() production, test.c's format_bytes_human() test-only); utils.c/utils.h is already linked into every component that would ever need this, with zero new build-system surface.
• Trade-offs: xce_repl_units_mode_t (scheduler-private, repl_internal.h) and the new xce_units_mode_t (shared, utils.h) are two independently-defined enums with the same three semantic values, rather than one shared type — deliberate, since promoting the scheduler's own enum into shared-lib scope would be a layering violation in the other direction (a src/lib type must not be defined in terms of a scheduler-internal concept, nor should the scheduler's already-tested, already-working xce_repl_units_mode_t plumbing need to change for this). A small translate_units_mode() switch at the one new call site (repl_info.c) bridges the two; a Revalidation Trigger in design.md flags this mapping for a future maintainer.
• Follow-up: none.

Existing test fixtures were asserting on the bug, not a locked contract

• Context: test_repl_limit_table.c (built under Requirement Set 1) has multiple fixtures using "mem" as the limit's constrained resource, with assertions like strstr(captured, "accounted=4.00").
• Finding: these assertions currently pass BECAUSE of the resource-name argument bug (the classifier is never actually asked about "mem"), not because "4.00" is a deliberately-chosen display contract. Fixing the bug will legitimately change this output to a human-formatted form.
• Implication: this is expected, not a regression to avoid — design.md's Testing Strategy requires the implementer to prove this via a temporary revert (old assertion fails for the right reason once the bug is fixed) before finalizing the updated assertion, the same RED→GREEN discipline as any other bug fix in this codebase.

References

• .wolf/buglog.json bug-1784–bug-1790 (gitea#46 limit-matching chain, 2026-09-07) — internal, not a URL.
• .wolf/buglog.json bug-1593 (info resource GLOBAL/SHARED-vs-host-based display fix, 2026-08-19) — internal, not a URL.
• .wolf/buglog.json bug-1685 (limit plugin double-free; also the evidence that privdata->limits is name-keyed) — internal, not a URL.
• .wolf/buglog.json bug-1762 (build-wiring precedent for white-box REPL test files) — internal, not a URL.
• .wolf/buglog.json bug-1599 (bare-printf regression scanner false-positive) — internal, not a URL.
• .kiro/steering/resource-accounting.md — reservation-vs-actual-usage policy, scoped to XCE_RESF_DYNAMIC; confirmed not applicable to this design.