myxce/.kiro/specs/scheduler-sim-diagnostics/research.md

Research & Design Decisions

Summary

• Feature: scheduler-sim-diagnostics — Requirement set 1 (info limit REPL command)
• Discovery Scope: Extension (extends scheduler-core's existing REPL info command family and reuses the limit plugin's existing library-level formatters)
• Key Findings:
  • The REPL info command family is a single dispatch function (repl_cmd_info(), repl_info.c) with one else if branch per subcommand — adding limit is a mechanical extension of an existing, well-established pattern.
  • The limit plugin's private state (limit_plugin_data_t.limits/.instances) is genuinely inaccessible outside src/scheduler/src/plugins/limit.c today — no header declares the type — confirming the brief's stated gap. Two new one-line accessor functions close it.
  • Two library-level formatters already exist and already do most of Requirement 1's work: fmt_limit() and fmt_consumer()/fmt_group() (src/lib/xcelimit.c, src/lib/group.c), including correct inclusion/exclusion membership rendering (fmt_group() already prefixes excluded members with ~). These are adopted directly rather than rebuilt.
  • The live-usage read path (GLOBAL/SHARED single value vs. per-host breakout) is architecturally identical to the pattern repl_info_resource_one() already implements for ordinary resources (bug-1593) — the limit's synthetic capacity resource (vres->vres) is a real, already-registered xce_resource_t carrying the same XCE_RESF_GLOBAL/XCE_RESF_SHARED flags, so the same shared-vs-per-host read logic applies unmodified.

Research Log

Limit plugin data model and matching semantics

• Context: Requirement 1 AC1/AC5/AC6 need to know exactly what "scope dimensions" and "restricted membership" mean at the code level, since an earlier requirements-review pass surfaced a scope-vs-runtime ambiguity.
• Sources Consulted: src/include/xce/types.h (xce_limit_t, xce_limit_config_t, xce_consumer_t, xce_vres_t), src/lib/xcelimit.c (match_limit, build_vres_name, is_host_based_limit, instantiate_limit), src/scheduler/src/plugins/limit.c (match_job_host_limit_, match_job_global_limit_, update_limit_res_value, limit_prepare_allocation), .wolf/buglog.json bug-1784 through bug-1790 (the gitea#46 limit-matching bug chain, 2026-09-07).
• Findings:
  • xce_consumer_type_t has exactly four values (CONSUMER_USER, CONSUMER_QUEUE, CONSUMER_PROJECT, CONSUMER_HOST) — the four scope dimensions in Requirement 1 AC1 are the complete set, no fifth dimension exists.
  • is_host_based_limit() forces per-host live-instance tracking whenever the limit's value is a percentage of a resource that is not XCE_RESF_GLOBAL/XCE_RESF_SHARED — independent of whether CONSUMER_HOST was configured. This is real, current behavior (not hypothetical), confirmed via source and via bug-1789/1790's live-verified fix history for the equivalent allocation-side computation.
  • xce_group_t supports both members and ex_members (exclusion). Confirmed as real production configuration via bug-1789's own note describing a live-configured limit ("limit1... all users except rshen").
  • fmt_group() (src/lib/group.c:388) already renders both lists correctly (~name prefix for exclusions, all/[] shortcuts for the builtin groups) and is already extern-declared in group.h. fmt_consumer()/fmt_limit() (src/lib/xcelimit.c) already compose fmt_group() into a full per-limit configuration dump.
  • The bug-1789/1790 chain (global limits under-counting usage against per-host resources) is fully closed as of 2026-09-07 — the underlying accounting this feature displays is trustworthy across all resource/scope combinations by the time this spec is implemented.
• Implications: Requirement 1's "configured setting" display can adopt fmt_limit() directly instead of re-deriving formatting logic; only the effective-host-scope annotation (Req 1.6) and the live-instance/usage rendering (Req 2, Req 3) are genuinely new code.

REPL command dispatch and existing info resource precedent

• Context: Need the concrete integration point for a new info limit subcommand and a working precedent for the GLOBAL/SHARED-vs-per-host live-value read split Requirement 2/3 require.
• Sources Consulted: src/scheduler/src/repl.c (repl_dispatch), src/scheduler/src/repl_info.c (repl_cmd_info, repl_info_resource, repl_info_resource_one), src/scheduler/src/plugin.c (plugin_register, plugin_register_core), src/scheduler/include/plugin.h, bug-1593 (repl_info_resource_one's original GLOBAL/SHARED-vs-host-based display bug and fix).
• Findings:
  • repl_cmd_info() is a flat strcmp chain on tokens[1]; every existing subcommand (job, queue, worker, resource, ...) follows the identical <name>|all argument shape Requirement 4 asks for.
  • repl_info_resource_one() (post bug-1593) reads a GLOBAL/SHARED resource's value from sim->core.sharedInst/sharedInUse at r->idx, and explicitly declines to fabricate a value for a host-based resource, deferring to info worker <name> instead. This is the exact pattern Requirement 2/3's per-instance read path reuses.
  • Plugins are registered into core->plugins, a name-keyed handle (key_add(core->plugins, plugin->name, plugin)); the limit plugin registers under the fixed name "xce_sched_limit" (limit.c:557). No plugin is ever renamed or re-registered under a different name at runtime.
  • Synthetic limit-capacity resources (vres->vres, created by add_vres()) are inserted into the same core.resmgr table info resource/info worker already iterate, flagged XCE_RESF_LIMIT (already a named flag in repl_resource_flags_str's table). A GLOBAL-scoped limit's synthetic resource can therefore already appear in info resource all/info resource <name> today, labeled with the LIMIT flag — this is pre-existing behavior, unrelated to and unmodified by this spec.
• Implications: New command wiring is a single new branch in repl_cmd_info() plus one new static function repl_info_limit() in repl_info.c, following repl_info_resource()'s two-level (_all iterates, _one prints) shape.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Direct struct exposure	Move limit_plugin_data_t into a shared header, let repl_info.c include it directly	Fewer functions	Breaks the plugin's existing data encapsulation; every other plugin's private state stays file-local — this would be the only exception	Rejected
Opaque query API (iterator_t out-params, per-limit filtered iteration)	New multi-function query interface returning filtered iterators	Nice call-site ergonomics	New abstraction with only one consumer today; no existing accessor in this codebase does filtered iteration — the rest (sched_get_job, core->plugins, core->jobs) hand back a raw kvmgr/xlist handle and let the caller iter_make/iter_next	Rejected — over-built for current scope
Selected: two raw-handle accessors	limit_get_configured_limits(core) / limit_get_usage_instances(core) return the plugin's existing var (kvmgr/xlist) handles, or NULL if the plugin isn't registered; caller iterates with the same iter_make/iter_next/iter_destroy idiom used everywhere else in this codebase	Matches every existing accessor pattern (core->jobs, core->queues, core->plugins); minimal new surface (2 functions); no new types	Caller must filter usage instances by vres->limit == target_limit itself (pointer comparison) since there is no per-limit index — acceptable, this table is small (configured-limit count × scope-instance count, not job count)	Selected

Design Decisions

Decision: Plugin accessor shape

• Context: limit_plugin_data_t (limits, instances) is a file-local type in limit.c; Requirement 1/2/3 need read access to both from repl_info.c.
• Alternatives Considered:
  1. Expose the struct itself in a shared header.
  2. Build a filtered query API (per-name lookup, per-limit instance iteration) as new plugin-owned functions.
  3. Expose the two raw var handles as accessor functions, encapsulating the plugin-name lookup ("xce_sched_limit") inside the accessor itself so repl_info.c never needs to know it.
• Selected Approach: (3). limit_get_configured_limits(xce_sched_core_t* core) and limit_get_usage_instances(xce_sched_core_t* core), both declared in src/scheduler/include/plugin.h next to register_limit, defined in limit.c. Each does its own key_find(core->plugins, "xce_sched_limit", &plugin) and returns plugin->data's relevant field (or NULL if the plugin is not registered or not yet initialized).
• Rationale: Matches the codebase's existing accessor idiom exactly (raw handle + shared iterator trio), keeps the plugin-name string fully inside limit.c (the one place that already owns it via register_limit()), and adds the minimum surface needed — no speculative per-limit-name lookup function, since both AC1's "specific name" and AC2's "all" cases are satisfiable by iterating the same handle and filtering in the caller.
• Trade-offs: repl_info_limit() finds a limit by name via a direct key_find() (privdata->limits is a name-keyed kvmgr — confirmed by bug-1685's duplicate-name -EEXIST handling, so this is O(1)-ish, not a scan) and a linear scan + pointer-equality filter to find that limit's usage instances in privdata->instances (keyed by vresname, not by owning limit — no index avoids this scan; accepted, same scale argument as below).
• Follow-up: None — this is a stable, minimal boundary; a future capability under this same open-ended spec (e.g. decision-reasoning trace) does not need to touch this accessor.

Decision: Reuse fmt_limit()/fmt_group() for configured-setting display

• Context: Requirement 1 AC1/AC5 need to print a limit's name, per-dimension scope, and inclusion/exclusion membership.
• Alternatives Considered:
  1. Write new formatting code in repl_info.c mirroring the other repl_info_*_one() functions' inline fprintf style.
  2. Call the existing fmt_limit() (which internally calls fmt_consumer()/fmt_group()) into a stack buffer and print it, then append the REPL-specific effective-scope note and live-usage lines the library formatter cannot know about.
• Selected Approach: (2).
• Rationale: fmt_limit() already produces correct, exclusion-aware output and is already used/tested at the library level (src/lib/testsuite/test_limit.c); duplicating that logic in repl_info.c would both waste effort and risk drifting out of sync with the library's own definition of "configured setting." Build-vs-adopt: adopt.
• Trade-offs: fmt_limit()'s output format (multi-line, PER_USER = [...] style) is fixed by the library; the REPL wraps it with a [sim] prefix per line rather than reformatting it, consistent with how other multi-line info output is already prefixed in this codebase (e.g. info jobs).
• Follow-up: None.

Decision: Effective-scope annotation is REPL-layer, not library-layer

• Context: Requirement 1 AC6 (implicit host-scoping for a percent-based limit on a non-shared resource) is not something fmt_limit() knows about — it only reflects the configured consumers[] array.
• Alternatives Considered:
  1. Change fmt_limit() itself to call is_host_based_limit() and inject a synthetic PER_HOST line.
  2. Keep fmt_limit() unchanged (it is a general-purpose library formatter used elsewhere, e.g. by any future non-REPL consumer that wants the raw configured shape) and have repl_info_limit() call is_host_based_limit() itself, appending an explanatory note only when consumers[CONSUMER_HOST] == NULL and the check returns true.
• Selected Approach: (2).
• Rationale: is_host_based_limit() is already extern-declared in xcelimit.h and used exactly this way (a yes/no runtime check) elsewhere; the "effective scope" concept is specific to what an operator inspecting the REPL needs to know, not a general property of the limit's configured shape — keeping it at the REPL layer avoids changing a shared library formatter's contract for one caller's needs.
• Trade-offs: None material.
• Follow-up: None.

Risks & Mitigations

• Risk: Linear scan over privdata->instances to filter by vres->limit == target_limit (this table is keyed by vresname, not by owning limit, so no index avoids the scan) could be surprising if a deployment ever configures an unusually large number of distinct scope instances (e.g. thousands of users × per-user limits). — Mitigation: Accepted for this spec; the existing match_job_host_limit_/match_job_global_limit_ matching path already does a key_find per job per limit against the same table, so this table's size is already load-bearing elsewhere in the scheduler's hot path — a read-only REPL scan over it is not a new scaling concern this spec introduces.
• Risk: Because is_host_based_limit() takes the plugin's resmgr handle (a var), calling it from repl_info.c requires reaching sim->core.resmgr — already public (used directly by repl_info_resource() today), so no new exposure needed. — Mitigation: N/A, confirmed no new risk.
• Risk: build_vres_name() never encodes host in the vresname it builds — an implementer could reasonably assume "one xce_vres_t per (user, queue, project, host)" and design storage/iteration around a cardinality that does not exist. — Mitigation: design.md's Data Models section states the correct cardinality explicitly (one xce_vres_t per (user, queue, project); host is produced by a per-worker fan-out over that single xce_vres_t's shared resource index, not by additional xce_vres_t instances) and cites this finding.
• Risk: Build wiring for the new white-box test — test_repl_resource_table.c's existing #include set omits plugins/limit.c and resource.c; naively adding a limit-info test there, or to a careless new file, reproduces the exact undefined-reference class bug-1762 already hit once for a sibling REPL feature. — Mitigation: design.md's Testing Strategy calls this out explicitly and directs a new sibling test file (test_repl_limit_table.c) with its own white-box include set, real (not stubbed) resource.c since the test needs non-trivial accounted-usage values.

Additional Findings (design review pass)

• xce_sched_worker_t carries two separate zarrays per resource idx: .resvals (capacity) and .resInUse (accounted usage) — confirmed via sched_check_host_satisfy_() (resource.c), which reads both the identical way this design's per-host instance display needs to.
• instantiate_limit() and update_limit_res_value() run back-to-back inside the same match_job_host_limit_/match_job_global_limit_ call (plugins/limit.c) — a matched xce_vres_t's capacity entry is therefore always populated by the time the xce_vres_t itself is observable via the new accessors; only its in-use entry can legitimately be absent (genuine zero usage, not "no instance").
• resource-accounting.md's reservation-vs-actual-usage reconciliation is explicitly scoped to resources flagged both XCE_RESF_DYNAMIC and XCE_RESF_CONSUMABLE. A limit's synthetic capacity resource (add_vres()) is flagged XCE_RESF_LIMIT | XCE_RESF_CONSUMABLE | XCE_RESF_RELEASE plus GLOBAL|SHARED or DEFAULT_HOST — never XCE_RESF_DYNAMIC — so that steering document does not apply here; the capacity/in-use pair this design reads is an exact scheduler-computed accounting number, not an OS-reconciled estimate.
• src/scheduler/testsuite/test_no_bare_printf.c regression-tests every file under src/scheduler/src/ for bare printf() calls (must use fprintf(stdout, ...)); bug-1599 documents a known scanner false-positive on comment prose, not an exemption for real code.

References

• .wolf/buglog.json bug-1784–bug-1790 (gitea#46 limit-matching chain, 2026-09-07) — internal, not a URL.
• .wolf/buglog.json bug-1593 (info resource GLOBAL/SHARED-vs-host-based display fix, 2026-08-19) — internal, not a URL.
• .wolf/buglog.json bug-1685 (limit plugin double-free; also the evidence that privdata->limits is name-keyed) — internal, not a URL.
• .wolf/buglog.json bug-1762 (build-wiring precedent for white-box REPL test files) — internal, not a URL.
• .wolf/buglog.json bug-1599 (bare-printf regression scanner false-positive) — internal, not a URL.
• .kiro/steering/resource-accounting.md — reservation-vs-actual-usage policy, scoped to XCE_RESF_DYNAMIC; confirmed not applicable to this design.