Research: resourcemap / distributions / hostBased

Status: scope decision pending — do not generate requirements.md until a scope option below is chosen

What triggered this spec

xceadmin config build warns resources.yaml: field 'resourcemap' on '<name>' is schema-valid but has no ClusterConfig destination -- ignored for every resource entry that uses resourcemap. The initial framing (a quick capnp-field-uncomment-and-wire task) turned out to understate the real scope once the C-side consumers were traced — see findings below.

Findings (verified against code, file:line cited)

res_distribution_t (include/xce/types.h:127-131): {value: xce_val_t*, nhosts: int, hosts: char**} — maps 1:1 to one resourcemap YAML entry {value, location: [...]}. Commented-out capnp counterpart: src/schema/controller.capnp:195-198,208-209.
xce_resource_t.hbased_defined is typed xce_val_t* (a value, not a bool flag) — plausibly a fallback/default value for hosts not covered by any explicit distributions entry (matches the location: [default] sentinel used by dynres1/strres1/staticres3/host_quota). Inference only — nothing in the codebase reads this field to confirm.
The fields are dead on the C side too, not just uncommented in capnp. resobj_init() (src/lib/resobj.c:37-51) explicitly zeroes dst->distributions/dst->hbased_defined with /* FIXME: */ comments, even under a hypothetical populated input. scheduler.c:787-813's resource update handler copies only valtype/ordtype/flags/interval, with a comment explaicitly excluding these two as "deep-structs." No other file in scheduler/, worker/, controller/, lib/ reads res_distribution_t, ->distributions, or ->hbased_defined at all (repo-wide grep, zero other hits).
A separate, fully-live per-host resource-value pipeline already exists, independent of resourcemap, via hosts.yaml's host-level resource: [{name, value}] list. Full trace: mapping.py:_build_host_config → HostConfig.resources (capnp) → controller apply_join_state() → mark_host_res() (src/controller/worker.c:163-221) → update_val_res() writes w->host.resvals[idx] → pushed to scheduler via collect_resval()/Worker.resvals → schedobj.c:286-304 zarray_put(&(worker->resvals), ...) → read by sched_check_host_satisfy_() (src/scheduler/src/resource.c:330,357-379) during job placement.
mark_host_res() (worker.c:190-193) actively rejects a resource flagged XCE_RESF_SHARED: if (r->flags & XCE_RESF_SHARED) { ERROR(...); return -EINVAL; }. So "shared pool" and "per-host static value" are mutually exclusive today, by construction.
XCE_RESF_SHARED/GLOBAL pooling machinery (sched->sharedInst, sched_check_shared_satisfy_(), src/scheduler/src/resource.c:232-282) is real and live, but exclusively serves job-limit virtual resources (src/lib/xcelimit.c:399-404's add_vres()), never a physical resources.yaml resource. The scheduler plumbing for "N consumers draw from one pooled value" is architecturally reusable, just not reachable from resourcemap today.
interval/dynamic-resource semantics have zero enforcement logic anywhere — last_report is never read; interval is only copied, never acted on. The only real dynamic-reporting mechanism is src/worker/src/resource_sync.c's 11 hardcoded built-in load indices (init_builtin_load_indices(), lines 59-94). A plugin hook exists to register a custom dynamic resource (resource_sync_register_load_index(), line 104) but nothing calls it from any YAML-driven path — a user-defined dynamic resource (dynres1/dynres2-style) has no live way to ever acquire a value today.
src/configs/resources.yaml's own example comments are internally inconsistent about "share" semantics: staticres1 (one resourcemap entry, two hosts in location:) says "host1 and host2 share the 100 res"; staticres2 (two separate entries, one host each) says "each of host1 and host2 shares the 100 res" — opposite shapes, same word. Not a reliable semantic source as-is.

Three candidate scopes (sizes verified against the above, not guessed)

Static replication to N hosts (staticres3-style: one resourcemap declaration → the same static value written into N named hosts' per-host resvals, reusing the existing mark_host_res()/update_val_res() path). Pure DRY/ergonomics win — functionally redundant with just repeating {name, value} across N hosts.yaml host entries, since hosts.yaml already does the "one host, one static value" case live end-to-end. Small: capnp field un-comment + mapping.py translation + a controller-side loop over the host list at config-load/push time. No scheduler changes, no new runtime concept.
Static replication + shared pool (adds staticres1/staticres4-style: N hosts collectively draw down ONE pooled value). Requires: allowing XCE_RESF_SHARED on a physical resource (currently mark_host_res refuses this), wiring sched->sharedInst for resources.yaml-sourced resources (today limits-only), and defining job-placement semantics for a shared physical resource (distinct from the existing shared limit semantics — a resource being consumed to zero is a scheduling constraint, not a rejection). Real new capability, touches job-placement code.
All three, plus dynamic per-host custom resource reporting (dynres1/dynres2-style). Needs a new worker-side mechanism: something must call resource_sync_register_load_index() (or an equivalent) per YAML-declared dynamic resource, decide how/when it's collected per host, and thread it into the same resvals pipewith proper interval-based refresh — none of which exists today even as a partial scaffold. Largest scope.

Decision

Not yet made — user chose to record the options and decide later rather than commit now. Do not proceed to requirements.md generation until one of the three scopes (or a variant) is chosen. When resumed, re-run /kiro-spec-requirements resource-distribution-mapping with the chosen scope stated explicitly, since the EARS requirements (and everything downstream) differ substantially by option.

6.1 KiB Raw Blame History