add resource distribution map project

.kiro/specs/resource-distribution-mapping/requirements.md

@@ -0,0 +1,7 @@
+# Requirements Document
+
+## Project Description (Input)
+Map resources.yaml's `resourcemap` field (per-host static/dynamic resource value assignments) through to the controller. Currently xcecommon.mapping._build_resource cannot map it because controller.capnp's Resource struct has its distributions/hostBased fields commented out (controller.capnp:208-209, corresponding to xce_resource_t's distributions/hbased_defined in include/xce/types.h). Scope: (1) design and un-comment/wire the capnp Resource.distributions (List(ResourceDistribution)) and Resource.hostBased fields, (2) regenerate generated capnp bindings, (3) implement xcecommon.mapping's YAML resourcemap -> capnp distributions/hostBased translation (YAML shape: list of {value, location: [hostnames]} entries -- see src/configs/resources.yaml for real examples covering dynamic hostbased, static shared, and static per-host-distinct resource patterns), (4) remove resourcemap from build.py's _UNMAPPED_BUT_VALID_ITEM_KEYS once mapped, (5) verify round-trip against the C side (res_distribution_t / xce_resource_t consumers, e.g. scheduler.c/worker.c resource distribution logic) to confirm the encoding actually matches what the C code expects at runtime, not just that it compiles. This is a real capnp schema change (adds wire-format fields) affecting the ClusterConfig contract other components already depend on -- needs design review, not a blind field un-comment.
+
+## Requirements
+<!-- BLOCKED: scope decision pending, see research.md -- do not generate requirements until one of the three candidate scopes there is chosen. Initial framing (a simple capnp-field-uncomment) understated the real scope once the C-side consumers were traced: the fields are dead end-to-end (not just uncommented capnp), a separate live per-host mechanism already exists via hosts.yaml, and two of the three plausible resourcemap semantics (shared pool, dynamic per-host reporting) require genuinely new controller/scheduler/worker runtime capability, not just a mapping fix. -->

.kiro/specs/resource-distribution-mapping/research.md

@@ -0,0 +1,30 @@
+# Research: resourcemap / distributions / hostBased
+
+## Status: scope decision pending — do not generate requirements.md until a scope option below is chosen
+
+## What triggered this spec
+
+`xceadmin config build` warns `resources.yaml: field 'resourcemap' on '<name>' is schema-valid but has no ClusterConfig destination -- ignored` for every resource entry that uses `resourcemap`. The initial framing (a quick capnp-field-uncomment-and-wire task) turned out to understate the real scope once the C-side consumers were traced — see findings below.
+
+## Findings (verified against code, file:line cited)
+
+- `res_distribution_t` (`include/xce/types.h:127-131`): `{value: xce_val_t*, nhosts: int, hosts: char**}` — maps 1:1 to one `resourcemap` YAML entry `{value, location: [...]}`. Commented-out capnp counterpart: `src/schema/controller.capnp:195-198,208-209`.
+- `xce_resource_t.hbased_defined` is typed `xce_val_t*` (a value, not a bool flag) — plausibly a fallback/default value for hosts not covered by any explicit `distributions` entry (matches the `location: [default]` sentinel used by `dynres1`/`strres1`/`staticres3`/`host_quota`). Inference only — nothing in the codebase reads this field to confirm.
+- **The fields are dead on the C side too, not just uncommented in capnp.** `resobj_init()` (`src/lib/resobj.c:37-51`) explicitly zeroes `dst->distributions`/`dst->hbased_defined` with `/* FIXME: */` comments, even under a hypothetical populated input. `scheduler.c:787-813`'s resource update handler copies only `valtype`/`ordtype`/`flags`/`interval`, with a comment explaicitly excluding these two as "deep-structs." No other file in scheduler/, worker/, controller/, lib/ reads `res_distribution_t`, `->distributions`, or `->hbased_defined` at all (repo-wide grep, zero other hits).
+- **A separate, fully-live per-host resource-value pipeline already exists**, independent of `resourcemap`, via `hosts.yaml`'s host-level `resource: [{name, value}]` list. Full trace: `mapping.py:_build_host_config` → `HostConfig.resources` (capnp) → controller `apply_join_state()` → `mark_host_res()` (`src/controller/worker.c:163-221`) → `update_val_res()` writes `w->host.resvals[idx]` → pushed to scheduler via `collect_resval()`/`Worker.resvals` → `schedobj.c:286-304` `zarray_put(&(worker->resvals), ...)` → read by `sched_check_host_satisfy_()` (`src/scheduler/src/resource.c:330,357-379`) during job placement.
+- `mark_host_res()` (`worker.c:190-193`) **actively rejects** a resource flagged `XCE_RESF_SHARED`: `if (r->flags & XCE_RESF_SHARED) { ERROR(...); return -EINVAL; }`. So "shared pool" and "per-host static value" are mutually exclusive today, by construction.
+- `XCE_RESF_SHARED`/`GLOBAL` pooling machinery (`sched->sharedInst`, `sched_check_shared_satisfy_()`, `src/scheduler/src/resource.c:232-282`) is real and live, but exclusively serves job-limit virtual resources (`src/lib/xcelimit.c:399-404`'s `add_vres()`), never a physical `resources.yaml` resource. The scheduler plumbing for "N consumers draw from one pooled value" is architecturally reusable, just not reachable from `resourcemap` today.
+- `interval`/dynamic-resource semantics have **zero enforcement logic anywhere** — `last_report` is never read; `interval` is only copied, never acted on. The only real dynamic-reporting mechanism is `src/worker/src/resource_sync.c`'s 11 hardcoded built-in load indices (`init_builtin_load_indices()`, lines 59-94). A plugin hook exists to register a *custom* dynamic resource (`resource_sync_register_load_index()`, line 104) but nothing calls it from any YAML-driven path — a user-defined dynamic resource (`dynres1`/`dynres2`-style) has no live way to ever acquire a value today.
+- `src/configs/resources.yaml`'s own example comments are internally inconsistent about "share" semantics: `staticres1` (one `resourcemap` entry, two hosts in `location:`) says "host1 and host2 **share** the 100 res"; `staticres2` (two separate entries, one host each) says "each of host1 and host2 **shares** the 100 res" — opposite shapes, same word. Not a reliable semantic source as-is.
+
+## Three candidate scopes (sizes verified against the above, not guessed)
+
+1. **Static replication to N hosts** (`staticres3`-style: one `resourcemap` declaration → the same static value written into N named hosts' per-host `resvals`, reusing the existing `mark_host_res()`/`update_val_res()` path). Pure DRY/ergonomics win — functionally redundant with just repeating `{name, value}` across N `hosts.yaml` host entries, since `hosts.yaml` already does the "one host, one static value" case live end-to-end. Small: capnp field un-comment + mapping.py translation + a controller-side loop over the host list at config-load/push time. No scheduler changes, no new runtime concept.
+
+2. **Static replication + shared pool** (adds `staticres1`/`staticres4`-style: N hosts collectively draw down ONE pooled value). Requires: allowing `XCE_RESF_SHARED` on a physical resource (currently `mark_host_res` refuses this), wiring `sched->sharedInst` for resources.yaml-sourced resources (today limits-only), and defining job-placement semantics for a shared physical resource (distinct from the existing shared *limit* semantics — a resource being consumed to zero is a scheduling constraint, not a rejection). Real new capability, touches job-placement code.
+
+3. **All three, plus dynamic per-host custom resource reporting** (`dynres1`/`dynres2`-style). Needs a new worker-side mechanism: something must call `resource_sync_register_load_index()` (or an equivalent) per YAML-declared dynamic resource, decide how/when it's collected per host, and thread it into the same `resvals` pipewith proper interval-based refresh — none of which exists today even as a partial scaffold. Largest scope.
+
+## Decision
+
+Not yet made — user chose to record the options and decide later rather than commit now. **Do not proceed to requirements.md generation until one of the three scopes (or a variant) is chosen.** When resumed, re-run `/kiro-spec-requirements resource-distribution-mapping` with the chosen scope stated explicitly, since the EARS requirements (and everything downstream) differ substantially by option.

.kiro/specs/resource-distribution-mapping/spec.json

@@ -0,0 +1,22 @@
+{
+  "feature_name": "resource-distribution-mapping",
+  "created_at": "2026-07-23T08:25:47Z",
+  "updated_at": "2026-07-23T08:25:47Z",
+  "language": "en",
+  "phase": "initialized",
+  "approvals": {
+    "requirements": {
+      "generated": false,
+      "approved": false
+    },
+    "design": {
+      "generated": false,
+      "approved": false
+    },
+    "tasks": {
+      "generated": false,
+      "approved": false
+    }
+  },
+  "ready_for_implementation": false
+}