myxce/.kiro/specs/resval-id-decoupling/research.md

# Gap Analysis: resval-id-decoupling

> Generated: 2026-05-12T07:25:00-07:00
> Feature: resval-id-decoupling

## Executive Summary

- **Scope**: Three call sites in the controller use `collect_resval_with_idx()` to produce index-keyed resource values sent to two consumers (scheduler and API server). Fixing all three brings every cross-subsystem resval path into alignment with the already-correct `Data_data_resvals` dataset path.
- **Existing Infrastructure**: `xce_name_val_t` (`NameValue` in capnp) already exists in `types.h` and has a **complete capnp codec** (`encode_NameValue_list` / `decode_NameValue_list`) in the generated code — it was defined for this purpose but never wired in at the resval-transfer boundary.
- **Wire format impact**: `Conf4Sched.resvals` (field @6) and `Worker.resvals` (field @5) in `controller.capnp` currently declare `List(ValueWithIdx)`. Changing these to `List(NameValue)` is a **breaking wire change** — no intermediate backward-compatibility window unless two field slots are used.
- **Build chain**: The capnp schema in `src/schema/controller.capnp` drives generation of `src/build/lib/controller.capnp.[ch]` via `meson` + `capnpc-c`. A schema edit automatically regenerates the C codecs on the next build.
- **Key insight**: `makeResource_()` in `resobj.c:137` always overwrites `resobj->idx = rc` (from `key_add`) regardless of the `idx` transmitted in the `Resource` wire struct — so the controller's idx is lost at the scheduler's resmgr and the assumption that they match is never enforced.

---

## Existing Components Inventory

### Types and Structs

| Symbol | Location | Role | Status |
|--------|----------|------|--------|
| `xce_val_with_idx_t` | `src/include/xce/types.h:38` | `{int idx, xce_val_t* val}` — current cross-subsystem carrier | **Replace at boundaries** |
| `xce_name_val_t` | `src/include/xce/types.h:43` | `{char* name, xce_val_t* val}` — name-keyed carrier | **Ready to use** |
| `xce_resval_t` | `src/include/xce/types.h:48` | `{char* name, char* host, xce_val_t* val}` — used by Path C | Already correct |
| `xce_conf4sched_t.resvals` | `src/include/typemap.h:50` | `xce_val_with_idx_t**` — shared resvals in initial config | **Change to `xce_name_val_t**`** |
| `xce_worker_t.resvals_to_sched` | `src/include/xce/types.h:217` | `xce_val_with_idx_t**` — per-worker resvals | **Change to `xce_name_val_t**`** |

### Controller — Call Sites (Sender Side)

| File | Line | Function | Purpose | Change needed |
|------|------|----------|---------|---------------|
| `src/controller/dataex.c` | 62 | `collect_resval_with_idx()` | Iterates `zarray_t`, emits `{idx, val}` | Replace/augment with name-keyed variant |
| `src/controller/dataex.c` | 363 | `collect_shared_resvals()` | Calls `collect_resval_with_idx(&c->sharedinst)` | Change to name-keyed collection |
| `src/controller/dataex.c` | 141–163 | `collect_resvals_workers()` | Calls `collect_resval_with_idx(&w->resvals)` per worker | Change to name-keyed collection |
| `src/controller/sched.c` | 118–120 | `xcesched_push_configs()` | Puts shared resvals into `conf4sched->resvals` | Change field type; update free |
| `src/controller/sched.c` | 442–444 | `xcesched_push_workerlist()` | Puts worker resvals into `w->host.resvals_to_sched` | Change field type; update free |
| `src/controller/apiserver.c` | 367–369 | `apiserver_worker_change()` | Same pattern as sched for API server path | Change field type; update free |

### Free/cleanup Sites

| File | Line | Call | Must be updated |
|------|------|------|-----------------|
| `src/controller/sched.c` | 145 | `free_resval_with_idx_list(&conf4sched->resvals, ...)` | → `free_NameValue_list` or equivalent |
| `src/controller/sched.c` | 226, 463 | `free_resval_with_idx_list(&w->resvals_to_sched, ...)` | Same |
| `src/controller/apiserver.c` | 51, 397–399 | `free_resval_with_idx_list(...)` | Same |
| `src/controller/dataex.c` | 404 | `free_resval_with_idx_list(...)` | Same |
| `src/controller/reply.c` | 66 | `free_resval_with_idx_list(...)` | Same |
| `src/controller/opget.c` | 128 | `free_resval_with_idx_list(...)` | Same |
| `src/include/utils.h` | 130 | `free_resval_with_idx_list` declaration | Update or keep alongside new |

### Scheduler — Consumer Side

| File | Line | Code | Change needed |
|------|------|------|---------------|
| `src/scheduler/src/scheduler.c` | 574–586 | `sched_init_sharedInst()` — uses `idx` directly from `config->resvals[i]->idx` | Change to name-lookup in `sched->resmgr` |
| `src/scheduler/src/schedobj.c` | 274–280 | `make_sched_worker()` — uses `idx` directly from `w->resvals_to_sched[i]->idx` | Change to name-lookup |

### Capnp Schema and Generated Code

| Asset | Location | Current | Change |
|-------|----------|---------|--------|
| `Conf4Sched.resvals` | `src/schema/controller.capnp:230` | `List(ValueWithIdx)` | → `List(NameValue)` |
| `Worker.resvals` | `src/schema/controller.capnp:129` | `List(ValueWithIdx)` | → `List(NameValue)` |
| `controller.capnp.c/h` | `src/build/lib/` | Auto-generated — regenerated by `ninja -C build` | Automatically updated after schema edit |
| `encode_NameValue_list` | `src/build/lib/controller.capnp.c:3402` | Already exists, unused at these boundaries | Wire in to replace `encode_ValueWithIdx_list` |
| `decode_NameValue_list` | `src/build/lib/controller.capnp.c:3426` | Already exists, unused at these boundaries | Wire in to replace `decode_ValueWithIdx_list` |
| `free_NameValue_list` | `src/build/lib/controller.capnp.c:3458` | Already exists | Replace `free_ValueWithIdx_list` at these call sites |

### API Server (Go frontend) — Impact

The Go API server (`src/frontend/apiserver/backend/clib.go`) includes `controller.capnp.h` and works with `xce_worker_t*` via CGo. It accesses `resvals_to_sched` in `apiserver_worker_change()` (controller side) which populates `xce_worker_t.resvals_to_sched`. The Go frontend's `cResvalToDistribution` in `convert.go:273` already processes `xce_resval_t` (name-keyed), not `xce_val_with_idx_t`. The API server impact is limited to updating any code that reads `resvals_to_sched` field — if it accesses it at all on the Go side (currently no direct access found).

---

## Root Cause Deep Dive

The zarray-based stores (`c->sharedinst`, `w->host.resvals`) use `r->idx` as the slot key. `r->idx` is assigned by `key_add()` in `makeResource_` (`resobj.c:137`) — it is the slot returned by the internal container, which depends on insertion order. The controller and scheduler each call `makeResource()` independently (controller at cluster init, scheduler at `sched_init_resources()`), with no guarantee of identical ordering.

Crucially, the `Resource` capnp struct **does transmit `r->idx` on the wire** (`encode_Resource` at `build/lib/controller.capnp.c:3889`), but `decode_Resource` writes it into a temporary `xce_resource_t`, which is then passed to `makeResource()`, which **immediately overwrites** the decoded `idx` with the local `key_add` return value (`resobj->idx = rc`). The transmitted idx is therefore discarded at the scheduler.

---

## Implementation Options

### Option A — Replace `ValueWithIdx` with `NameValue` at both capnp fields (Recommended)

**What changes:**
1. `controller.capnp`: change `Conf4Sched.resvals @6` and `Worker.resvals @5` from `List(ValueWithIdx)` to `List(NameValue)`
2. Rebuild capnp — generated `encode_Conf4Sched`/`decode_Conf4Sched` and worker codec automatically use `NameValue` functions
3. `typemap.h`: `xce_conf4sched_t.resvals` → `xce_name_val_t**`
4. `types.h`: `xce_worker_t.resvals_to_sched` → `xce_name_val_t**`
5. Controller: new `collect_resval_by_name(zarray_t* vals, var resmgr, xlist_t* l)` that walks `resmgr` entries, reads the value by `r->idx` from the array, and emits `{name, val}` records
6. Scheduler `sched_init_sharedInst()`: iterate name-val list, `key_find(sched->resmgr, name)` → `r->idx` → `zarray_put`
7. Scheduler `make_sched_worker()`: same pattern
8. Update all free/cleanup call sites to use `free_NameValue_list`

**Pros:** Cleanest — removes the misleading `idx` field entirely; `NameValue` codec already exists; aligns with Path C's design; no backward compatibility concerns given no public release
**Cons:** None significant — wire-format break is a non-issue as there is no released version requiring compatibility

**Affected files (estimate):** 8 C files, 1 capnp schema, 0 Go files

---

### Option B — Add `name` field to `xce_val_with_idx_t` / `ValueWithIdx` ~~(Additive wire change)~~ [REJECTED]

Ruled out: backward compatibility is not a requirement for this project at its current stage. Option B's only advantage over Option A is a softer wire transition — without that constraint, it is strictly worse: it keeps a semantically dead `idx` field, requires more code changes, and leaves misleading structure in the API.

---

### Option C — Use Controller's idx as canonical; stop overwriting in `makeResource_` [REJECTED]

Ruled out: does not fix the design — just moves the source of truth without eliminating the hidden identity assumption. A resource deleted and re-added in the controller gets a new idx, and the scheduler has no way to know its cached idx is stale.

---

## Recommendation

**Option A** is the selected approach. Backward compatibility is not a constraint (no public release), so the wire-format change to `List(NameValue)` is straightforward. The `NameValue` codec already exists in the generated capnp code and is fully functional — this is primarily a wiring exercise. The semantic change aligns the initial-config and worker-data paths with the already-correct `Data_data_resvals` update path (Path C).

---

## Files to Change (Option A)

| File | Change Type |
|------|-------------|
| `src/schema/controller.capnp` | Schema edit: 2 field type changes |
| `src/include/xce/types.h` | Struct field type change: `resvals_to_sched` |
| `src/include/typemap.h` | Struct field type change: `resvals` |
| `src/controller/dataex.c` | New `collect_resval_by_name()` function; update `collect_shared_resvals()` and `collect_resvals_workers()`; **remove** `collect_resval_with_idx` and `collect_resval_with_idx_` |
| `src/controller/sched.c` | Update 3 call sites (populate and free) |
| `src/controller/apiserver.c` | Update 2 call sites (populate and free) |
| `src/controller/reply.c` | Update free call |
| `src/controller/opget.c` | Update free call |
| `src/scheduler/src/scheduler.c` | Update `sched_init_sharedInst()` |
| `src/scheduler/src/schedobj.c` | Update `make_sched_worker()` |
| `src/include/utils.h` | Remove `free_resval_with_idx_list` declaration; add `free_name_val_list` declaration |
| `src/lib/utils.c` | Remove `free_resval_with_idx_list`; add `free_name_val_list(xce_name_val_t***, int)` |
| `src/controller/include/cluster.h` | Remove `collect_resval_with_idx` declaration |

**Not changed:** `src/scheduler/src/handler.c` (Path C untouched), `src/frontend/apiserver/` (Go API server does not directly access `resvals_to_sched` in Go code), capnp generated files (auto-regenerated by `ninja -C build`).

---

## Decisions (Closed Questions)

1. **New collector function signature**: `collect_resval_by_name` is controller-internal logic — add it to `dataex.c` alongside `collect_resval_with_idx`. It needs both the `zarray_t` (values indexed by `r->idx`) and the `resmgr` (name→idx mapping) as parameters.

2. **Handling unknown resources in scheduler**: **Warn and skip.** When `sched_init_sharedInst` or `make_sched_worker` encounters a name not in `sched->resmgr`, emit a `WARN` log and continue. This is safe because the resource list always arrives in the same config push before resvals are applied (see decision 3).

3. **Ordering guarantee**: **Confirmed — current behavior, must be preserved.** `sched_init_resources()` is called before `sched_init_sharedInst()` in `sched_init_config()` (`scheduler.c:603-609`). The design must not reorder these calls.

4. **Removal of `collect_resval_with_idx`**: **Remove entirely.** No cross-subsystem callers will remain after this change, and intra-subsystem callers (e.g., `worker.c` reading from `sharedinst` for API get-resval) access the zarray directly by `r->idx` without going through this function. Delete `collect_resval_with_idx`, `collect_resval_with_idx_`, and remove the declaration from `src/controller/include/cluster.h`.

5. **`free_name_val_list` utility**: **Add to `utils.c`/`utils.h`.** A new `free_name_val_list(xce_name_val_t***, int)` function mirroring `free_resval_with_idx_list` is needed. The capnp-generated `free_NameValue_list` is an option but couples cleanup to the serialization layer — a standalone utility in `utils.c` is cleaner and consistent with the existing pattern.

---

# Cross-Spec Gap Audit (2026-06-08)

> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch.

## Analysis Summary

- **resval-id-decoupling** is **~100% complete**. All 8 task groups, 22 subtasks (1.1–8.3) are `[x]` in tasks.md.
- **No remaining gaps**: `collect_resval_with_idx` and `free_resval_with_idx_list` are removed (verified by grep: zero matches). `collect_resval` (name-keyed, iterates resmgr not zarray) replaces it. The capnp schema (`Conf4Sched.resvals @6`, `Worker.resvals @5`) now use `List(NameValue)`. The scheduler resolves by name via `key_find(sched->resmgr, name)` → `r->idx`. The Go API server has `cNameValToWorkerResval` in `convert.go` for the new `xce_name_val_t**` type.
- **Task approval gap**: spec.json shows `tasks.approved: false`. All tasks are implemented but the tasks document was never formally approved. This is a process artifact — the implementation matches the approved design.
- **Cross-spec impact**: `scheduler-controller-msg-gap` extends the same scheduler handlers (`sched_init_sharedInst`, `make_sched_worker`) that were modified by this spec. The two specs are consistent — both use name-based resource resolution.
- **Recommendation**: Mark tasks as approved in spec.json, or run `/kiro-spec-status resval-id-decoupling` to sync.

---

# Gap Analysis — Validation Pass (2026-06-09)

## Method

Independent re-verification of every requirement (requirements.md R1–R6) against the current code on branch `persistent_ha`, via grep + direct read of the controller, scheduler, libxce, capnp schema, and Go API-server sources. Prior audit (2026-06-08) claimed ~100% completion; this pass confirms that claim and adds two residual findings it missed.

## Current State Summary

- The feature is **implemented end-to-end**. `collect_resval_with_idx` and `free_resval_with_idx_list` no longer exist anywhere in `src/` (zero grep matches). The replacement `collect_resval(var resmgr, zarray_t*, xlist_t*)` lives at `src/controller/dataex.c:23` and iterates the resmgr (names) rather than the zarray (slots), exactly as designed.
- Wire format: `src/schema/controller.capnp` — `Worker.resvals @5` (line 138) and `Conf4Sched.resvals @6` (line 239) are both `List(NameValue)`. Field ordinals preserved.
- All tasks in tasks.md are `[x]` (1.1–8.3), but `spec.json` still shows `tasks.approved: false` (phase: `tasks-generated`). Implementation landed despite the unapproved tasks document — a process artifact, not a code gap.

## Requirement-to-Asset Map

| Req | EARS | Asset(s) | Status |
|-----|------|----------|--------|
| R1 — name-keyed shared resvals in initial config | R1-1 | `controller.capnp:239` `List(NameValue)`; `src/controller/dataex.c:381` `collect_resval(c->resmgr, &c->sharedinst, l)`; `src/controller/sched.c:120` casts to `xce_name_val_t**` | **Implemented** |
| | R1-2 | `src/scheduler/src/scheduler.c:574-599` `sched_init_sharedInst`: `key_find(sched->resmgr, nv->name)` → `zarray_put(&sched->sharedInst, r->idx, val)` | **Implemented** |
| | R1-3 | `scheduler.c:587-592`: WARN + `continue` on unknown name; NULL name/entry also skipped | **Implemented** |
| | R1-4 | Same name→`r->idx` resolution as Path C handler (`scheduler.c:782-797`) | **Implemented** (by inspection; no equivalence test exists — see R6) |
| R2 — name-keyed per-worker resvals to scheduler | R2-1 | `controller.capnp:138` `List(NameValue)`; `dataex.c:151-166` `collect_resvals_workers(resmgr,...)`; `sched.c:442` | **Implemented** |
| | R2-2 | `src/scheduler/src/schedobj.c:274-292` `make_sched_worker`: name lookup → `zarray_put(&worker->resvals, r->idx, val)` | **Implemented** |
| | R2-3 | `schedobj.c:283-288`: WARN + skip, worker construction continues | **Implemented** |
| | R2-4 | Resolution happens at each receipt against the live `sched->resmgr`; no index is cached across messages | **Implemented** (by design; no automated test for resource add/remove mid-run) |
| R3 — name-keyed worker resvals to API server | R3-1 | `src/controller/apiserver.c:367-369`: same `collect_resval` + `xce_name_val_t**` path as the scheduler push | **Implemented** |
| | R3-2 | `src/frontend/apiserver/backend/convert.go:100-105` iterates `resvals_to_sched`, `convert.go:283` `cNameValToWorkerResval` (name-keyed, nil-checked); `convert_pure.go:210`; `models/hosts.go:94` `ResourceValues` | **Implemented** |
| R4 — Path C unchanged | R4-1 | `src/scheduler/src/handler.c:82-103` (`Data_data_resvals` → `XCE_SCHED_OP_UPDATE_RESVALS_FULL`) and `scheduler.c:782-800` untouched, still `xce_resval_t` name-keyed | **Implemented** |
| | R4-2 | `xce_conf4sched_t.resvals` (`typemap.h:62`) and `xce_worker_t.resvals_to_sched` (`types.h:219`) are `xce_name_val_t**`; codec auto-generated; `free_name_val_list` in `src/lib/utils.c:388` / `utils.h:130`; all controller free sites converted | **Implemented** |
| R5 — remove index-keyed call sites | R5-1 | Zero cross-subsystem index-keyed transfers remain (grep-verified); both receivers resolve by name | **Implemented** |
| | R5-2 | `xce_val_with_idx_t` typedef (`src/include/xce/types.h:38-41`) and capnp `struct ValueWithIdx` (`controller.capnp:176`) still exist with **zero users and no scoping comment** | **Partial** (SHOULD-level: dead type lingers, contrary to "removed or restricted ... with a comment") |
| R6 — no regression / correctness under divergent indices | R6-1 | Correct by code inspection (name resolution is order-independent); **no automated test** registers resources in different orders and asserts `sharedInst` contents | **Partial** (code Implemented, test Missing) |
| | R6-2 | Existing controller/scheduler suites passed at task 8.2; same-order behavior preserved | **Implemented** |
| | R6-3 | `calc_wanted_*` paths read `sharedInst`/`worker->resvals` via scheduler-side `r->idx` (scheduler testsuite exercises these at zarray level) | **Implemented** |

### Adjacent finding (Constraint — out of spec scope, same defect class)

`XCE_SCHED_OP_UPDATE_RESVALS` (per-host running update, `handler.c:231-251` → `scheduler.c:801-816`) receives name-keyed `xce_resval_t` records but stores them with a **hardcoded slot**: `zarray_put(&w->resvals, 0, rv->val)` — it finds the worker by `rv->host` but never resolves `rv->name` to `r->idx`. Any per-host update for a resource whose scheduler idx ≠ 0 lands in the wrong slot (or clobbers slot 0). This path was not enumerated in this spec (Paths A/B/C only) and belongs to the `scheduler-controller-msg-gap` surface, but it violates the spirit of EARS-R5-1. Flagged for follow-up, not a regression of this feature.

## Implementation Approach Options (for residual gaps)

### Option A — Minimal cleanup within this spec (Recommended)
Delete the dead `xce_val_with_idx_t` typedef and capnp `ValueWithIdx` struct (or add the scoping comment EARS-R5-2 asks for); add one scheduler testsuite case that feeds `sched_init_sharedInst`/`make_sched_worker` a name-val list against a resmgr populated in a different order and asserts slot contents (closes R6-1 test gap).
- ✅ Closes both Partial items with ~2 files + 1 test; no wire change (struct ordinals of NameValue untouched)
- ❌ Touches a shared header (`types.h`) — full rebuild needed to confirm nothing else references the type

### Option B — Fold residuals into `scheduler-controller-msg-gap`
Fix the `zarray_put(..., 0, ...)` slot bug together with the R5-2/R6-1 residuals in the adjacent spec, since it already owns the per-host update handlers.
- ✅ Single coherent change set for all per-host resval handling; this spec can be closed as-is
- ❌ R5-2/R6-1 residue stays attached to the wrong spec's history; risk of being dropped if that spec stalls

### Option C — Accept and close
Mark tasks approved, record R5-2 (SHOULD) and the missing R6-1 test as known accepted debt.
- ✅ Zero effort; all MUST-level acceptance criteria are demonstrably met
- ❌ Dead type invites accidental reuse; the divergent-order scenario — the entire point of the feature — remains untested by automation

## Effort & Risk

- **Effort: S** — residual work is a dead-type deletion plus one table-driven C test; the feature itself requires no further code.
- **Risk: Low** — all changes verified in place and grep-confirmed; the only Medium-risk item is the adjacent slot-0 bug, which is pre-existing and outside this spec.

## Recommendations

1. Approve tasks in `spec.json` (`tasks.approved: true`) to close the process gap — implementation matches the approved design.
2. Take Option A: remove (or comment-scope) `xce_val_with_idx_t` / `ValueWithIdx`, and add the divergent-registration-order test for `sched_init_sharedInst` / `make_sched_worker` (TDD: the test also documents EARS-R6-1 permanently).
3. File the `XCE_SCHED_OP_UPDATE_RESVALS` slot-0 defect against `scheduler-controller-msg-gap` (or a standalone fix) — it is the last index-style hazard in the resval surface.

## Research Needed

- **RN-1**: Confirm `XCE_SCHED_OP_UPDATE_RESVALS` (`scheduler.c:814`) is a live path (who sends `schedUpdateResvals` capnp field @8, and with which resources) before fixing the hardcoded slot 0 — it may be stub/dead code.
- **RN-2**: Verify no out-of-tree or generated consumer (e.g., `src/build/lib/controller.capnp.c`) still references `ValueWithIdx` codecs before deleting the schema struct; a clean `ninja -C src/build` after removal settles this.
- **RN-3**: EARS-R2-4 (resource add/remove mid-run) has no test harness today; determine whether the scheduler testsuite can drive `sched_handle_resources` followed by a worker push without a live controller.