myxce/.kiro/specs/resval-id-decoupling/research.md
Rongsong Shen 20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00

303 lines
29 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Gap Analysis: resval-id-decoupling
> Generated: 2026-05-12T07:25:00-07:00
> Feature: resval-id-decoupling
## Executive Summary
- **Scope**: Three call sites in the controller use `collect_resval_with_idx()` to produce index-keyed resource values sent to two consumers (scheduler and API server). Fixing all three brings every cross-subsystem resval path into alignment with the already-correct `Data_data_resvals` dataset path.
- **Existing Infrastructure**: `xce_name_val_t` (`NameValue` in capnp) already exists in `types.h` and has a **complete capnp codec** (`encode_NameValue_list` / `decode_NameValue_list`) in the generated code — it was defined for this purpose but never wired in at the resval-transfer boundary.
- **Wire format impact**: `Conf4Sched.resvals` (field @6) and `Worker.resvals` (field @5) in `controller.capnp` currently declare `List(ValueWithIdx)`. Changing these to `List(NameValue)` is a **breaking wire change** — no intermediate backward-compatibility window unless two field slots are used.
- **Build chain**: The capnp schema in `src/schema/controller.capnp` drives generation of `src/build/lib/controller.capnp.[ch]` via `meson` + `capnpc-c`. A schema edit automatically regenerates the C codecs on the next build.
- **Key insight**: `makeResource_()` in `resobj.c:137` always overwrites `resobj->idx = rc` (from `key_add`) regardless of the `idx` transmitted in the `Resource` wire struct — so the controller's idx is lost at the scheduler's resmgr and the assumption that they match is never enforced.
---
## Existing Components Inventory
### Types and Structs
| Symbol | Location | Role | Status |
|--------|----------|------|--------|
| `xce_val_with_idx_t` | `src/include/xce/types.h:38` | `{int idx, xce_val_t* val}` — current cross-subsystem carrier | **Replace at boundaries** |
| `xce_name_val_t` | `src/include/xce/types.h:43` | `{char* name, xce_val_t* val}` — name-keyed carrier | **Ready to use** |
| `xce_resval_t` | `src/include/xce/types.h:48` | `{char* name, char* host, xce_val_t* val}` — used by Path C | Already correct |
| `xce_conf4sched_t.resvals` | `src/include/typemap.h:50` | `xce_val_with_idx_t**` — shared resvals in initial config | **Change to `xce_name_val_t**`** |
| `xce_worker_t.resvals_to_sched` | `src/include/xce/types.h:217` | `xce_val_with_idx_t**` — per-worker resvals | **Change to `xce_name_val_t**`** |
### Controller — Call Sites (Sender Side)
| File | Line | Function | Purpose | Change needed |
|------|------|----------|---------|---------------|
| `src/controller/dataex.c` | 62 | `collect_resval_with_idx()` | Iterates `zarray_t`, emits `{idx, val}` | Replace/augment with name-keyed variant |
| `src/controller/dataex.c` | 363 | `collect_shared_resvals()` | Calls `collect_resval_with_idx(&c->sharedinst)` | Change to name-keyed collection |
| `src/controller/dataex.c` | 141–163 | `collect_resvals_workers()` | Calls `collect_resval_with_idx(&w->resvals)` per worker | Change to name-keyed collection |
| `src/controller/sched.c` | 118–120 | `xcesched_push_configs()` | Puts shared resvals into `conf4sched->resvals` | Change field type; update free |
| `src/controller/sched.c` | 442–444 | `xcesched_push_workerlist()` | Puts worker resvals into `w->host.resvals_to_sched` | Change field type; update free |
| `src/controller/apiserver.c` | 367–369 | `apiserver_worker_change()` | Same pattern as sched for API server path | Change field type; update free |
### Free/cleanup Sites
| File | Line | Call | Must be updated |
|------|------|------|-----------------|
| `src/controller/sched.c` | 145 | `free_resval_with_idx_list(&conf4sched->resvals, ...)` | → `free_NameValue_list` or equivalent |
| `src/controller/sched.c` | 226, 463 | `free_resval_with_idx_list(&w->resvals_to_sched, ...)` | Same |
| `src/controller/apiserver.c` | 51, 397–399 | `free_resval_with_idx_list(...)` | Same |
| `src/controller/dataex.c` | 404 | `free_resval_with_idx_list(...)` | Same |
| `src/controller/reply.c` | 66 | `free_resval_with_idx_list(...)` | Same |
| `src/controller/opget.c` | 128 | `free_resval_with_idx_list(...)` | Same |
| `src/include/utils.h` | 130 | `free_resval_with_idx_list` declaration | Update or keep alongside new |
### Scheduler — Consumer Side
| File | Line | Code | Change needed |
|------|------|------|---------------|
| `src/scheduler/src/scheduler.c` | 574–586 | `sched_init_sharedInst()` — uses `idx` directly from `config->resvals[i]->idx` | Change to name-lookup in `sched->resmgr` |
| `src/scheduler/src/schedobj.c` | 274–280 | `make_sched_worker()` — uses `idx` directly from `w->resvals_to_sched[i]->idx` | Change to name-lookup |
### Capnp Schema and Generated Code
| Asset | Location | Current | Change |
|-------|----------|---------|--------|
| `Conf4Sched.resvals` | `src/schema/controller.capnp:230` | `List(ValueWithIdx)` | → `List(NameValue)` |
| `Worker.resvals` | `src/schema/controller.capnp:129` | `List(ValueWithIdx)` | → `List(NameValue)` |
| `controller.capnp.c/h` | `src/build/lib/` | Auto-generated — regenerated by `ninja -C build` | Automatically updated after schema edit |
| `encode_NameValue_list` | `src/build/lib/controller.capnp.c:3402` | Already exists, unused at these boundaries | Wire in to replace `encode_ValueWithIdx_list` |
| `decode_NameValue_list` | `src/build/lib/controller.capnp.c:3426` | Already exists, unused at these boundaries | Wire in to replace `decode_ValueWithIdx_list` |
| `free_NameValue_list` | `src/build/lib/controller.capnp.c:3458` | Already exists | Replace `free_ValueWithIdx_list` at these call sites |
### API Server (Go frontend) — Impact
The Go API server (`src/frontend/apiserver/backend/clib.go`) includes `controller.capnp.h` and works with `xce_worker_t*` via CGo. It accesses `resvals_to_sched` in `apiserver_worker_change()` (controller side) which populates `xce_worker_t.resvals_to_sched`. The Go frontend's `cResvalToDistribution` in `convert.go:273` already processes `xce_resval_t` (name-keyed), not `xce_val_with_idx_t`. The API server impact is limited to updating any code that reads `resvals_to_sched` field — if it accesses it at all on the Go side (currently no direct access found).
---
## Root Cause Deep Dive
The zarray-based stores (`c->sharedinst`, `w->host.resvals`) use `r->idx` as the slot key. `r->idx` is assigned by `key_add()` in `makeResource_` (`resobj.c:137`) — it is the slot returned by the internal container, which depends on insertion order. The controller and scheduler each call `makeResource()` independently (controller at cluster init, scheduler at `sched_init_resources()`), with no guarantee of identical ordering.
Crucially, the `Resource` capnp struct **does transmit `r->idx` on the wire** (`encode_Resource` at `build/lib/controller.capnp.c:3889`), but `decode_Resource` writes it into a temporary `xce_resource_t`, which is then passed to `makeResource()`, which **immediately overwrites** the decoded `idx` with the local `key_add` return value (`resobj->idx = rc`). The transmitted idx is therefore discarded at the scheduler.
---
## Implementation Options
### Option A — Replace `ValueWithIdx` with `NameValue` at both capnp fields (Recommended)
**What changes:**
1. `controller.capnp`: change `Conf4Sched.resvals @6` and `Worker.resvals @5` from `List(ValueWithIdx)` to `List(NameValue)`
2. Rebuild capnp — generated `encode_Conf4Sched`/`decode_Conf4Sched` and worker codec automatically use `NameValue` functions
3. `typemap.h`: `xce_conf4sched_t.resvals` → `xce_name_val_t**`
4. `types.h`: `xce_worker_t.resvals_to_sched` → `xce_name_val_t**`
5. Controller: new `collect_resval_by_name(zarray_t* vals, var resmgr, xlist_t* l)` that walks `resmgr` entries, reads the value by `r->idx` from the array, and emits `{name, val}` records
6. Scheduler `sched_init_sharedInst()`: iterate name-val list, `key_find(sched->resmgr, name)` → `r->idx` → `zarray_put`
7. Scheduler `make_sched_worker()`: same pattern
8. Update all free/cleanup call sites to use `free_NameValue_list`
**Pros:** Cleanest — removes the misleading `idx` field entirely; `NameValue` codec already exists; aligns with Path C's design; no backward compatibility concerns given no public release
**Cons:** None significant — wire-format break is a non-issue as there is no released version requiring compatibility
**Affected files (estimate):** 8 C files, 1 capnp schema, 0 Go files
---
### Option B — Add `name` field to `xce_val_with_idx_t` / `ValueWithIdx` ~~(Additive wire change)~~ [REJECTED]
Ruled out: backward compatibility is not a requirement for this project at its current stage. Option B's only advantage over Option A is a softer wire transition — without that constraint, it is strictly worse: it keeps a semantically dead `idx` field, requires more code changes, and leaves misleading structure in the API.
---
### Option C — Use Controller's idx as canonical; stop overwriting in `makeResource_` [REJECTED]
Ruled out: does not fix the design — just moves the source of truth without eliminating the hidden identity assumption. A resource deleted and re-added in the controller gets a new idx, and the scheduler has no way to know its cached idx is stale.
---
## Recommendation
**Option A** is the selected approach. Backward compatibility is not a constraint (no public release), so the wire-format change to `List(NameValue)` is straightforward. The `NameValue` codec already exists in the generated capnp code and is fully functional — this is primarily a wiring exercise. The semantic change aligns the initial-config and worker-data paths with the already-correct `Data_data_resvals` update path (Path C).
---
## Files to Change (Option A)
| File | Change Type |
|------|-------------|
| `src/schema/controller.capnp` | Schema edit: 2 field type changes |
| `src/include/xce/types.h` | Struct field type change: `resvals_to_sched` |
| `src/include/typemap.h` | Struct field type change: `resvals` |
| `src/controller/dataex.c` | New `collect_resval_by_name()` function; update `collect_shared_resvals()` and `collect_resvals_workers()`; **remove** `collect_resval_with_idx` and `collect_resval_with_idx_` |
| `src/controller/sched.c` | Update 3 call sites (populate and free) |
| `src/controller/apiserver.c` | Update 2 call sites (populate and free) |
| `src/controller/reply.c` | Update free call |
| `src/controller/opget.c` | Update free call |
| `src/scheduler/src/scheduler.c` | Update `sched_init_sharedInst()` |
| `src/scheduler/src/schedobj.c` | Update `make_sched_worker()` |
| `src/include/utils.h` | Remove `free_resval_with_idx_list` declaration; add `free_name_val_list` declaration |
| `src/lib/utils.c` | Remove `free_resval_with_idx_list`; add `free_name_val_list(xce_name_val_t***, int)` |
| `src/controller/include/cluster.h` | Remove `collect_resval_with_idx` declaration |
**Not changed:** `src/scheduler/src/handler.c` (Path C untouched), `src/frontend/apiserver/` (Go API server does not directly access `resvals_to_sched` in Go code), capnp generated files (auto-regenerated by `ninja -C build`).
---
## Decisions (Closed Questions)
1. **New collector function signature**: `collect_resval_by_name` is controller-internal logic — add it to `dataex.c` alongside `collect_resval_with_idx`. It needs both the `zarray_t` (values indexed by `r->idx`) and the `resmgr` (name→idx mapping) as parameters.
2. **Handling unknown resources in scheduler**: **Warn and skip.** When `sched_init_sharedInst` or `make_sched_worker` encounters a name not in `sched->resmgr`, emit a `WARN` log and continue. This is safe because the resource list always arrives in the same config push before resvals are applied (see decision 3).
3. **Ordering guarantee**: **Confirmed — current behavior, must be preserved.** `sched_init_resources()` is called before `sched_init_sharedInst()` in `sched_init_config()` (`scheduler.c:603-609`). The design must not reorder these calls.
4. **Removal of `collect_resval_with_idx`**: **Remove entirely.** No cross-subsystem callers will remain after this change, and intra-subsystem callers (e.g., `worker.c` reading from `sharedinst` for API get-resval) access the zarray directly by `r->idx` without going through this function. Delete `collect_resval_with_idx`, `collect_resval_with_idx_`, and remove the declaration from `src/controller/include/cluster.h`.
5. **`free_name_val_list` utility**: **Add to `utils.c`/`utils.h`.** A new `free_name_val_list(xce_name_val_t***, int)` function mirroring `free_resval_with_idx_list` is needed. The capnp-generated `free_NameValue_list` is an option but couples cleanup to the serialization layer — a standalone utility in `utils.c` is cleaner and consistent with the existing pattern.
---
# Cross-Spec Gap Audit (2026-06-08)
> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch.
## Analysis Summary
- **resval-id-decoupling** is **~100% complete**. All 8 task groups, 22 subtasks (1.1–8.3) are `[x]` in tasks.md.
- **No remaining gaps**: `collect_resval_with_idx` and `free_resval_with_idx_list` are removed (verified by grep: zero matches). `collect_resval` (name-keyed, iterates resmgr not zarray) replaces it. The capnp schema (`Conf4Sched.resvals @6`, `Worker.resvals @5`) now use `List(NameValue)`. The scheduler resolves by name via `key_find(sched->resmgr, name)` → `r->idx`. The Go API server has `cNameValToWorkerResval` in `convert.go` for the new `xce_name_val_t**` type.
- **Task approval gap**: spec.json shows `tasks.approved: false`. All tasks are implemented but the tasks document was never formally approved. This is a process artifact — the implementation matches the approved design.
- **Cross-spec impact**: `scheduler-controller-msg-gap` extends the same scheduler handlers (`sched_init_sharedInst`, `make_sched_worker`) that were modified by this spec. The two specs are consistent — both use name-based resource resolution.
- **Recommendation**: Mark tasks as approved in spec.json, or run `/kiro-spec-status resval-id-decoupling` to sync.
---
# Gap Analysis — Validation Pass (2026-06-09)
## Method
Independent re-verification of every requirement (requirements.md R1–R6) against the current code on branch `persistent_ha`, via grep + direct read of the controller, scheduler, libxce, capnp schema, and Go API-server sources. Prior audit (2026-06-08) claimed ~100% completion; this pass confirms that claim and adds two residual findings it missed.
## Current State Summary
- The feature is **implemented end-to-end**. `collect_resval_with_idx` and `free_resval_with_idx_list` no longer exist anywhere in `src/` (zero grep matches). The replacement `collect_resval(var resmgr, zarray_t*, xlist_t*)` lives at `src/controller/dataex.c:23` and iterates the resmgr (names) rather than the zarray (slots), exactly as designed.
- Wire format: `src/schema/controller.capnp` — `Worker.resvals @5` (line 138) and `Conf4Sched.resvals @6` (line 239) are both `List(NameValue)`. Field ordinals preserved.
- All tasks in tasks.md are `[x]` (1.1–8.3), but `spec.json` still shows `tasks.approved: false` (phase: `tasks-generated`). Implementation landed despite the unapproved tasks document — a process artifact, not a code gap.
## Requirement-to-Asset Map
| Req | EARS | Asset(s) | Status |
|-----|------|----------|--------|
| R1 — name-keyed shared resvals in initial config | R1-1 | `controller.capnp:239` `List(NameValue)`; `src/controller/dataex.c:381` `collect_resval(c->resmgr, &c->sharedinst, l)`; `src/controller/sched.c:120` casts to `xce_name_val_t**` | **Implemented** |
| | R1-2 | `src/scheduler/src/scheduler.c:574-599` `sched_init_sharedInst`: `key_find(sched->resmgr, nv->name)` → `zarray_put(&sched->sharedInst, r->idx, val)` | **Implemented** |
| | R1-3 | `scheduler.c:587-592`: WARN + `continue` on unknown name; NULL name/entry also skipped | **Implemented** |
| | R1-4 | Same name→`r->idx` resolution as Path C handler (`scheduler.c:782-797`) | **Implemented** (by inspection; no equivalence test exists — see R6) |
| R2 — name-keyed per-worker resvals to scheduler | R2-1 | `controller.capnp:138` `List(NameValue)`; `dataex.c:151-166` `collect_resvals_workers(resmgr,...)`; `sched.c:442` | **Implemented** |
| | R2-2 | `src/scheduler/src/schedobj.c:274-292` `make_sched_worker`: name lookup → `zarray_put(&worker->resvals, r->idx, val)` | **Implemented** |
| | R2-3 | `schedobj.c:283-288`: WARN + skip, worker construction continues | **Implemented** |
| | R2-4 | Resolution happens at each receipt against the live `sched->resmgr`; no index is cached across messages | **Implemented** (by design; no automated test for resource add/remove mid-run) |
| R3 — name-keyed worker resvals to API server | R3-1 | `src/controller/apiserver.c:367-369`: same `collect_resval` + `xce_name_val_t**` path as the scheduler push | **Implemented** |
| | R3-2 | `src/frontend/apiserver/backend/convert.go:100-105` iterates `resvals_to_sched`, `convert.go:283` `cNameValToWorkerResval` (name-keyed, nil-checked); `convert_pure.go:210`; `models/hosts.go:94` `ResourceValues` | **Implemented** |
| R4 — Path C unchanged | R4-1 | `src/scheduler/src/handler.c:82-103` (`Data_data_resvals` → `XCE_SCHED_OP_UPDATE_RESVALS_FULL`) and `scheduler.c:782-800` untouched, still `xce_resval_t` name-keyed | **Implemented** |
| | R4-2 | `xce_conf4sched_t.resvals` (`typemap.h:62`) and `xce_worker_t.resvals_to_sched` (`types.h:219`) are `xce_name_val_t**`; codec auto-generated; `free_name_val_list` in `src/lib/utils.c:388` / `utils.h:130`; all controller free sites converted | **Implemented** |
| R5 — remove index-keyed call sites | R5-1 | Zero cross-subsystem index-keyed transfers remain (grep-verified); both receivers resolve by name | **Implemented** |
| | R5-2 | `xce_val_with_idx_t` typedef (`src/include/xce/types.h:38-41`) and capnp `struct ValueWithIdx` (`controller.capnp:176`) still exist with **zero users and no scoping comment** | **Partial** (SHOULD-level: dead type lingers, contrary to "removed or restricted ... with a comment") |
| R6 — no regression / correctness under divergent indices | R6-1 | Correct by code inspection (name resolution is order-independent); **no automated test** registers resources in different orders and asserts `sharedInst` contents | **Partial** (code Implemented, test Missing) |
| | R6-2 | Existing controller/scheduler suites passed at task 8.2; same-order behavior preserved | **Implemented** |
| | R6-3 | `calc_wanted_*` paths read `sharedInst`/`worker->resvals` via scheduler-side `r->idx` (scheduler testsuite exercises these at zarray level) | **Implemented** |
### Adjacent finding (Constraint — out of spec scope, same defect class)
`XCE_SCHED_OP_UPDATE_RESVALS` (per-host running update, `handler.c:231-251` → `scheduler.c:801-816`) receives name-keyed `xce_resval_t` records but stores them with a **hardcoded slot**: `zarray_put(&w->resvals, 0, rv->val)` — it finds the worker by `rv->host` but never resolves `rv->name` to `r->idx`. Any per-host update for a resource whose scheduler idx ≠ 0 lands in the wrong slot (or clobbers slot 0). This path was not enumerated in this spec (Paths A/B/C only) and belongs to the `scheduler-controller-msg-gap` surface, but it violates the spirit of EARS-R5-1. Flagged for follow-up, not a regression of this feature.
## Implementation Approach Options (for residual gaps)
### Option A — Minimal cleanup within this spec (Recommended)
Delete the dead `xce_val_with_idx_t` typedef and capnp `ValueWithIdx` struct (or add the scoping comment EARS-R5-2 asks for); add one scheduler testsuite case that feeds `sched_init_sharedInst`/`make_sched_worker` a name-val list against a resmgr populated in a different order and asserts slot contents (closes R6-1 test gap).
- ✅ Closes both Partial items with ~2 files + 1 test; no wire change (struct ordinals of NameValue untouched)
- ❌ Touches a shared header (`types.h`) — full rebuild needed to confirm nothing else references the type
### Option B — Fold residuals into `scheduler-controller-msg-gap`
Fix the `zarray_put(..., 0, ...)` slot bug together with the R5-2/R6-1 residuals in the adjacent spec, since it already owns the per-host update handlers.
- ✅ Single coherent change set for all per-host resval handling; this spec can be closed as-is
- ❌ R5-2/R6-1 residue stays attached to the wrong spec's history; risk of being dropped if that spec stalls
### Option C — Accept and close
Mark tasks approved, record R5-2 (SHOULD) and the missing R6-1 test as known accepted debt.
- ✅ Zero effort; all MUST-level acceptance criteria are demonstrably met
- ❌ Dead type invites accidental reuse; the divergent-order scenario — the entire point of the feature — remains untested by automation
## Effort & Risk
- **Effort: S** — residual work is a dead-type deletion plus one table-driven C test; the feature itself requires no further code.
- **Risk: Low** — all changes verified in place and grep-confirmed; the only Medium-risk item is the adjacent slot-0 bug, which is pre-existing and outside this spec.
## Recommendations
1. Approve tasks in `spec.json` (`tasks.approved: true`) to close the process gap — implementation matches the approved design.
2. Take Option A: remove (or comment-scope) `xce_val_with_idx_t` / `ValueWithIdx`, and add the divergent-registration-order test for `sched_init_sharedInst` / `make_sched_worker` (TDD: the test also documents EARS-R6-1 permanently).
3. File the `XCE_SCHED_OP_UPDATE_RESVALS` slot-0 defect against `scheduler-controller-msg-gap` (or a standalone fix) — it is the last index-style hazard in the resval surface.
## Research Needed
- **RN-1**: Confirm `XCE_SCHED_OP_UPDATE_RESVALS` (`scheduler.c:814`) is a live path (who sends `schedUpdateResvals` capnp field @8, and with which resources) before fixing the hardcoded slot 0 — it may be stub/dead code.
- **RN-2**: Verify no out-of-tree or generated consumer (e.g., `src/build/lib/controller.capnp.c`) still references `ValueWithIdx` codecs before deleting the schema struct; a clean `ninja -C src/build` after removal settles this.
- **RN-3**: EARS-R2-4 (resource add/remove mid-run) has no test harness today; determine whether the scheduler testsuite can drive `sched_handle_resources` followed by a worker push without a live controller.
---
## Gap Re-Audit (2026-06-18)
> POST-IMPLEMENTATION GAP AUDIT. Tasks 24/24 `[x]`, `spec.json.phase: implemented`.
> Independent re-verification of R1–R6 against current `persistent_ha` code via codegraph + grep + verbatim reads. Supersedes the 2026-06-09 validation pass; records two status changes since then.
### Changes since the 2026-06-09 pass
1. **Process gap CLOSED.** `spec.json` now shows `phase: implemented` and `approvals.tasks.approved: true` (was `false`/`tasks-generated`). The prior "tasks unapproved" finding no longer applies.
2. **Adjacent slot-0 defect FIXED.** `XCE_SCHED_OP_UPDATE_RESVALS` (per-host running update) at `src/scheduler/src/scheduler.c:816-834` now resolves `rv->name` via `key_find(sched->resmgr, ...)` and stores with `zarray_put(&w->resvals, r->idx, rv->val)` (line 831) — the hardcoded `zarray_put(..., 0, ...)` flagged on 2026-06-09 is gone. The last index-style hazard in the resval surface is closed. (Belongs to `scheduler-controller-msg-gap` scope, but worth recording: the defect class this spec targets is now eliminated across ALL resval paths A/B/C and both running-update handlers.)
### Requirement-to-Asset Map
| Req | EARS | Asset(s) — file:line | Tag |
|-----|------|----------------------|-----|
| R1 — name-keyed shared resvals in initial config | R1-1 | `src/schema/controller.capnp:241` `resvals @6: List(NameValue)`; `src/controller/dataex.c:381` `collect_resval(c->resmgr, &c->sharedinst, l)`; `src/controller/sched.c` cast to `xce_name_val_t**` | MET |
| | R1-2 | `src/scheduler/src/scheduler.c:587-595` `sched_init_sharedInst`: `key_find(sched->resmgr, nv->name)` → `zarray_put(&sched->sharedInst, r->idx, val)` | MET |
| | R1-3 | `scheduler.c:583-592`: NULL nv/name skipped; unknown name → `WARN` + `continue`, `sharedInst` untouched | MET |
| | R1-4 | Same name→`r->idx` resolution as Path C (`scheduler.c:797-812`, `UPDATE_RESVALS_FULL`) | MET (by inspection; no equivalence test — see R6-1) |
| R2 — name-keyed per-worker resvals to scheduler | R2-1 | `controller.capnp:140` `resvals @5: List(NameValue)`; `dataex.c:374` `collect_resvals_workers(c->resmgr, l)`; `sched.c:477` `collect_resval(c->resmgr, &w->host.resvals, ...)` | MET |
| | R2-2 | `src/scheduler/src/schedobj.c:274-292` `make_sched_worker`: name lookup → `zarray_put(&worker->resvals, r->idx, val)` (line 291) | MET |
| | R2-3 | `schedobj.c:279-288`: NULL skip; unknown name → `WARN` + skip; worker construction continues | MET |
| | R2-4 | Resolution runs against live `sched->resmgr` on every receipt; no idx cached across messages (`schedobj.c:274-292`) | MET (by design; no mid-run add/remove test — RN-3) |
| R3 — name-keyed worker resvals to API server | R3-1 | `src/controller/apiserver.c` `apiserver_worker_change`: same `collect_resval` + `xce_name_val_t**` path as scheduler push | MET |
| | R3-2 | `src/frontend/apiserver/backend/convert.go:100-105` iterates `worker.resvals_to_sched` (`unsafe.Slice`), `convert.go:283` `cNameValToWorkerResval` (name-keyed, nil-checked); `convert_pure.go:210` `ResourceValues: r.resvals`; `models/hosts.go:94` `ResourceValues []ResourceDistributionInfo` | MET |
| R4 — Path C unchanged | R4-1 | `src/scheduler/src/handler.c` (`Data_data_resvals` → `XCE_SCHED_OP_UPDATE_RESVALS_FULL`) and `scheduler.c:797-814` untouched, still `xce_resval_t` name-keyed | MET |
| | R4-2 | `xce_conf4sched_t.resvals` (`typemap.h`) and `xce_worker_t.resvals_to_sched` (`types.h`) are `xce_name_val_t**`; codec auto-generated; `free_name_val_list` (`src/lib/utils.c:388`); all 7 controller free sites converted (codegraph: 7 callers across apiserver.c, dataex.c, opget.c, reply.c, sched.c) | MET |
| R5 — remove index-keyed call sites | R5-1 | Zero cross-subsystem index-keyed transfers remain (grep: `collect_resval_with_idx`/`free_resval_with_idx_list` = 0 matches in src/ excl. build/); both receivers resolve by name | MET |
| | R5-2 | `xce_val_with_idx_t` typedef (`src/include/xce/types.h:38-41`) and capnp `struct ValueWithIdx` (`controller.capnp:178`) still present — zero non-generated users, **no scoping comment** | Missing (SHOULD-level) |
| R6 — no regression / correctness under divergent indices | R6-1 | Correct by code inspection (name resolution is order-independent at `scheduler.c:587` and `schedobj.c:283`); scheduler testsuite (`src/scheduler/testsuite/test.c`) has **zero** references to `sched_init_sharedInst` / `make_sched_worker` / divergent resval order | Missing (test only; code MET) |
| | R6-2 | Same-order behavior preserved; controller/scheduler suites passed at task 8.2 | MET |
| | R6-3 | `calc_wanted_*` read `sharedInst` / `worker->resvals` via scheduler-side `r->idx`; all writers (`sched_init_sharedInst`, `make_sched_worker`, both UPDATE_RESVALS handlers) now key by `r->idx` | MET |
**Score: 21/23 EARS MET. 2 Missing — both non-MUST (R5-2 SHOULD; R6-1 test-only, code present).** All MUST-level acceptance criteria are demonstrably satisfied.
### Residual Gaps
- **R5-2 (SHOULD):** Dead `xce_val_with_idx_t` / `struct ValueWithIdx` linger without the "removed or restricted to intra-subsystem use, with a comment explaining its scope" that EARS-R5-2 requests. Invites accidental reuse of the very index-keyed shape this feature eliminated.
- **R6-1 (test gap):** The feature's entire raison d'être — correctness when controller and scheduler register resources in *different* orders — has no automated test. Code is order-independent by construction, but the invariant is unguarded against regression.
- **Carry-forward note:** RN-3 (EARS-R2-4 mid-run resource add/remove) still has no harness. Lower priority than R6-1.
### Options for Open Work
**Option A — Close residuals in-spec (Recommended).**
Delete `xce_val_with_idx_t` (types.h:38-41) + `struct ValueWithIdx` (controller.capnp:178) [or add the EARS-R5-2 scoping comment if retained for tooling], and add one table-driven scheduler testsuite case feeding `sched_init_sharedInst` / `make_sched_worker` a name-val list against a resmgr populated in reversed order, asserting `sharedInst` / `worker->resvals` slot contents.
- Effort: **S** (~2 files + 1 test). Risk: **Low**.
- Caveat: deleting from `types.h` is a shared-header edit — requires a clean `nix develop --command ninja -C src/build` to confirm no generated/out-of-tree consumer references `ValueWithIdx` (RN-2).
**Option B — Defer to a cleanup/test sweep across HA specs.**
Bundle R5-2 + R6-1 + RN-3 with similar dead-type/test debt in adjacent specs.
- Effort: **S-M**. Risk: **Low**. Cost: residue stays detached from this spec's history; may be dropped if the sweep stalls.
**Option C — Accept and close as-is.**
Record R5-2 and the missing R6-1 test as known accepted debt; all MUST criteria met, slot-0 hazard already fixed, spec.json fully approved.
- Effort: **none**. Risk: **Low-Medium** — dead type invites reuse; the feature's core scenario remains unverified by automation.
### Verdict
Feature is **functionally complete and CLOSED for all MUST requirements** (R1-1..R1-3, R2-1..R2-4, R3-1/R3-2, R4-1/R4-2, R5-1, R6-2/R6-3). Two non-blocking residuals remain (R5-2 dead-type cleanup; R6-1 divergent-order test). Recommend **Option A** to fully satisfy the SHOULD clause and lock the core invariant under test. Effort **S**, Risk **Low**.