myxce/.kiro/specs/resval-id-decoupling/research.md
Rongsong Shen 20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00

29 KiB
Raw Blame History

Gap Analysis: resval-id-decoupling

Generated: 2026-05-12T07:25:00-07:00 Feature: resval-id-decoupling

Executive Summary

  • Scope: Three call sites in the controller use collect_resval_with_idx() to produce index-keyed resource values sent to two consumers (scheduler and API server). Fixing all three brings every cross-subsystem resval path into alignment with the already-correct Data_data_resvals dataset path.
  • Existing Infrastructure: xce_name_val_t (NameValue in capnp) already exists in types.h and has a complete capnp codec (encode_NameValue_list / decode_NameValue_list) in the generated code — it was defined for this purpose but never wired in at the resval-transfer boundary.
  • Wire format impact: Conf4Sched.resvals (field @6) and Worker.resvals (field @5) in controller.capnp currently declare List(ValueWithIdx). Changing these to List(NameValue) is a breaking wire change — no intermediate backward-compatibility window unless two field slots are used.
  • Build chain: The capnp schema in src/schema/controller.capnp drives generation of src/build/lib/controller.capnp.[ch] via meson + capnpc-c. A schema edit automatically regenerates the C codecs on the next build.
  • Key insight: makeResource_() in resobj.c:137 always overwrites resobj->idx = rc (from key_add) regardless of the idx transmitted in the Resource wire struct — so the controller's idx is lost at the scheduler's resmgr and the assumption that they match is never enforced.

Existing Components Inventory

Types and Structs

Symbol Location Role Status
xce_val_with_idx_t src/include/xce/types.h:38 {int idx, xce_val_t* val} — current cross-subsystem carrier Replace at boundaries
xce_name_val_t src/include/xce/types.h:43 {char* name, xce_val_t* val} — name-keyed carrier Ready to use
xce_resval_t src/include/xce/types.h:48 {char* name, char* host, xce_val_t* val} — used by Path C Already correct
xce_conf4sched_t.resvals src/include/typemap.h:50 xce_val_with_idx_t** — shared resvals in initial config Change to xce_name_val_t**
xce_worker_t.resvals_to_sched src/include/xce/types.h:217 xce_val_with_idx_t** — per-worker resvals Change to xce_name_val_t**

Controller — Call Sites (Sender Side)

File Line Function Purpose Change needed
src/controller/dataex.c 62 collect_resval_with_idx() Iterates zarray_t, emits {idx, val} Replace/augment with name-keyed variant
src/controller/dataex.c 363 collect_shared_resvals() Calls collect_resval_with_idx(&c->sharedinst) Change to name-keyed collection
src/controller/dataex.c 141–163 collect_resvals_workers() Calls collect_resval_with_idx(&w->resvals) per worker Change to name-keyed collection
src/controller/sched.c 118–120 xcesched_push_configs() Puts shared resvals into conf4sched->resvals Change field type; update free
src/controller/sched.c 442–444 xcesched_push_workerlist() Puts worker resvals into w->host.resvals_to_sched Change field type; update free
src/controller/apiserver.c 367–369 apiserver_worker_change() Same pattern as sched for API server path Change field type; update free

Free/cleanup Sites

File Line Call Must be updated
src/controller/sched.c 145 free_resval_with_idx_list(&conf4sched->resvals, ...) → free_NameValue_list or equivalent
src/controller/sched.c 226, 463 free_resval_with_idx_list(&w->resvals_to_sched, ...) Same
src/controller/apiserver.c 51, 397–399 free_resval_with_idx_list(...) Same
src/controller/dataex.c 404 free_resval_with_idx_list(...) Same
src/controller/reply.c 66 free_resval_with_idx_list(...) Same
src/controller/opget.c 128 free_resval_with_idx_list(...) Same
src/include/utils.h 130 free_resval_with_idx_list declaration Update or keep alongside new

Scheduler — Consumer Side

File Line Code Change needed
src/scheduler/src/scheduler.c 574–586 sched_init_sharedInst() — uses idx directly from config->resvals[i]->idx Change to name-lookup in sched->resmgr
src/scheduler/src/schedobj.c 274–280 make_sched_worker() — uses idx directly from w->resvals_to_sched[i]->idx Change to name-lookup

Capnp Schema and Generated Code

Asset Location Current Change
Conf4Sched.resvals src/schema/controller.capnp:230 List(ValueWithIdx) → List(NameValue)
Worker.resvals src/schema/controller.capnp:129 List(ValueWithIdx) → List(NameValue)
controller.capnp.c/h src/build/lib/ Auto-generated — regenerated by ninja -C build Automatically updated after schema edit
encode_NameValue_list src/build/lib/controller.capnp.c:3402 Already exists, unused at these boundaries Wire in to replace encode_ValueWithIdx_list
decode_NameValue_list src/build/lib/controller.capnp.c:3426 Already exists, unused at these boundaries Wire in to replace decode_ValueWithIdx_list
free_NameValue_list src/build/lib/controller.capnp.c:3458 Already exists Replace free_ValueWithIdx_list at these call sites

API Server (Go frontend) — Impact

The Go API server (src/frontend/apiserver/backend/clib.go) includes controller.capnp.h and works with xce_worker_t* via CGo. It accesses resvals_to_sched in apiserver_worker_change() (controller side) which populates xce_worker_t.resvals_to_sched. The Go frontend's cResvalToDistribution in convert.go:273 already processes xce_resval_t (name-keyed), not xce_val_with_idx_t. The API server impact is limited to updating any code that reads resvals_to_sched field — if it accesses it at all on the Go side (currently no direct access found).


Root Cause Deep Dive

The zarray-based stores (c->sharedinst, w->host.resvals) use r->idx as the slot key. r->idx is assigned by key_add() in makeResource_ (resobj.c:137) — it is the slot returned by the internal container, which depends on insertion order. The controller and scheduler each call makeResource() independently (controller at cluster init, scheduler at sched_init_resources()), with no guarantee of identical ordering.

Crucially, the Resource capnp struct does transmit r->idx on the wire (encode_Resource at build/lib/controller.capnp.c:3889), but decode_Resource writes it into a temporary xce_resource_t, which is then passed to makeResource(), which immediately overwrites the decoded idx with the local key_add return value (resobj->idx = rc). The transmitted idx is therefore discarded at the scheduler.


Implementation Options

What changes:

  1. controller.capnp: change Conf4Sched.resvals @6 and Worker.resvals @5 from List(ValueWithIdx) to List(NameValue)
  2. Rebuild capnp — generated encode_Conf4Sched/decode_Conf4Sched and worker codec automatically use NameValue functions
  3. typemap.h: xce_conf4sched_t.resvals → xce_name_val_t**
  4. types.h: xce_worker_t.resvals_to_sched → xce_name_val_t**
  5. Controller: new collect_resval_by_name(zarray_t* vals, var resmgr, xlist_t* l) that walks resmgr entries, reads the value by r->idx from the array, and emits {name, val} records
  6. Scheduler sched_init_sharedInst(): iterate name-val list, key_find(sched->resmgr, name) → r->idx → zarray_put
  7. Scheduler make_sched_worker(): same pattern
  8. Update all free/cleanup call sites to use free_NameValue_list

Pros: Cleanest — removes the misleading idx field entirely; NameValue codec already exists; aligns with Path C's design; no backward compatibility concerns given no public release Cons: None significant — wire-format break is a non-issue as there is no released version requiring compatibility

Affected files (estimate): 8 C files, 1 capnp schema, 0 Go files


Option B — Add name field to xce_val_with_idx_t / ValueWithIdx (Additive wire change) [REJECTED]

Ruled out: backward compatibility is not a requirement for this project at its current stage. Option B's only advantage over Option A is a softer wire transition — without that constraint, it is strictly worse: it keeps a semantically dead idx field, requires more code changes, and leaves misleading structure in the API.


Option C — Use Controller's idx as canonical; stop overwriting in makeResource_ [REJECTED]

Ruled out: does not fix the design — just moves the source of truth without eliminating the hidden identity assumption. A resource deleted and re-added in the controller gets a new idx, and the scheduler has no way to know its cached idx is stale.


Recommendation

Option A is the selected approach. Backward compatibility is not a constraint (no public release), so the wire-format change to List(NameValue) is straightforward. The NameValue codec already exists in the generated capnp code and is fully functional — this is primarily a wiring exercise. The semantic change aligns the initial-config and worker-data paths with the already-correct Data_data_resvals update path (Path C).


Files to Change (Option A)

File Change Type
src/schema/controller.capnp Schema edit: 2 field type changes
src/include/xce/types.h Struct field type change: resvals_to_sched
src/include/typemap.h Struct field type change: resvals
src/controller/dataex.c New collect_resval_by_name() function; update collect_shared_resvals() and collect_resvals_workers(); remove collect_resval_with_idx and collect_resval_with_idx_
src/controller/sched.c Update 3 call sites (populate and free)
src/controller/apiserver.c Update 2 call sites (populate and free)
src/controller/reply.c Update free call
src/controller/opget.c Update free call
src/scheduler/src/scheduler.c Update sched_init_sharedInst()
src/scheduler/src/schedobj.c Update make_sched_worker()
src/include/utils.h Remove free_resval_with_idx_list declaration; add free_name_val_list declaration
src/lib/utils.c Remove free_resval_with_idx_list; add free_name_val_list(xce_name_val_t***, int)
src/controller/include/cluster.h Remove collect_resval_with_idx declaration

Not changed: src/scheduler/src/handler.c (Path C untouched), src/frontend/apiserver/ (Go API server does not directly access resvals_to_sched in Go code), capnp generated files (auto-regenerated by ninja -C build).


Decisions (Closed Questions)

  1. New collector function signature: collect_resval_by_name is controller-internal logic — add it to dataex.c alongside collect_resval_with_idx. It needs both the zarray_t (values indexed by r->idx) and the resmgr (name→idx mapping) as parameters.

  2. Handling unknown resources in scheduler: Warn and skip. When sched_init_sharedInst or make_sched_worker encounters a name not in sched->resmgr, emit a WARN log and continue. This is safe because the resource list always arrives in the same config push before resvals are applied (see decision 3).

  3. Ordering guarantee: Confirmed — current behavior, must be preserved. sched_init_resources() is called before sched_init_sharedInst() in sched_init_config() (scheduler.c:603-609). The design must not reorder these calls.

  4. Removal of collect_resval_with_idx: Remove entirely. No cross-subsystem callers will remain after this change, and intra-subsystem callers (e.g., worker.c reading from sharedinst for API get-resval) access the zarray directly by r->idx without going through this function. Delete collect_resval_with_idx, collect_resval_with_idx_, and remove the declaration from src/controller/include/cluster.h.

  5. free_name_val_list utility: Add to utils.c/utils.h. A new free_name_val_list(xce_name_val_t***, int) function mirroring free_resval_with_idx_list is needed. The capnp-generated free_NameValue_list is an option but couples cleanup to the serialization layer — a standalone utility in utils.c is cleaner and consistent with the existing pattern.


Cross-Spec Gap Audit (2026-06-08)

Batch /kiro-validate-gap across 9 specs on persistent_ha branch.

Analysis Summary

  • resval-id-decoupling is ~100% complete. All 8 task groups, 22 subtasks (1.1–8.3) are [x] in tasks.md.
  • No remaining gaps: collect_resval_with_idx and free_resval_with_idx_list are removed (verified by grep: zero matches). collect_resval (name-keyed, iterates resmgr not zarray) replaces it. The capnp schema (Conf4Sched.resvals @6, Worker.resvals @5) now use List(NameValue). The scheduler resolves by name via key_find(sched->resmgr, name) → r->idx. The Go API server has cNameValToWorkerResval in convert.go for the new xce_name_val_t** type.
  • Task approval gap: spec.json shows tasks.approved: false. All tasks are implemented but the tasks document was never formally approved. This is a process artifact — the implementation matches the approved design.
  • Cross-spec impact: scheduler-controller-msg-gap extends the same scheduler handlers (sched_init_sharedInst, make_sched_worker) that were modified by this spec. The two specs are consistent — both use name-based resource resolution.
  • Recommendation: Mark tasks as approved in spec.json, or run /kiro-spec-status resval-id-decoupling to sync.

Gap Analysis — Validation Pass (2026-06-09)

Method

Independent re-verification of every requirement (requirements.md R1–R6) against the current code on branch persistent_ha, via grep + direct read of the controller, scheduler, libxce, capnp schema, and Go API-server sources. Prior audit (2026-06-08) claimed ~100% completion; this pass confirms that claim and adds two residual findings it missed.

Current State Summary

  • The feature is implemented end-to-end. collect_resval_with_idx and free_resval_with_idx_list no longer exist anywhere in src/ (zero grep matches). The replacement collect_resval(var resmgr, zarray_t*, xlist_t*) lives at src/controller/dataex.c:23 and iterates the resmgr (names) rather than the zarray (slots), exactly as designed.
  • Wire format: src/schema/controller.capnp — Worker.resvals @5 (line 138) and Conf4Sched.resvals @6 (line 239) are both List(NameValue). Field ordinals preserved.
  • All tasks in tasks.md are [x] (1.1–8.3), but spec.json still shows tasks.approved: false (phase: tasks-generated). Implementation landed despite the unapproved tasks document — a process artifact, not a code gap.

Requirement-to-Asset Map

Req EARS Asset(s) Status
R1 — name-keyed shared resvals in initial config R1-1 controller.capnp:239 List(NameValue); src/controller/dataex.c:381 collect_resval(c->resmgr, &c->sharedinst, l); src/controller/sched.c:120 casts to xce_name_val_t** Implemented
R1-2 src/scheduler/src/scheduler.c:574-599 sched_init_sharedInst: key_find(sched->resmgr, nv->name) → zarray_put(&sched->sharedInst, r->idx, val) Implemented
R1-3 scheduler.c:587-592: WARN + continue on unknown name; NULL name/entry also skipped Implemented
R1-4 Same name→r->idx resolution as Path C handler (scheduler.c:782-797) Implemented (by inspection; no equivalence test exists — see R6)
R2 — name-keyed per-worker resvals to scheduler R2-1 controller.capnp:138 List(NameValue); dataex.c:151-166 collect_resvals_workers(resmgr,...); sched.c:442 Implemented
R2-2 src/scheduler/src/schedobj.c:274-292 make_sched_worker: name lookup → zarray_put(&worker->resvals, r->idx, val) Implemented
R2-3 schedobj.c:283-288: WARN + skip, worker construction continues Implemented
R2-4 Resolution happens at each receipt against the live sched->resmgr; no index is cached across messages Implemented (by design; no automated test for resource add/remove mid-run)
R3 — name-keyed worker resvals to API server R3-1 src/controller/apiserver.c:367-369: same collect_resval + xce_name_val_t** path as the scheduler push Implemented
R3-2 src/frontend/apiserver/backend/convert.go:100-105 iterates resvals_to_sched, convert.go:283 cNameValToWorkerResval (name-keyed, nil-checked); convert_pure.go:210; models/hosts.go:94 ResourceValues Implemented
R4 — Path C unchanged R4-1 src/scheduler/src/handler.c:82-103 (Data_data_resvals → XCE_SCHED_OP_UPDATE_RESVALS_FULL) and scheduler.c:782-800 untouched, still xce_resval_t name-keyed Implemented
R4-2 xce_conf4sched_t.resvals (typemap.h:62) and xce_worker_t.resvals_to_sched (types.h:219) are xce_name_val_t**; codec auto-generated; free_name_val_list in src/lib/utils.c:388 / utils.h:130; all controller free sites converted Implemented
R5 — remove index-keyed call sites R5-1 Zero cross-subsystem index-keyed transfers remain (grep-verified); both receivers resolve by name Implemented
R5-2 xce_val_with_idx_t typedef (src/include/xce/types.h:38-41) and capnp struct ValueWithIdx (controller.capnp:176) still exist with zero users and no scoping comment Partial (SHOULD-level: dead type lingers, contrary to "removed or restricted ... with a comment")
R6 — no regression / correctness under divergent indices R6-1 Correct by code inspection (name resolution is order-independent); no automated test registers resources in different orders and asserts sharedInst contents Partial (code Implemented, test Missing)
R6-2 Existing controller/scheduler suites passed at task 8.2; same-order behavior preserved Implemented
R6-3 calc_wanted_* paths read sharedInst/worker->resvals via scheduler-side r->idx (scheduler testsuite exercises these at zarray level) Implemented

Adjacent finding (Constraint — out of spec scope, same defect class)

XCE_SCHED_OP_UPDATE_RESVALS (per-host running update, handler.c:231-251 → scheduler.c:801-816) receives name-keyed xce_resval_t records but stores them with a hardcoded slot: zarray_put(&w->resvals, 0, rv->val) — it finds the worker by rv->host but never resolves rv->name to r->idx. Any per-host update for a resource whose scheduler idx ≠ 0 lands in the wrong slot (or clobbers slot 0). This path was not enumerated in this spec (Paths A/B/C only) and belongs to the scheduler-controller-msg-gap surface, but it violates the spirit of EARS-R5-1. Flagged for follow-up, not a regression of this feature.

Implementation Approach Options (for residual gaps)

Delete the dead xce_val_with_idx_t typedef and capnp ValueWithIdx struct (or add the scoping comment EARS-R5-2 asks for); add one scheduler testsuite case that feeds sched_init_sharedInst/make_sched_worker a name-val list against a resmgr populated in a different order and asserts slot contents (closes R6-1 test gap).

  • ✅ Closes both Partial items with ~2 files + 1 test; no wire change (struct ordinals of NameValue untouched)
  • ❌ Touches a shared header (types.h) — full rebuild needed to confirm nothing else references the type

Option B — Fold residuals into scheduler-controller-msg-gap

Fix the zarray_put(..., 0, ...) slot bug together with the R5-2/R6-1 residuals in the adjacent spec, since it already owns the per-host update handlers.

  • ✅ Single coherent change set for all per-host resval handling; this spec can be closed as-is
  • ❌ R5-2/R6-1 residue stays attached to the wrong spec's history; risk of being dropped if that spec stalls

Option C — Accept and close

Mark tasks approved, record R5-2 (SHOULD) and the missing R6-1 test as known accepted debt.

  • ✅ Zero effort; all MUST-level acceptance criteria are demonstrably met
  • ❌ Dead type invites accidental reuse; the divergent-order scenario — the entire point of the feature — remains untested by automation

Effort & Risk

  • Effort: S — residual work is a dead-type deletion plus one table-driven C test; the feature itself requires no further code.
  • Risk: Low — all changes verified in place and grep-confirmed; the only Medium-risk item is the adjacent slot-0 bug, which is pre-existing and outside this spec.

Recommendations

  1. Approve tasks in spec.json (tasks.approved: true) to close the process gap — implementation matches the approved design.
  2. Take Option A: remove (or comment-scope) xce_val_with_idx_t / ValueWithIdx, and add the divergent-registration-order test for sched_init_sharedInst / make_sched_worker (TDD: the test also documents EARS-R6-1 permanently).
  3. File the XCE_SCHED_OP_UPDATE_RESVALS slot-0 defect against scheduler-controller-msg-gap (or a standalone fix) — it is the last index-style hazard in the resval surface.

Research Needed

  • RN-1: Confirm XCE_SCHED_OP_UPDATE_RESVALS (scheduler.c:814) is a live path (who sends schedUpdateResvals capnp field @8, and with which resources) before fixing the hardcoded slot 0 — it may be stub/dead code.
  • RN-2: Verify no out-of-tree or generated consumer (e.g., src/build/lib/controller.capnp.c) still references ValueWithIdx codecs before deleting the schema struct; a clean ninja -C src/build after removal settles this.
  • RN-3: EARS-R2-4 (resource add/remove mid-run) has no test harness today; determine whether the scheduler testsuite can drive sched_handle_resources followed by a worker push without a live controller.

Gap Re-Audit (2026-06-18)

POST-IMPLEMENTATION GAP AUDIT. Tasks 24/24 [x], spec.json.phase: implemented. Independent re-verification of R1–R6 against current persistent_ha code via codegraph + grep + verbatim reads. Supersedes the 2026-06-09 validation pass; records two status changes since then.

Changes since the 2026-06-09 pass

  1. Process gap CLOSED. spec.json now shows phase: implemented and approvals.tasks.approved: true (was false/tasks-generated). The prior "tasks unapproved" finding no longer applies.
  2. Adjacent slot-0 defect FIXED. XCE_SCHED_OP_UPDATE_RESVALS (per-host running update) at src/scheduler/src/scheduler.c:816-834 now resolves rv->name via key_find(sched->resmgr, ...) and stores with zarray_put(&w->resvals, r->idx, rv->val) (line 831) — the hardcoded zarray_put(..., 0, ...) flagged on 2026-06-09 is gone. The last index-style hazard in the resval surface is closed. (Belongs to scheduler-controller-msg-gap scope, but worth recording: the defect class this spec targets is now eliminated across ALL resval paths A/B/C and both running-update handlers.)

Requirement-to-Asset Map

Req EARS Asset(s) — file:line Tag
R1 — name-keyed shared resvals in initial config R1-1 src/schema/controller.capnp:241 resvals @6: List(NameValue); src/controller/dataex.c:381 collect_resval(c->resmgr, &c->sharedinst, l); src/controller/sched.c cast to xce_name_val_t** MET
R1-2 src/scheduler/src/scheduler.c:587-595 sched_init_sharedInst: key_find(sched->resmgr, nv->name) → zarray_put(&sched->sharedInst, r->idx, val) MET
R1-3 scheduler.c:583-592: NULL nv/name skipped; unknown name → WARN + continue, sharedInst untouched MET
R1-4 Same name→r->idx resolution as Path C (scheduler.c:797-812, UPDATE_RESVALS_FULL) MET (by inspection; no equivalence test — see R6-1)
R2 — name-keyed per-worker resvals to scheduler R2-1 controller.capnp:140 resvals @5: List(NameValue); dataex.c:374 collect_resvals_workers(c->resmgr, l); sched.c:477 collect_resval(c->resmgr, &w->host.resvals, ...) MET
R2-2 src/scheduler/src/schedobj.c:274-292 make_sched_worker: name lookup → zarray_put(&worker->resvals, r->idx, val) (line 291) MET
R2-3 schedobj.c:279-288: NULL skip; unknown name → WARN + skip; worker construction continues MET
R2-4 Resolution runs against live sched->resmgr on every receipt; no idx cached across messages (schedobj.c:274-292) MET (by design; no mid-run add/remove test — RN-3)
R3 — name-keyed worker resvals to API server R3-1 src/controller/apiserver.c apiserver_worker_change: same collect_resval + xce_name_val_t** path as scheduler push MET
R3-2 src/frontend/apiserver/backend/convert.go:100-105 iterates worker.resvals_to_sched (unsafe.Slice), convert.go:283 cNameValToWorkerResval (name-keyed, nil-checked); convert_pure.go:210 ResourceValues: r.resvals; models/hosts.go:94 ResourceValues []ResourceDistributionInfo MET
R4 — Path C unchanged R4-1 src/scheduler/src/handler.c (Data_data_resvals → XCE_SCHED_OP_UPDATE_RESVALS_FULL) and scheduler.c:797-814 untouched, still xce_resval_t name-keyed MET
R4-2 xce_conf4sched_t.resvals (typemap.h) and xce_worker_t.resvals_to_sched (types.h) are xce_name_val_t**; codec auto-generated; free_name_val_list (src/lib/utils.c:388); all 7 controller free sites converted (codegraph: 7 callers across apiserver.c, dataex.c, opget.c, reply.c, sched.c) MET
R5 — remove index-keyed call sites R5-1 Zero cross-subsystem index-keyed transfers remain (grep: collect_resval_with_idx/free_resval_with_idx_list = 0 matches in src/ excl. build/); both receivers resolve by name MET
R5-2 xce_val_with_idx_t typedef (src/include/xce/types.h:38-41) and capnp struct ValueWithIdx (controller.capnp:178) still present — zero non-generated users, no scoping comment Missing (SHOULD-level)
R6 — no regression / correctness under divergent indices R6-1 Correct by code inspection (name resolution is order-independent at scheduler.c:587 and schedobj.c:283); scheduler testsuite (src/scheduler/testsuite/test.c) has zero references to sched_init_sharedInst / make_sched_worker / divergent resval order Missing (test only; code MET)
R6-2 Same-order behavior preserved; controller/scheduler suites passed at task 8.2 MET
R6-3 calc_wanted_* read sharedInst / worker->resvals via scheduler-side r->idx; all writers (sched_init_sharedInst, make_sched_worker, both UPDATE_RESVALS handlers) now key by r->idx MET

Score: 21/23 EARS MET. 2 Missing — both non-MUST (R5-2 SHOULD; R6-1 test-only, code present). All MUST-level acceptance criteria are demonstrably satisfied.

Residual Gaps

  • R5-2 (SHOULD): Dead xce_val_with_idx_t / struct ValueWithIdx linger without the "removed or restricted to intra-subsystem use, with a comment explaining its scope" that EARS-R5-2 requests. Invites accidental reuse of the very index-keyed shape this feature eliminated.
  • R6-1 (test gap): The feature's entire raison d'être — correctness when controller and scheduler register resources in different orders — has no automated test. Code is order-independent by construction, but the invariant is unguarded against regression.
  • Carry-forward note: RN-3 (EARS-R2-4 mid-run resource add/remove) still has no harness. Lower priority than R6-1.

Options for Open Work

Option A — Close residuals in-spec (Recommended). Delete xce_val_with_idx_t (types.h:38-41) + struct ValueWithIdx (controller.capnp:178) [or add the EARS-R5-2 scoping comment if retained for tooling], and add one table-driven scheduler testsuite case feeding sched_init_sharedInst / make_sched_worker a name-val list against a resmgr populated in reversed order, asserting sharedInst / worker->resvals slot contents.

  • Effort: S (~2 files + 1 test). Risk: Low.
  • Caveat: deleting from types.h is a shared-header edit — requires a clean nix develop --command ninja -C src/build to confirm no generated/out-of-tree consumer references ValueWithIdx (RN-2).

Option B — Defer to a cleanup/test sweep across HA specs. Bundle R5-2 + R6-1 + RN-3 with similar dead-type/test debt in adjacent specs.

  • Effort: S-M. Risk: Low. Cost: residue stays detached from this spec's history; may be dropped if the sweep stalls.

Option C — Accept and close as-is. Record R5-2 and the missing R6-1 test as known accepted debt; all MUST criteria met, slot-0 hazard already fixed, spec.json fully approved.

  • Effort: none. Risk: Low-Medium — dead type invites reuse; the feature's core scenario remains unverified by automation.

Verdict

Feature is functionally complete and CLOSED for all MUST requirements (R1-1..R1-3, R2-1..R2-4, R3-1/R3-2, R4-1/R4-2, R5-1, R6-2/R6-3). Two non-blocking residuals remain (R5-2 dead-type cleanup; R6-1 divergent-order test). Recommend Option A to fully satisfy the SHOULD clause and lock the core invariant under test. Effort S, Risk Low.