Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
303 lines
29 KiB
Markdown
303 lines
29 KiB
Markdown
# Gap Analysis: resval-id-decoupling
|
||
|
||
> Generated: 2026-05-12T07:25:00-07:00
|
||
> Feature: resval-id-decoupling
|
||
|
||
## Executive Summary
|
||
|
||
- **Scope**: Three call sites in the controller use `collect_resval_with_idx()` to produce index-keyed resource values sent to two consumers (scheduler and API server). Fixing all three brings every cross-subsystem resval path into alignment with the already-correct `Data_data_resvals` dataset path.
|
||
- **Existing Infrastructure**: `xce_name_val_t` (`NameValue` in capnp) already exists in `types.h` and has a **complete capnp codec** (`encode_NameValue_list` / `decode_NameValue_list`) in the generated code — it was defined for this purpose but never wired in at the resval-transfer boundary.
|
||
- **Wire format impact**: `Conf4Sched.resvals` (field @6) and `Worker.resvals` (field @5) in `controller.capnp` currently declare `List(ValueWithIdx)`. Changing these to `List(NameValue)` is a **breaking wire change** — no intermediate backward-compatibility window unless two field slots are used.
|
||
- **Build chain**: The capnp schema in `src/schema/controller.capnp` drives generation of `src/build/lib/controller.capnp.[ch]` via `meson` + `capnpc-c`. A schema edit automatically regenerates the C codecs on the next build.
|
||
- **Key insight**: `makeResource_()` in `resobj.c:137` always overwrites `resobj->idx = rc` (from `key_add`) regardless of the `idx` transmitted in the `Resource` wire struct — so the controller's idx is lost at the scheduler's resmgr and the assumption that they match is never enforced.
|
||
|
||
---
|
||
|
||
## Existing Components Inventory
|
||
|
||
### Types and Structs
|
||
|
||
| Symbol | Location | Role | Status |
|
||
|--------|----------|------|--------|
|
||
| `xce_val_with_idx_t` | `src/include/xce/types.h:38` | `{int idx, xce_val_t* val}` — current cross-subsystem carrier | **Replace at boundaries** |
|
||
| `xce_name_val_t` | `src/include/xce/types.h:43` | `{char* name, xce_val_t* val}` — name-keyed carrier | **Ready to use** |
|
||
| `xce_resval_t` | `src/include/xce/types.h:48` | `{char* name, char* host, xce_val_t* val}` — used by Path C | Already correct |
|
||
| `xce_conf4sched_t.resvals` | `src/include/typemap.h:50` | `xce_val_with_idx_t**` — shared resvals in initial config | **Change to `xce_name_val_t**`** |
|
||
| `xce_worker_t.resvals_to_sched` | `src/include/xce/types.h:217` | `xce_val_with_idx_t**` — per-worker resvals | **Change to `xce_name_val_t**`** |
|
||
|
||
### Controller — Call Sites (Sender Side)
|
||
|
||
| File | Line | Function | Purpose | Change needed |
|
||
|------|------|----------|---------|---------------|
|
||
| `src/controller/dataex.c` | 62 | `collect_resval_with_idx()` | Iterates `zarray_t`, emits `{idx, val}` | Replace/augment with name-keyed variant |
|
||
| `src/controller/dataex.c` | 363 | `collect_shared_resvals()` | Calls `collect_resval_with_idx(&c->sharedinst)` | Change to name-keyed collection |
|
||
| `src/controller/dataex.c` | 141–163 | `collect_resvals_workers()` | Calls `collect_resval_with_idx(&w->resvals)` per worker | Change to name-keyed collection |
|
||
| `src/controller/sched.c` | 118–120 | `xcesched_push_configs()` | Puts shared resvals into `conf4sched->resvals` | Change field type; update free |
|
||
| `src/controller/sched.c` | 442–444 | `xcesched_push_workerlist()` | Puts worker resvals into `w->host.resvals_to_sched` | Change field type; update free |
|
||
| `src/controller/apiserver.c` | 367–369 | `apiserver_worker_change()` | Same pattern as sched for API server path | Change field type; update free |
|
||
|
||
### Free/cleanup Sites
|
||
|
||
| File | Line | Call | Must be updated |
|
||
|------|------|------|-----------------|
|
||
| `src/controller/sched.c` | 145 | `free_resval_with_idx_list(&conf4sched->resvals, ...)` | → `free_NameValue_list` or equivalent |
|
||
| `src/controller/sched.c` | 226, 463 | `free_resval_with_idx_list(&w->resvals_to_sched, ...)` | Same |
|
||
| `src/controller/apiserver.c` | 51, 397–399 | `free_resval_with_idx_list(...)` | Same |
|
||
| `src/controller/dataex.c` | 404 | `free_resval_with_idx_list(...)` | Same |
|
||
| `src/controller/reply.c` | 66 | `free_resval_with_idx_list(...)` | Same |
|
||
| `src/controller/opget.c` | 128 | `free_resval_with_idx_list(...)` | Same |
|
||
| `src/include/utils.h` | 130 | `free_resval_with_idx_list` declaration | Update or keep alongside new |
|
||
|
||
### Scheduler — Consumer Side
|
||
|
||
| File | Line | Code | Change needed |
|
||
|------|------|------|---------------|
|
||
| `src/scheduler/src/scheduler.c` | 574–586 | `sched_init_sharedInst()` — uses `idx` directly from `config->resvals[i]->idx` | Change to name-lookup in `sched->resmgr` |
|
||
| `src/scheduler/src/schedobj.c` | 274–280 | `make_sched_worker()` — uses `idx` directly from `w->resvals_to_sched[i]->idx` | Change to name-lookup |
|
||
|
||
### Capnp Schema and Generated Code
|
||
|
||
| Asset | Location | Current | Change |
|
||
|-------|----------|---------|--------|
|
||
| `Conf4Sched.resvals` | `src/schema/controller.capnp:230` | `List(ValueWithIdx)` | → `List(NameValue)` |
|
||
| `Worker.resvals` | `src/schema/controller.capnp:129` | `List(ValueWithIdx)` | → `List(NameValue)` |
|
||
| `controller.capnp.c/h` | `src/build/lib/` | Auto-generated — regenerated by `ninja -C build` | Automatically updated after schema edit |
|
||
| `encode_NameValue_list` | `src/build/lib/controller.capnp.c:3402` | Already exists, unused at these boundaries | Wire in to replace `encode_ValueWithIdx_list` |
|
||
| `decode_NameValue_list` | `src/build/lib/controller.capnp.c:3426` | Already exists, unused at these boundaries | Wire in to replace `decode_ValueWithIdx_list` |
|
||
| `free_NameValue_list` | `src/build/lib/controller.capnp.c:3458` | Already exists | Replace `free_ValueWithIdx_list` at these call sites |
|
||
|
||
### API Server (Go frontend) — Impact
|
||
|
||
The Go API server (`src/frontend/apiserver/backend/clib.go`) includes `controller.capnp.h` and works with `xce_worker_t*` via CGo. It accesses `resvals_to_sched` in `apiserver_worker_change()` (controller side) which populates `xce_worker_t.resvals_to_sched`. The Go frontend's `cResvalToDistribution` in `convert.go:273` already processes `xce_resval_t` (name-keyed), not `xce_val_with_idx_t`. The API server impact is limited to updating any code that reads `resvals_to_sched` field — if it accesses it at all on the Go side (currently no direct access found).
|
||
|
||
---
|
||
|
||
## Root Cause Deep Dive
|
||
|
||
The zarray-based stores (`c->sharedinst`, `w->host.resvals`) use `r->idx` as the slot key. `r->idx` is assigned by `key_add()` in `makeResource_` (`resobj.c:137`) — it is the slot returned by the internal container, which depends on insertion order. The controller and scheduler each call `makeResource()` independently (controller at cluster init, scheduler at `sched_init_resources()`), with no guarantee of identical ordering.
|
||
|
||
Crucially, the `Resource` capnp struct **does transmit `r->idx` on the wire** (`encode_Resource` at `build/lib/controller.capnp.c:3889`), but `decode_Resource` writes it into a temporary `xce_resource_t`, which is then passed to `makeResource()`, which **immediately overwrites** the decoded `idx` with the local `key_add` return value (`resobj->idx = rc`). The transmitted idx is therefore discarded at the scheduler.
|
||
|
||
---
|
||
|
||
## Implementation Options
|
||
|
||
### Option A — Replace `ValueWithIdx` with `NameValue` at both capnp fields (Recommended)
|
||
|
||
**What changes:**
|
||
1. `controller.capnp`: change `Conf4Sched.resvals @6` and `Worker.resvals @5` from `List(ValueWithIdx)` to `List(NameValue)`
|
||
2. Rebuild capnp — generated `encode_Conf4Sched`/`decode_Conf4Sched` and worker codec automatically use `NameValue` functions
|
||
3. `typemap.h`: `xce_conf4sched_t.resvals` → `xce_name_val_t**`
|
||
4. `types.h`: `xce_worker_t.resvals_to_sched` → `xce_name_val_t**`
|
||
5. Controller: new `collect_resval_by_name(zarray_t* vals, var resmgr, xlist_t* l)` that walks `resmgr` entries, reads the value by `r->idx` from the array, and emits `{name, val}` records
|
||
6. Scheduler `sched_init_sharedInst()`: iterate name-val list, `key_find(sched->resmgr, name)` → `r->idx` → `zarray_put`
|
||
7. Scheduler `make_sched_worker()`: same pattern
|
||
8. Update all free/cleanup call sites to use `free_NameValue_list`
|
||
|
||
**Pros:** Cleanest — removes the misleading `idx` field entirely; `NameValue` codec already exists; aligns with Path C's design; no backward compatibility concerns given no public release
|
||
**Cons:** None significant — wire-format break is a non-issue as there is no released version requiring compatibility
|
||
|
||
**Affected files (estimate):** 8 C files, 1 capnp schema, 0 Go files
|
||
|
||
---
|
||
|
||
### Option B — Add `name` field to `xce_val_with_idx_t` / `ValueWithIdx` ~~(Additive wire change)~~ [REJECTED]
|
||
|
||
Ruled out: backward compatibility is not a requirement for this project at its current stage. Option B's only advantage over Option A is a softer wire transition — without that constraint, it is strictly worse: it keeps a semantically dead `idx` field, requires more code changes, and leaves misleading structure in the API.
|
||
|
||
---
|
||
|
||
### Option C — Use Controller's idx as canonical; stop overwriting in `makeResource_` [REJECTED]
|
||
|
||
Ruled out: does not fix the design — just moves the source of truth without eliminating the hidden identity assumption. A resource deleted and re-added in the controller gets a new idx, and the scheduler has no way to know its cached idx is stale.
|
||
|
||
---
|
||
|
||
## Recommendation
|
||
|
||
**Option A** is the selected approach. Backward compatibility is not a constraint (no public release), so the wire-format change to `List(NameValue)` is straightforward. The `NameValue` codec already exists in the generated capnp code and is fully functional — this is primarily a wiring exercise. The semantic change aligns the initial-config and worker-data paths with the already-correct `Data_data_resvals` update path (Path C).
|
||
|
||
---
|
||
|
||
## Files to Change (Option A)
|
||
|
||
| File | Change Type |
|
||
|------|-------------|
|
||
| `src/schema/controller.capnp` | Schema edit: 2 field type changes |
|
||
| `src/include/xce/types.h` | Struct field type change: `resvals_to_sched` |
|
||
| `src/include/typemap.h` | Struct field type change: `resvals` |
|
||
| `src/controller/dataex.c` | New `collect_resval_by_name()` function; update `collect_shared_resvals()` and `collect_resvals_workers()`; **remove** `collect_resval_with_idx` and `collect_resval_with_idx_` |
|
||
| `src/controller/sched.c` | Update 3 call sites (populate and free) |
|
||
| `src/controller/apiserver.c` | Update 2 call sites (populate and free) |
|
||
| `src/controller/reply.c` | Update free call |
|
||
| `src/controller/opget.c` | Update free call |
|
||
| `src/scheduler/src/scheduler.c` | Update `sched_init_sharedInst()` |
|
||
| `src/scheduler/src/schedobj.c` | Update `make_sched_worker()` |
|
||
| `src/include/utils.h` | Remove `free_resval_with_idx_list` declaration; add `free_name_val_list` declaration |
|
||
| `src/lib/utils.c` | Remove `free_resval_with_idx_list`; add `free_name_val_list(xce_name_val_t***, int)` |
|
||
| `src/controller/include/cluster.h` | Remove `collect_resval_with_idx` declaration |
|
||
|
||
**Not changed:** `src/scheduler/src/handler.c` (Path C untouched), `src/frontend/apiserver/` (Go API server does not directly access `resvals_to_sched` in Go code), capnp generated files (auto-regenerated by `ninja -C build`).
|
||
|
||
---
|
||
|
||
## Decisions (Closed Questions)
|
||
|
||
1. **New collector function signature**: `collect_resval_by_name` is controller-internal logic — add it to `dataex.c` alongside `collect_resval_with_idx`. It needs both the `zarray_t` (values indexed by `r->idx`) and the `resmgr` (name→idx mapping) as parameters.
|
||
|
||
2. **Handling unknown resources in scheduler**: **Warn and skip.** When `sched_init_sharedInst` or `make_sched_worker` encounters a name not in `sched->resmgr`, emit a `WARN` log and continue. This is safe because the resource list always arrives in the same config push before resvals are applied (see decision 3).
|
||
|
||
3. **Ordering guarantee**: **Confirmed — current behavior, must be preserved.** `sched_init_resources()` is called before `sched_init_sharedInst()` in `sched_init_config()` (`scheduler.c:603-609`). The design must not reorder these calls.
|
||
|
||
4. **Removal of `collect_resval_with_idx`**: **Remove entirely.** No cross-subsystem callers will remain after this change, and intra-subsystem callers (e.g., `worker.c` reading from `sharedinst` for API get-resval) access the zarray directly by `r->idx` without going through this function. Delete `collect_resval_with_idx`, `collect_resval_with_idx_`, and remove the declaration from `src/controller/include/cluster.h`.
|
||
|
||
5. **`free_name_val_list` utility**: **Add to `utils.c`/`utils.h`.** A new `free_name_val_list(xce_name_val_t***, int)` function mirroring `free_resval_with_idx_list` is needed. The capnp-generated `free_NameValue_list` is an option but couples cleanup to the serialization layer — a standalone utility in `utils.c` is cleaner and consistent with the existing pattern.
|
||
|
||
---
|
||
|
||
# Cross-Spec Gap Audit (2026-06-08)
|
||
|
||
> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch.
|
||
|
||
## Analysis Summary
|
||
|
||
- **resval-id-decoupling** is **~100% complete**. All 8 task groups, 22 subtasks (1.1–8.3) are `[x]` in tasks.md.
|
||
- **No remaining gaps**: `collect_resval_with_idx` and `free_resval_with_idx_list` are removed (verified by grep: zero matches). `collect_resval` (name-keyed, iterates resmgr not zarray) replaces it. The capnp schema (`Conf4Sched.resvals @6`, `Worker.resvals @5`) now use `List(NameValue)`. The scheduler resolves by name via `key_find(sched->resmgr, name)` → `r->idx`. The Go API server has `cNameValToWorkerResval` in `convert.go` for the new `xce_name_val_t**` type.
|
||
- **Task approval gap**: spec.json shows `tasks.approved: false`. All tasks are implemented but the tasks document was never formally approved. This is a process artifact — the implementation matches the approved design.
|
||
- **Cross-spec impact**: `scheduler-controller-msg-gap` extends the same scheduler handlers (`sched_init_sharedInst`, `make_sched_worker`) that were modified by this spec. The two specs are consistent — both use name-based resource resolution.
|
||
- **Recommendation**: Mark tasks as approved in spec.json, or run `/kiro-spec-status resval-id-decoupling` to sync.
|
||
|
||
---
|
||
|
||
# Gap Analysis — Validation Pass (2026-06-09)
|
||
|
||
## Method
|
||
|
||
Independent re-verification of every requirement (requirements.md R1–R6) against the current code on branch `persistent_ha`, via grep + direct read of the controller, scheduler, libxce, capnp schema, and Go API-server sources. Prior audit (2026-06-08) claimed ~100% completion; this pass confirms that claim and adds two residual findings it missed.
|
||
|
||
## Current State Summary
|
||
|
||
- The feature is **implemented end-to-end**. `collect_resval_with_idx` and `free_resval_with_idx_list` no longer exist anywhere in `src/` (zero grep matches). The replacement `collect_resval(var resmgr, zarray_t*, xlist_t*)` lives at `src/controller/dataex.c:23` and iterates the resmgr (names) rather than the zarray (slots), exactly as designed.
|
||
- Wire format: `src/schema/controller.capnp` — `Worker.resvals @5` (line 138) and `Conf4Sched.resvals @6` (line 239) are both `List(NameValue)`. Field ordinals preserved.
|
||
- All tasks in tasks.md are `[x]` (1.1–8.3), but `spec.json` still shows `tasks.approved: false` (phase: `tasks-generated`). Implementation landed despite the unapproved tasks document — a process artifact, not a code gap.
|
||
|
||
## Requirement-to-Asset Map
|
||
|
||
| Req | EARS | Asset(s) | Status |
|
||
|-----|------|----------|--------|
|
||
| R1 — name-keyed shared resvals in initial config | R1-1 | `controller.capnp:239` `List(NameValue)`; `src/controller/dataex.c:381` `collect_resval(c->resmgr, &c->sharedinst, l)`; `src/controller/sched.c:120` casts to `xce_name_val_t**` | **Implemented** |
|
||
| | R1-2 | `src/scheduler/src/scheduler.c:574-599` `sched_init_sharedInst`: `key_find(sched->resmgr, nv->name)` → `zarray_put(&sched->sharedInst, r->idx, val)` | **Implemented** |
|
||
| | R1-3 | `scheduler.c:587-592`: WARN + `continue` on unknown name; NULL name/entry also skipped | **Implemented** |
|
||
| | R1-4 | Same name→`r->idx` resolution as Path C handler (`scheduler.c:782-797`) | **Implemented** (by inspection; no equivalence test exists — see R6) |
|
||
| R2 — name-keyed per-worker resvals to scheduler | R2-1 | `controller.capnp:138` `List(NameValue)`; `dataex.c:151-166` `collect_resvals_workers(resmgr,...)`; `sched.c:442` | **Implemented** |
|
||
| | R2-2 | `src/scheduler/src/schedobj.c:274-292` `make_sched_worker`: name lookup → `zarray_put(&worker->resvals, r->idx, val)` | **Implemented** |
|
||
| | R2-3 | `schedobj.c:283-288`: WARN + skip, worker construction continues | **Implemented** |
|
||
| | R2-4 | Resolution happens at each receipt against the live `sched->resmgr`; no index is cached across messages | **Implemented** (by design; no automated test for resource add/remove mid-run) |
|
||
| R3 — name-keyed worker resvals to API server | R3-1 | `src/controller/apiserver.c:367-369`: same `collect_resval` + `xce_name_val_t**` path as the scheduler push | **Implemented** |
|
||
| | R3-2 | `src/frontend/apiserver/backend/convert.go:100-105` iterates `resvals_to_sched`, `convert.go:283` `cNameValToWorkerResval` (name-keyed, nil-checked); `convert_pure.go:210`; `models/hosts.go:94` `ResourceValues` | **Implemented** |
|
||
| R4 — Path C unchanged | R4-1 | `src/scheduler/src/handler.c:82-103` (`Data_data_resvals` → `XCE_SCHED_OP_UPDATE_RESVALS_FULL`) and `scheduler.c:782-800` untouched, still `xce_resval_t` name-keyed | **Implemented** |
|
||
| | R4-2 | `xce_conf4sched_t.resvals` (`typemap.h:62`) and `xce_worker_t.resvals_to_sched` (`types.h:219`) are `xce_name_val_t**`; codec auto-generated; `free_name_val_list` in `src/lib/utils.c:388` / `utils.h:130`; all controller free sites converted | **Implemented** |
|
||
| R5 — remove index-keyed call sites | R5-1 | Zero cross-subsystem index-keyed transfers remain (grep-verified); both receivers resolve by name | **Implemented** |
|
||
| | R5-2 | `xce_val_with_idx_t` typedef (`src/include/xce/types.h:38-41`) and capnp `struct ValueWithIdx` (`controller.capnp:176`) still exist with **zero users and no scoping comment** | **Partial** (SHOULD-level: dead type lingers, contrary to "removed or restricted ... with a comment") |
|
||
| R6 — no regression / correctness under divergent indices | R6-1 | Correct by code inspection (name resolution is order-independent); **no automated test** registers resources in different orders and asserts `sharedInst` contents | **Partial** (code Implemented, test Missing) |
|
||
| | R6-2 | Existing controller/scheduler suites passed at task 8.2; same-order behavior preserved | **Implemented** |
|
||
| | R6-3 | `calc_wanted_*` paths read `sharedInst`/`worker->resvals` via scheduler-side `r->idx` (scheduler testsuite exercises these at zarray level) | **Implemented** |
|
||
|
||
### Adjacent finding (Constraint — out of spec scope, same defect class)
|
||
|
||
`XCE_SCHED_OP_UPDATE_RESVALS` (per-host running update, `handler.c:231-251` → `scheduler.c:801-816`) receives name-keyed `xce_resval_t` records but stores them with a **hardcoded slot**: `zarray_put(&w->resvals, 0, rv->val)` — it finds the worker by `rv->host` but never resolves `rv->name` to `r->idx`. Any per-host update for a resource whose scheduler idx ≠ 0 lands in the wrong slot (or clobbers slot 0). This path was not enumerated in this spec (Paths A/B/C only) and belongs to the `scheduler-controller-msg-gap` surface, but it violates the spirit of EARS-R5-1. Flagged for follow-up, not a regression of this feature.
|
||
|
||
## Implementation Approach Options (for residual gaps)
|
||
|
||
### Option A — Minimal cleanup within this spec (Recommended)
|
||
Delete the dead `xce_val_with_idx_t` typedef and capnp `ValueWithIdx` struct (or add the scoping comment EARS-R5-2 asks for); add one scheduler testsuite case that feeds `sched_init_sharedInst`/`make_sched_worker` a name-val list against a resmgr populated in a different order and asserts slot contents (closes R6-1 test gap).
|
||
- ✅ Closes both Partial items with ~2 files + 1 test; no wire change (struct ordinals of NameValue untouched)
|
||
- ❌ Touches a shared header (`types.h`) — full rebuild needed to confirm nothing else references the type
|
||
|
||
### Option B — Fold residuals into `scheduler-controller-msg-gap`
|
||
Fix the `zarray_put(..., 0, ...)` slot bug together with the R5-2/R6-1 residuals in the adjacent spec, since it already owns the per-host update handlers.
|
||
- ✅ Single coherent change set for all per-host resval handling; this spec can be closed as-is
|
||
- ❌ R5-2/R6-1 residue stays attached to the wrong spec's history; risk of being dropped if that spec stalls
|
||
|
||
### Option C — Accept and close
|
||
Mark tasks approved, record R5-2 (SHOULD) and the missing R6-1 test as known accepted debt.
|
||
- ✅ Zero effort; all MUST-level acceptance criteria are demonstrably met
|
||
- ❌ Dead type invites accidental reuse; the divergent-order scenario — the entire point of the feature — remains untested by automation
|
||
|
||
## Effort & Risk
|
||
|
||
- **Effort: S** — residual work is a dead-type deletion plus one table-driven C test; the feature itself requires no further code.
|
||
- **Risk: Low** — all changes verified in place and grep-confirmed; the only Medium-risk item is the adjacent slot-0 bug, which is pre-existing and outside this spec.
|
||
|
||
## Recommendations
|
||
|
||
1. Approve tasks in `spec.json` (`tasks.approved: true`) to close the process gap — implementation matches the approved design.
|
||
2. Take Option A: remove (or comment-scope) `xce_val_with_idx_t` / `ValueWithIdx`, and add the divergent-registration-order test for `sched_init_sharedInst` / `make_sched_worker` (TDD: the test also documents EARS-R6-1 permanently).
|
||
3. File the `XCE_SCHED_OP_UPDATE_RESVALS` slot-0 defect against `scheduler-controller-msg-gap` (or a standalone fix) — it is the last index-style hazard in the resval surface.
|
||
|
||
## Research Needed
|
||
|
||
- **RN-1**: Confirm `XCE_SCHED_OP_UPDATE_RESVALS` (`scheduler.c:814`) is a live path (who sends `schedUpdateResvals` capnp field @8, and with which resources) before fixing the hardcoded slot 0 — it may be stub/dead code.
|
||
- **RN-2**: Verify no out-of-tree or generated consumer (e.g., `src/build/lib/controller.capnp.c`) still references `ValueWithIdx` codecs before deleting the schema struct; a clean `ninja -C src/build` after removal settles this.
|
||
- **RN-3**: EARS-R2-4 (resource add/remove mid-run) has no test harness today; determine whether the scheduler testsuite can drive `sched_handle_resources` followed by a worker push without a live controller.
|
||
|
||
---
|
||
|
||
## Gap Re-Audit (2026-06-18)
|
||
|
||
> POST-IMPLEMENTATION GAP AUDIT. Tasks 24/24 `[x]`, `spec.json.phase: implemented`.
|
||
> Independent re-verification of R1–R6 against current `persistent_ha` code via codegraph + grep + verbatim reads. Supersedes the 2026-06-09 validation pass; records two status changes since then.
|
||
|
||
### Changes since the 2026-06-09 pass
|
||
|
||
1. **Process gap CLOSED.** `spec.json` now shows `phase: implemented` and `approvals.tasks.approved: true` (was `false`/`tasks-generated`). The prior "tasks unapproved" finding no longer applies.
|
||
2. **Adjacent slot-0 defect FIXED.** `XCE_SCHED_OP_UPDATE_RESVALS` (per-host running update) at `src/scheduler/src/scheduler.c:816-834` now resolves `rv->name` via `key_find(sched->resmgr, ...)` and stores with `zarray_put(&w->resvals, r->idx, rv->val)` (line 831) — the hardcoded `zarray_put(..., 0, ...)` flagged on 2026-06-09 is gone. The last index-style hazard in the resval surface is closed. (Belongs to `scheduler-controller-msg-gap` scope, but worth recording: the defect class this spec targets is now eliminated across ALL resval paths A/B/C and both running-update handlers.)
|
||
|
||
### Requirement-to-Asset Map
|
||
|
||
| Req | EARS | Asset(s) — file:line | Tag |
|
||
|-----|------|----------------------|-----|
|
||
| R1 — name-keyed shared resvals in initial config | R1-1 | `src/schema/controller.capnp:241` `resvals @6: List(NameValue)`; `src/controller/dataex.c:381` `collect_resval(c->resmgr, &c->sharedinst, l)`; `src/controller/sched.c` cast to `xce_name_val_t**` | MET |
|
||
| | R1-2 | `src/scheduler/src/scheduler.c:587-595` `sched_init_sharedInst`: `key_find(sched->resmgr, nv->name)` → `zarray_put(&sched->sharedInst, r->idx, val)` | MET |
|
||
| | R1-3 | `scheduler.c:583-592`: NULL nv/name skipped; unknown name → `WARN` + `continue`, `sharedInst` untouched | MET |
|
||
| | R1-4 | Same name→`r->idx` resolution as Path C (`scheduler.c:797-812`, `UPDATE_RESVALS_FULL`) | MET (by inspection; no equivalence test — see R6-1) |
|
||
| R2 — name-keyed per-worker resvals to scheduler | R2-1 | `controller.capnp:140` `resvals @5: List(NameValue)`; `dataex.c:374` `collect_resvals_workers(c->resmgr, l)`; `sched.c:477` `collect_resval(c->resmgr, &w->host.resvals, ...)` | MET |
|
||
| | R2-2 | `src/scheduler/src/schedobj.c:274-292` `make_sched_worker`: name lookup → `zarray_put(&worker->resvals, r->idx, val)` (line 291) | MET |
|
||
| | R2-3 | `schedobj.c:279-288`: NULL skip; unknown name → `WARN` + skip; worker construction continues | MET |
|
||
| | R2-4 | Resolution runs against live `sched->resmgr` on every receipt; no idx cached across messages (`schedobj.c:274-292`) | MET (by design; no mid-run add/remove test — RN-3) |
|
||
| R3 — name-keyed worker resvals to API server | R3-1 | `src/controller/apiserver.c` `apiserver_worker_change`: same `collect_resval` + `xce_name_val_t**` path as scheduler push | MET |
|
||
| | R3-2 | `src/frontend/apiserver/backend/convert.go:100-105` iterates `worker.resvals_to_sched` (`unsafe.Slice`), `convert.go:283` `cNameValToWorkerResval` (name-keyed, nil-checked); `convert_pure.go:210` `ResourceValues: r.resvals`; `models/hosts.go:94` `ResourceValues []ResourceDistributionInfo` | MET |
|
||
| R4 — Path C unchanged | R4-1 | `src/scheduler/src/handler.c` (`Data_data_resvals` → `XCE_SCHED_OP_UPDATE_RESVALS_FULL`) and `scheduler.c:797-814` untouched, still `xce_resval_t` name-keyed | MET |
|
||
| | R4-2 | `xce_conf4sched_t.resvals` (`typemap.h`) and `xce_worker_t.resvals_to_sched` (`types.h`) are `xce_name_val_t**`; codec auto-generated; `free_name_val_list` (`src/lib/utils.c:388`); all 7 controller free sites converted (codegraph: 7 callers across apiserver.c, dataex.c, opget.c, reply.c, sched.c) | MET |
|
||
| R5 — remove index-keyed call sites | R5-1 | Zero cross-subsystem index-keyed transfers remain (grep: `collect_resval_with_idx`/`free_resval_with_idx_list` = 0 matches in src/ excl. build/); both receivers resolve by name | MET |
|
||
| | R5-2 | `xce_val_with_idx_t` typedef (`src/include/xce/types.h:38-41`) and capnp `struct ValueWithIdx` (`controller.capnp:178`) still present — zero non-generated users, **no scoping comment** | Missing (SHOULD-level) |
|
||
| R6 — no regression / correctness under divergent indices | R6-1 | Correct by code inspection (name resolution is order-independent at `scheduler.c:587` and `schedobj.c:283`); scheduler testsuite (`src/scheduler/testsuite/test.c`) has **zero** references to `sched_init_sharedInst` / `make_sched_worker` / divergent resval order | Missing (test only; code MET) |
|
||
| | R6-2 | Same-order behavior preserved; controller/scheduler suites passed at task 8.2 | MET |
|
||
| | R6-3 | `calc_wanted_*` read `sharedInst` / `worker->resvals` via scheduler-side `r->idx`; all writers (`sched_init_sharedInst`, `make_sched_worker`, both UPDATE_RESVALS handlers) now key by `r->idx` | MET |
|
||
|
||
**Score: 21/23 EARS MET. 2 Missing — both non-MUST (R5-2 SHOULD; R6-1 test-only, code present).** All MUST-level acceptance criteria are demonstrably satisfied.
|
||
|
||
### Residual Gaps
|
||
|
||
- **R5-2 (SHOULD):** Dead `xce_val_with_idx_t` / `struct ValueWithIdx` linger without the "removed or restricted to intra-subsystem use, with a comment explaining its scope" that EARS-R5-2 requests. Invites accidental reuse of the very index-keyed shape this feature eliminated.
|
||
- **R6-1 (test gap):** The feature's entire raison d'être — correctness when controller and scheduler register resources in *different* orders — has no automated test. Code is order-independent by construction, but the invariant is unguarded against regression.
|
||
- **Carry-forward note:** RN-3 (EARS-R2-4 mid-run resource add/remove) still has no harness. Lower priority than R6-1.
|
||
|
||
### Options for Open Work
|
||
|
||
**Option A — Close residuals in-spec (Recommended).**
|
||
Delete `xce_val_with_idx_t` (types.h:38-41) + `struct ValueWithIdx` (controller.capnp:178) [or add the EARS-R5-2 scoping comment if retained for tooling], and add one table-driven scheduler testsuite case feeding `sched_init_sharedInst` / `make_sched_worker` a name-val list against a resmgr populated in reversed order, asserting `sharedInst` / `worker->resvals` slot contents.
|
||
- Effort: **S** (~2 files + 1 test). Risk: **Low**.
|
||
- Caveat: deleting from `types.h` is a shared-header edit — requires a clean `nix develop --command ninja -C src/build` to confirm no generated/out-of-tree consumer references `ValueWithIdx` (RN-2).
|
||
|
||
**Option B — Defer to a cleanup/test sweep across HA specs.**
|
||
Bundle R5-2 + R6-1 + RN-3 with similar dead-type/test debt in adjacent specs.
|
||
- Effort: **S-M**. Risk: **Low**. Cost: residue stays detached from this spec's history; may be dropped if the sweep stalls.
|
||
|
||
**Option C — Accept and close as-is.**
|
||
Record R5-2 and the missing R6-1 test as known accepted debt; all MUST criteria met, slot-0 hazard already fixed, spec.json fully approved.
|
||
- Effort: **none**. Risk: **Low-Medium** — dead type invites reuse; the feature's core scenario remains unverified by automation.
|
||
|
||
### Verdict
|
||
|
||
Feature is **functionally complete and CLOSED for all MUST requirements** (R1-1..R1-3, R2-1..R2-4, R3-1/R3-2, R4-1/R4-2, R5-1, R6-2/R6-3). Two non-blocking residuals remain (R5-2 dead-type cleanup; R6-1 divergent-order test). Recommend **Option A** to fully satisfy the SHOULD clause and lock the core invariant under test. Effort **S**, Risk **Low**.
|