docs(specs): gap-analysis validation pass across all 9 HA-family specs

Appended "Gap Analysis — Validation Pass (2026-06-09)" to each spec's
research.md with requirement-to-asset maps, status tags, and effort/risk.

Key outcomes:
- ha-repl-hardening: REOPENED with task series 7.1-7.7 — replication
  payload data plane is missing (snapshot chunks and WAL catch-up ship
  header-only; standbys converge on LSN without holding entity state)
- worker-join-refactor, resval-id-decoupling: marked implemented
  (all criteria re-verified, full test suite green)
- buglog: 335/336 (scheduler dequeue, open), 337/338/339 (fixed in
  previous commit); cerebrum Decision Log corrected (term-only vote
  deny was recorded as intentional but was a regression)

.kiro/specs/scheduler-controller-msg-gap/research.md

@@ -109,4 +109,100 @@ All 19 NotifyData event tags:
 - **No remaining gaps**: All 8 requirements have verified implementation. The 6 previously stubbed notify/dataset handlers (`schedUpdateJob`, `schedUpdateQueue`, `schedUpdateResvals`, `schedJobRusages`, `Data_data_resources`, `Data_data_resvals`) are wired with full business logic. Dead code (`pushmsg` branches, `sched_handle_jobpush`, `sched_handle_workerpush`) removed. Symbolic logging via `xce_notify_event_name()` and `xce_sched_op_name()` is in place.
 - **Optional gap**: Task 9.9 (verify WARN log when handler returns non-zero) is marked optional — requires log capture infrastructure not yet present in the test harness.
 - **Cross-spec relationship**: `resval-id-decoupling` uses the same pattern (name-based resource lookup via `sched->resmgr`) in `sched_init_sharedInst`. The `XCE_SCHED_OP_UPDATE_RESVALS_FULL` handler added in this spec also resolves by name — consistency is maintained.
-- **Recommendation**: The spec can be marked `implemented`. Task 9.9 can be closed as deferred.
+- **Recommendation**: The spec can be marked `implemented`. Task 9.9 can be closed as deferred.
+---
+
+# Gap Analysis — Validation Pass (2026-06-09)
+
+> Method: per `.claude/skills/kiro-validate-gap/rules/gap-analysis.md`. Every requirement re-verified against current source on `persistent_ha` (handler.c, scheduler.c, schedobj.c, plugin.c, helper.c, resobj.c, controller sched.c, testsuite/test.c). This pass **revises the 2026-06-08 audit's "~100% complete" conclusion**: handler wiring is complete, but four acceptance criteria are only partially met, two latent bugs exist in the dequeue paths, payload memory is leaked for the new op types, and the Task 9 test checkboxes substantially overstate actual coverage.
+
+## 1. Current State Summary
+
+- All 7 controller-emitted NotifyData events (`src/controller/sched.c`: schedStartJob, schedExitJob, schedFailJob, schedUpdateJob, schedJobRusages, schedUpdateResvals, schedUpdateQueue) have non-stub scheduler handlers in `src/scheduler/src/handler.c` (`sched_handle_notify`, lines 275–323). Unknown events return `-EINVAL`.
+- Both previously-empty dataset cases (`Data_data_resources`, `Data_data_resvals`) are wired in `sched_handle_dataset` (handler.c:130–144) with ownership transfer (`dataset->nrows = 0; dataset->rows = NULL`).
+- Dead `Request_body_pushmsg` branch and `sched_handle_jobpush`/`sched_handle_workerpush` are fully removed; the only remaining reference is a comment in testsuite/test.c:459. Unknown `body_which` returns `-EINVAL` (handler.c:349–351).
+- The enqueue→dequeue two-phase pattern is followed: all new handlers enqueue `xce_sched_update_request_t` via `enq_pend_update()`; `sched_update()` (scheduler.c:727–845) processes the five new ops on the main thread.
+- Note: no controller code currently **pushes** `Data_data_resources`/`Data_data_resvals` to the scheduler — `src/controller/sched.c` pushes only schedjobs/schedworkers; `src/controller/opget.c` serves these tables only as get-responses. The Req 1 handlers are forward-looking (harmless, but untriggered in production today).
+
+## 2. Requirement-to-Asset Map
+
+| Req / AC | Asset(s) | Status | Notes |
+|---|---|---|---|
+| 1 / UB-1.1 (resources add **or update**) | handler.c:58–79 `sched_handle_resources`; scheduler.c:770–781 `XCE_SCHED_OP_UPDATE_RESOURCES` | **Partial** | Dequeue calls `makeResource()`, but `makeResource_` (src/lib/resobj.c:108–140) returns NULL with `EEXIST` when the name already exists — **update semantics are missing**; re-pushed catalogs silently fail to modify existing resources. Return value is also ignored (no log), violating the error-handling steering ("never silently swallow errors"). |
+| 1 / UB-1.2 (resvals → sharedInst by index) | handler.c:81–102; scheduler.c:782–799 `UPDATE_RESVALS_FULL` | Implemented | Resolves resource by name via `key_find(sched->resmgr,…)`, puts at `r->idx` — consistent with `sched_init_sharedInst` and the resval-id-decoupling spec. Unknown resources WARN-and-skip. |
+| 1 / UB-1.3 (skip NULL rows) | scheduler.c:773–775, 786–788 | Implemented | `continue` on NULL entries in both loops. |
+| 1 / UB-1.4 (-ENOMEM, free partial) | handler.c calloc guards | Implemented | Handlers return `-ENOMEM`; no partial allocations to free at enqueue level. |
+| 2 / UB-2.1, UB-2.2 (dead code gone) | handler.c:325–362 | **Implemented** | `Request_body_pushmsg` case and both stub functions absent from source; `grep` finds no definitions anywhere in src/scheduler/. |
+| 3 / UB-3.1 (update mutable fields: state, priority, queue, resreq) | scheduler.c:707–725 `sched_update_existing_job` | **Partial** | Only `job->state` is copied from the update payload. Priority, queue, and resource-request fields named in the requirement are not updated. `plugin_job_update()` is called. |
+| 3 / UB-3.2 (-ENOENT unknown jobid) | scheduler.c:716–718 | Implemented | Wait — caveat: `sched_update_existing_job` exists but the `XCE_SCHED_OP_UPDATE_JOB` dequeue case routes to `sched_update_jobs()` (scheduler.c:754–756), which is the bulk create path. Whether a notify-driven single-job update reaches `sched_update_existing_job` needs verification (see Research Needed R-2). |
+| 3 / UB-3.3 (-EINVAL NULL payload) | handler.c:180–182 | Implemented | |
+| 4 / UB-4.1, 4.2, 4.3 (queue update) | schedobj.c:485–511 `sched_update_existing_queue`; scheduler.c:767–769 | Implemented | status/priority/accept_interval/sched_delay/resreq all updated; resreq freed + reallocated; `-ENOENT` on unknown queue; `-EINVAL` on NULL; `plugin_queue_update()` called. |
+| 5 / UB-5.1 (per-worker resvals update) | scheduler.c:801–818 `UPDATE_RESVALS` | **Partial — bug** | `zarray_put(&w->resvals, 0, rv->val)` writes **index 0 unconditionally**. Worker resvals are indexed by resource idx (`zarray_put(&(worker->resvals), r->idx, val)`, schedobj.c:291) and read back via `zarray_get(&w->resvals, i)` in resource.c:161. The handler must resolve `rv->name` → `sched->resmgr` → `r->idx` and put at that index; as written, every update clobbers slot 0 regardless of resource. |
+| 5 / UB-5.2 (skip unknown host) | scheduler.c:808–811 | Implemented | `sched_get_worker()` NULL → `continue`. |
+| 5 / UB-5.3 (zero values → return 0) | handler.c:235–237 | Implemented | NULL or `count == 0` short-circuits before enqueue. |
+| 6 / UB-6.1 (**store** rusage on job + plugin) | scheduler.c:819–831 `JOB_RUSAGE`; plugin.c:361 `plugin_job_rusage_update` | **Partial** | Plugin hook is implemented and called, but the rusage data is **not stored on the job object** as the requirement states — after the plugin call the payload is dropped (and leaked, see §3). |
+| 6 / UB-6.2 (-ENOENT unknown jobid) | scheduler.c:822–824 | Implemented | |
+| 6 / UB-6.3 (-EINVAL NULL rusage) | handler.c:257–259 | Implemented | |
+| 7 / UB-7.1 (WARN with op + errno) | scheduler.c:838–841 | Implemented | `WARN("%s: op %s failed: %s (%d)", …, sched_op_name(req->op), strerror(-rc), rc)` after the switch. |
+| 7 / UB-7.2 (unknown notify → -EINVAL) | handler.c:315–317 | Implemented (non-regression confirmed) | |
+| 7 / UB-7.3 (unknown body_which → -EINVAL) | handler.c:349–351 | Implemented (non-regression confirmed) | |
+| 8 / UB-8.1 (op name in req handler log) | handler.c:330 | Implemented | `TRACE("%s op = %s", __func__, xce_request_op_name(request))`. |
+| 8 / UB-8.2 (dataset table name) | handler.c:109 | Implemented | `xce_dataset_table(dataset->table)`. |
+| 8 / UB-8.3 (notify event name) | handler.c:279 | Implemented | `xce_notify_event_name(notify->event)`. |
+| 8 / UB-8.4 (UNKNOWN(%d) fallback) | src/lib/helper.c:367–415 | Implemented | All 19 NotifyData tags covered + `snprintf(buf, …, "UNKNOWN(%d)", event)`. |
+| 8 / UB-8.5 (symbolic op name in WARN) | scheduler.c:685–706 `sched_op_name` | Implemented — **deviation** | Implemented as **static `sched_op_name()` in scheduler.c**, not `xce_sched_op_name()` in `src/lib/helper.c`/`src/include/helper.h` as design Component 8 and tasks 1.1/1.3 (both `[x]`) state. The deviation is arguably correct (the enum is scheduler-internal; libxce cannot see `xce_sched_op_t`), but tasks.md and design.md no longer reflect reality. Uses a non-thread-safe `static char buf[32]` — acceptable since only the main thread calls it; same pattern as the helper. |
+| 9 (tests) | src/scheduler/testsuite/test.c | **Partial — checkboxes overstate coverage** | See §4. |
+
+**Memory-ownership constraint (design risk realized):** `sched_handle_notify` nullifies `notify->data.*` after successful enqueue (handler.c:297–315), transferring ownership from `free_Request()` to the update request — but `sched_pend_update_killer` (scheduler.c:311–321) has empty `break;` for all five new ops and the dequeue cases never free the payloads. Result: every `schedUpdateQueue` leaks an `xce_queue_t`, every `schedUpdateResvals` leaks the `xce_resval_list_t` (entries + name/host strings; only `rv->val` survives into the zarray — and on the index-0 bug path even that gets clobbered without a free), every `schedJobRusages` leaks the rusages struct, and every resources/resvals dataset leaks the row array and row structs (`makeInstance` copies; originals are never freed). The `UPDATE_JOB` path is fine (existing killer frees via `free_Job_ptr`). The design's High-severity risk row predicted exactly this; the mitigation was only half-applied (transfer done, release missing).
+
+## 3. Identified Gaps (priority order)
+
+1. **G-1 (bug, correctness): UPDATE_RESVALS writes worker resval index 0 unconditionally** — scheduler.c:813. Breaks scheduling decisions that read per-resource values. (Req 5)
+2. **G-2 (bug, correctness): UPDATE_RESOURCES cannot update an existing resource** — `makeResource` → `EEXIST` → NULL, return ignored, no log. "Add or update" is only "add". (Req 1 / UB-1.1)
+3. **G-3 (leak): killer/dequeue never free new-op payloads** despite ownership transfer (queue, resval_list, rusages, resource/resval rows). Slow per-message leak in a long-lived daemon. (Design risk #1)
+4. **G-4 (requirement shortfall): job update copies only `state`** — priority/queue/resreq not applied. (Req 3 / UB-3.1)
+5. **G-5 (requirement shortfall): rusage not stored on job object** — only the plugin hook sees it. Requires a rusage field on `xce_sched_job_t` (none exists today). (Req 6 / UB-6.1)
+6. **G-6 (test debt): tasks 9.2, 9.3, 9.4, 9.6, 9.7 are checked `[x]` but absent from test.c.** The only new tests are: 9.1 notify-event-name (real), an enum compile-presence check, 9.5 reduced to a NULL-guard exercised against a **local stub** of `sched_update_existing_queue` defined inside test.c:25–34 (the real schedobj.c implementation is never linked/tested), and 9.8 reduced to `TEST_PASS()` with a comment. `xce_sched_op_name`/`sched_op_name` has no test (9.1 claims it does). None of the enqueue handlers, dequeue cases, or `sched_update_existing_job` are tested.
+7. **G-7 (doc drift): design Component 8 / tasks 1.1, 1.3 say `xce_sched_op_name` lives in lib/helper** — it is a static in scheduler.c. Update spec docs or move the function.
+
+## 4. Implementation Approach Options
+
+**Option A — Extend in place (minimal correctness patch).** Fix G-1 (resolve `rv->name` → resmgr idx, mirroring schedobj.c:284–291), G-2 (on `EEXIST` find existing resource and mutate fields, or add an `updateResource()` beside `makeResource()` in resobj.c; log NULL returns), G-3 (add `free_*` calls in `sched_pend_update_killer` per op, reusing the deep-free helpers in src/lib/request.c), G-4 (copy priority/queue/resreq in `sched_update_existing_job`), and G-7 (doc edit). Defer G-5/G-6.
+- ✅ Smallest diff, all in files already touched by this spec; restores correctness fast.
+- ❌ Leaves UB-6.1 unmet and test checkboxes false; test debt persists.
+
+**Option B — New test program + extracted ops module.** Create `src/scheduler/testsuite/test_msg_handlers.c` (links scheduler objects + generated capnp headers, which test.c cannot include — the stated reason 9.8 became a comment), and optionally extract the five dequeue case bodies into `src/scheduler/src/msgops.c` with non-static functions so they are unit-testable without including scheduler.c.
+- ✅ Makes every acceptance criterion genuinely testable; honest Task 9; aligns with testing steering (tiered test_*.c programs already exist in other daemons).
+- ❌ Larger diff; new file + meson.build change; refactor risk in scheduler.c.
+
+**Option C — Hybrid (recommended).** Phase 1: Option A bug/leak fixes (G-1…G-4, G-7) with TDD where linkable. Phase 2: Option B test program covering handler enqueue + dequeue paths, replacing the stub-based 9.5 and the TEST_PASS 9.8 with real assertions; un-check or re-implement tasks 9.2–9.8 accordingly. Phase 3 (optional, needs decision): G-5 — add a rusages field to `xce_sched_job_t` or formally amend Req 6 to "deliver to plugins only".
+- ✅ Correctness first, honesty in tracking, requirement closure staged.
+- ❌ Two-phase coordination; Phase 3 needs a requirements decision.
+
+## 5. Effort & Risk
+
+| Work item | Effort | Risk | Justification |
+|---|---|---|---|
+| G-1 resvals index fix | S | Low | One-line lookup mirroring existing schedobj.c pattern; resmgr API already used two cases above. |
+| G-2 resource update-on-EEXIST | S–M | Medium | Touching shared libxce resobj.c affects controller too; scheduler-side find-and-mutate is the safer S variant. |
+| G-3 killer frees | M | Medium | Must trace ownership per type against free_* helpers in request.c; wrong free = double-free under ASAN, wrong no-op = leak persists. |
+| G-4 job field updates | S | Low | Copy 3 more fields + string realloc, same shape as queue update. |
+| G-5 store rusage on job | M | Medium | New field on `xce_sched_job_t` + lifetime/free in `kill_sched_job`; or amend requirement (zero code). |
+| G-6 real test program | M | Medium | New testsuite binary + meson wiring; capnp header dependency is the known friction that derailed Task 9 the first time. |
+| G-7 doc sync | S | Low | Edit design.md/tasks.md only. |
+| **Total (Option C, Phases 1–2)** | **M (≈4–6 days)** | **Medium** | All within established patterns; risk concentrated in memory-ownership work under ASAN. |
+
+## 6. Recommendations for Design/Implementation Follow-up
+
+1. Treat G-1 and G-2 as **defects against the already-approved spec** — fix before marking the spec `implemented`; both silently corrupt or ignore controller state updates.
+2. Adopt **Option C**: in-place fixes first (TDD per project rules — failing test in the new test program first where possible), then the dedicated `test_msg_handlers.c` to make Task 9 honest. Un-check tasks 9.2–9.8 until real tests exist, or annotate them with the stub caveat.
+3. Resolve G-3 with the design's own mitigation table: request struct owns the payload after nullification ⇒ `sched_pend_update_killer` must deep-free each new op's payload (reuse `free_Queue_ptr`/`free_Resource_ptr`-style helpers from src/lib/request.c if exported; otherwise add small free helpers).
+4. Decide G-5 explicitly: either add the job-side rusage storage Req 6 promises, or amend UB-6.1 — do not leave the requirement and code disagreeing.
+5. Sync design.md Component 8 and tasks 1.1/1.3 with the actual `sched_op_name` location (G-7).
+
+## 7. Research Needed
+
+- **R-1:** Which `free_*` deep-free helpers in src/lib/request.c are exported and safe for `xce_queue_t`, `xce_resval_list_t`, `xce_job_rusages_t`, `xce_resource_t*[]` — confirm against ASAN before wiring into the killer.
+- **R-2:** Confirm the notify-driven `schedUpdateJob` actually reaches `sched_update_existing_job()`: the dequeue `XCE_SCHED_OP_UPDATE_JOB` case calls `sched_update_jobs()` (bulk path); verify `sched_update_jobs` performs update-in-place for existing jobids (design Component 5 said it should) rather than duplicating jobs.
+- **R-3:** Is there a planned controller-side emitter for `Data_data_resources`/`Data_data_resvals` pushes to the scheduler (e.g., post-failover config resync in the persistent_ha work)? If not, decide whether Req 1's handlers stay forward-looking or get an emitter spec.
+- **R-4:** Whether `makeInstance(&resobj_type, r)` deep-copies all of `xce_resource_t` (name string included) — determines whether resource row payloads can be freed immediately after `makeResource` returns.