myxce/.kiro/specs/scheduler-controller-msg-gap/research.md

# Research Log: scheduler-controller-msg-gap

> Feature: scheduler-controller-msg-gap
> Created: 2026-05-11

## Discovery Type

Extension (light discovery) — extending existing handler code in an established C codebase.

## Summary

The scheduler's `handler.c` dispatcher processes controller messages via `scheduler_req_handler`. Six message cases are stubbed out or empty-bodied. The `vtask_entry()` assembly layer in `src/lib/xcectx.c` rewrites all `pushmsg` into `datamsg`, making the `Request_body_pushmsg` branch dead code. Existing plugin hooks (`plugin_job_update`, `plugin_queue_update`, `plugin_worker_update`, `plugin_job_state_change`) exist for notifying plugins of state changes. A `plugin_job_rusage_update` hook does not yet exist and must be added. The `src/lib/helper.c` file already provides `xce_request_op_name()`, `xce_request_msg_name()`, and `xce_dataset_table()` — a new `xce_notify_event_name()` function is needed.

## Research Log

### Topic 1: Plugin Hook System

**Source:** `src/scheduler/include/plugin.h`, `src/scheduler/src/plugin.c`

**Findings:**
- Hooks follow a `plugin_*_update(void* sched, xce_sched_*_t*)` pattern
- Called after state mutation to notify registered plugins
- Hooks iterate registered plugins and call their function pointers
- `plugin_job_state_change(void*, xce_sched_job_t*, uint32_t)` takes an extra state parameter
- No `plugin_job_rusage_update` hook exists yet — must be added

**Implication:** New handlers should call the corresponding plugin hook after mutating scheduler state. The rusage handler needs a new hook typedef and global function.

### Topic 2: Object Conversion Functions

**Source:** `src/scheduler/src/schedobj.c`

**Findings:**
- `make_sched_job(void* arg, xce_job_t* j)` — maps jobid, times, state, user/queue/project pointers, resreq, mintasks/maxtasks, allocs, accounts
- `make_sched_queue(void* arg, xce_queue_t* q)` — maps name, attrs, status, priority, accept_interval, sched_delay, resreq
- `make_sched_worker(void* arg, xce_worker_t* w)` — maps name, status, idx, resvals zarray from `w->resvals_to_sched`
- `kill_sched_*` functions free their respective objects

**Implication:** Job/queue/worker update handlers should use `make_sched_*` for create-or-update semantics. For updates (existing objects), we need update-in-place logic since `make_sched_*` always creates new objects.

### Topic 3: Resource and Shared Instance Initialization

**Source:** `src/scheduler/src/scheduler.c`

**Findings:**
- `sched_init_resources()` — iterates `config->resources[]`, calls `makeResource(sched->resmgr, r)` for each, warns on NULL/duplicate
- `sched_init_sharedInst()` — iterates `config->resvals[]`, uses `MOVE_PTR` to steal val pointers into `sched->sharedInst` zarray
- Both called from `sched_init_config()` during initial config push
- For runtime updates, we need incremental versions that can add new or update existing resources

**Implication:** Dataset handlers for `Data_data_resources` and `Data_data_resvals` need new update functions that can both create and modify existing entries.

### Topic 4: Existing Helper Functions

**Source:** `src/lib/helper.c`, `src/include/helper.h`

**Findings:**
- `xce_request_op_name(xce_request_t* req)` — returns string for OPCode enum
- `xce_request_msg_name(xce_request_t* req)` — returns string for Request body tag
- `xce_dataset_table(int table)` — returns string for Data table type enum
- All return `"unknown"` for unrecognized values
- `xce_notify_event_name()` does NOT exist yet — must be created
- Signature pattern: `char* func_name(enumerated_type_or_struct_ptr)` returning `char*`

**Implication:** Create `xce_notify_event_name(int event)` following same pattern. Also consider `char* xce_sched_op_name(xce_sched_op_t op)` for the update request op enum.

### Topic 5: Cap'n Proto Enum Values (NotifyData)

**Source:** `src/schema/controller.capnp`

**Findings:**
All 19 NotifyData event tags:
- `jobPidInfo(0)`, `jobFailInfo(1)`, `jobDispatchFail(2)`, `jobFinishInfo(3)`, `schedStartJob(4)`, `schedExitJob(5)`, `schedFailJob(6)`, `schedUpdateJob(7)`, `schedUpdateResvals(8)`, `stoppedJob(9)`, `resumedJob(10)`, `ckptinfo(11)`, `schedUpdateQueue(12)`, `taskPidInfo(13)`, `taskFailInfo(14)`, `taskFinish(15)`, `postDoneJob(16)`, `schedJobRusages(17)`, `updateJob(18)`

**Implication:** The `xce_notify_event_name()` function must cover all 19 values. Only the scheduler-relevant ones need handler implementations (schedUpdateJob, schedUpdateQueue, schedUpdateResvals, schedJobRusages are the gap items).

### Topic 6: Dead Code Analysis

**Source:** `src/lib/xcectx.c` lines 93-106, `src/scheduler/src/handler.c` lines 172-197

**Findings:**
- `vtask_entry()` accumulates `pushmsg` rows into `ctx->dataset`, then rewrites `body_which = Request_body_datamsg` and `body.datamsg = ctx->dataset`
- The `Request_body_pushmsg` case in `scheduler_req_handler` (lines 185-198) calls `sched_handle_jobpush()` and `sched_handle_workerpush()`, both of which are stubs returning 0
- These functions can never be reached via normal message flow

**Implication:** Safe to remove the pushmsg case branch and both stub functions.

## Architecture Decisions

1. **Handler pattern:** New notify handlers follow the same enqueue-to-pend-updates pattern as `sched_on_job_state_change` — create a `xce_sched_update_request_t`, set the op, enqueue via `enq_pend_update()`, process in `sched_update()` main loop.

2. **Update-in-place for job/queue/worker:** For NotifyData update events, we need update semantics (find existing, mutate fields) rather than create semantics (make_sched_*). New `sched_update_job`, `sched_update_queue`, `sched_update_worker` functions will handle this.

3. **New XCE_SCHED_OP enum values:** Need `XCE_SCHED_OP_UPDATE_RESVALS` for resource value updates and `XCE_SCHED_OP_JOB_RUSAGE` for rusage updates.

4. **Plugin hook for rusage:** Add `plugin_job_rusage_hook_t` typedef and `plugin_job_rusage_update()` global function following existing patterns.

5. **Helper function:** Add `xce_notify_event_name(int)` and `xce_sched_op_name(xce_sched_op_t)` to `src/lib/helper.c` and `src/include/helper.h`.

---

# Cross-Spec Gap Audit (2026-06-08)

> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch.

## Analysis Summary

- **scheduler-controller-msg-gap** is **~100% complete**. All 9 task groups, 28 subtasks (1.1–9.9) are `[x]` (9.9 optional).
- **No remaining gaps**: All 8 requirements have verified implementation. The 6 previously stubbed notify/dataset handlers (`schedUpdateJob`, `schedUpdateQueue`, `schedUpdateResvals`, `schedJobRusages`, `Data_data_resources`, `Data_data_resvals`) are wired with full business logic. Dead code (`pushmsg` branches, `sched_handle_jobpush`, `sched_handle_workerpush`) removed. Symbolic logging via `xce_notify_event_name()` and `xce_sched_op_name()` is in place.
- **Optional gap**: Task 9.9 (verify WARN log when handler returns non-zero) is marked optional — requires log capture infrastructure not yet present in the test harness.
- **Cross-spec relationship**: `resval-id-decoupling` uses the same pattern (name-based resource lookup via `sched->resmgr`) in `sched_init_sharedInst`. The `XCE_SCHED_OP_UPDATE_RESVALS_FULL` handler added in this spec also resolves by name — consistency is maintained.
- **Recommendation**: The spec can be marked `implemented`. Task 9.9 can be closed as deferred.
---

# Gap Analysis — Validation Pass (2026-06-09)

> Method: per `.claude/skills/kiro-validate-gap/rules/gap-analysis.md`. Every requirement re-verified against current source on `persistent_ha` (handler.c, scheduler.c, schedobj.c, plugin.c, helper.c, resobj.c, controller sched.c, testsuite/test.c). This pass **revises the 2026-06-08 audit's "~100% complete" conclusion**: handler wiring is complete, but four acceptance criteria are only partially met, two latent bugs exist in the dequeue paths, payload memory is leaked for the new op types, and the Task 9 test checkboxes substantially overstate actual coverage.

## 1. Current State Summary

- All 7 controller-emitted NotifyData events (`src/controller/sched.c`: schedStartJob, schedExitJob, schedFailJob, schedUpdateJob, schedJobRusages, schedUpdateResvals, schedUpdateQueue) have non-stub scheduler handlers in `src/scheduler/src/handler.c` (`sched_handle_notify`, lines 275–323). Unknown events return `-EINVAL`.
- Both previously-empty dataset cases (`Data_data_resources`, `Data_data_resvals`) are wired in `sched_handle_dataset` (handler.c:130–144) with ownership transfer (`dataset->nrows = 0; dataset->rows = NULL`).
- Dead `Request_body_pushmsg` branch and `sched_handle_jobpush`/`sched_handle_workerpush` are fully removed; the only remaining reference is a comment in testsuite/test.c:459. Unknown `body_which` returns `-EINVAL` (handler.c:349–351).
- The enqueue→dequeue two-phase pattern is followed: all new handlers enqueue `xce_sched_update_request_t` via `enq_pend_update()`; `sched_update()` (scheduler.c:727–845) processes the five new ops on the main thread.
- Note: no controller code currently **pushes** `Data_data_resources`/`Data_data_resvals` to the scheduler — `src/controller/sched.c` pushes only schedjobs/schedworkers; `src/controller/opget.c` serves these tables only as get-responses. The Req 1 handlers are forward-looking (harmless, but untriggered in production today).

## 2. Requirement-to-Asset Map

| Req / AC | Asset(s) | Status | Notes |
|---|---|---|---|
| 1 / UB-1.1 (resources add **or update**) | handler.c:58–79 `sched_handle_resources`; scheduler.c:770–781 `XCE_SCHED_OP_UPDATE_RESOURCES` | **Partial** | Dequeue calls `makeResource()`, but `makeResource_` (src/lib/resobj.c:108–140) returns NULL with `EEXIST` when the name already exists — **update semantics are missing**; re-pushed catalogs silently fail to modify existing resources. Return value is also ignored (no log), violating the error-handling steering ("never silently swallow errors"). |
| 1 / UB-1.2 (resvals → sharedInst by index) | handler.c:81–102; scheduler.c:782–799 `UPDATE_RESVALS_FULL` | Implemented | Resolves resource by name via `key_find(sched->resmgr,…)`, puts at `r->idx` — consistent with `sched_init_sharedInst` and the resval-id-decoupling spec. Unknown resources WARN-and-skip. |
| 1 / UB-1.3 (skip NULL rows) | scheduler.c:773–775, 786–788 | Implemented | `continue` on NULL entries in both loops. |
| 1 / UB-1.4 (-ENOMEM, free partial) | handler.c calloc guards | Implemented | Handlers return `-ENOMEM`; no partial allocations to free at enqueue level. |
| 2 / UB-2.1, UB-2.2 (dead code gone) | handler.c:325–362 | **Implemented** | `Request_body_pushmsg` case and both stub functions absent from source; `grep` finds no definitions anywhere in src/scheduler/. |
| 3 / UB-3.1 (update mutable fields: state, priority, queue, resreq) | scheduler.c:707–725 `sched_update_existing_job` | **Partial** | Only `job->state` is copied from the update payload. Priority, queue, and resource-request fields named in the requirement are not updated. `plugin_job_update()` is called. |
| 3 / UB-3.2 (-ENOENT unknown jobid) | scheduler.c:716–718 | Implemented | Wait — caveat: `sched_update_existing_job` exists but the `XCE_SCHED_OP_UPDATE_JOB` dequeue case routes to `sched_update_jobs()` (scheduler.c:754–756), which is the bulk create path. Whether a notify-driven single-job update reaches `sched_update_existing_job` needs verification (see Research Needed R-2). |
| 3 / UB-3.3 (-EINVAL NULL payload) | handler.c:180–182 | Implemented | |
| 4 / UB-4.1, 4.2, 4.3 (queue update) | schedobj.c:485–511 `sched_update_existing_queue`; scheduler.c:767–769 | Implemented | status/priority/accept_interval/sched_delay/resreq all updated; resreq freed + reallocated; `-ENOENT` on unknown queue; `-EINVAL` on NULL; `plugin_queue_update()` called. |
| 5 / UB-5.1 (per-worker resvals update) | scheduler.c:801–818 `UPDATE_RESVALS` | **Partial — bug** | `zarray_put(&w->resvals, 0, rv->val)` writes **index 0 unconditionally**. Worker resvals are indexed by resource idx (`zarray_put(&(worker->resvals), r->idx, val)`, schedobj.c:291) and read back via `zarray_get(&w->resvals, i)` in resource.c:161. The handler must resolve `rv->name` → `sched->resmgr` → `r->idx` and put at that index; as written, every update clobbers slot 0 regardless of resource. |
| 5 / UB-5.2 (skip unknown host) | scheduler.c:808–811 | Implemented | `sched_get_worker()` NULL → `continue`. |
| 5 / UB-5.3 (zero values → return 0) | handler.c:235–237 | Implemented | NULL or `count == 0` short-circuits before enqueue. |
| 6 / UB-6.1 (**store** rusage on job + plugin) | scheduler.c:819–831 `JOB_RUSAGE`; plugin.c:361 `plugin_job_rusage_update` | **Partial** | Plugin hook is implemented and called, but the rusage data is **not stored on the job object** as the requirement states — after the plugin call the payload is dropped (and leaked, see §3). |
| 6 / UB-6.2 (-ENOENT unknown jobid) | scheduler.c:822–824 | Implemented | |
| 6 / UB-6.3 (-EINVAL NULL rusage) | handler.c:257–259 | Implemented | |
| 7 / UB-7.1 (WARN with op + errno) | scheduler.c:838–841 | Implemented | `WARN("%s: op %s failed: %s (%d)", …, sched_op_name(req->op), strerror(-rc), rc)` after the switch. |
| 7 / UB-7.2 (unknown notify → -EINVAL) | handler.c:315–317 | Implemented (non-regression confirmed) | |
| 7 / UB-7.3 (unknown body_which → -EINVAL) | handler.c:349–351 | Implemented (non-regression confirmed) | |
| 8 / UB-8.1 (op name in req handler log) | handler.c:330 | Implemented | `TRACE("%s op = %s", __func__, xce_request_op_name(request))`. |
| 8 / UB-8.2 (dataset table name) | handler.c:109 | Implemented | `xce_dataset_table(dataset->table)`. |
| 8 / UB-8.3 (notify event name) | handler.c:279 | Implemented | `xce_notify_event_name(notify->event)`. |
| 8 / UB-8.4 (UNKNOWN(%d) fallback) | src/lib/helper.c:367–415 | Implemented | All 19 NotifyData tags covered + `snprintf(buf, …, "UNKNOWN(%d)", event)`. |
| 8 / UB-8.5 (symbolic op name in WARN) | scheduler.c:685–706 `sched_op_name` | Implemented — **deviation** | Implemented as **static `sched_op_name()` in scheduler.c**, not `xce_sched_op_name()` in `src/lib/helper.c`/`src/include/helper.h` as design Component 8 and tasks 1.1/1.3 (both `[x]`) state. The deviation is arguably correct (the enum is scheduler-internal; libxce cannot see `xce_sched_op_t`), but tasks.md and design.md no longer reflect reality. Uses a non-thread-safe `static char buf[32]` — acceptable since only the main thread calls it; same pattern as the helper. |
| 9 (tests) | src/scheduler/testsuite/test.c | **Partial — checkboxes overstate coverage** | See §4. |

**Memory-ownership constraint (design risk realized):** `sched_handle_notify` nullifies `notify->data.*` after successful enqueue (handler.c:297–315), transferring ownership from `free_Request()` to the update request — but `sched_pend_update_killer` (scheduler.c:311–321) has empty `break;` for all five new ops and the dequeue cases never free the payloads. Result: every `schedUpdateQueue` leaks an `xce_queue_t`, every `schedUpdateResvals` leaks the `xce_resval_list_t` (entries + name/host strings; only `rv->val` survives into the zarray — and on the index-0 bug path even that gets clobbered without a free), every `schedJobRusages` leaks the rusages struct, and every resources/resvals dataset leaks the row array and row structs (`makeInstance` copies; originals are never freed). The `UPDATE_JOB` path is fine (existing killer frees via `free_Job_ptr`). The design's High-severity risk row predicted exactly this; the mitigation was only half-applied (transfer done, release missing).

## 3. Identified Gaps (priority order)

1. **G-1 (bug, correctness): UPDATE_RESVALS writes worker resval index 0 unconditionally** — scheduler.c:813. Breaks scheduling decisions that read per-resource values. (Req 5)
2. **G-2 (bug, correctness): UPDATE_RESOURCES cannot update an existing resource** — `makeResource` → `EEXIST` → NULL, return ignored, no log. "Add or update" is only "add". (Req 1 / UB-1.1)
3. **G-3 (leak): killer/dequeue never free new-op payloads** despite ownership transfer (queue, resval_list, rusages, resource/resval rows). Slow per-message leak in a long-lived daemon. (Design risk #1)
4. **G-4 (requirement shortfall): job update copies only `state`** — priority/queue/resreq not applied. (Req 3 / UB-3.1)
5. **G-5 (requirement shortfall): rusage not stored on job object** — only the plugin hook sees it. Requires a rusage field on `xce_sched_job_t` (none exists today). (Req 6 / UB-6.1)
6. **G-6 (test debt): tasks 9.2, 9.3, 9.4, 9.6, 9.7 are checked `[x]` but absent from test.c.** The only new tests are: 9.1 notify-event-name (real), an enum compile-presence check, 9.5 reduced to a NULL-guard exercised against a **local stub** of `sched_update_existing_queue` defined inside test.c:25–34 (the real schedobj.c implementation is never linked/tested), and 9.8 reduced to `TEST_PASS()` with a comment. `xce_sched_op_name`/`sched_op_name` has no test (9.1 claims it does). None of the enqueue handlers, dequeue cases, or `sched_update_existing_job` are tested.
7. **G-7 (doc drift): design Component 8 / tasks 1.1, 1.3 say `xce_sched_op_name` lives in lib/helper** — it is a static in scheduler.c. Update spec docs or move the function.

## 4. Implementation Approach Options

**Option A — Extend in place (minimal correctness patch).** Fix G-1 (resolve `rv->name` → resmgr idx, mirroring schedobj.c:284–291), G-2 (on `EEXIST` find existing resource and mutate fields, or add an `updateResource()` beside `makeResource()` in resobj.c; log NULL returns), G-3 (add `free_*` calls in `sched_pend_update_killer` per op, reusing the deep-free helpers in src/lib/request.c), G-4 (copy priority/queue/resreq in `sched_update_existing_job`), and G-7 (doc edit). Defer G-5/G-6.
- ✅ Smallest diff, all in files already touched by this spec; restores correctness fast.
- ❌ Leaves UB-6.1 unmet and test checkboxes false; test debt persists.

**Option B — New test program + extracted ops module.** Create `src/scheduler/testsuite/test_msg_handlers.c` (links scheduler objects + generated capnp headers, which test.c cannot include — the stated reason 9.8 became a comment), and optionally extract the five dequeue case bodies into `src/scheduler/src/msgops.c` with non-static functions so they are unit-testable without including scheduler.c.
- ✅ Makes every acceptance criterion genuinely testable; honest Task 9; aligns with testing steering (tiered test_*.c programs already exist in other daemons).
- ❌ Larger diff; new file + meson.build change; refactor risk in scheduler.c.

**Option C — Hybrid (recommended).** Phase 1: Option A bug/leak fixes (G-1…G-4, G-7) with TDD where linkable. Phase 2: Option B test program covering handler enqueue + dequeue paths, replacing the stub-based 9.5 and the TEST_PASS 9.8 with real assertions; un-check or re-implement tasks 9.2–9.8 accordingly. Phase 3 (optional, needs decision): G-5 — add a rusages field to `xce_sched_job_t` or formally amend Req 6 to "deliver to plugins only".
- ✅ Correctness first, honesty in tracking, requirement closure staged.
- ❌ Two-phase coordination; Phase 3 needs a requirements decision.

## 5. Effort & Risk

| Work item | Effort | Risk | Justification |
|---|---|---|---|
| G-1 resvals index fix | S | Low | One-line lookup mirroring existing schedobj.c pattern; resmgr API already used two cases above. |
| G-2 resource update-on-EEXIST | S–M | Medium | Touching shared libxce resobj.c affects controller too; scheduler-side find-and-mutate is the safer S variant. |
| G-3 killer frees | M | Medium | Must trace ownership per type against free_* helpers in request.c; wrong free = double-free under ASAN, wrong no-op = leak persists. |
| G-4 job field updates | S | Low | Copy 3 more fields + string realloc, same shape as queue update. |
| G-5 store rusage on job | M | Medium | New field on `xce_sched_job_t` + lifetime/free in `kill_sched_job`; or amend requirement (zero code). |
| G-6 real test program | M | Medium | New testsuite binary + meson wiring; capnp header dependency is the known friction that derailed Task 9 the first time. |
| G-7 doc sync | S | Low | Edit design.md/tasks.md only. |
| **Total (Option C, Phases 1–2)** | **M (≈4–6 days)** | **Medium** | All within established patterns; risk concentrated in memory-ownership work under ASAN. |

## 6. Recommendations for Design/Implementation Follow-up

1. Treat G-1 and G-2 as **defects against the already-approved spec** — fix before marking the spec `implemented`; both silently corrupt or ignore controller state updates.
2. Adopt **Option C**: in-place fixes first (TDD per project rules — failing test in the new test program first where possible), then the dedicated `test_msg_handlers.c` to make Task 9 honest. Un-check tasks 9.2–9.8 until real tests exist, or annotate them with the stub caveat.
3. Resolve G-3 with the design's own mitigation table: request struct owns the payload after nullification ⇒ `sched_pend_update_killer` must deep-free each new op's payload (reuse `free_Queue_ptr`/`free_Resource_ptr`-style helpers from src/lib/request.c if exported; otherwise add small free helpers).
4. Decide G-5 explicitly: either add the job-side rusage storage Req 6 promises, or amend UB-6.1 — do not leave the requirement and code disagreeing.
5. Sync design.md Component 8 and tasks 1.1/1.3 with the actual `sched_op_name` location (G-7).

## 7. Research Needed

- **R-1:** Which `free_*` deep-free helpers in src/lib/request.c are exported and safe for `xce_queue_t`, `xce_resval_list_t`, `xce_job_rusages_t`, `xce_resource_t*[]` — confirm against ASAN before wiring into the killer.
- **R-2:** Confirm the notify-driven `schedUpdateJob` actually reaches `sched_update_existing_job()`: the dequeue `XCE_SCHED_OP_UPDATE_JOB` case calls `sched_update_jobs()` (bulk path); verify `sched_update_jobs` performs update-in-place for existing jobids (design Component 5 said it should) rather than duplicating jobs.
- **R-3:** Is there a planned controller-side emitter for `Data_data_resources`/`Data_data_resvals` pushes to the scheduler (e.g., post-failover config resync in the persistent_ha work)? If not, decide whether Req 1's handlers stay forward-looking or get an emitter spec.
- **R-4:** Whether `makeInstance(&resobj_type, r)` deep-copies all of `xce_resource_t` (name string included) — determines whether resource row payloads can be freed immediately after `makeResource` returns.