myxce/.kiro/specs/scheduler-controller-msg-gap/research.md

Research Log: scheduler-controller-msg-gap

Feature: scheduler-controller-msg-gap Created: 2026-05-11

Discovery Type

Extension (light discovery) — extending existing handler code in an established C codebase.

Summary

The scheduler's handler.c dispatcher processes controller messages via scheduler_req_handler. Six message cases are stubbed out or empty-bodied. The vtask_entry() assembly layer in src/lib/xcectx.c rewrites all pushmsg into datamsg, making the Request_body_pushmsg branch dead code. Existing plugin hooks (plugin_job_update, plugin_queue_update, plugin_worker_update, plugin_job_state_change) exist for notifying plugins of state changes. A plugin_job_rusage_update hook does not yet exist and must be added. The src/lib/helper.c file already provides xce_request_op_name(), xce_request_msg_name(), and xce_dataset_table() — a new xce_notify_event_name() function is needed.

Research Log

Topic 1: Plugin Hook System

Source: src/scheduler/include/plugin.h, src/scheduler/src/plugin.c

Findings:

• Hooks follow a plugin_*_update(void* sched, xce_sched_*_t*) pattern
• Called after state mutation to notify registered plugins
• Hooks iterate registered plugins and call their function pointers
• plugin_job_state_change(void*, xce_sched_job_t*, uint32_t) takes an extra state parameter
• No plugin_job_rusage_update hook exists yet — must be added

Implication: New handlers should call the corresponding plugin hook after mutating scheduler state. The rusage handler needs a new hook typedef and global function.

Topic 2: Object Conversion Functions

Source: src/scheduler/src/schedobj.c

Findings:

• make_sched_job(void* arg, xce_job_t* j) — maps jobid, times, state, user/queue/project pointers, resreq, mintasks/maxtasks, allocs, accounts
• make_sched_queue(void* arg, xce_queue_t* q) — maps name, attrs, status, priority, accept_interval, sched_delay, resreq
• make_sched_worker(void* arg, xce_worker_t* w) — maps name, status, idx, resvals zarray from w->resvals_to_sched
• kill_sched_* functions free their respective objects

Implication: Job/queue/worker update handlers should use make_sched_* for create-or-update semantics. For updates (existing objects), we need update-in-place logic since make_sched_* always creates new objects.

Topic 3: Resource and Shared Instance Initialization

Source: src/scheduler/src/scheduler.c

Findings:

• sched_init_resources() — iterates config->resources[], calls makeResource(sched->resmgr, r) for each, warns on NULL/duplicate
• sched_init_sharedInst() — iterates config->resvals[], uses MOVE_PTR to steal val pointers into sched->sharedInst zarray
• Both called from sched_init_config() during initial config push
• For runtime updates, we need incremental versions that can add new or update existing resources

Implication: Dataset handlers for Data_data_resources and Data_data_resvals need new update functions that can both create and modify existing entries.

Topic 4: Existing Helper Functions

Source: src/lib/helper.c, src/include/helper.h

Findings:

• xce_request_op_name(xce_request_t* req) — returns string for OPCode enum
• xce_request_msg_name(xce_request_t* req) — returns string for Request body tag
• xce_dataset_table(int table) — returns string for Data table type enum
• All return "unknown" for unrecognized values
• xce_notify_event_name() does NOT exist yet — must be created
• Signature pattern: char* func_name(enumerated_type_or_struct_ptr) returning char*

Implication: Create xce_notify_event_name(int event) following same pattern. Also consider char* xce_sched_op_name(xce_sched_op_t op) for the update request op enum.

Topic 5: Cap'n Proto Enum Values (NotifyData)

Source: src/schema/controller.capnp

Findings: All 19 NotifyData event tags:

• jobPidInfo(0), jobFailInfo(1), jobDispatchFail(2), jobFinishInfo(3), schedStartJob(4), schedExitJob(5), schedFailJob(6), schedUpdateJob(7), schedUpdateResvals(8), stoppedJob(9), resumedJob(10), ckptinfo(11), schedUpdateQueue(12), taskPidInfo(13), taskFailInfo(14), taskFinish(15), postDoneJob(16), schedJobRusages(17), updateJob(18)

Implication: The xce_notify_event_name() function must cover all 19 values. Only the scheduler-relevant ones need handler implementations (schedUpdateJob, schedUpdateQueue, schedUpdateResvals, schedJobRusages are the gap items).

Topic 6: Dead Code Analysis

Source: src/lib/xcectx.c lines 93-106, src/scheduler/src/handler.c lines 172-197

Findings:

• vtask_entry() accumulates pushmsg rows into ctx->dataset, then rewrites body_which = Request_body_datamsg and body.datamsg = ctx->dataset
• The Request_body_pushmsg case in scheduler_req_handler (lines 185-198) calls sched_handle_jobpush() and sched_handle_workerpush(), both of which are stubs returning 0
• These functions can never be reached via normal message flow

Implication: Safe to remove the pushmsg case branch and both stub functions.

Architecture Decisions

1. Handler pattern: New notify handlers follow the same enqueue-to-pend-updates pattern as sched_on_job_state_change — create a xce_sched_update_request_t, set the op, enqueue via enq_pend_update(), process in sched_update() main loop.

2. Update-in-place for job/queue/worker: For NotifyData update events, we need update semantics (find existing, mutate fields) rather than create semantics (make_sched_*). New sched_update_job, sched_update_queue, sched_update_worker functions will handle this.

3. New XCE_SCHED_OP enum values: Need XCE_SCHED_OP_UPDATE_RESVALS for resource value updates and XCE_SCHED_OP_JOB_RUSAGE for rusage updates.

4. Plugin hook for rusage: Add plugin_job_rusage_hook_t typedef and plugin_job_rusage_update() global function following existing patterns.

5. Helper function: Add xce_notify_event_name(int) and xce_sched_op_name(xce_sched_op_t) to src/lib/helper.c and src/include/helper.h.

---

Cross-Spec Gap Audit (2026-06-08)

Batch /kiro-validate-gap across 9 specs on persistent_ha branch.

Analysis Summary

• scheduler-controller-msg-gap is ~100% complete. All 9 task groups, 28 subtasks (1.1–9.9) are [x] (9.9 optional).
• No remaining gaps: All 8 requirements have verified implementation. The 6 previously stubbed notify/dataset handlers (schedUpdateJob, schedUpdateQueue, schedUpdateResvals, schedJobRusages, Data_data_resources, Data_data_resvals) are wired with full business logic. Dead code (pushmsg branches, sched_handle_jobpush, sched_handle_workerpush) removed. Symbolic logging via xce_notify_event_name() and xce_sched_op_name() is in place.
• Optional gap: Task 9.9 (verify WARN log when handler returns non-zero) is marked optional — requires log capture infrastructure not yet present in the test harness.
• Cross-spec relationship: resval-id-decoupling uses the same pattern (name-based resource lookup via sched->resmgr) in sched_init_sharedInst. The XCE_SCHED_OP_UPDATE_RESVALS_FULL handler added in this spec also resolves by name — consistency is maintained.
• Recommendation: The spec can be marked implemented. Task 9.9 can be closed as deferred.

---

Gap Analysis — Validation Pass (2026-06-09)

Method: per .claude/skills/kiro-validate-gap/rules/gap-analysis.md. Every requirement re-verified against current source on persistent_ha (handler.c, scheduler.c, schedobj.c, plugin.c, helper.c, resobj.c, controller sched.c, testsuite/test.c). This pass revises the 2026-06-08 audit's "~100% complete" conclusion: handler wiring is complete, but four acceptance criteria are only partially met, two latent bugs exist in the dequeue paths, payload memory is leaked for the new op types, and the Task 9 test checkboxes substantially overstate actual coverage.

1. Current State Summary

• All 7 controller-emitted NotifyData events (src/controller/sched.c: schedStartJob, schedExitJob, schedFailJob, schedUpdateJob, schedJobRusages, schedUpdateResvals, schedUpdateQueue) have non-stub scheduler handlers in src/scheduler/src/handler.c (sched_handle_notify, lines 275–323). Unknown events return -EINVAL.
• Both previously-empty dataset cases (Data_data_resources, Data_data_resvals) are wired in sched_handle_dataset (handler.c:130–144) with ownership transfer (dataset->nrows = 0; dataset->rows = NULL).
• Dead Request_body_pushmsg branch and sched_handle_jobpush/sched_handle_workerpush are fully removed; the only remaining reference is a comment in testsuite/test.c:459. Unknown body_which returns -EINVAL (handler.c:349–351).
• The enqueue→dequeue two-phase pattern is followed: all new handlers enqueue xce_sched_update_request_t via enq_pend_update(); sched_update() (scheduler.c:727–845) processes the five new ops on the main thread.
• Note: no controller code currently pushes Data_data_resources/Data_data_resvals to the scheduler — src/controller/sched.c pushes only schedjobs/schedworkers; src/controller/opget.c serves these tables only as get-responses. The Req 1 handlers are forward-looking (harmless, but untriggered in production today).

2. Requirement-to-Asset Map

Req / AC	Asset(s)	Status	Notes
1 / UB-1.1 (resources add or update)	handler.c:58–79 sched_handle_resources; scheduler.c:770–781 XCE_SCHED_OP_UPDATE_RESOURCES	Partial	Dequeue calls makeResource(), but makeResource_ (src/lib/resobj.c:108–140) returns NULL with EEXIST when the name already exists — update semantics are missing; re-pushed catalogs silently fail to modify existing resources. Return value is also ignored (no log), violating the error-handling steering ("never silently swallow errors").
1 / UB-1.2 (resvals → sharedInst by index)	handler.c:81–102; scheduler.c:782–799 UPDATE_RESVALS_FULL	Implemented	Resolves resource by name via key_find(sched->resmgr,…), puts at r->idx — consistent with sched_init_sharedInst and the resval-id-decoupling spec. Unknown resources WARN-and-skip.
1 / UB-1.3 (skip NULL rows)	scheduler.c:773–775, 786–788	Implemented	continue on NULL entries in both loops.
1 / UB-1.4 (-ENOMEM, free partial)	handler.c calloc guards	Implemented	Handlers return -ENOMEM; no partial allocations to free at enqueue level.
2 / UB-2.1, UB-2.2 (dead code gone)	handler.c:325–362	Implemented	Request_body_pushmsg case and both stub functions absent from source; grep finds no definitions anywhere in src/scheduler/.
3 / UB-3.1 (update mutable fields: state, priority, queue, resreq)	scheduler.c:707–725 sched_update_existing_job	Partial	Only job->state is copied from the update payload. Priority, queue, and resource-request fields named in the requirement are not updated. plugin_job_update() is called.
3 / UB-3.2 (-ENOENT unknown jobid)	scheduler.c:716–718	Implemented	Wait — caveat: sched_update_existing_job exists but the XCE_SCHED_OP_UPDATE_JOB dequeue case routes to sched_update_jobs() (scheduler.c:754–756), which is the bulk create path. Whether a notify-driven single-job update reaches sched_update_existing_job needs verification (see Research Needed R-2).
3 / UB-3.3 (-EINVAL NULL payload)	handler.c:180–182	Implemented
4 / UB-4.1, 4.2, 4.3 (queue update)	schedobj.c:485–511 sched_update_existing_queue; scheduler.c:767–769	Implemented	status/priority/accept_interval/sched_delay/resreq all updated; resreq freed + reallocated; -ENOENT on unknown queue; -EINVAL on NULL; plugin_queue_update() called.
5 / UB-5.1 (per-worker resvals update)	scheduler.c:801–818 UPDATE_RESVALS	Partial — bug	zarray_put(&w->resvals, 0, rv->val) writes index 0 unconditionally. Worker resvals are indexed by resource idx (zarray_put(&(worker->resvals), r->idx, val), schedobj.c:291) and read back via zarray_get(&w->resvals, i) in resource.c:161. The handler must resolve rv->name → sched->resmgr → r->idx and put at that index; as written, every update clobbers slot 0 regardless of resource.
5 / UB-5.2 (skip unknown host)	scheduler.c:808–811	Implemented	sched_get_worker() NULL → continue.
5 / UB-5.3 (zero values → return 0)	handler.c:235–237	Implemented	NULL or count == 0 short-circuits before enqueue.
6 / UB-6.1 (store rusage on job + plugin)	scheduler.c:819–831 JOB_RUSAGE; plugin.c:361 plugin_job_rusage_update	Partial	Plugin hook is implemented and called, but the rusage data is not stored on the job object as the requirement states — after the plugin call the payload is dropped (and leaked, see §3).
6 / UB-6.2 (-ENOENT unknown jobid)	scheduler.c:822–824	Implemented
6 / UB-6.3 (-EINVAL NULL rusage)	handler.c:257–259	Implemented
7 / UB-7.1 (WARN with op + errno)	scheduler.c:838–841	Implemented	WARN("%s: op %s failed: %s (%d)", …, sched_op_name(req->op), strerror(-rc), rc) after the switch.
7 / UB-7.2 (unknown notify → -EINVAL)	handler.c:315–317	Implemented (non-regression confirmed)
7 / UB-7.3 (unknown body_which → -EINVAL)	handler.c:349–351	Implemented (non-regression confirmed)
8 / UB-8.1 (op name in req handler log)	handler.c:330	Implemented	TRACE("%s op = %s", __func__, xce_request_op_name(request)).
8 / UB-8.2 (dataset table name)	handler.c:109	Implemented	xce_dataset_table(dataset->table).
8 / UB-8.3 (notify event name)	handler.c:279	Implemented	xce_notify_event_name(notify->event).
8 / UB-8.4 (UNKNOWN(%d) fallback)	src/lib/helper.c:367–415	Implemented	All 19 NotifyData tags covered + snprintf(buf, …, "UNKNOWN(%d)", event).
8 / UB-8.5 (symbolic op name in WARN)	scheduler.c:685–706 sched_op_name	Implemented — deviation	Implemented as static sched_op_name() in scheduler.c, not xce_sched_op_name() in src/lib/helper.c/src/include/helper.h as design Component 8 and tasks 1.1/1.3 (both [x]) state. The deviation is arguably correct (the enum is scheduler-internal; libxce cannot see xce_sched_op_t), but tasks.md and design.md no longer reflect reality. Uses a non-thread-safe static char buf[32] — acceptable since only the main thread calls it; same pattern as the helper.
9 (tests)	src/scheduler/testsuite/test.c	Partial — checkboxes overstate coverage	See §4.

Memory-ownership constraint (design risk realized): sched_handle_notify nullifies notify->data.* after successful enqueue (handler.c:297–315), transferring ownership from free_Request() to the update request — but sched_pend_update_killer (scheduler.c:311–321) has empty break; for all five new ops and the dequeue cases never free the payloads. Result: every schedUpdateQueue leaks an xce_queue_t, every schedUpdateResvals leaks the xce_resval_list_t (entries + name/host strings; only rv->val survives into the zarray — and on the index-0 bug path even that gets clobbered without a free), every schedJobRusages leaks the rusages struct, and every resources/resvals dataset leaks the row array and row structs (makeInstance copies; originals are never freed). The UPDATE_JOB path is fine (existing killer frees via free_Job_ptr). The design's High-severity risk row predicted exactly this; the mitigation was only half-applied (transfer done, release missing).

3. Identified Gaps (priority order)

1. G-1 (bug, correctness): UPDATE_RESVALS writes worker resval index 0 unconditionally — scheduler.c:813. Breaks scheduling decisions that read per-resource values. (Req 5)
2. G-2 (bug, correctness): UPDATE_RESOURCES cannot update an existing resource — makeResource → EEXIST → NULL, return ignored, no log. "Add or update" is only "add". (Req 1 / UB-1.1)
3. G-3 (leak): killer/dequeue never free new-op payloads despite ownership transfer (queue, resval_list, rusages, resource/resval rows). Slow per-message leak in a long-lived daemon. (Design risk #1)
4. G-4 (requirement shortfall): job update copies only state — priority/queue/resreq not applied. (Req 3 / UB-3.1)
5. G-5 (requirement shortfall): rusage not stored on job object — only the plugin hook sees it. Requires a rusage field on xce_sched_job_t (none exists today). (Req 6 / UB-6.1)
6. G-6 (test debt): tasks 9.2, 9.3, 9.4, 9.6, 9.7 are checked [x] but absent from test.c. The only new tests are: 9.1 notify-event-name (real), an enum compile-presence check, 9.5 reduced to a NULL-guard exercised against a local stub of sched_update_existing_queue defined inside test.c:25–34 (the real schedobj.c implementation is never linked/tested), and 9.8 reduced to TEST_PASS() with a comment. xce_sched_op_name/sched_op_name has no test (9.1 claims it does). None of the enqueue handlers, dequeue cases, or sched_update_existing_job are tested.
7. G-7 (doc drift): design Component 8 / tasks 1.1, 1.3 say xce_sched_op_name lives in lib/helper — it is a static in scheduler.c. Update spec docs or move the function.

4. Implementation Approach Options

Option A — Extend in place (minimal correctness patch). Fix G-1 (resolve rv->name → resmgr idx, mirroring schedobj.c:284–291), G-2 (on EEXIST find existing resource and mutate fields, or add an updateResource() beside makeResource() in resobj.c; log NULL returns), G-3 (add free_* calls in sched_pend_update_killer per op, reusing the deep-free helpers in src/lib/request.c), G-4 (copy priority/queue/resreq in sched_update_existing_job), and G-7 (doc edit). Defer G-5/G-6.

• ✅ Smallest diff, all in files already touched by this spec; restores correctness fast.
• ❌ Leaves UB-6.1 unmet and test checkboxes false; test debt persists.

Option B — New test program + extracted ops module. Create src/scheduler/testsuite/test_msg_handlers.c (links scheduler objects + generated capnp headers, which test.c cannot include — the stated reason 9.8 became a comment), and optionally extract the five dequeue case bodies into src/scheduler/src/msgops.c with non-static functions so they are unit-testable without including scheduler.c.

• ✅ Makes every acceptance criterion genuinely testable; honest Task 9; aligns with testing steering (tiered test_*.c programs already exist in other daemons).
• ❌ Larger diff; new file + meson.build change; refactor risk in scheduler.c.

Option C — Hybrid (recommended). Phase 1: Option A bug/leak fixes (G-1…G-4, G-7) with TDD where linkable. Phase 2: Option B test program covering handler enqueue + dequeue paths, replacing the stub-based 9.5 and the TEST_PASS 9.8 with real assertions; un-check or re-implement tasks 9.2–9.8 accordingly. Phase 3 (optional, needs decision): G-5 — add a rusages field to xce_sched_job_t or formally amend Req 6 to "deliver to plugins only".

• ✅ Correctness first, honesty in tracking, requirement closure staged.
• ❌ Two-phase coordination; Phase 3 needs a requirements decision.

5. Effort & Risk

Work item	Effort	Risk	Justification
G-1 resvals index fix	S	Low	One-line lookup mirroring existing schedobj.c pattern; resmgr API already used two cases above.
G-2 resource update-on-EEXIST	S–M	Medium	Touching shared libxce resobj.c affects controller too; scheduler-side find-and-mutate is the safer S variant.
G-3 killer frees	M	Medium	Must trace ownership per type against free_* helpers in request.c; wrong free = double-free under ASAN, wrong no-op = leak persists.
G-4 job field updates	S	Low	Copy 3 more fields + string realloc, same shape as queue update.
G-5 store rusage on job	M	Medium	New field on xce_sched_job_t + lifetime/free in kill_sched_job; or amend requirement (zero code).
G-6 real test program	M	Medium	New testsuite binary + meson wiring; capnp header dependency is the known friction that derailed Task 9 the first time.
G-7 doc sync	S	Low	Edit design.md/tasks.md only.
Total (Option C, Phases 1–2)	M (≈4–6 days)	Medium	All within established patterns; risk concentrated in memory-ownership work under ASAN.

6. Recommendations for Design/Implementation Follow-up

1. Treat G-1 and G-2 as defects against the already-approved spec — fix before marking the spec implemented; both silently corrupt or ignore controller state updates.
2. Adopt Option C: in-place fixes first (TDD per project rules — failing test in the new test program first where possible), then the dedicated test_msg_handlers.c to make Task 9 honest. Un-check tasks 9.2–9.8 until real tests exist, or annotate them with the stub caveat.
3. Resolve G-3 with the design's own mitigation table: request struct owns the payload after nullification ⇒ sched_pend_update_killer must deep-free each new op's payload (reuse free_Queue_ptr/free_Resource_ptr-style helpers from src/lib/request.c if exported; otherwise add small free helpers).
4. Decide G-5 explicitly: either add the job-side rusage storage Req 6 promises, or amend UB-6.1 — do not leave the requirement and code disagreeing.
5. Sync design.md Component 8 and tasks 1.1/1.3 with the actual sched_op_name location (G-7).

7. Research Needed

• R-1: Which free_* deep-free helpers in src/lib/request.c are exported and safe for xce_queue_t, xce_resval_list_t, xce_job_rusages_t, xce_resource_t*[] — confirm against ASAN before wiring into the killer.
• R-2: Confirm the notify-driven schedUpdateJob actually reaches sched_update_existing_job(): the dequeue XCE_SCHED_OP_UPDATE_JOB case calls sched_update_jobs() (bulk path); verify sched_update_jobs performs update-in-place for existing jobids (design Component 5 said it should) rather than duplicating jobs.
• R-3: Is there a planned controller-side emitter for Data_data_resources/Data_data_resvals pushes to the scheduler (e.g., post-failover config resync in the persistent_ha work)? If not, decide whether Req 1's handlers stay forward-looking or get an emitter spec.
• R-4: Whether makeInstance(&resobj_type, r) deep-copies all of xce_resource_t (name string included) — determines whether resource row payloads can be freed immediately after makeResource returns.