myxce/.kiro/specs/scheduler-controller-msg-gap/research.md

# Research Log: scheduler-controller-msg-gap

> Feature: scheduler-controller-msg-gap
> Created: 2026-05-11

## Discovery Type

Extension (light discovery) — extending existing handler code in an established C codebase.

## Summary

The scheduler's `handler.c` dispatcher processes controller messages via `scheduler_req_handler`. Six message cases are stubbed out or empty-bodied. The `vtask_entry()` assembly layer in `src/lib/xcectx.c` rewrites all `pushmsg` into `datamsg`, making the `Request_body_pushmsg` branch dead code. Existing plugin hooks (`plugin_job_update`, `plugin_queue_update`, `plugin_worker_update`, `plugin_job_state_change`) exist for notifying plugins of state changes. A `plugin_job_rusage_update` hook does not yet exist and must be added. The `src/lib/helper.c` file already provides `xce_request_op_name()`, `xce_request_msg_name()`, and `xce_dataset_table()` — a new `xce_notify_event_name()` function is needed.

## Research Log

### Topic 1: Plugin Hook System

**Source:** `src/scheduler/include/plugin.h`, `src/scheduler/src/plugin.c`

**Findings:**
- Hooks follow a `plugin_*_update(void* sched, xce_sched_*_t*)` pattern
- Called after state mutation to notify registered plugins
- Hooks iterate registered plugins and call their function pointers
- `plugin_job_state_change(void*, xce_sched_job_t*, uint32_t)` takes an extra state parameter
- No `plugin_job_rusage_update` hook exists yet — must be added

**Implication:** New handlers should call the corresponding plugin hook after mutating scheduler state. The rusage handler needs a new hook typedef and global function.

### Topic 2: Object Conversion Functions

**Source:** `src/scheduler/src/schedobj.c`

**Findings:**
- `make_sched_job(void* arg, xce_job_t* j)` — maps jobid, times, state, user/queue/project pointers, resreq, mintasks/maxtasks, allocs, accounts
- `make_sched_queue(void* arg, xce_queue_t* q)` — maps name, attrs, status, priority, accept_interval, sched_delay, resreq
- `make_sched_worker(void* arg, xce_worker_t* w)` — maps name, status, idx, resvals zarray from `w->resvals_to_sched`
- `kill_sched_*` functions free their respective objects

**Implication:** Job/queue/worker update handlers should use `make_sched_*` for create-or-update semantics. For updates (existing objects), we need update-in-place logic since `make_sched_*` always creates new objects.

### Topic 3: Resource and Shared Instance Initialization

**Source:** `src/scheduler/src/scheduler.c`

**Findings:**
- `sched_init_resources()` — iterates `config->resources[]`, calls `makeResource(sched->resmgr, r)` for each, warns on NULL/duplicate
- `sched_init_sharedInst()` — iterates `config->resvals[]`, uses `MOVE_PTR` to steal val pointers into `sched->sharedInst` zarray
- Both called from `sched_init_config()` during initial config push
- For runtime updates, we need incremental versions that can add new or update existing resources

**Implication:** Dataset handlers for `Data_data_resources` and `Data_data_resvals` need new update functions that can both create and modify existing entries.

### Topic 4: Existing Helper Functions

**Source:** `src/lib/helper.c`, `src/include/helper.h`

**Findings:**
- `xce_request_op_name(xce_request_t* req)` — returns string for OPCode enum
- `xce_request_msg_name(xce_request_t* req)` — returns string for Request body tag
- `xce_dataset_table(int table)` — returns string for Data table type enum
- All return `"unknown"` for unrecognized values
- `xce_notify_event_name()` does NOT exist yet — must be created
- Signature pattern: `char* func_name(enumerated_type_or_struct_ptr)` returning `char*`

**Implication:** Create `xce_notify_event_name(int event)` following same pattern. Also consider `char* xce_sched_op_name(xce_sched_op_t op)` for the update request op enum.

### Topic 5: Cap'n Proto Enum Values (NotifyData)

**Source:** `src/schema/controller.capnp`

**Findings:**
All 19 NotifyData event tags:
- `jobPidInfo(0)`, `jobFailInfo(1)`, `jobDispatchFail(2)`, `jobFinishInfo(3)`, `schedStartJob(4)`, `schedExitJob(5)`, `schedFailJob(6)`, `schedUpdateJob(7)`, `schedUpdateResvals(8)`, `stoppedJob(9)`, `resumedJob(10)`, `ckptinfo(11)`, `schedUpdateQueue(12)`, `taskPidInfo(13)`, `taskFailInfo(14)`, `taskFinish(15)`, `postDoneJob(16)`, `schedJobRusages(17)`, `updateJob(18)`

**Implication:** The `xce_notify_event_name()` function must cover all 19 values. Only the scheduler-relevant ones need handler implementations (schedUpdateJob, schedUpdateQueue, schedUpdateResvals, schedJobRusages are the gap items).

### Topic 6: Dead Code Analysis

**Source:** `src/lib/xcectx.c` lines 93-106, `src/scheduler/src/handler.c` lines 172-197

**Findings:**
- `vtask_entry()` accumulates `pushmsg` rows into `ctx->dataset`, then rewrites `body_which = Request_body_datamsg` and `body.datamsg = ctx->dataset`
- The `Request_body_pushmsg` case in `scheduler_req_handler` (lines 185-198) calls `sched_handle_jobpush()` and `sched_handle_workerpush()`, both of which are stubs returning 0
- These functions can never be reached via normal message flow

**Implication:** Safe to remove the pushmsg case branch and both stub functions.

## Architecture Decisions

1. **Handler pattern:** New notify handlers follow the same enqueue-to-pend-updates pattern as `sched_on_job_state_change` — create a `xce_sched_update_request_t`, set the op, enqueue via `enq_pend_update()`, process in `sched_update()` main loop.

2. **Update-in-place for job/queue/worker:** For NotifyData update events, we need update semantics (find existing, mutate fields) rather than create semantics (make_sched_*). New `sched_update_job`, `sched_update_queue`, `sched_update_worker` functions will handle this.

3. **New XCE_SCHED_OP enum values:** Need `XCE_SCHED_OP_UPDATE_RESVALS` for resource value updates and `XCE_SCHED_OP_JOB_RUSAGE` for rusage updates.

4. **Plugin hook for rusage:** Add `plugin_job_rusage_hook_t` typedef and `plugin_job_rusage_update()` global function following existing patterns.

5. **Helper function:** Add `xce_notify_event_name(int)` and `xce_sched_op_name(xce_sched_op_t)` to `src/lib/helper.c` and `src/include/helper.h`.

---

# Cross-Spec Gap Audit (2026-06-08)

> Batch `/kiro-validate-gap` across 9 specs on persistent_ha branch.

## Analysis Summary

- **scheduler-controller-msg-gap** is **~100% complete**. All 9 task groups, 28 subtasks (1.1–9.9) are `[x]` (9.9 optional).
- **No remaining gaps**: All 8 requirements have verified implementation. The 6 previously stubbed notify/dataset handlers (`schedUpdateJob`, `schedUpdateQueue`, `schedUpdateResvals`, `schedJobRusages`, `Data_data_resources`, `Data_data_resvals`) are wired with full business logic. Dead code (`pushmsg` branches, `sched_handle_jobpush`, `sched_handle_workerpush`) removed. Symbolic logging via `xce_notify_event_name()` and `xce_sched_op_name()` is in place.
- **Optional gap**: Task 9.9 (verify WARN log when handler returns non-zero) is marked optional — requires log capture infrastructure not yet present in the test harness.
- **Cross-spec relationship**: `resval-id-decoupling` uses the same pattern (name-based resource lookup via `sched->resmgr`) in `sched_init_sharedInst`. The `XCE_SCHED_OP_UPDATE_RESVALS_FULL` handler added in this spec also resolves by name — consistency is maintained.
- **Recommendation**: The spec can be marked `implemented`. Task 9.9 can be closed as deferred.