myxce/.kiro/specs/scheduler-controller-msg-gap/research.md

Research Log: scheduler-controller-msg-gap

Feature: scheduler-controller-msg-gap Created: 2026-05-11

Discovery Type

Extension (light discovery) — extending existing handler code in an established C codebase.

Summary

The scheduler's handler.c dispatcher processes controller messages via scheduler_req_handler. Six message cases are stubbed out or empty-bodied. The vtask_entry() assembly layer in src/lib/xcectx.c rewrites all pushmsg into datamsg, making the Request_body_pushmsg branch dead code. Existing plugin hooks (plugin_job_update, plugin_queue_update, plugin_worker_update, plugin_job_state_change) exist for notifying plugins of state changes. A plugin_job_rusage_update hook does not yet exist and must be added. The src/lib/helper.c file already provides xce_request_op_name(), xce_request_msg_name(), and xce_dataset_table() — a new xce_notify_event_name() function is needed.

Research Log

Topic 1: Plugin Hook System

Source: src/scheduler/include/plugin.h, src/scheduler/src/plugin.c

Findings:

• Hooks follow a plugin_*_update(void* sched, xce_sched_*_t*) pattern
• Called after state mutation to notify registered plugins
• Hooks iterate registered plugins and call their function pointers
• plugin_job_state_change(void*, xce_sched_job_t*, uint32_t) takes an extra state parameter
• No plugin_job_rusage_update hook exists yet — must be added

Implication: New handlers should call the corresponding plugin hook after mutating scheduler state. The rusage handler needs a new hook typedef and global function.

Topic 2: Object Conversion Functions

Source: src/scheduler/src/schedobj.c

Findings:

• make_sched_job(void* arg, xce_job_t* j) — maps jobid, times, state, user/queue/project pointers, resreq, mintasks/maxtasks, allocs, accounts
• make_sched_queue(void* arg, xce_queue_t* q) — maps name, attrs, status, priority, accept_interval, sched_delay, resreq
• make_sched_worker(void* arg, xce_worker_t* w) — maps name, status, idx, resvals zarray from w->resvals_to_sched
• kill_sched_* functions free their respective objects

Implication: Job/queue/worker update handlers should use make_sched_* for create-or-update semantics. For updates (existing objects), we need update-in-place logic since make_sched_* always creates new objects.

Topic 3: Resource and Shared Instance Initialization

Source: src/scheduler/src/scheduler.c

Findings:

• sched_init_resources() — iterates config->resources[], calls makeResource(sched->resmgr, r) for each, warns on NULL/duplicate
• sched_init_sharedInst() — iterates config->resvals[], uses MOVE_PTR to steal val pointers into sched->sharedInst zarray
• Both called from sched_init_config() during initial config push
• For runtime updates, we need incremental versions that can add new or update existing resources

Implication: Dataset handlers for Data_data_resources and Data_data_resvals need new update functions that can both create and modify existing entries.

Topic 4: Existing Helper Functions

Source: src/lib/helper.c, src/include/helper.h

Findings:

• xce_request_op_name(xce_request_t* req) — returns string for OPCode enum
• xce_request_msg_name(xce_request_t* req) — returns string for Request body tag
• xce_dataset_table(int table) — returns string for Data table type enum
• All return "unknown" for unrecognized values
• xce_notify_event_name() does NOT exist yet — must be created
• Signature pattern: char* func_name(enumerated_type_or_struct_ptr) returning char*

Implication: Create xce_notify_event_name(int event) following same pattern. Also consider char* xce_sched_op_name(xce_sched_op_t op) for the update request op enum.

Topic 5: Cap'n Proto Enum Values (NotifyData)

Source: src/schema/controller.capnp

Findings: All 19 NotifyData event tags:

• jobPidInfo(0), jobFailInfo(1), jobDispatchFail(2), jobFinishInfo(3), schedStartJob(4), schedExitJob(5), schedFailJob(6), schedUpdateJob(7), schedUpdateResvals(8), stoppedJob(9), resumedJob(10), ckptinfo(11), schedUpdateQueue(12), taskPidInfo(13), taskFailInfo(14), taskFinish(15), postDoneJob(16), schedJobRusages(17), updateJob(18)

Implication: The xce_notify_event_name() function must cover all 19 values. Only the scheduler-relevant ones need handler implementations (schedUpdateJob, schedUpdateQueue, schedUpdateResvals, schedJobRusages are the gap items).

Topic 6: Dead Code Analysis

Source: src/lib/xcectx.c lines 93-106, src/scheduler/src/handler.c lines 172-197

Findings:

• vtask_entry() accumulates pushmsg rows into ctx->dataset, then rewrites body_which = Request_body_datamsg and body.datamsg = ctx->dataset
• The Request_body_pushmsg case in scheduler_req_handler (lines 185-198) calls sched_handle_jobpush() and sched_handle_workerpush(), both of which are stubs returning 0
• These functions can never be reached via normal message flow

Implication: Safe to remove the pushmsg case branch and both stub functions.

Architecture Decisions

1. Handler pattern: New notify handlers follow the same enqueue-to-pend-updates pattern as sched_on_job_state_change — create a xce_sched_update_request_t, set the op, enqueue via enq_pend_update(), process in sched_update() main loop.

2. Update-in-place for job/queue/worker: For NotifyData update events, we need update semantics (find existing, mutate fields) rather than create semantics (make_sched_*). New sched_update_job, sched_update_queue, sched_update_worker functions will handle this.

3. New XCE_SCHED_OP enum values: Need XCE_SCHED_OP_UPDATE_RESVALS for resource value updates and XCE_SCHED_OP_JOB_RUSAGE for rusage updates.

4. Plugin hook for rusage: Add plugin_job_rusage_hook_t typedef and plugin_job_rusage_update() global function following existing patterns.

5. Helper function: Add xce_notify_event_name(int) and xce_sched_op_name(xce_sched_op_t) to src/lib/helper.c and src/include/helper.h.

---

Cross-Spec Gap Audit (2026-06-08)

Batch /kiro-validate-gap across 9 specs on persistent_ha branch.

Analysis Summary

• scheduler-controller-msg-gap is ~100% complete. All 9 task groups, 28 subtasks (1.1–9.9) are [x] (9.9 optional).
• No remaining gaps: All 8 requirements have verified implementation. The 6 previously stubbed notify/dataset handlers (schedUpdateJob, schedUpdateQueue, schedUpdateResvals, schedJobRusages, Data_data_resources, Data_data_resvals) are wired with full business logic. Dead code (pushmsg branches, sched_handle_jobpush, sched_handle_workerpush) removed. Symbolic logging via xce_notify_event_name() and xce_sched_op_name() is in place.
• Optional gap: Task 9.9 (verify WARN log when handler returns non-zero) is marked optional — requires log capture infrastructure not yet present in the test harness.
• Cross-spec relationship: resval-id-decoupling uses the same pattern (name-based resource lookup via sched->resmgr) in sched_init_sharedInst. The XCE_SCHED_OP_UPDATE_RESVALS_FULL handler added in this spec also resolves by name — consistency is maintained.
• Recommendation: The spec can be marked implemented. Task 9.9 can be closed as deferred.