myxce/.kiro/specs/scheduler-controller-msg-gap/design.md

Design: scheduler-controller-msg-gap

Feature: scheduler-controller-msg-gap Created: 2026-05-11 Status: Design Generated

Architecture Pattern & Boundary Map

Pattern: Extension of existing message handler infrastructure. The scheduler uses a two-phase processing model:

1. Enqueue phase (handler.c, runs on IO thread) — lightweight: validate, allocate request struct, enqueue into pend_updates[0]
2. Dequeue phase (scheduler.c sched_update(), runs on main thread) — heavyweight: mutate scheduler state, call plugin hooks, trigger scheduling sessions

All new handlers follow this pattern. No architectural boundary changes needed.

Scope boundary: Only src/scheduler/ source files, src/lib/helper.c, and src/include/helper.h are modified. No controller-side changes. No Cap'n Proto schema changes.

Technology Stack & Alignment

Aspect	Choice	Rationale
Language	C99	Matches existing codebase convention
IPC protocol	Cap'n Proto (existing)	No schema changes — only handler gaps filled
Build system	Meson (existing)	No build changes needed
Threading model	Same as existing — handler enqueues, main thread dequeues	Preserves thread safety guarantees
Logging	Existing TRACE/INFO/WARN/ERROR macros	Consistent with codebase
Plugin hooks	Follow existing plugin_*_update() pattern	Consistency with established plugin API

Components & Interface Contracts

Component 1: handler.c — Notify handlers (Req 3, 4, 5, 6)

Extend sched_handle_notify() to process the four currently-stubbed NotifyData events.

sched_handle_sched_update_job

Signature: static int sched_handle_sched_update_job(
                xce_scheduler_t* sched,
                xce_job_t* job)

Behavior:
  - If job == NULL, return -EINVAL
  - Enqueue XCE_SCHED_OP_UPDATE_JOB with job pointer
  - On calloc failure, return -ENOMEM
  - Return 0 on success

Requirement traceability: 3.1, 3.2, 3.3

sched_handle_sched_update_queue

Signature: static int sched_handle_sched_update_queue(
                xce_scheduler_t* sched,
                xce_queue_t* queue)

Behavior:
  - If queue == NULL, return -EINVAL
  - Enqueue XCE_SCHED_OP_UPDATE_QUEUE with queue pointer
  - On calloc failure, return -ENOMEM
  - Return 0 on success

Requirement traceability: 4.1, 4.2, 4.3

sched_handle_sched_update_resvals

Signature: static int sched_handle_sched_update_resvals(
                xce_scheduler_t* sched,
                xce_resval_list_t* resvals)

Behavior:
  - If resvals == NULL or count == 0, return 0
  - Enqueue XCE_SCHED_OP_UPDATE_RESVALS with resvals pointer
  - On calloc failure, return -ENOMEM
  - Return 0 on success

Requirement traceability: 5.1, 5.2, 5.3

sched_handle_sched_job_rusage

Signature: static int sched_handle_sched_job_rusage(
                xce_scheduler_t* sched,
                xce_job_rusages_t* rusages)

Behavior:
  - If rusages == NULL, return -EINVAL
  - Enqueue XCE_SCHED_OP_JOB_RUSAGE with rusages pointer
  - On calloc failure, return -ENOMEM
  - Return 0 on success

Requirement traceability: 6.1, 6.2, 6.3

sched_handle_notify switch changes

The existing switch in sched_handle_notify() gains four new cases:

NotifyData event	Handler function	Previous behavior
NotifyData_data_schedUpdateJob	sched_handle_sched_update_job(sched, notify->data.sched_update_job)	TRACE log only
NotifyData_data_schedUpdateQueue	sched_handle_sched_update_queue(sched, notify->data.sched_update_queue)	Empty break
NotifyData_data_schedUpdateResvals	sched_handle_sched_update_resvals(sched, notify->data.sched_update_resvals)	Empty break
NotifyData_data_schedJobRusages	sched_handle_sched_job_rusage(sched, notify->data.sched_job_rusages)	Empty break

Component 2: handler.c — Dataset handlers (Req 1)

sched_handle_resources

Signature: static int sched_handle_resources(
                xce_scheduler_t* sched,
                xce_resource_data_t* resdata)

Behavior:
  - If resdata == NULL or resdata->count == 0, return 0
  - Enqueue XCE_SCHED_OP_UPDATE_RESOURCES with resdata pointer
  - On calloc failure, return -ENOMEM
  - Return 0 on success

Requirement traceability: 1.1, 1.3, 1.4

sched_handle_resvals

Signature: static int sched_handle_resvals(
                xce_scheduler_t* sched,
                xce_resval_data_t* resvaldata)

Behavior:
  - If resvaldata == NULL or resvaldata->count == 0, return 0
  - Enqueue XCE_SCHED_OP_UPDATE_RESVALS_FULL with resvaldata pointer
  - On calloc failure, return -ENOMEM
  - Return 0 on success

Requirement traceability: 1.2, 1.3, 1.4

sched_handle_dataset switch changes

Data table	Handler function	Previous behavior
Data_data_resources	sched_handle_resources(sched, dataset->data.resources)	Empty break
Data_data_resvals	sched_handle_resvals(sched, dataset->data.resvals)	Empty break

Component 3: scheduler.h — New op types and update request fields

Extend xce_sched_op_t and xce_sched_update_request_t:

typedef enum {
    XCE_SCHED_OP_NONE = 0,
    XCE_SCHED_OP_UPDATE_CONFIG,
    XCE_SCHED_OP_UPDATE_JOB,
    XCE_SCHED_OP_JSTATE_CHANGE,
    XCE_SCHED_OP_UPDATE_WORKER,
    XCE_SCHED_OP_UPDATE_QUEUE,        // NEW
    XCE_SCHED_OP_UPDATE_RESOURCES,    // NEW
    XCE_SCHED_OP_UPDATE_RESVALS_FULL, // NEW — from dataset bulk resvals
    XCE_SCHED_OP_UPDATE_RESVALS,      // NEW — from notify incremental resvals
    XCE_SCHED_OP_JOB_RUSAGE           // NEW
} xce_sched_op_t;

Extend xce_sched_update_request_t union:

struct {
    xce_queue_t*          queue;        // for UPDATE_QUEUE
} queueupdate;
struct {
    xce_resource_data_t*  resdata;      // for UPDATE_RESOURCES
} resourceupdate;
struct {
    xce_resval_data_t*    resvaldata;  // for UPDATE_RESVALS_FULL
} resvalfullupdate;
struct {
    xce_resval_list_t*    resvals;      // for UPDATE_RESVALS (notify)
} resvalupdate;
struct {
    xce_job_rusages_t*    rusages;      // for JOB_RUSAGE
} job_rusage;

Component 4: scheduler.c — Dequeue processing in sched_update()

Extend the switch (req->op) in sched_update() with new cases:

XCE_SCHED_OP_UPDATE_QUEUE

• Find existing queue by name with sched_get_queue()
• If found: update mutable fields (status, priority, accept_interval, sched_delay, resreq)
• If not found: return -ENOENT — queues are created only via the initial Conf4Sched config push, never via notify
• Call plugin_queue_update()

Requirement traceability: 4.1, 4.2

XCE_SCHED_OP_UPDATE_RESOURCES

• Iterate resdata->resources[]
• For each: call makeResource(sched->resmgr, r) to add or update
• NULL entries skipped (1.3)

Requirement traceability: 1.1

XCE_SCHED_OP_UPDATE_RESVALS_FULL

• Iterate resvaldata->vals[]
• For each: find resource by name in sched->resmgr, update value
• Unknown resources skipped with WARN log

Requirement traceability: 1.2

XCE_SCHED_OP_UPDATE_RESVALS

• Iterate resvals->vals[]
• For each: find worker by host name via sched_get_worker()
• If found: update worker's resvals zarray at the resource index
• Unknown hosts skipped (5.2)

Requirement traceability: 5.1, 5.2, 5.3

XCE_SCHED_OP_JOB_RUSAGE

• Find job by jobid via sched_get_job()
• If not found: return -ENOENT (6.2)
• Store rusage data on job object
• Call plugin_job_rusage_update()

Requirement traceability: 6.1, 6.2

Component 5: scheduler.c — Job update-in-place (Req 3)

New function: sched_update_existing_job

Signature: static int sched_update_existing_job(
                xce_scheduler_t* sched,
                xce_job_t* updated)

Behavior:
  - Find existing job by jobid via sched_get_job()
  - If not found: return -ENOENT — jobs are created only via
    the Data_data_schedjobs dataset push, never via notify
  - If found: update mutable fields (state, priority, allocs)
  - Call plugin_job_update()
  - Return 0 on success, negative on error

Requirement traceability: 3.1, 3.2, 3.3

This is wired into the existing XCE_SCHED_OP_UPDATE_JOB case — no new op type needed since sched_on_jobs() already creates new jobs. The change is to make sched_update_jobs() also handle job updates for existing jobids by replacing rather than just adding.

Component 6: plugin.h / plugin.c — New rusage hook (Req 6)

Add to plugin_hooks_t:

typedef int (*plugin_job_rusage_hook_t)(void*, xce_sched_job_t*, xce_job_rusages_t*);

Add to plugin_hooks_t struct:

plugin_job_rusage_hook_t  rusage_hook;

Add global function:

int plugin_job_rusage_update(void* sched, xce_sched_job_t* job, xce_job_rusages_t* rusages);

This follows the exact pattern of plugin_job_state_change() — iterate registered plugins, call their hook if non-NULL.

Component 7: handler.c — Dead code removal (Req 2)

Remove from scheduler_req_handler():

• The entire Request_body_pushmsg case block (lines 185–198)
• The static functions sched_handle_jobpush() and sched_handle_workerpush()

These are unreachable because vtask_entry() in src/lib/xcectx.c rewrites all pushmsg into datamsg.

Requirement traceability: 2.1, 2.2

Component 8: Name-lookup functions (Req 8)

Layering correction (bug-765 G-7, 2026-06-18): these two helpers live in DIFFERENT layers. xce_notify_event_name(int) is lib-level and lives in src/lib/helper.c / src/include/helper.h. The scheduler-op name lookup operates on xce_sched_op_t, a scheduler-only type (scheduler.h); src/lib must not include scheduler headers (no layering inversion), so it is a static sched_op_name(xce_sched_op_t) in src/scheduler/src/scheduler.c, NOT xce_sched_op_name in lib/helper. The original "both in helper.c" text below is superseded for the scheduler-op helper.

New function: xce_notify_event_name (lib — src/lib/helper.c)

Signature: char* xce_notify_event_name(int event)

Behavior: Switch onNotifyData_data_* enum values, return symbolic string.
Fallback: "UNKNOWN(%d)" with numeric substitution.

Matches pattern of xce_dataset_table() — takes int, returns char*.

Scheduler-op name lookup: static sched_op_name (scheduler — src/scheduler/src/scheduler.c)

Signature: static char* sched_op_name(xce_sched_op_t op)

Behavior: Switch on XCE_SCHED_OP_* enum values, return symbolic string.
Fallback: "UNKNOWN(%d)" with numeric substitution.

Kept in the scheduler layer (NOT lib/helper as xce_sched_op_name) because xce_sched_op_t is a scheduler-only type and src/lib must not depend on scheduler headers — see the layering correction above. Required for Requirement 7's error logging in sched_update().

Changes to handler.c logging

Current	Changed to
TRACE("%s table = %d", __func__, dataset->table)	TRACE("%s table = %s", __func__, xce_dataset_table(dataset->table))
TRACE("%s...", __func__) (in stub handlers)	INFO("%s event = %s jobid = %s", __func__, xce_notify_event_name(notify->event), jobidString)

Requirement traceability: 8.1, 8.2, 8.3, 8.4

Component 9: scheduler.c — Error logging in sched_update() (Req 7)

In the sched_update() main loop, after the switch (req->op) block, when rc < 0:

WARN("%s: op %s failed: %s (%d)", __func__,
     sched_op_name(req->op), strerror(-rc), rc);

Requirement traceability: 7.1

Component 10: schedobj.c — Queue update function (Req 4)

New function: sched_update_existing_queue

Signature: int sched_update_existing_queue(
                xce_scheduler_t* sched,
                xce_queue_t* updated)

Behavior:
  - Find existing queue by name via sched_get_queue()
  - If not found: return -ENOENT — queues are created only via
    the initial Conf4Sched config push, never via notify
  - If found: update mutable fields:
      sched_queue->status = updated->status
      sched_queue->priority = updated->priority
      sched_queue->accept_interval = updated->accept_interval
      sched_queue->sched_delay = updated->sched_delay
      // resreq is a string field — reallocate and replace
  - Call plugin_queue_update()
  - Return 0 on success, negative on error

Requirement traceability: 4.1, 4.2

Component 11: sched_pend_update_killer updates (scheduler.c)

Extend the sched_pend_update_killer destructor to handle new op types:

case XCE_SCHED_OP_UPDATE_QUEUE:
    // queue pointer belongs to scheduler, no free needed
    break;
case XCE_SCHED_OP_UPDATE_RESOURCES:
    // resource_data rows belong to scheduler, no free needed
    break;
case XCE_SCHED_OP_UPDATE_RESVALS_FULL:
    // resval_data rows owned by capnp decode, freed by free_Request
    break;
case XCE_SCHED_OP_UPDATE_RESVALS:
    // resval_list owned by capnp decode, freed by free_Request
    break;
case XCE_SCHED_OP_JOB_RUSAGE:
    // rusage struct owned by capnp decode, freed by free_Request
    break;

Important: Memory ownership must be traced carefully. For XCE_SCHED_OP_UPDATE_JOB and XCE_SCHED_OP_UPDATE_WORKER, the existing pattern nullifies dataset->nrows and dataset->rows after enqueue to transfer ownership. New queue/rusage/resvals handlers should follow the same pattern — the request struct owns the data, and free_Request() in xcectx.c handles cleanup.

Data Flow Diagrams

Notify message processing flow

Controller (sched.c)                    Scheduler (handler.c)              Scheduler (scheduler.c)
─────────────────────                   ─────────────────────              ─────────────────────
xce_notify_req()
    │
    ├── schedStartJob ──────────► sched_job_state_change() ──► enq_pend_update(JSTATE_CHANGE)
    │                                                                     │
    ├── schedExitJob ───────────► sched_job_state_change() ──►           │
    │                                                                     │
    ├── schedFailJob ───────────► sched_job_state_change() ──►           │
    │                                                                     │
    ├── schedUpdateJob ────────► sched_handle_sched_update_job() ─► enq_pend_update(UPDATE_JOB)     [NEW]
    │                                                                     │
    ├── schedUpdateQueue ──────► sched_handle_sched_update_queue() ─► enq_pend_update(UPDATE_QUEUE)  [NEW]
    │                                                                     │
    ├── schedUpdateResvals ────► sched_handle_sched_update_resvals() ─► enq_pend_update(UPDATE_RESVALS) [NEW]
    │                                                                     │
    └── schedJobRusages ───────► sched_handle_sched_job_rusage() ──► enq_pend_update(JOB_RUSAGE)     [NEW]
                                                                          │
                                                                          ▼
                                                                    sched_update() main loop
                                                                      ├── UPDATE_JOB ──► sched_update_existing_job()
                                                                      ├── UPDATE_QUEUE ──► sched_update_existing_queue()
                                                                      ├── UPDATE_RESVALS ──► update worker resvals
                                                                      ├── UPDATE_RESOURCES ──► update resmgr [NEW]
                                                                      ├── UPDATE_RESVALS_FULL ──► update sharedInst [NEW]
                                                                      └── JOB_RUSAGE ──► store on job + plugin hook [NEW]

Dataset message processing flow

Controller (dataex.c)                    Scheduler (handler.c)              Scheduler (scheduler.c)
────────────────────                    ─────────────────────              ─────────────────────
xce_push_data_req()
    │
    ├── Data_data_schedjobs ──► sched_on_jobs() ──────────► enq_pend_update(UPDATE_JOB)
    │                                                                     │
    ├── Data_data_schedworkers ► sched_on_workers() ───────► enq_pend_update(UPDATE_WORKER)
    │                                                                     │
    ├── Data_data_resources ──► sched_handle_resources() ─► enq_pend_update(UPDATE_RESOURCES)     [NEW]
    │                                                                     │
    └── Data_data_resvals ────► sched_handle_resvals() ───► enq_pend_update(UPDATE_RESVALS_FULL) [NEW]
                                                                          │
                                                                          ▼
                                                                    sched_update() main loop

File Change Summary

File	Change Type	Requirements
src/scheduler/src/handler.c	Modified — add handlers, remove dead code, update logging	1, 2, 3, 4, 5, 6, 8
src/scheduler/include/scheduler.h	Modified — extend enums and structs	1, 3, 4, 5, 6
src/scheduler/src/scheduler.c	Modified — add dequeue cases, error logging, update functions	1, 3, 4, 5, 6, 7
src/scheduler/include/plugin.h	Modified — add rusage hook typedef and global function	6
src/scheduler/src/plugin.c	Modified — add plugin_job_rusage_update() implementation	6
src/scheduler/src/schedobj.c	Modified — add sched_update_existing_queue()	4
src/lib/helper.c	Modified — add xce_notify_event_name() (lib-level, int arg); the scheduler-op name lookup is a static sched_op_name() in scheduler.c instead — see Component 8 layering note	8
src/include/helper.h	Modified — declare xce_notify_event_name()	8
src/scheduler/testsuite/test.c	Modified — add unit tests for all new handlers	All

Build Order

1. src/include/helper.h — add declarations (no dependencies)
2. src/lib/helper.c — add implementations (depends on step 1)
3. src/scheduler/include/scheduler.h — extend enums and structs (depends on typemap headers)
4. src/scheduler/include/plugin.h — add rusage hook typedef (depends on xce/types.h)
5. src/scheduler/src/plugin.c — add plugin_job_rusage_update() (depends on step 4)
6. src/scheduler/src/schedobj.c — add sched_update_existing_queue() (depends on step 3)
7. src/scheduler/src/scheduler.c — add dequeue processing, error logging, update functions (depends on steps 3–6)
8. src/scheduler/src/handler.c — add handlers, remove dead code, update logging (depends on steps 1–7)
9. src/scheduler/testsuite/test.c — add unit tests (depends on all above)

Risks & Mitigations

Risk	Severity	Mitigation
Memory ownership of queued data — capnp-decoded structs freed before dequeue processes them	High	Follow existing pattern: for dataset paths, nullify dataset->nrows/dataset->rows after enqueue to transfer ownership. For notify paths, copy essential data into the request struct rather than holding pointers to capnp memory.
sched_update_existing_job must handle concurrent modification — handler thread enqueues while main thread processes	Medium	Already mitigated by existing double-buffer swap in sched_update(). The handler thread only touches pend_updates[0] while the main thread drains pend_updates[1].
New UPDATE_QUEUE op requires queue name lookup — potential lock contention	Low	sched_get_queue() already uses read lock. Queue updates are infrequent.
sched_op_name (static, scheduler.c) must handle future enum additions	Low	Follow xce_dataset_table pattern — return "UNKNOWN(%d)" for unrecognized values.