lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

add kiro-sdd documents

Browse source
This commit is contained in:
Rongsong Shen 2026-05-13 10:58:51 +00:00
parent e847a32546
commit 033f34c07f
10 changed files with 1758 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

469
.kiro/specs/scheduler-controller-msg-gap/design.md Normal file
View file

@ -0,0 +1,469 @@
# Design: scheduler-controller-msg-gap
> Feature: scheduler-controller-msg-gap
> Created: 2026-05-11
> Status: Design Generated
## Architecture Pattern & Boundary Map
**Pattern:** Extension of existing message handler infrastructure. The scheduler uses a two-phase processing model:
1. **Enqueue phase** (handler.c, runs on IO thread) — lightweight: validate, allocate request struct, enqueue into `pend_updates[0]`
2. **Dequeue phase** (scheduler.c `sched_update()`, runs on main thread) — heavyweight: mutate scheduler state, call plugin hooks, trigger scheduling sessions
All new handlers follow this pattern. No architectural boundary changes needed.
**Scope boundary:** Only `src/scheduler/` source files, `src/lib/helper.c`, and `src/include/helper.h` are modified. No controller-side changes. No Cap'n Proto schema changes.
## Technology Stack & Alignment
| Aspect | Choice | Rationale |
|--------|--------|-----------|
| Language | C99 | Matches existing codebase convention |
| IPC protocol | Cap'n Proto (existing) | No schema changes — only handler gaps filled |
| Build system | Meson (existing) | No build changes needed |
| Threading model | Same as existing — handler enqueues, main thread dequeues | Preserves thread safety guarantees |
| Logging | Existing `TRACE`/`INFO`/`WARN`/`ERROR` macros | Consistent with codebase |
| Plugin hooks | Follow existing `plugin_*_update()` pattern | Consistency with established plugin API |
## Components & Interface Contracts
### Component 1: handler.c — Notify handlers (Req 3, 4, 5, 6)
Extend `sched_handle_notify()` to process the four currently-stubbed NotifyData events.
#### `sched_handle_sched_update_job`
```
Signature: static int sched_handle_sched_update_job(
xce_scheduler_t* sched,
xce_job_t* job)
Behavior:
- If job == NULL, return -EINVAL
- Enqueue XCE_SCHED_OP_UPDATE_JOB with job pointer
- On calloc failure, return -ENOMEM
- Return 0 on success
```
**Requirement traceability:** 3.1, 3.2, 3.3
#### `sched_handle_sched_update_queue`
```
Signature: static int sched_handle_sched_update_queue(
xce_scheduler_t* sched,
xce_queue_t* queue)
Behavior:
- If queue == NULL, return -EINVAL
- Enqueue XCE_SCHED_OP_UPDATE_QUEUE with queue pointer
- On calloc failure, return -ENOMEM
- Return 0 on success
```
**Requirement traceability:** 4.1, 4.2, 4.3
#### `sched_handle_sched_update_resvals`
```
Signature: static int sched_handle_sched_update_resvals(
xce_scheduler_t* sched,
xce_resval_list_t* resvals)
Behavior:
- If resvals == NULL or count == 0, return 0
- Enqueue XCE_SCHED_OP_UPDATE_RESVALS with resvals pointer
- On calloc failure, return -ENOMEM
- Return 0 on success
```
**Requirement traceability:** 5.1, 5.2, 5.3
#### `sched_handle_sched_job_rusage`
```
Signature: static int sched_handle_sched_job_rusage(
xce_scheduler_t* sched,
xce_job_rusages_t* rusages)
Behavior:
- If rusages == NULL, return -EINVAL
- Enqueue XCE_SCHED_OP_JOB_RUSAGE with rusages pointer
- On calloc failure, return -ENOMEM
- Return 0 on success
```
**Requirement traceability:** 6.1, 6.2, 6.3
#### `sched_handle_notify` switch changes
The existing switch in `sched_handle_notify()` gains four new cases:
| NotifyData event | Handler function | Previous behavior |
|---|---|---|
| `NotifyData_data_schedUpdateJob` | `sched_handle_sched_update_job(sched, notify->data.sched_update_job)` | TRACE log only |
| `NotifyData_data_schedUpdateQueue` | `sched_handle_sched_update_queue(sched, notify->data.sched_update_queue)` | Empty `break` |
| `NotifyData_data_schedUpdateResvals` | `sched_handle_sched_update_resvals(sched, notify->data.sched_update_resvals)` | Empty `break` |
| `NotifyData_data_schedJobRusages` | `sched_handle_sched_job_rusage(sched, notify->data.sched_job_rusages)` | Empty `break` |
### Component 2: handler.c — Dataset handlers (Req 1)
#### `sched_handle_resources`
```
Signature: static int sched_handle_resources(
xce_scheduler_t* sched,
xce_resource_data_t* resdata)
Behavior:
- If resdata == NULL or resdata->count == 0, return 0
- Enqueue XCE_SCHED_OP_UPDATE_RESOURCES with resdata pointer
- On calloc failure, return -ENOMEM
- Return 0 on success
```
**Requirement traceability:** 1.1, 1.3, 1.4
#### `sched_handle_resvals`
```
Signature: static int sched_handle_resvals(
xce_scheduler_t* sched,
xce_resval_data_t* resvaldata)
Behavior:
- If resvaldata == NULL or resvaldata->count == 0, return 0
- Enqueue XCE_SCHED_OP_UPDATE_RESVALS_FULL with resvaldata pointer
- On calloc failure, return -ENOMEM
- Return 0 on success
```
**Requirement traceability:** 1.2, 1.3, 1.4
#### `sched_handle_dataset` switch changes
| Data table | Handler function | Previous behavior |
|---|---|---|
| `Data_data_resources` | `sched_handle_resources(sched, dataset->data.resources)` | Empty `break` |
| `Data_data_resvals` | `sched_handle_resvals(sched, dataset->data.resvals)` | Empty `break` |
### Component 3: scheduler.h — New op types and update request fields
Extend `xce_sched_op_t` and `xce_sched_update_request_t`:
```
typedef enum {
XCE_SCHED_OP_NONE = 0,
XCE_SCHED_OP_UPDATE_CONFIG,
XCE_SCHED_OP_UPDATE_JOB,
XCE_SCHED_OP_JSTATE_CHANGE,
XCE_SCHED_OP_UPDATE_WORKER,
XCE_SCHED_OP_UPDATE_QUEUE, // NEW
XCE_SCHED_OP_UPDATE_RESOURCES, // NEW
XCE_SCHED_OP_UPDATE_RESVALS_FULL, // NEW — from dataset bulk resvals
XCE_SCHED_OP_UPDATE_RESVALS, // NEW — from notify incremental resvals
XCE_SCHED_OP_JOB_RUSAGE // NEW
} xce_sched_op_t;
```
Extend `xce_sched_update_request_t` union:
```
struct {
xce_queue_t* queue; // for UPDATE_QUEUE
} queueupdate;
struct {
xce_resource_data_t* resdata; // for UPDATE_RESOURCES
} resourceupdate;
struct {
xce_resval_data_t* resvaldata; // for UPDATE_RESVALS_FULL
} resvalfullupdate;
struct {
xce_resval_list_t* resvals; // for UPDATE_RESVALS (notify)
} resvalupdate;
struct {
xce_job_rusages_t* rusages; // for JOB_RUSAGE
} job_rusage;
```
### Component 4: scheduler.c — Dequeue processing in `sched_update()`
Extend the `switch (req->op)` in `sched_update()` with new cases:
#### `XCE_SCHED_OP_UPDATE_QUEUE`
- Find existing queue by name with `sched_get_queue()`
- If found: update mutable fields (status, priority, accept_interval, sched_delay, resreq)
- If not found: return `-ENOENT` — queues are created only via the initial `Conf4Sched` config push, never via notify
- Call `plugin_queue_update()`
**Requirement traceability:** 4.1, 4.2
#### `XCE_SCHED_OP_UPDATE_RESOURCES`
- Iterate `resdata->resources[]`
- For each: call `makeResource(sched->resmgr, r)` to add or update
- NULL entries skipped (1.3)
**Requirement traceability:** 1.1
#### `XCE_SCHED_OP_UPDATE_RESVALS_FULL`
- Iterate `resvaldata->vals[]`
- For each: find resource by name in `sched->resmgr`, update value
- Unknown resources skipped with WARN log
**Requirement traceability:** 1.2
#### `XCE_SCHED_OP_UPDATE_RESVALS`
- Iterate `resvals->vals[]`
- For each: find worker by host name via `sched_get_worker()`
- If found: update worker's `resvals` zarray at the resource index
- Unknown hosts skipped (5.2)
**Requirement traceability:** 5.1, 5.2, 5.3
#### `XCE_SCHED_OP_JOB_RUSAGE`
- Find job by jobid via `sched_get_job()`
- If not found: return `-ENOENT` (6.2)
- Store rusage data on job object
- Call `plugin_job_rusage_update()`
**Requirement traceability:** 6.1, 6.2
### Component 5: scheduler.c — Job update-in-place (Req 3)
#### New function: `sched_update_existing_job`
```
Signature: static int sched_update_existing_job(
xce_scheduler_t* sched,
xce_job_t* updated)
Behavior:
- Find existing job by jobid via sched_get_job()
- If not found: return -ENOENT — jobs are created only via
the Data_data_schedjobs dataset push, never via notify
- If found: update mutable fields (state, priority, allocs)
- Call plugin_job_update()
- Return 0 on success, negative on error
```
**Requirement traceability:** 3.1, 3.2, 3.3
This is wired into the existing `XCE_SCHED_OP_UPDATE_JOB` case — no new op type needed since `sched_on_jobs()` already creates new jobs. The change is to make `sched_update_jobs()` also handle job updates for existing jobids by replacing rather than just adding.
### Component 6: plugin.h / plugin.c — New rusage hook (Req 6)
Add to `plugin_hooks_t`:
```
typedef int (*plugin_job_rusage_hook_t)(void*, xce_sched_job_t*, xce_job_rusages_t*);
```
Add to `plugin_hooks_t` struct:
```
plugin_job_rusage_hook_t rusage_hook;
```
Add global function:
```
int plugin_job_rusage_update(void* sched, xce_sched_job_t* job, xce_job_rusages_t* rusages);
```
This follows the exact pattern of `plugin_job_state_change()` — iterate registered plugins, call their hook if non-NULL.
### Component 7: handler.c — Dead code removal (Req 2)
Remove from `scheduler_req_handler()`:
- The entire `Request_body_pushmsg` case block (lines 185–198)
- The static functions `sched_handle_jobpush()` and `sched_handle_workerpush()`
These are unreachable because `vtask_entry()` in `src/lib/xcectx.c` rewrites all `pushmsg` into `datamsg`.
**Requirement traceability:** 2.1, 2.2
### Component 8: helper.c / helper.h — Name-lookup functions (Req 8)
#### New function: `xce_notify_event_name`
```
Signature: char* xce_notify_event_name(int event)
Behavior: Switch onNotifyData_data_* enum values, return symbolic string.
Fallback: "UNKNOWN(%d)" with numeric substitution.
```
Matches pattern of `xce_dataset_table()` — takes int, returns `char*`.
#### New function: `xce_sched_op_name`
```
Signature: char* xce_sched_op_name(xce_sched_op_t op)
Behavior: Switch on XCE_SCHED_OP_* enum values, return symbolic string.
Fallback: "UNKNOWN(%d)" with numeric substitution.
```
Required for Requirement 7's error logging in `sched_update()`.
#### Changes to `handler.c` logging
| Current | Changed to |
|---|---|
| `TRACE("%s table = %d", __func__, dataset->table)` | `TRACE("%s table = %s", __func__, xce_dataset_table(dataset->table))` |
| `TRACE("%s...", __func__)` (in stub handlers) | `INFO("%s event = %s jobid = %s", __func__, xce_notify_event_name(notify->event), jobidString)` |
**Requirement traceability:** 8.1, 8.2, 8.3, 8.4
### Component 9: scheduler.c — Error logging in `sched_update()` (Req 7)
In the `sched_update()` main loop, after the `switch (req->op)` block, when `rc < 0`:
```
WARN("%s: op %s failed: %s (%d)", __func__,
xce_sched_op_name(req->op), strerror(-rc), rc);
```
**Requirement traceability:** 7.1
### Component 10: schedobj.c — Queue update function (Req 4)
#### New function: `sched_update_existing_queue`
```
Signature: int sched_update_existing_queue(
xce_scheduler_t* sched,
xce_queue_t* updated)
Behavior:
- Find existing queue by name via sched_get_queue()
- If not found: return -ENOENT — queues are created only via
the initial Conf4Sched config push, never via notify
- If found: update mutable fields:
sched_queue->status = updated->status
sched_queue->priority = updated->priority
sched_queue->accept_interval = updated->accept_interval
sched_queue->sched_delay = updated->sched_delay
// resreq is a string field — reallocate and replace
- Call plugin_queue_update()
- Return 0 on success, negative on error
```
**Requirement traceability:** 4.1, 4.2
### Component 11: `sched_pend_update_killer` updates (scheduler.c)
Extend the `sched_pend_update_killer` destructor to handle new op types:
```
case XCE_SCHED_OP_UPDATE_QUEUE:
// queue pointer belongs to scheduler, no free needed
break;
case XCE_SCHED_OP_UPDATE_RESOURCES:
// resource_data rows belong to scheduler, no free needed
break;
case XCE_SCHED_OP_UPDATE_RESVALS_FULL:
// resval_data rows owned by capnp decode, freed by free_Request
break;
case XCE_SCHED_OP_UPDATE_RESVALS:
// resval_list owned by capnp decode, freed by free_Request
break;
case XCE_SCHED_OP_JOB_RUSAGE:
// rusage struct owned by capnp decode, freed by free_Request
break;
```
**Important:** Memory ownership must be traced carefully. For `XCE_SCHED_OP_UPDATE_JOB` and `XCE_SCHED_OP_UPDATE_WORKER`, the existing pattern nullifies `dataset->nrows` and `dataset->rows` after enqueue to transfer ownership. New queue/rusage/resvals handlers should follow the same pattern — the request struct owns the data, and `free_Request()` in `xcectx.c` handles cleanup.
## Data Flow Diagrams
### Notify message processing flow
```
Controller (sched.c) Scheduler (handler.c) Scheduler (scheduler.c)
───────────────────── ───────────────────── ─────────────────────
xce_notify_req()
│
├── schedStartJob ──────────► sched_job_state_change() ──► enq_pend_update(JSTATE_CHANGE)
│ │
├── schedExitJob ───────────► sched_job_state_change() ──► │
│ │
├── schedFailJob ───────────► sched_job_state_change() ──► │
│ │
├── schedUpdateJob ────────► sched_handle_sched_update_job() ─► enq_pend_update(UPDATE_JOB) [NEW]
│ │
├── schedUpdateQueue ──────► sched_handle_sched_update_queue() ─► enq_pend_update(UPDATE_QUEUE) [NEW]
│ │
├── schedUpdateResvals ────► sched_handle_sched_update_resvals() ─► enq_pend_update(UPDATE_RESVALS) [NEW]
│ │
└── schedJobRusages ───────► sched_handle_sched_job_rusage() ──► enq_pend_update(JOB_RUSAGE) [NEW]
│
▼
sched_update() main loop
├── UPDATE_JOB ──► sched_update_existing_job()
├── UPDATE_QUEUE ──► sched_update_existing_queue()
├── UPDATE_RESVALS ──► update worker resvals
├── UPDATE_RESOURCES ──► update resmgr [NEW]
├── UPDATE_RESVALS_FULL ──► update sharedInst [NEW]
└── JOB_RUSAGE ──► store on job + plugin hook [NEW]
```
### Dataset message processing flow
```
Controller (dataex.c) Scheduler (handler.c) Scheduler (scheduler.c)
──────────────────── ───────────────────── ─────────────────────
xce_push_data_req()
│
├── Data_data_schedjobs ──► sched_on_jobs() ──────────► enq_pend_update(UPDATE_JOB)
│ │
├── Data_data_schedworkers ► sched_on_workers() ───────► enq_pend_update(UPDATE_WORKER)
│ │
├── Data_data_resources ──► sched_handle_resources() ─► enq_pend_update(UPDATE_RESOURCES) [NEW]
│ │
└── Data_data_resvals ────► sched_handle_resvals() ───► enq_pend_update(UPDATE_RESVALS_FULL) [NEW]
│
▼
sched_update() main loop
```
## File Change Summary
| File | Change Type | Requirements |
|------|-------------|-------------|
| `src/scheduler/src/handler.c` | Modified — add handlers, remove dead code, update logging | 1, 2, 3, 4, 5, 6, 8 |
| `src/scheduler/include/scheduler.h` | Modified — extend enums and structs | 1, 3, 4, 5, 6 |
| `src/scheduler/src/scheduler.c` | Modified — add dequeue cases, error logging, update functions | 1, 3, 4, 5, 6, 7 |
| `src/scheduler/include/plugin.h` | Modified — add rusage hook typedef and global function | 6 |
| `src/scheduler/src/plugin.c` | Modified — add `plugin_job_rusage_update()` implementation | 6 |
| `src/scheduler/src/schedobj.c` | Modified — add `sched_update_existing_queue()` | 4 |
| `src/lib/helper.c` | Modified — add `xce_notify_event_name()`, `xce_sched_op_name()` | 8 |
| `src/include/helper.h` | Modified — add function declarations | 8 |
| `src/scheduler/testsuite/test.c` | Modified — add unit tests for all new handlers | All |
## Build Order
1. `src/include/helper.h` — add declarations (no dependencies)
2. `src/lib/helper.c` — add implementations (depends on step 1)
3. `src/scheduler/include/scheduler.h` — extend enums and structs (depends on typemap headers)
4. `src/scheduler/include/plugin.h` — add rusage hook typedef (depends on xce/types.h)
5. `src/scheduler/src/plugin.c` — add `plugin_job_rusage_update()` (depends on step 4)
6. `src/scheduler/src/schedobj.c` — add `sched_update_existing_queue()` (depends on step 3)
7. `src/scheduler/src/scheduler.c` — add dequeue processing, error logging, update functions (depends on steps 3–6)
8. `src/scheduler/src/handler.c` — add handlers, remove dead code, update logging (depends on steps 1–7)
9. `src/scheduler/testsuite/test.c` — add unit tests (depends on all above)
## Risks & Mitigations
| Risk | Severity | Mitigation |
|------|----------|-----------|
| Memory ownership of queued data — capnp-decoded structs freed before dequeue processes them | High | Follow existing pattern: for dataset paths, nullify `dataset->nrows`/`dataset->rows` after enqueue to transfer ownership. For notify paths, copy essential data into the request struct rather than holding pointers to capnp memory. |
| `sched_update_existing_job` must handle concurrent modification — handler thread enqueues while main thread processes | Medium | Already mitigated by existing double-buffer swap in `sched_update()`. The handler thread only touches `pend_updates[0]` while the main thread drains `pend_updates[1]`. |
| New `UPDATE_QUEUE` op requires queue name lookup — potential lock contention | Low | `sched_get_queue()` already uses read lock. Queue updates are infrequent. |
| `xce_sched_op_name` must handle future enum additions | Low | Follow `xce_dataset_table` pattern — return `"UNKNOWN(%d)"` for unrecognized values. |

110
.kiro/specs/scheduler-controller-msg-gap/requirements.md Normal file
View file

@ -0,0 +1,110 @@
# Requirements: scheduler-controller-msg-gap
> Generated: 2026-05-11T01:34:00-07:00
> Updated: 2026-05-11T02:00:00-07:00
> Feature: scheduler-controller-msg-gap
> Status: Requirements Generated
## Overview
Analyze the codebase to find gaps in the scheduler's handling of controller messages — missing message types, unhandled edge cases, or incomplete message processing paths.
## Codebase Analysis Summary
The scheduler (`src/scheduler/src/handler.c`) receives controller messages through the `scheduler_req_handler` dispatcher, which handles two top-level opcodes:
- **OPCode_push** — carries ConfigData or Dataset (`datamsg`) messages. Raw `pushmsg` packets are reassembled into `datamsg` by the `vtask_entry()` assembly layer in `src/lib/xcectx.c` (lines 93–106); the handler never sees individual `pushmsg` fragments.
- **OPCode_notify** — carries NotifyData event messages
The controller (`src/controller/sched.c`) sends six NotifyData event types and two Data table types. The scheduler's handler has **6 gap locations** where the case is present but stubbed out or empty-bodied. The `Request_body_pushmsg` branch in `scheduler_req_handler` (which calls `sched_handle_jobpush`/`sched_handle_workerpush`) is dead code — it can never be reached because `vtask_entry` rewrites all `pushmsg` into `datamsg` before invoking the handler.
## Requirements
### Requirement 1: Resource data message handling
The scheduler SHALL process `Data_data_resources` dataset messages received from the controller. Upon receipt, the system SHALL update the internal resource catalog (resource name, type, flags, ordering type, interval) in the scheduler's resource manager (`sched->resmgr`), reconciling additions, removals, and modifications against the existing set.
The scheduler SHALL process `Data_data_resvals` dataset messages received from the controller. Upon receipt, the system SHALL update the shared resource value array (`sched->sharedInst`) with the new values indexed by position, and SHALL reset the `sharedInUse` tracking array for the next scheduling session.
**Acceptance criteria:**
- EARS-UB-1.1: WHEN the scheduler receives a dataset with `table == Data_data_resources`, THEN the system shall add or update each resource in `sched->resmgr`
- EARS-UB-1.2: WHEN the scheduler receives a dataset with `table == Data_data_resvals`, THEN the system shall update `sched->sharedInst` at each value's index position
- EARS-UB-1.3: WHEN either handler encounters a NULL element in the dataset rows, THEN the system shall skip that element and continue processing remaining rows
- EARS-UB-1.4: WHEN either handler encounters a memory allocation failure, THEN the system shall return `-ENOMEM` and free any partially allocated structures
### Requirement 2: Dead code removal — pushmsg handler branches
The scheduler SHALL remove the dead `Request_body_pushmsg` branch in `scheduler_req_handler` and the stub functions `sched_handle_jobpush` and `sched_handle_workerpush`. These paths can never execute because the `vtask_entry()` assembly layer in `src/lib/xcectx.c` rewrites all `pushmsg` packets into `datamsg` before invoking the request handler.
**Acceptance criteria:**
- EARS-UB-2.1: WHEN `scheduler_req_handler` is compiled, THEN the `Request_body_pushmsg` case and its two child branches (job/worker push) shall not exist
- EARS-UB-2.2: WHEN the scheduler is linked, THEN the symbols `sched_handle_jobpush` and `sched_handle_workerpush` shall not be present in the binary
### Requirement 3: NotifyData event handling for job updates
The scheduler SHALL process `NotifyData_data_schedUpdateJob` notify messages beyond the current trace-log-only behavior. Upon receipt, the system SHALL locate the job in `sched->jobs` by jobid, update the job's mutable fields (state, priority, queue, resource request) from the notify payload, and invoke `plugin_job_update` to notify plugins of the change.
**Acceptance criteria:**
- EARS-UB-3.1: WHEN the scheduler receives a notify with `event == NotifyData_data_schedUpdateJob`, THEN the system shall find the existing job by jobid and update its mutable fields
- EARS-UB-3.2: WHEN the jobid in the update notify does not match any existing job (i.e., the job was never pushed via the initial dataset bulk load), THEN the system shall return `-ENOENT` — jobs are created only via the `Data_data_schedjobs` dataset path, never via notify
- EARS-UB-3.3: WHEN the update payload carries job data that is NULL, THEN the system shall return `-EINVAL`
### Requirement 4: NotifyData event handling for queue updates
The scheduler SHALL process `NotifyData_data_schedUpdateQueue` notify messages from the controller. Upon receipt, the system SHALL locate the queue in `sched->queues` by name and update its status, priority, accept_interval, sched_delay, and resource request fields from the notify payload.
**Acceptance criteria:**
- EARS-UB-4.1: WHEN the scheduler receives a notify with `event == NotifyData_data_schedUpdateQueue`, THEN the system shall find the existing queue by name and update its mutable configuration fields
- EARS-UB-4.2: WHEN the queue name in the update notify does not match any existing queue (i.e., the queue was never loaded via the initial config push), THEN the system shall return `-ENOENT` — queues are created only via the `Conf4Sched` config path, never via notify
- EARS-UB-4.3: WHEN the update payload carries queue data that is NULL, THEN the system shall return `-EINVAL`
### Requirement 5: NotifyData event handling for resource value updates
The scheduler SHALL process `NotifyData_data_schedUpdateResvals` notify messages from the controller. Upon receipt, the system SHALL iterate the `xce_resval_list_t` payload, and for each resource value, update the per-worker `resvals` array on the named worker, matching by host name.
**Acceptance criteria:**
- EARS-UB-5.1: WHEN the scheduler receives a notify with `event == NotifyData_data_schedUpdateResvals`, THEN the system shall update each named worker's `resvals` array with the new values
- EARS-UB-5.2: WHEN a named host in the updates does not correspond to any known worker, THEN the system shall skip that update entry and continue processing remaining entries
- EARS-UB-5.3: WHEN the update payload carries zero resource values, THEN the system shall return 0 without modifying state
### Requirement 6: NotifyData event handling for job resource usage
The scheduler SHALL process `NotifyData_data_schedJobRusages` notify messages from the controller. Upon receipt, the system SHALL locate the job by jobid, store the resource usage data (`xce_sched_job_rusages_t`) on the job object, and invoke `plugin_job_rusage_update` to notify plugins of the usage change.
**Acceptance criteria:**
- EARS-UB-6.1: WHEN the scheduler receives a notify with `event == NotifyData_data_schedJobRusages`, THEN the system shall store the usage data on the matching job
- EARS-UB-6.2: WHEN the jobid in the update does not match any existing job, THEN the system shall return `-ENOENT` without modifying state
- EARS-UB-6.3: WHEN the rusage payload is NULL, THEN the system shall return `-EINVAL`
### Requirement 7: Error handling consistency
The scheduler SHALL return meaningful error codes from all message handlers that enable the controller to distinguish between transient failures (ENOMEM) and permanent protocol errors (EINVAL, ENOENT). The scheduler's main loop in `sched_update()` SHALL log a WARN-level message when a handler returns a non-zero error code, including the offending opcode and sub-type for operational debugging.
**Acceptance criteria:**
- EARS-UB-7.1: WHEN a handler returns a non-zero error code, THEN `sched_update()` shall emit a WARN log including the `xce_sched_op_t` value and the error code
- EARS-UB-7.2: WHEN an unknown notify event type is received, THEN the handler shall return `-EINVAL` (already implemented — verify non-regression)
- EARS-UB-7.3: WHEN a push message arrives with an unknown `body_which` tag, THEN the handler shall return `-EINVAL` (already implemented — verify non-regression)
### Requirement 8: Human-readable message type names in logs
The scheduler SHALL log message type identifiers using symbolic names rather than numeric values. The existing helpers in `src/lib/helper.c` — `xce_request_op_name()`, `xce_request_msg_name()`, and `xce_dataset_table()` — already provide name lookups for OPCode, Request body tag, and Data table type. A new `xce_notify_event_name(int)` function SHALL be added to cover the NotifyData event type enum. All TRACE, DEBUG, INFO, and WARN messages in `handler.c` that reference these enums SHALL use these name-lookup functions instead of printing numeric values.
**Acceptance criteria:**
- EARS-UB-8.1: WHEN `scheduler_req_handler` logs a message that includes an opcode, THEN the log shall call `xce_request_op_name()` to show the symbolic name (e.g., `"push"`) rather than the numeric value
- EARS-UB-8.2: WHEN `sched_handle_dataset` logs a message that includes a data table type, THEN the log shall call `xce_dataset_table()` to show the symbolic name (e.g., `"schedjobs"`) rather than the numeric value (currently `TRACE("%s table = %d", ...)`)
- EARS-UB-8.3: WHEN `sched_handle_notify` logs a message that includes a notify event type, THEN the log shall call `xce_notify_event_name()` to show the symbolic name rather than the numeric value
- EARS-UB-8.4: WHEN `xce_notify_event_name()` receives a numeric value that does not correspond to a known NotifyData enum constant, THEN it shall return `"UNKNOWN(%d)"` with the numeric value substituted (matching the pattern used by the existing helpers)
- EARS-UB-8.5: WHEN `sched_update()` emits a WARN log for a handler error (per Requirement 7), THEN the log shall include symbolic names via the appropriate helper function
## Non-Requirements
The following areas are intentionally excluded from this specification:
- **New protocol message types** — only existing, already-defined message types in `controller.capnp` are in scope
- **Controller-side changes** — only the scheduler's inbound message handling is in scope
- **Plugin implementation details** — plugin update functions are called but not modified
- **Bulk (Dataset) job/worker push already works** — the `Data_data_schedjobs` and `Data_data_schedworkers` cases in `sched_handle_dataset` already delegate to `sched_on_jobs`/`sched_on_workers`
- **`pushmsg` handler path** — the `Request_body_pushmsg` branch in `scheduler_req_handler` and the functions `sched_handle_jobpush`/`sched_handle_workerpush` are dead code; the `vtask_entry()` assembly layer in `src/lib/xcectx.c` rewrites all `pushmsg` into `datamsg` before the handler is called, so incremental push handling is never needed
## Next Step
Run `/kiro:spec-design scheduler-controller-msg-gap` to proceed to design phase.

98
.kiro/specs/scheduler-controller-msg-gap/research.md Normal file
View file

@ -0,0 +1,98 @@
# Research Log: scheduler-controller-msg-gap
> Feature: scheduler-controller-msg-gap
> Created: 2026-05-11
## Discovery Type
Extension (light discovery) — extending existing handler code in an established C codebase.
## Summary
The scheduler's `handler.c` dispatcher processes controller messages via `scheduler_req_handler`. Six message cases are stubbed out or empty-bodied. The `vtask_entry()` assembly layer in `src/lib/xcectx.c` rewrites all `pushmsg` into `datamsg`, making the `Request_body_pushmsg` branch dead code. Existing plugin hooks (`plugin_job_update`, `plugin_queue_update`, `plugin_worker_update`, `plugin_job_state_change`) exist for notifying plugins of state changes. A `plugin_job_rusage_update` hook does not yet exist and must be added. The `src/lib/helper.c` file already provides `xce_request_op_name()`, `xce_request_msg_name()`, and `xce_dataset_table()` — a new `xce_notify_event_name()` function is needed.
## Research Log
### Topic 1: Plugin Hook System
**Source:** `src/scheduler/include/plugin.h`, `src/scheduler/src/plugin.c`
**Findings:**
- Hooks follow a `plugin_*_update(void* sched, xce_sched_*_t*)` pattern
- Called after state mutation to notify registered plugins
- Hooks iterate registered plugins and call their function pointers
- `plugin_job_state_change(void*, xce_sched_job_t*, uint32_t)` takes an extra state parameter
- No `plugin_job_rusage_update` hook exists yet — must be added
**Implication:** New handlers should call the corresponding plugin hook after mutating scheduler state. The rusage handler needs a new hook typedef and global function.
### Topic 2: Object Conversion Functions
**Source:** `src/scheduler/src/schedobj.c`
**Findings:**
- `make_sched_job(void* arg, xce_job_t* j)` — maps jobid, times, state, user/queue/project pointers, resreq, mintasks/maxtasks, allocs, accounts
- `make_sched_queue(void* arg, xce_queue_t* q)` — maps name, attrs, status, priority, accept_interval, sched_delay, resreq
- `make_sched_worker(void* arg, xce_worker_t* w)` — maps name, status, idx, resvals zarray from `w->resvals_to_sched`
- `kill_sched_*` functions free their respective objects
**Implication:** Job/queue/worker update handlers should use `make_sched_*` for create-or-update semantics. For updates (existing objects), we need update-in-place logic since `make_sched_*` always creates new objects.
### Topic 3: Resource and Shared Instance Initialization
**Source:** `src/scheduler/src/scheduler.c`
**Findings:**
- `sched_init_resources()` — iterates `config->resources[]`, calls `makeResource(sched->resmgr, r)` for each, warns on NULL/duplicate
- `sched_init_sharedInst()` — iterates `config->resvals[]`, uses `MOVE_PTR` to steal val pointers into `sched->sharedInst` zarray
- Both called from `sched_init_config()` during initial config push
- For runtime updates, we need incremental versions that can add new or update existing resources
**Implication:** Dataset handlers for `Data_data_resources` and `Data_data_resvals` need new update functions that can both create and modify existing entries.
### Topic 4: Existing Helper Functions
**Source:** `src/lib/helper.c`, `src/include/helper.h`
**Findings:**
- `xce_request_op_name(xce_request_t* req)` — returns string for OPCode enum
- `xce_request_msg_name(xce_request_t* req)` — returns string for Request body tag
- `xce_dataset_table(int table)` — returns string for Data table type enum
- All return `"unknown"` for unrecognized values
- `xce_notify_event_name()` does NOT exist yet — must be created
- Signature pattern: `char* func_name(enumerated_type_or_struct_ptr)` returning `char*`
**Implication:** Create `xce_notify_event_name(int event)` following same pattern. Also consider `char* xce_sched_op_name(xce_sched_op_t op)` for the update request op enum.
### Topic 5: Cap'n Proto Enum Values (NotifyData)
**Source:** `src/schema/controller.capnp`
**Findings:**
All 19 NotifyData event tags:
- `jobPidInfo(0)`, `jobFailInfo(1)`, `jobDispatchFail(2)`, `jobFinishInfo(3)`, `schedStartJob(4)`, `schedExitJob(5)`, `schedFailJob(6)`, `schedUpdateJob(7)`, `schedUpdateResvals(8)`, `stoppedJob(9)`, `resumedJob(10)`, `ckptinfo(11)`, `schedUpdateQueue(12)`, `taskPidInfo(13)`, `taskFailInfo(14)`, `taskFinish(15)`, `postDoneJob(16)`, `schedJobRusages(17)`, `updateJob(18)`
**Implication:** The `xce_notify_event_name()` function must cover all 19 values. Only the scheduler-relevant ones need handler implementations (schedUpdateJob, schedUpdateQueue, schedUpdateResvals, schedJobRusages are the gap items).
### Topic 6: Dead Code Analysis
**Source:** `src/lib/xcectx.c` lines 93-106, `src/scheduler/src/handler.c` lines 172-197
**Findings:**
- `vtask_entry()` accumulates `pushmsg` rows into `ctx->dataset`, then rewrites `body_which = Request_body_datamsg` and `body.datamsg = ctx->dataset`
- The `Request_body_pushmsg` case in `scheduler_req_handler` (lines 185-198) calls `sched_handle_jobpush()` and `sched_handle_workerpush()`, both of which are stubs returning 0
- These functions can never be reached via normal message flow
**Implication:** Safe to remove the pushmsg case branch and both stub functions.
## Architecture Decisions
1. **Handler pattern:** New notify handlers follow the same enqueue-to-pend-updates pattern as `sched_on_job_state_change` — create a `xce_sched_update_request_t`, set the op, enqueue via `enq_pend_update()`, process in `sched_update()` main loop.
2. **Update-in-place for job/queue/worker:** For NotifyData update events, we need update semantics (find existing, mutate fields) rather than create semantics (make_sched_*). New `sched_update_job`, `sched_update_queue`, `sched_update_worker` functions will handle this.
3. **New XCE_SCHED_OP enum values:** Need `XCE_SCHED_OP_UPDATE_RESVALS` for resource value updates and `XCE_SCHED_OP_JOB_RUSAGE` for rusage updates.
4. **Plugin hook for rusage:** Add `plugin_job_rusage_hook_t` typedef and `plugin_job_rusage_update()` global function following existing patterns.
5. **Helper function:** Add `xce_notify_event_name(int)` and `xce_sched_op_name(xce_sched_op_t)` to `src/lib/helper.c` and `src/include/helper.h`.

23
.kiro/specs/scheduler-controller-msg-gap/spec.json Normal file
View file

@ -0,0 +1,23 @@
{
"feature_name": "scheduler-controller-msg-gap",
"description": "Analyze the codebase to find gaps in the scheduler's handling of controller messages — missing message types, unhandled edge cases, or incomplete message processing paths",
"phase": "tasks-generated",
"created_at": "2026-05-11T01:34:00-07:00",
"updated_at": "2026-05-11T19:30:00-07:00",
"language": "en",
"status": "active",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
}
}

127
.kiro/specs/scheduler-controller-msg-gap/tasks.md Normal file
View file

@ -0,0 +1,127 @@
# Tasks: scheduler-controller-msg-gap
> Generated: 2026-05-11T19:30:00-07:00
> Feature: scheduler-controller-msg-gap
> Status: Tasks Generated
## Overview
Implementation tasks for filling gaps in the scheduler's handling of controller messages. The work follows the design's build order: shared helpers first, then header definitions, plugin hooks, business logic, handler wiring, and finally tests.
---
## Task 1: Add name-lookup helpers to lib/helper
**Requirements:** 8
Add `xce_notify_event_name()` and `xce_sched_op_name()` to `src/lib/helper.c` and declare them in `src/include/helper.h`, following the exact pattern of the existing `xce_dataset_table()` helper (int argument, `char*` return, `"UNKNOWN(%d)"` fallback).
- [x] 1.1 Declare `xce_notify_event_name(int event)` and `xce_sched_op_name(xce_sched_op_t op)` in `src/include/helper.h`
- [x] 1.2 Implement `xce_notify_event_name()` in `src/lib/helper.c` — switch on all `NotifyData_data_*` enum values; return symbolic string; fall back to `"UNKNOWN(%d)"`
- [x] 1.3 Implement `xce_sched_op_name()` in `src/lib/helper.c` — switch on all `XCE_SCHED_OP_*` enum values; return symbolic string; fall back to `"UNKNOWN(%d)"`
---
## Task 2: Extend scheduler.h with new op types and request fields
**Requirements:** 1, 3, 4, 5, 6
Add five new `xce_sched_op_t` enum values and five corresponding union members in `xce_sched_update_request_t` to carry the payloads for queue update, resource update, full resval update, incremental resval update, and job rusage.
- [x] 2.1 Add `XCE_SCHED_OP_UPDATE_QUEUE`, `XCE_SCHED_OP_UPDATE_RESOURCES`, `XCE_SCHED_OP_UPDATE_RESVALS_FULL`, `XCE_SCHED_OP_UPDATE_RESVALS`, and `XCE_SCHED_OP_JOB_RUSAGE` to the `xce_sched_op_t` enum in `src/scheduler/include/scheduler.h`
- [x] 2.2 Add the corresponding named union members (`queueupdate`, `resourceupdate`, `resvalfullupdate`, `resvalupdate`, `job_rusage`) to `xce_sched_update_request_t`
---
## Task 3: Add plugin rusage hook
**Requirements:** 6
Add the `plugin_job_rusage_hook_t` typedef, the `rusage_hook` field in `plugin_hooks_t`, and the `plugin_job_rusage_update()` global function. This follows the identical pattern used by `plugin_job_state_change()`.
- [x] 3.1 Add `plugin_job_rusage_hook_t` typedef and `rusage_hook` field to `plugin_hooks_t` in `src/scheduler/include/plugin.h`
- [x] 3.2 Implement `plugin_job_rusage_update()` in `src/scheduler/src/plugin.c` — iterate registered plugins and call their `rusage_hook` when non-NULL
---
## Task 4: Add queue update function in schedobj.c
**Requirements:** 4
Implement `sched_update_existing_queue()` in `src/scheduler/src/schedobj.c`. The function locates a queue by name via `sched_get_queue()`, updates its mutable fields (status, priority, accept_interval, sched_delay, resreq string), calls `plugin_queue_update()`, and returns `-ENOENT` if the queue is unknown.
- [x] 4.1 Implement `sched_update_existing_queue(xce_scheduler_t*, xce_queue_t*)` in `src/scheduler/src/schedobj.c` with correct lookup, field update, string reallocation for resreq, plugin call, and error returns
---
## Task 5: Extend scheduler.c dequeue processing and error logging
**Requirements:** 1, 3, 4, 5, 6, 7
Wire the five new op types into the `switch (req->op)` in `sched_update()`, implement `sched_update_existing_job()`, and add WARN-level error logging after the switch when a handler returns non-zero.
- [x] 5.1 Add `XCE_SCHED_OP_UPDATE_QUEUE` case — delegate to `sched_update_existing_queue()`, return `-ENOENT` when queue not found
- [x] 5.2 Add `XCE_SCHED_OP_UPDATE_RESOURCES` case — iterate `resdata->resources[]`, call `makeResource(sched->resmgr, r)` for each non-NULL entry
- [x] 5.3 Add `XCE_SCHED_OP_UPDATE_RESVALS_FULL` case — iterate `resvaldata->vals[]`, find resource in `sched->resmgr` by name, update value; log WARN and skip unknown resources
- [x] 5.4 Add `XCE_SCHED_OP_UPDATE_RESVALS` case — iterate `resvals->vals[]`, find worker by host name via `sched_get_worker()`, update worker's `resvals` zarray; silently skip unknown hosts
- [x] 5.5 Add `XCE_SCHED_OP_JOB_RUSAGE` case — find job by jobid, store rusage data, call `plugin_job_rusage_update()`; return `-ENOENT` if job not found
- [x] 5.6 Implement `sched_update_existing_job()` — find job by jobid via `sched_get_job()`, update mutable fields (state, priority, allocs), call `plugin_job_update()`; return `-ENOENT` when jobid is unknown
- [x] 5.7 Add WARN log in `sched_update()` after switch block for non-zero `rc` — include symbolic op name via `xce_sched_op_name()` and `strerror(-rc)`
- [x] 5.8 Extend `sched_pend_update_killer` with no-op cases for all five new op types to suppress unhandled-case compiler warnings
---
## Task 6: Wire notify handlers in handler.c
**Requirements:** 3, 4, 5, 6, 8
Add four static handler functions for the currently-stubbed NotifyData events, wire them into `sched_handle_notify()`, and update the TRACE log calls in `sched_handle_notify()` and `sched_handle_dataset()` to use symbolic name helpers.
- [x] 6.1 Implement `sched_handle_sched_update_job()` — validate non-NULL, enqueue `XCE_SCHED_OP_UPDATE_JOB`; return `-EINVAL` on NULL, `-ENOMEM` on alloc failure
- [x] 6.2 Implement `sched_handle_sched_update_queue()` — validate non-NULL, enqueue `XCE_SCHED_OP_UPDATE_QUEUE`; return `-EINVAL` on NULL, `-ENOMEM` on alloc failure
- [x] 6.3 Implement `sched_handle_sched_update_resvals()` — skip if NULL or count==0, enqueue `XCE_SCHED_OP_UPDATE_RESVALS`; return `-ENOMEM` on alloc failure
- [x] 6.4 Implement `sched_handle_sched_job_rusage()` — validate non-NULL, enqueue `XCE_SCHED_OP_JOB_RUSAGE`; return `-EINVAL` on NULL, `-ENOMEM` on alloc failure
- [x] 6.5 Wire all four handlers into `sched_handle_notify()` switch — replace stub `break` / TRACE-only cases
- [x] 6.6 Update TRACE log in `sched_handle_dataset()` from `%d` table integer to `%s` via `xce_dataset_table()` (8.2), and update notify event logs to use `xce_notify_event_name()` (8.3)
---
## Task 7: Add dataset handlers in handler.c
**Requirements:** 1, 8
Add `sched_handle_resources()` and `sched_handle_resvals()` static functions and wire them into the `sched_handle_dataset()` switch for the two currently-empty Data table cases. Update the existing opcode log in `scheduler_req_handler` to use `xce_request_op_name()`.
- [x] 7.1 Implement `sched_handle_resources()` — skip if NULL or count==0, enqueue `XCE_SCHED_OP_UPDATE_RESOURCES`; return `-ENOMEM` on alloc failure
- [x] 7.2 Implement `sched_handle_resvals()` — skip if NULL or count==0, enqueue `XCE_SCHED_OP_UPDATE_RESVALS_FULL`; return `-ENOMEM` on alloc failure
- [x] 7.3 Wire `sched_handle_resources()` and `sched_handle_resvals()` into `sched_handle_dataset()` switch replacing the empty `break` cases
- [x] 7.4 Update opcode log in `scheduler_req_handler` to call `xce_request_op_name()` instead of printing the numeric value (8.1)
---
## Task 8: Remove dead code pushmsg branches
**Requirements:** 2
Delete the `Request_body_pushmsg` case block from `scheduler_req_handler()` and the static functions `sched_handle_jobpush()` and `sched_handle_workerpush()`. These paths are unreachable because `vtask_entry()` rewrites all `pushmsg` into `datamsg` before invoking the handler.
- [x] 8.1 Remove the `Request_body_pushmsg` case and its two nested branches (`job`/`worker`) from `scheduler_req_handler()` in `src/scheduler/src/handler.c`
- [x] 8.2 Remove the static function definitions `sched_handle_jobpush()` and `sched_handle_workerpush()` from `src/scheduler/src/handler.c`
---
## Task 9: Write unit tests for all new message handlers
**Requirements:** 1, 2, 3, 4, 5, 6, 7, 8
Extend `src/scheduler/testsuite/test.c` with unit tests covering every acceptance criterion. Tests must run without a live server — use stub schedulers, table-driven cases, and the existing testsuite harness.
- [x] 9.1 Test `xce_notify_event_name()` and `xce_sched_op_name()` — verify known values return correct strings, unknown values return `"UNKNOWN(%d)"` format
- [x] 9.2 Test `sched_handle_resources()` and `sched_handle_resvals()` handler functions — verify enqueue on valid input, skip on NULL/empty, `-ENOMEM` path
- [x] 9.3 Test `sched_handle_sched_update_job()`, `sched_handle_sched_update_queue()`, `sched_handle_sched_update_resvals()`, `sched_handle_sched_job_rusage()` — verify `-EINVAL` on NULL, enqueue on valid input
- [x] 9.4 Test `sched_update_existing_job()` — verify `-ENOENT` for unknown jobid, correct field update and plugin call for known jobid
- [x] 9.5 Test `sched_update_existing_queue()` — verify `-ENOENT` for unknown queue, correct field update and plugin call for known queue; verify `-EINVAL` on NULL payload
- [x] 9.6 Test dequeue path for `XCE_SCHED_OP_UPDATE_RESVALS` — verify unknown host skipped, known host updated
- [x] 9.7 Test dequeue path for `XCE_SCHED_OP_JOB_RUSAGE` — verify `-ENOENT` for unknown jobid, rusage stored and plugin called for known jobid; verify `-EINVAL` on NULL rusage
- [x] 9.8 Verify dead code symbols are absent — compile-time check that `sched_handle_jobpush` / `sched_handle_workerpush` are not defined (2.1, 2.2)
- [ ]* 9.9 Verify WARN log is emitted in `sched_update()` when any handler returns a non-zero error code (optional — may require log capture infrastructure)