myxce/.kiro/specs/scheduler-core/research.md

# Research & Design Decisions: scheduler-core

## Summary
- **Feature**: `scheduler-core`
- **Discovery Scope**: Complex Integration (extension of an existing C daemon, new wire protocol, new cross-component opcode)
- **Key Findings**:
  - `requirements.md` is unusually design-complete for a requirements doc — most C-level contracts (struct layouts, thread model, capnp wire framing) are already normative, not left to design. Design's real job here is: (1) the still-deferred `ReplayEvent`/`ReplayDecision` payload capnp schemas, (2) the concrete File Structure Plan, (3) closing two boundary tensions the requirements doc surfaces but doesn't resolve.
  - Almost every payload type `ReplayEvent.body`'s union needs (`Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `QueueConf`, `SchedJobRusages`) **already exists verbatim** in `src/schema/controller.capnp` — the requirements author already picked names that match the live schema. Only `JobStateChangeEvent` and `JobUpdateEvent` are genuinely new. This is a Build-vs-Adopt win: reuse, don't reinvent.
  - The scheduler component's real file layout (`src/scheduler/src/{decision.c,handler.c,main.c,plugin.c,scheduler.c,plugins/{fcfs.c,limit.c,parjob_allocator.c,seqjob_allocator.c},resource.c,schedobj.c}`) is substantially larger than `.wolf/anatomy.md`'s cached listing (which only knew `handler.c`, `main.c`, `scheduler.c`). File Structure Plan below is built from the live `find`, not the stale cache.

## Research Log

### Existing scheduler component layout
- **Context**: Need concrete file paths for the File Structure Plan; `structure.md` gives the pattern (`include/`, `src/`, `testsuite/`, plugin subdir), but not the current file set.
- **Sources Consulted**: `find src/scheduler -name '*.c' -o -name '*.h'`, `src/scheduler/testsuite/meson.build`.
- **Findings**: `include/`: `decision.h`, `plugin.h`, `sched/types.h`, `scheduler.h`. `src/`: `decision.c`, `handler.c`, `main.c`, `plugin.c`, `scheduler.c`, `resource.c`, `schedobj.c`, `plugins/{fcfs.c,limit.c,parjob_allocator.c,seqjob_allocator.c}`. `testsuite/`: `test.c` (runner harness), `test_handlers.c`, `test_job_array_expand.c`, `test_job_array_wire_e2e.c`, `test_job_update.c` — standalone TUs, several `#include ../src/*.c` directly (white-box pattern) rather than linking.
- **Implications**: New core/driver files (`xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t`) are new `src/` files, not modifications of `scheduler.c` wholesale — `scheduler.c` keeps today's `xce_scheduler_t`-owning code during Phase 1, then Phase 2 extracts state into a new `core.c`. Test files for the new pieces follow the existing standalone-TU-with-`#include`, one-binary-per-unit convention already established in `testsuite/meson.build`.

### Controller-side opcode dispatch location and pattern
- **Context**: AC 10 requires a new `OPCode_schedAdmin` case; need to confirm which file owns `myreq_handler` and the exact style of an existing opcode case for consistency.
- **Sources Consulted**: `grep -rln myreq_handler src/controller/*.c`, `cluster.c` around `OPCode_fetchfile` and `OPCode_upgradeConfig`.
- **Findings**: `myreq_handler` lives in `src/controller/cluster.c` (also referenced from `ha.c`/`ha_convergence.c`, but the dispatch switch itself is in `cluster.c`). The closest structural precedent is `OPCode_upgradeConfig` (`cluster.c:670-705`): gated by `ha_is_primary(&c->ha)` (reject standby with `-EAGAIN`, `RequestReply_data_nodata`), then delegates to a `handle_*()` function. **No RBAC/auth check is visible in that case block** — the comment trail (`cluster.c:672-693`, bug-fix 2026-07-25) discusses primary/standby gating exhaustively but never mentions an authentication check at this opcode. If `OPCode_upgradeConfig` has one, it happens earlier (TLS handshake / connection-accept time) or not at all at this layer.
- **Implications**: AC 10's claim "same RBAC role as config push" cannot be verified against a concrete gate in the code as it stands — carried into design.md as an **Open Question**, not asserted as a resolved fact. `OPCode_schedAdmin`'s case block should structurally mirror `OPCode_upgradeConfig` (primary-only gate, `handle_*()` delegation, explicit reply on every path) since that is the only comparable admin-opcode precedent in the file.

### xceadmin CLI extension pattern
- **Context**: AC 10's `xceadmin scheduler record start/stop/status` and AC 20's `xceadmin scheduler dump` need a concrete module plan consistent with the existing `config push`/`config build` implementation.
- **Sources Consulted**: `src/xceadmin/src/xceadmin/{cli.py,push.py,build.py,discovery.py}`, `src/xceadmin/src/xcecommon/{wire.py,schema.py}`.
- **Findings**: `cli.py` is a thin `click` layer only (`@cli.group()` per resource, `@group.command()` per verb) — all business logic lives in per-resource modules (`push.py`, `build.py`) that a) load the schema once via `xcecommon.schema.load_schema()`, b) build/decode capnp messages, c) go through `xcecommon.wire.connect`/`send_request` for the socket, and d) for request/reply opcodes, route through `xceadmin.discovery.follow_redirects` + `xceadmin.discovery.decode_reply` (which unwraps the `Request{opcode: OPCode_reply, body: replymsg}` envelope every reply is wrapped in — a real bug fixed 2026-07-30, documented in `discovery.py`'s own docstring). Every raising path uses one exception hierarchy (`XceAdminError` subclasses), caught once in `cli.py`.
- **Implications**: A new `xceadmin/scheduler.py` module is the natural home for `record_start()`, `record_stop()`, `record_status()`, and `dump()` — same shape as `push_config()`/`probe_config()`, reusing `discovery.follow_redirects`/`decode_reply` (this is a request/reply opcode per AC 10, so unlike `probe_config()` it DOES need redirect-following). `dump()` differs from the other three: it operates on a local `.replay` file only, no network call — it decodes a capnp stream with `xcecommon.schema.load_schema()`'s module object and formats each message type to text; it belongs in the same module for CLI cohesion but has no wire dependency.

### Reusable capnp payload types for ReplayEvent / SnapshotData
- **Context**: Requirements AC 4 explicitly defers the exact capnp schema for event payloads to design; AC 20's `ReplayEvent.body` union names `Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `JobStateChangeEvent`, `JobUpdateEvent`, `QueueConf`, `SchedJobRusages` without defining the two `*Event` types.
- **Sources Consulted**: `grep -n '^struct ' src/schema/controller.capnp`.
- **Findings**: `Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `QueueConf`, `SchedJobRusages` **all already exist** in `controller.capnp` with those exact names (the requirements author evidently cross-checked the schema before naming the union members). `JobStateChangeEvent` and `JobUpdateEvent` do not exist. The existing `Job` struct (`controller.capnp:288`) already carries `jobid` + `state`, and `JobAlloc`/`JobPendReason` exist for allocation/pend-reason payloads that the new `DispatchDecision`/`PendDecision` need.
- **Implications**: Design only needs to define 2 new event payload structs and 3 new decision payload structs (reusing `JobAlloc` and `JobPendReason` internally), not 9. See Data Models section of `design.md`.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Event-sourced core with pluggable driver (selected) | Core holds all scheduling state, driven purely by typed events + a pull-based clock; two driver implementations (normal, simulation) translate their respective input sources to the same event stream | Normal↔sim equivalence becomes a structural property, not a discipline; picker/allocator threading is unchanged from today, minimizing regression surface | Requires strict discipline (no wall-clock calls, no controller calls) inside the core — a single violation breaks the central invariant silently until someone runs `--sim-verify` | Already fully specified in requirements.md; design.md's job is to make the boundary and remaining schema concrete, not to re-litigate the pattern |
| Mode-flagged single scheduler (rejected in requirements) | Keep `xce_scheduler_t` as-is, add `if (sim_mode)` branches for clock/IO | No new types, smallest diff | Directly violates the central invariant requirements establishes (normal/sim would share code paths only by discipline, not by construction) — rejected by requirements.md Req 5 AC 1, not re-opened here | — |

## Design Decisions

### Decision: New capnp structs for JOB_STATE_CHANGE / JOB_UPDATE events
- **Context**: AC 4 specifies the field set (jobid, old_state, new_state, exit_code for state-change; jobid, changed_fields, full job record for update) but no capnp shape exists.
- **Alternatives Considered**:
  1. Reuse `Job` alone and let the driver infer old/new state from two full records — rejected, doubles payload size for a two-field delta and loses `exit_code`.
  2. Define thin new structs that carry only the delta plus embed `Job` where a full record is genuinely needed — selected.
- **Selected Approach**: `JobStateChangeEvent{jobid, oldState, newState, exitCode}`; `JobUpdateEvent{jobid, changedFields, job: Job}` (embeds the existing `Job` struct, not a re-derived one).
- **Rationale**: Matches AC 4's field table exactly; embedding `Job` for the update case (where the driver already has to build a full record) avoids a second job-shape definition; the state-change case stays flat since AC 4 requires no other job fields.
- **Trade-offs**: `JobUpdateEvent` duplicates the full `Job` payload even for a single-field priority change — acceptable, matches AC 4's explicit field table (`job: xce_job_t*` full record), not a design deviation.
- **Follow-up**: None — field set is normative from requirements, only the capnp wrapper is new.

### Decision: Decision payload structs (DispatchDecision / PendDecision / PreemptDecision) — and a boundary note on PreemptDecision
- **Context**: AC 5 fully specifies fields for all three decision types. Separately, the Boundary Context assigns "the preemption decision type" to the `scheduler-preemption` sibling spec.
- **Alternatives Considered**:
  1. Omit `PreemptDecision` entirely from this design, leave a placeholder in the `ReplayDecision.body` union for `scheduler-preemption` to add later — rejected, AC 5's decision output table is a Requirement 5 AC in *this* spec, and the `on_decision` callback signature / `ReplayDecision` union must compile today (Phase 2) before `scheduler-preemption` exists.
  2. Define `PreemptDecision`'s wire shape now per AC 5's field table, treat victim-selection *semantics* (not the wire shape) as what `scheduler-preemption` owns — selected.
- **Selected Approach**: `PreemptDecision{jobid, victims: List(PreemptVictim{jobid, workerid}), reason: JobPendReason}` defined in this spec's schema additions.
- **Rationale**: The wire struct is a passive data shape with no scheduling logic in it; nothing about defining it here binds `scheduler-preemption` to any victim-selection algorithm. Leaving the union member undefined would make `xce_decision_t`/`ReplayDecision` incomplete types this spec's own Phase 2 code needs to reference (`can_preempt()`'s caller, per requirements.md Req 1 AC 4's allocator pseudocode, already branches on a preempt outcome).
- **Trade-offs**: If `scheduler-preemption` later needs additional fields on `PreemptDecision`, that spec must extend this schema (a revalidation trigger, listed in design.md's Boundary Commitments) rather than owning it outright from scratch.
- **Follow-up**: Flag explicitly to whoever picks up `scheduler-preemption` that the wire shape is pre-committed here; the spec should confirm the field set is sufficient before implementation.

### Decision: `OPCode_schedAdmin` authorization mechanism — deferred to a follow-up spec (project owner call, 2026-08-02)
- **Context**: AC 10 states "same RBAC role as config push," but the only comparable opcode (`OPCode_upgradeConfig`) has no visible authorization check in `cluster.c` — only a primary/standby gate.
- **Alternatives Considered**:
  1. Assert a specific mechanism (e.g. "add a `cluster:control` RBAC check") without verifying one exists for `config push` today — rejected, would document a fictitious parity claim.
  2. Block Phase 3 until the implementer traces `config push`'s real auth boundary and applies it here too — rejected by project owner: not worth gating this spec's delivery on.
  3. Ship `OPCode_schedAdmin` at parity with `OPCode_upgradeConfig`'s current posture (primary-gate only, no RBAC check), document the gap explicitly, and track auth hardening as separate future work — **selected**.
- **Selected Approach**: Option 3. `design.md`'s Open Questions/Risks and Security Considerations sections record this as an accepted, deliberate gap rather than an unresolved blocker.
- **Rationale**: `config push`'s current authorization boundary could not be located in the controller-side handler during discovery. Rather than block this spec on retrofitting auth for two opcodes at once, the project owner chose to scope that work separately.
- **Trade-offs**: `OPCode_schedAdmin` (and, unchanged, `OPCode_upgradeConfig`) remain gated only by primary/standby routing until the follow-up lands — acceptable short-term since both are already TLS-connected, admin-tooling-only surfaces (xceadmin), not general network-reachable endpoints.
- **Follow-up**: Add an "auth hardening for admin opcodes (`OPCode_upgradeConfig`, `OPCode_schedAdmin`)" item to the project roadmap (see `design.md` Open Questions/Risks #1). Track as a future spec, not a task inside `scheduler-core`.

### Decision: `OPCode_schedAdmin`'s blocking cross-process wait — accepted as a boundary case, no admission reservation (project owner call, 2026-08-02)
- **Context**: Design review (2026-08-02) flagged that `myreq_handler` runs as a `vtask_t` handler (bounded 8-thread pool, `cluster.c:238`), and `OPCode_schedAdmin` blocks that thread up to 5s waiting on a cross-process reply from the scheduler daemon while holding `c->sched.lock` — unlike `OPCode_upgradeConfig`'s in-process, `COMMIT_WAIT_RESERVED`-admitted commit-barrier wait. `ha-async-io.md`'s vtaskmgr rule requires admission reservation for bounded blocking and forbids unbounded blocking on this pool; the `raft-drive-async` family fixed prior real violations of this exact rule (`config-push`'s busy-poll+join was one of them).
- **Alternatives Considered**:
  1. Add a `COMMIT_WAIT_RESERVED`-style admission reservation for `OPCode_schedAdmin` before Phase 3 ships — the design-review recommendation.
  2. Make the scheduler forward asynchronous (vtask completion callback instead of blocking wait) — larger structural change, deferred as unnecessary for current usage.
  3. Accept the blocking wait as a boundary case given expected request volume — **selected**.
- **Selected Approach**: Option 3. `xceadmin scheduler record`/`dump` commands are cluster-administrator-triggered, human-paced, low-frequency — not a request class expected to burst concurrently the way heartbeat/replication traffic does. No admission-reservation mechanism is added.
- **Rationale**: The `COMMIT_WAIT_RESERVED` precedent exists because commit-barrier waits are on the hot path of every job submission (high, uncontrolled frequency). Admin recording control is operator-initiated and rare by nature; the risk this pattern protects against in the commit-barrier case doesn't apply at comparable severity here.
- **Trade-offs**: If usage patterns change (e.g. automated/scripted recording control invoked at scale, or a monitoring tool polling `record status` frequently), this becomes a live risk without warning — no test or admission gate would catch it.
- **Follow-up**: If `scheduler record`/`dump` usage ever becomes automated or high-frequency, revisit with a `COMMIT_WAIT_RESERVED`-style admission reservation.
- **SUPERSEDES (2026-08-19, design-review follow-up)**: Option 3 above is no longer the shipped behavior. A retrospective design review flagged that `c->sched.lock` already serializes concurrent `schedAdmin` requests one at a time — a burst of N concurrent requests could occupy N vtask threads for up to N×5s each even under Option 3's "accepted, human-paced" framing, which this Decision's own Trade-offs bullet didn't fully account for. `vtaskmgr_blocking_acquire`/`vtaskmgr_blocking_release` (Option 1's mechanism, reusing `commit.c`'s existing `c->vtaskmgr` reservation rather than a new one) now wrap the whole forward+wait region; admission-exhausted requests fast-reject with `{ok=false, error="server busy, try again"}` instead of queuing. See `design.md`'s `SchedAdminOpcode` component ("Blocking-wait admission (hardened 2026-08-19)") and `requirements.md` Req 5 AC 10's own Amendment for the current, authoritative description.

## Risks & Mitigations
- **`OPCode_schedAdmin` ships without an RBAC/auth check beyond the primary/standby gate — accepted, not blocking.** Mitigation: documented explicitly in `design.md`; tracked as a future roadmap item (see Decision above), not a Phase 3 dependency.
- **`OPCode_schedAdmin` blocks a vtask thread for up to 5s per request — MITIGATED 2026-08-19.** Now gated by the same `vtaskmgr_blocking_acquire`/`_release` admission reservation `commit.c`'s commit-wait barrier shares (see the Decision's "SUPERSEDES" note above); admission-exhausted bursts fast-reject rather than occupying vtask threads.
- **`PreemptDecision` wire shape is pre-committed before `scheduler-preemption`'s own design exists** — mitigation: documented as a revalidation trigger; `scheduler-preemption` must review this shape early rather than discovering it mid-implementation.
- **`jbuf` capacity=1 determinism invariant depends on a `FIXME: make this configurable` comment never being resolved carelessly** — mitigation: already hard-pinned in requirements.md Req 5 AC 7 with an explicit Phase 2 task; design.md's File Structure Plan marks `xce_sched_core_init()` as the single place this is enforced.

## References
- `src/scheduler/CLAUDE.md`, `.kiro/steering/{structure.md,tech.md,ha-async-io.md}` — component layout and controller HA invariants.
- `src/schema/controller.capnp` — canonical wire schema; source of truth for reused struct names.
- `src/xceadmin/src/xceadmin/push.py`, `discovery.py` — reference pattern for the new `scheduler.py` xceadmin module.