myxce/.kiro/specs/scheduler-core/research.md

Research & Design Decisions: scheduler-core

Summary

• Feature: scheduler-core
• Discovery Scope: Complex Integration (extension of an existing C daemon, new wire protocol, new cross-component opcode)
• Key Findings:
  • requirements.md is unusually design-complete for a requirements doc — most C-level contracts (struct layouts, thread model, capnp wire framing) are already normative, not left to design. Design's real job here is: (1) the still-deferred ReplayEvent/ReplayDecision payload capnp schemas, (2) the concrete File Structure Plan, (3) closing two boundary tensions the requirements doc surfaces but doesn't resolve.
  • Almost every payload type ReplayEvent.body's union needs (Conf4Sched, SchedJobData, WorkerData, ResourceData, ResValList, QueueConf, SchedJobRusages) already exists verbatim in src/schema/controller.capnp — the requirements author already picked names that match the live schema. Only JobStateChangeEvent and JobUpdateEvent are genuinely new. This is a Build-vs-Adopt win: reuse, don't reinvent.
  • The scheduler component's real file layout (src/scheduler/src/{decision.c,handler.c,main.c,plugin.c,scheduler.c,plugins/{fcfs.c,limit.c,parjob_allocator.c,seqjob_allocator.c},resource.c,schedobj.c}) is substantially larger than .wolf/anatomy.md's cached listing (which only knew handler.c, main.c, scheduler.c). File Structure Plan below is built from the live find, not the stale cache.

Research Log

Existing scheduler component layout

• Context: Need concrete file paths for the File Structure Plan; structure.md gives the pattern (include/, src/, testsuite/, plugin subdir), but not the current file set.
• Sources Consulted: find src/scheduler -name '*.c' -o -name '*.h', src/scheduler/testsuite/meson.build.
• Findings: include/: decision.h, plugin.h, sched/types.h, scheduler.h. src/: decision.c, handler.c, main.c, plugin.c, scheduler.c, resource.c, schedobj.c, plugins/{fcfs.c,limit.c,parjob_allocator.c,seqjob_allocator.c}. testsuite/: test.c (runner harness), test_handlers.c, test_job_array_expand.c, test_job_array_wire_e2e.c, test_job_update.c — standalone TUs, several #include ../src/*.c directly (white-box pattern) rather than linking.
• Implications: New core/driver files (xce_sched_core_t, xce_sched_driver_t, xce_sched_sim_driver_t) are new src/ files, not modifications of scheduler.c wholesale — scheduler.c keeps today's xce_scheduler_t-owning code during Phase 1, then Phase 2 extracts state into a new core.c. Test files for the new pieces follow the existing standalone-TU-with-#include, one-binary-per-unit convention already established in testsuite/meson.build.

Controller-side opcode dispatch location and pattern

• Context: AC 10 requires a new OPCode_schedAdmin case; need to confirm which file owns myreq_handler and the exact style of an existing opcode case for consistency.
• Sources Consulted: grep -rln myreq_handler src/controller/*.c, cluster.c around OPCode_fetchfile and OPCode_upgradeConfig.
• Findings: myreq_handler lives in src/controller/cluster.c (also referenced from ha.c/ha_convergence.c, but the dispatch switch itself is in cluster.c). The closest structural precedent is OPCode_upgradeConfig (cluster.c:670-705): gated by ha_is_primary(&c->ha) (reject standby with -EAGAIN, RequestReply_data_nodata), then delegates to a handle_*() function. No RBAC/auth check is visible in that case block — the comment trail (cluster.c:672-693, bug-fix 2026-07-25) discusses primary/standby gating exhaustively but never mentions an authentication check at this opcode. If OPCode_upgradeConfig has one, it happens earlier (TLS handshake / connection-accept time) or not at all at this layer.
• Implications: AC 10's claim "same RBAC role as config push" cannot be verified against a concrete gate in the code as it stands — carried into design.md as an Open Question, not asserted as a resolved fact. OPCode_schedAdmin's case block should structurally mirror OPCode_upgradeConfig (primary-only gate, handle_*() delegation, explicit reply on every path) since that is the only comparable admin-opcode precedent in the file.

xceadmin CLI extension pattern

• Context: AC 10's xceadmin scheduler record start/stop/status and AC 20's xceadmin scheduler dump need a concrete module plan consistent with the existing config push/config build implementation.
• Sources Consulted: src/xceadmin/src/xceadmin/{cli.py,push.py,build.py,discovery.py}, src/xceadmin/src/xcecommon/{wire.py,schema.py}.
• Findings: cli.py is a thin click layer only (@cli.group() per resource, @group.command() per verb) — all business logic lives in per-resource modules (push.py, build.py) that a) load the schema once via xcecommon.schema.load_schema(), b) build/decode capnp messages, c) go through xcecommon.wire.connect/send_request for the socket, and d) for request/reply opcodes, route through xceadmin.discovery.follow_redirects + xceadmin.discovery.decode_reply (which unwraps the Request{opcode: OPCode_reply, body: replymsg} envelope every reply is wrapped in — a real bug fixed 2026-07-30, documented in discovery.py's own docstring). Every raising path uses one exception hierarchy (XceAdminError subclasses), caught once in cli.py.
• Implications: A new xceadmin/scheduler.py module is the natural home for record_start(), record_stop(), record_status(), and dump() — same shape as push_config()/probe_config(), reusing discovery.follow_redirects/decode_reply (this is a request/reply opcode per AC 10, so unlike probe_config() it DOES need redirect-following). dump() differs from the other three: it operates on a local .replay file only, no network call — it decodes a capnp stream with xcecommon.schema.load_schema()'s module object and formats each message type to text; it belongs in the same module for CLI cohesion but has no wire dependency.

Reusable capnp payload types for ReplayEvent / SnapshotData

• Context: Requirements AC 4 explicitly defers the exact capnp schema for event payloads to design; AC 20's ReplayEvent.body union names Conf4Sched, SchedJobData, WorkerData, ResourceData, ResValList, JobStateChangeEvent, JobUpdateEvent, QueueConf, SchedJobRusages without defining the two *Event types.
• Sources Consulted: grep -n '^struct ' src/schema/controller.capnp.
• Findings: Conf4Sched, SchedJobData, WorkerData, ResourceData, ResValList, QueueConf, SchedJobRusages all already exist in controller.capnp with those exact names (the requirements author evidently cross-checked the schema before naming the union members). JobStateChangeEvent and JobUpdateEvent do not exist. The existing Job struct (controller.capnp:288) already carries jobid + state, and JobAlloc/JobPendReason exist for allocation/pend-reason payloads that the new DispatchDecision/PendDecision need.
• Implications: Design only needs to define 2 new event payload structs and 3 new decision payload structs (reusing JobAlloc and JobPendReason internally), not 9. See Data Models section of design.md.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Event-sourced core with pluggable driver (selected)	Core holds all scheduling state, driven purely by typed events + a pull-based clock; two driver implementations (normal, simulation) translate their respective input sources to the same event stream	Normal↔sim equivalence becomes a structural property, not a discipline; picker/allocator threading is unchanged from today, minimizing regression surface	Requires strict discipline (no wall-clock calls, no controller calls) inside the core — a single violation breaks the central invariant silently until someone runs --sim-verify	Already fully specified in requirements.md; design.md's job is to make the boundary and remaining schema concrete, not to re-litigate the pattern
Mode-flagged single scheduler (rejected in requirements)	Keep xce_scheduler_t as-is, add if (sim_mode) branches for clock/IO	No new types, smallest diff	Directly violates the central invariant requirements establishes (normal/sim would share code paths only by discipline, not by construction) — rejected by requirements.md Req 5 AC 1, not re-opened here	—

Design Decisions

Decision: New capnp structs for JOB_STATE_CHANGE / JOB_UPDATE events

• Context: AC 4 specifies the field set (jobid, old_state, new_state, exit_code for state-change; jobid, changed_fields, full job record for update) but no capnp shape exists.
• Alternatives Considered:
  1. Reuse Job alone and let the driver infer old/new state from two full records — rejected, doubles payload size for a two-field delta and loses exit_code.
  2. Define thin new structs that carry only the delta plus embed Job where a full record is genuinely needed — selected.
• Selected Approach: JobStateChangeEvent{jobid, oldState, newState, exitCode}; JobUpdateEvent{jobid, changedFields, job: Job} (embeds the existing Job struct, not a re-derived one).
• Rationale: Matches AC 4's field table exactly; embedding Job for the update case (where the driver already has to build a full record) avoids a second job-shape definition; the state-change case stays flat since AC 4 requires no other job fields.
• Trade-offs: JobUpdateEvent duplicates the full Job payload even for a single-field priority change — acceptable, matches AC 4's explicit field table (job: xce_job_t* full record), not a design deviation.
• Follow-up: None — field set is normative from requirements, only the capnp wrapper is new.

Decision: Decision payload structs (DispatchDecision / PendDecision / PreemptDecision) — and a boundary note on PreemptDecision

• Context: AC 5 fully specifies fields for all three decision types. Separately, the Boundary Context assigns "the preemption decision type" to the scheduler-preemption sibling spec.
• Alternatives Considered:
  1. Omit PreemptDecision entirely from this design, leave a placeholder in the ReplayDecision.body union for scheduler-preemption to add later — rejected, AC 5's decision output table is a Requirement 5 AC in this spec, and the on_decision callback signature / ReplayDecision union must compile today (Phase 2) before scheduler-preemption exists.
  2. Define PreemptDecision's wire shape now per AC 5's field table, treat victim-selection semantics (not the wire shape) as what scheduler-preemption owns — selected.
• Selected Approach: PreemptDecision{jobid, victims: List(PreemptVictim{jobid, workerid}), reason: JobPendReason} defined in this spec's schema additions.
• Rationale: The wire struct is a passive data shape with no scheduling logic in it; nothing about defining it here binds scheduler-preemption to any victim-selection algorithm. Leaving the union member undefined would make xce_decision_t/ReplayDecision incomplete types this spec's own Phase 2 code needs to reference (can_preempt()'s caller, per requirements.md Req 1 AC 4's allocator pseudocode, already branches on a preempt outcome).
• Trade-offs: If scheduler-preemption later needs additional fields on PreemptDecision, that spec must extend this schema (a revalidation trigger, listed in design.md's Boundary Commitments) rather than owning it outright from scratch.
• Follow-up: Flag explicitly to whoever picks up scheduler-preemption that the wire shape is pre-committed here; the spec should confirm the field set is sufficient before implementation.

Decision: OPCode_schedAdmin authorization mechanism — deferred to a follow-up spec (project owner call, 2026-08-02)

• Context: AC 10 states "same RBAC role as config push," but the only comparable opcode (OPCode_upgradeConfig) has no visible authorization check in cluster.c — only a primary/standby gate.
• Alternatives Considered:
  1. Assert a specific mechanism (e.g. "add a cluster:control RBAC check") without verifying one exists for config push today — rejected, would document a fictitious parity claim.
  2. Block Phase 3 until the implementer traces config push's real auth boundary and applies it here too — rejected by project owner: not worth gating this spec's delivery on.
  3. Ship OPCode_schedAdmin at parity with OPCode_upgradeConfig's current posture (primary-gate only, no RBAC check), document the gap explicitly, and track auth hardening as separate future work — selected.
• Selected Approach: Option 3. design.md's Open Questions/Risks and Security Considerations sections record this as an accepted, deliberate gap rather than an unresolved blocker.
• Rationale: config push's current authorization boundary could not be located in the controller-side handler during discovery. Rather than block this spec on retrofitting auth for two opcodes at once, the project owner chose to scope that work separately.
• Trade-offs: OPCode_schedAdmin (and, unchanged, OPCode_upgradeConfig) remain gated only by primary/standby routing until the follow-up lands — acceptable short-term since both are already TLS-connected, admin-tooling-only surfaces (xceadmin), not general network-reachable endpoints.
• Follow-up: Add an "auth hardening for admin opcodes (OPCode_upgradeConfig, OPCode_schedAdmin)" item to the project roadmap (see design.md Open Questions/Risks #1). Track as a future spec, not a task inside scheduler-core.

Decision: OPCode_schedAdmin's blocking cross-process wait — accepted as a boundary case, no admission reservation (project owner call, 2026-08-02)

• Context: Design review (2026-08-02) flagged that myreq_handler runs as a vtask_t handler (bounded 8-thread pool, cluster.c:238), and OPCode_schedAdmin blocks that thread up to 5s waiting on a cross-process reply from the scheduler daemon while holding c->sched.lock — unlike OPCode_upgradeConfig's in-process, COMMIT_WAIT_RESERVED-admitted commit-barrier wait. ha-async-io.md's vtaskmgr rule requires admission reservation for bounded blocking and forbids unbounded blocking on this pool; the raft-drive-async family fixed prior real violations of this exact rule (config-push's busy-poll+join was one of them).
• Alternatives Considered:
  1. Add a COMMIT_WAIT_RESERVED-style admission reservation for OPCode_schedAdmin before Phase 3 ships — the design-review recommendation.
  2. Make the scheduler forward asynchronous (vtask completion callback instead of blocking wait) — larger structural change, deferred as unnecessary for current usage.
  3. Accept the blocking wait as a boundary case given expected request volume — selected.
• Selected Approach: Option 3. xceadmin scheduler record/dump commands are cluster-administrator-triggered, human-paced, low-frequency — not a request class expected to burst concurrently the way heartbeat/replication traffic does. No admission-reservation mechanism is added.
• Rationale: The COMMIT_WAIT_RESERVED precedent exists because commit-barrier waits are on the hot path of every job submission (high, uncontrolled frequency). Admin recording control is operator-initiated and rare by nature; the risk this pattern protects against in the commit-barrier case doesn't apply at comparable severity here.
• Trade-offs: If usage patterns change (e.g. automated/scripted recording control invoked at scale, or a monitoring tool polling record status frequently), this becomes a live risk without warning — no test or admission gate would catch it.
• Follow-up: If scheduler record/dump usage ever becomes automated or high-frequency, revisit with a COMMIT_WAIT_RESERVED-style admission reservation.
• SUPERSEDES (2026-08-19, design-review follow-up): Option 3 above is no longer the shipped behavior. A retrospective design review flagged that c->sched.lock already serializes concurrent schedAdmin requests one at a time — a burst of N concurrent requests could occupy N vtask threads for up to N×5s each even under Option 3's "accepted, human-paced" framing, which this Decision's own Trade-offs bullet didn't fully account for. vtaskmgr_blocking_acquire/vtaskmgr_blocking_release (Option 1's mechanism, reusing commit.c's existing c->vtaskmgr reservation rather than a new one) now wrap the whole forward+wait region; admission-exhausted requests fast-reject with {ok=false, error="server busy, try again"} instead of queuing. See design.md's SchedAdminOpcode component ("Blocking-wait admission (hardened 2026-08-19)") and requirements.md Req 5 AC 10's own Amendment for the current, authoritative description.

Risks & Mitigations

• OPCode_schedAdmin ships without an RBAC/auth check beyond the primary/standby gate — accepted, not blocking. Mitigation: documented explicitly in design.md; tracked as a future roadmap item (see Decision above), not a Phase 3 dependency.
• OPCode_schedAdmin blocks a vtask thread for up to 5s per request — MITIGATED 2026-08-19. Now gated by the same vtaskmgr_blocking_acquire/_release admission reservation commit.c's commit-wait barrier shares (see the Decision's "SUPERSEDES" note above); admission-exhausted bursts fast-reject rather than occupying vtask threads.
• PreemptDecision wire shape is pre-committed before scheduler-preemption's own design exists — mitigation: documented as a revalidation trigger; scheduler-preemption must review this shape early rather than discovering it mid-implementation.
• jbuf capacity=1 determinism invariant depends on a FIXME: make this configurable comment never being resolved carelessly — mitigation: already hard-pinned in requirements.md Req 5 AC 7 with an explicit Phase 2 task; design.md's File Structure Plan marks xce_sched_core_init() as the single place this is enforced.

References

• src/scheduler/CLAUDE.md, .kiro/steering/{structure.md,tech.md,ha-async-io.md} — component layout and controller HA invariants.
• src/schema/controller.capnp — canonical wire schema; source of truth for reused struct names.
• src/xceadmin/src/xceadmin/push.py, discovery.py — reference pattern for the new scheduler.py xceadmin module.