myxce/.kiro/specs/scheduler-primary-lifecycle/research.md

Summary

• Feature: scheduler-primary-lifecycle
• Discovery Scope: Extension (existing controller + scheduler components; no new daemon type, no new external dependency)
• Key Findings:
  • slowlane_submit() (src/controller/slowlane.c) already exists, is tested, and is the exact primitive this feature needs: a bounded, admission-capped executor with per-key in-flight dedup that runs submitted work off the Raft drive threads (iomgr/delaymgr). No new async-execution mechanism needs to be built.
  • XCE_CONTROLLER_HOOK_SCHEDULER / HOOK_DETAIL_SCHEDULER_JOIN / HOOK_DETAIL_SCHEDULER_DEAD (src/include/xce/cluster_def.h) are declared but never fired or consumed anywhere in src/controller/*.c — confirmed by exhaustive grep. They are dead enum values, not existing plumbing to build on.
  • The controller already tracks scheduler connection lifecycle via on_scheduler_join() / handle_scheduler_leave() (src/controller/sched.c) and a single cluster_t.sched.ep slot — this is reusable bookkeeping for "is my child scheduler currently joined," but it has no concept today of owning the scheduler process itself.
  • The scheduler's current rejoin logic (sched_needs_rejoin/sched_on_reconnect/sched_publish_check_rejoin, src/scheduler/src/scheduler.c) and its use of the general-purpose, candidate-rotating client library (xcecli_join, XCECLI_FAILOVER_CALL) both become dead weight under the new model and should be removed, not left dormant — the requirements (3.4) explicitly forbid the scheduler from ever trying another controller.
  • ha_dispatch_gate() (src/controller/ha.c:4573) already rejects scheduler write opcodes against a non-primary controller (CTRL_STANDBY's allow-list excludes job dispatch / pend-reason). Confirmed by reading the gate's full switch statement. This means the safety argument for this feature is already true today — the feature is solely about availability, not correctness.

Research Log

Where does controller promotion actually complete?

• Context: Need the concrete integration point for "start a scheduler on promotion."
• Sources Consulted: src/controller/ha.c, raft_cb_become_leader (line 1615), read via codegraph.
• Findings: raft_cb_become_leader is long and gated: election-timeout cancellation, safe-committed-LSN computation, first-boot cluster_init_phase2, WAL truncation-floor handling, then a sequence of promotion-refusal gates (recovery-truncated, apply-divergent, self-restart-pending, apply-resource-fenced, config-serve-fenced) that each return without completing promotion, staying alive per the project's never-self-terminate invariant. Only after every gate passes does the function proceed to authoritative promotion (role flip to CTRL_PRIMARY, ha_start_primary_heartbeat, repl_add_standby for reachable voters).
• Implications: The scheduler-start action must be triggered only at the tail of this function, after all refusal gates have passed — never speculatively earlier. A promotion attempt that gets refused by any gate must not start a scheduler.

Where does demotion happen?

• Context: Need the integration point for "signal my scheduler to exit."
• Sources Consulted: src/controller/ha.c, raft_cb_become_follower (line 2120), read via codegraph.
• Findings: raft_cb_become_follower is the Raft library's on_become_follower callback, invoked from multiple paths (losing an election, observing a higher term, an append-entries request from a legitimate newer leader). This is the single funnel for every graceful role-loss case named in Requirement 2.1 (voluntary step-down and higher-term discovery both route through it).
• Implications: One hook site covers all of Requirement 2's graceful-demotion triggers; no separate handling needed per trigger type.

Is there an existing async-safe way to do blocking work from the HA path?

• Context: .kiro/steering/ha-async-io.md's thread-role contract forbids blocking work on iomgr/delaymgr, and spawning/signaling/monitoring a child process is unavoidably blocking-ish (fork/exec, waitpid, kill).
• Sources Consulted: src/controller/slowlane.c, src/controller/include/slowlane.h, existing callers in ha_convergence.c, config_repl.c, persist.c.
• Findings: slowlane_submit(sl, key, fn, arg) already exists for exactly this purpose ("long blocking I/O off the Raft drive threads"), is non-blocking to the caller, runs fn on a dedicated worker thread with no lock held, and provides per-key in-flight dedup (-EEXIST on a duplicate submit of the same key while one is still running/queued). A test_slowlane.c suite already covers non-blocking submission, dedup, and admission-cap behavior.
• Implications: Build vs. Adopt — adopt. Scheduler start/stop/monitor actions are ordinary slowlane_submit() work items. The dedup key (scoped to "this controller's scheduler-management action") mechanically satisfies Requirement 1.2/1.3 ("at most one Scheduler Process," "shall not start a second") as a side effect of reusing an already-tested primitive, with no new synchronization to design or verify.

Is there an existing pattern for one XCE daemon supervising a child process?

• Context: Wanted to know if process_worker_manager.c/uv_ipc.c (the worker daemon's job-process supervision) is directly reusable.
• Sources Consulted: src/worker/src/process_worker_manager.c, src/worker/src/uv_ipc.c (libuv-based named-pipe IPC with a heartbeat callback between the worker daemon and its forked job-executor children).
• Findings: The pattern (fork a child, communicate over a local IPC channel, monitor a heartbeat) is directly analogous to what this feature needs between a controller and its scheduler child. The concrete code, however, is tightly coupled to job-execution concerns (cgroups, dbus, job exit-status plumbing) and to libuv, a dependency the controller does not currently carry.
• Implications: Build vs. Adopt — adopt the pattern, not the code. The controller gets its own small, purpose-built scheduler-process-manager component (fork/exec + reap + a liveness signal), following the same shape (parent owns child lifecycle, detects child exit, watches liveness) without pulling in libuv or the worker's job-execution-specific machinery. Simplification lens: a shared abstraction with the worker's job-process manager was considered and rejected — the two "child processes" (a job executable vs. a companion XCE daemon) differ enough (job exit codes/signals/cgroups vs. a single well-known daemon binary with its own wire protocol) that forcing a shared interface would add an abstraction layer with only one real implementation on each side.

How does the scheduler detect its own controller is dead or no longer primary?

• Context: Requirement 3 needs an active detection mechanism, not just reliance on OS-level parent-death propagation (discussed and explicitly not relied upon as the primary mechanism, since it doesn't cover "controller process dies while the host lives").
• Sources Consulted: src/lib/client.c (xcecli_join, XCECLI_FAILOVER_CALL), existing ctrlstatus/OPCode_ctrlstatus read-only status query (confirmed answerable in every controller role per ha_dispatch_gate), src/controller/sched.c.
• Findings: The scheduler already opens a persistent, Cap'n-Proto-framed TCP connection to a controller and already has a role-agnostic status query (ctrlstatus) available in every role. Connection loss (RDHUP) is already detected by the existing transport layer everywhere else in this codebase (workers, clients) — nothing new needs to be built to notice "my controller is unreachable." What is missing is (a) pinning the scheduler to exactly one controller address with no candidate list/rotation, and (b) a periodic check of that one controller's reported role, so a controller that demotes without the connection ever dropping (the common case — TCP stays up across a graceful step-down) is still detected.
• Implications: Build vs. Adopt — adopt the existing wire protocol and status query; build only the pin-to-one-controller startup mode and the periodic self-check loop (a much smaller version of the worker's checking_connection() shape, minus any rejoin/rotation behavior). The explicit graceful-exit signal (Requirement 2) rides the same existing connection as a new, small message case in the scheduler's existing request handler (scheduler_req_handler, src/scheduler/src/handler.c) rather than a separate channel.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Controller-owned child process (selected)	Primary controller forks/execs a scheduler bound to itself; scheduler self-exits on losing that specific controller	Reuses controller's already-hardened Raft election for "who's authoritative"; host death handled for free; no new election/lease protocol	Controller gains a new, bounded process-supervision responsibility; scheduler loses independent host placement	Matches decision reached in requirements discussion
Independent scheduler + periodic reconnect (rejected)	Keep scheduler as today, add an unconditional periodic xcecli_join() retry (worker-style checking_connection())	Smaller code change; preserves independent deployment	Does not solve scheduler-host-death or wedged-scheduler detection; still needs a second mechanism for those	Considered and superseded once host-death handling became an explicit requirement
Standby schedulers + controller-granted lease (rejected)	Multiple scheduler processes compete for a renewable lease from the primary controller	Handles wedge/host-death without host co-location; scheduler keeps independent placement	A second, parallel authority/liveness protocol to design, build, and prove correct, duplicating the controller's own Raft election in spirit	Rejected in favor of piggybacking on existing Raft election (simplification lens)
General per-host role-manager agent (deferred)	A new component per host starts/stops worker/controller/scheduler/apiserver by declared role and cluster state	Generalizes beyond just scheduler	New always-on process whose own supervision is unsolved; would need to watch controller election state itself, adding latency and an indirection layer	Explicitly deferred in requirements as a separate, independently-motivated initiative, not scope for this spec

Design Decisions

Decision: Use slowlane_submit() for all scheduler process management

• Context: Starting, monitoring, and signaling a child process from raft_cb_become_leader/raft_cb_become_follower must not block the Raft drive threads.
• Alternatives Considered:
  1. Spawn a brand-new dedicated pthread per promotion — works, but reinvents admission control and dedup that already exist.
  2. Do the fork/exec inline in the Raft callback — violates the thread-role contract outright (fork/exec/waitpid are blocking).
• Selected Approach: Submit scheduler-management actions (start on promotion, signal-and-reap on demotion, restart on detected wedge) to the existing SlowLane executor with a per-controller-instance dedup key.
• Rationale: Already built, already tested, already the established pattern for exactly this class of problem in this codebase (raft-drive-async spec).
• Trade-offs: None material — this is a pure reuse with no new abstraction.
• Follow-up: Confirm the SlowLane instance's max_inflight/n_workers sizing (set at slowlane_init() call sites in cluster.c) has headroom for scheduler-management tasks alongside its existing consumers (peer connect, snapshot transfer, config-push); this is a sizing check, not a new design.

Decision: Scheduler pins to exactly one controller address at startup, no candidate list

• Context: Requirement 3.4 forbids the scheduler from ever joining or continuing to operate against any controller other than the one that started it.
• Alternatives Considered:
  1. Keep the existing --peers full-candidate-list startup mode and just suppress rotation in the failover-call path — leaves a footgun where the shape of the config still implies "any of these are fine."
  2. Give the scheduler a single-controller startup mode entirely, with no candidate list construct at all.
• Selected Approach: Option 2. The controller passes its own address (not the full peer list) when it starts its scheduler.
• Rationale: Makes the "one controller owns one scheduler" invariant structurally true from the scheduler's own configuration, not just an unenforced behavioral convention.
• Trade-offs: The scheduler's startup contract changes (breaking change to its CLI/config surface); existing deployment tooling that passes a full peer list to xce_scheduler must be updated (see Requirement 6 / Integration & Migration Notes in design.md).
• Follow-up: Confirm during implementation whether any other in-tree consumer starts xce_scheduler with a multi-candidate list for a reason unrelated to this feature (e.g. a test fixture) — such usages need explicit review, not silent breakage.

Risks & Mitigations

• Risk: A rapid succession of promotions/demotions (chaotic failover) could thrash scheduler process spawn/kill cycles. — Mitigation: the SlowLane dedup key naturally coalesces overlapping start requests for the same controller instance into at most one in-flight action; a demotion always signals-and-reaps before that controller instance could ever legitimately be promoted again (a demoted node must win a fresh election first).
• Risk: The wedged-scheduler health check (Requirement 5) needs a "making progress" signal, not just an open connection — inventing a bespoke internal probe risks becoming its own source of false positives. — Mitigation: reuse the existing job/queue/config push traffic the controller already sends its scheduler as the progress signal (e.g., confirmation that a recent push was acted on), rather than adding a new synthetic heartbeat payload; finalize the exact signal during implementation against real traffic patterns.
• Risk: Existing tooling and specs (xce-deploy-ha, xce-test-ha's soak harness, any other spec assuming an independent, reconnecting scheduler) silently drift out of sync with the new model. — Mitigation: explicit File Structure Plan entries in design.md for every piece of tooling identified as depending on today's model (see design.md Integration & Migration Notes).

References

• .wolf/buglog.json bug-1760 — original live repro motivating this spec.
• .kiro/specs/raft-drive-async/design.md — original SlowLane design (cited verbatim in slowlane.h's own header comment).
• .kiro/steering/ha-async-io.md — thread-role contract and never-self-terminate invariant this design must respect.