myxce/.kiro/specs/scheduler-primary-lifecycle/requirements.md

Requirements Document

Project Description (Input)

Who has the problem: XCE cluster operators, and by extension every job that depends on the scheduler to be reachable after a controller failover.

Current situation: the scheduler (xce_scheduler) runs today as a fully independent, long-lived process that joins the controller cluster via xcecli_join(HostRole_scheduler) and publishes dispatch/pend-reason decisions via sched_publish_dispatch()/sched_publish_pend_reason(). Its only rejoin mechanism after a controller failover is reactive: sched_publish_check_rejoin() re-issues xcecli_join() only as a side effect of its own next successful publish call. After a real controller failover this creates a chicken-and-egg deadlock — the scheduler has nothing queued to publish (no new job ever reaches its picker, since the controller-side registration of "where the scheduler is" is live-connection-scoped state on whichever controller instance the scheduler last explicitly joined, and does not appear to carry over to a newly-elected primary) — so it can never publish anything to trigger its own rejoin. This was live-confirmed against a real 3-node cluster: after killing the primary, the scheduler showed zero log activity of any kind (no WARN/INFO/ERROR, no reconnect attempt) for 60+ seconds, while a job submitted to the new primary sat stuck PEND. Logged as bug-1760 in .wolf/buglog.json with full repro evidence. Separately, there is today no story at all for what happens if the scheduler's own host/process dies, is network-partitioned, or wedges internally (e.g. a disk stall blocking its recording I/O, or a deadlocked allocator thread) — no periodic health check, no standby, no automatic recovery.

What should change (scope decided across discussion, see rationale below): bind the scheduler process's lifecycle to the controller cluster's primary role, replacing its current independent-process-with-reactive-rejoin model entirely — including replacing its client-side candidate-rotation/rejoin logic, not just supplementing it.

• When a controller transitions to CTRL_PRIMARY (wins election / is promoted), it spawns and owns a scheduler process co-located on its own host (a genuine child process, not a signal to some independently-running scheduler).
• When that controller loses primary status gracefully (demoted, discovers a higher term, steps down), it explicitly signals its child scheduler to exit before completing the transition.
• The scheduler process itself actively monitors whether the specific controller instance that started it is still alive and still primary, and terminates itself the moment it can no longer confirm that — this is the primary detection mechanism (not a passive fallback), because it also covers cases OS-level parent-death propagation cannot guarantee (e.g. the controller process dying while its host survives). Co-located death (host crash, etc.) provides a redundant backstop for the same outcome. Critically, the scheduler does not attempt to rediscover or reconnect to whichever controller becomes the next primary — that job belongs entirely to the next primary's own act of spawning a fresh scheduler. This is a deliberate simplification: a scheduler's operating logic no longer needs any cross-primary candidate-rotation or rejoin behavior at all, because its lifetime is scoped to exactly one controller instance's primacy.
• The newly-elected primary spawns its own fresh scheduler exactly as it would on any other promotion, via the controller's existing, already-hardened Raft election — no new election/lease mechanism for schedulers.
• This eliminates: sched_needs_rejoin/sched_publish_check_rejoin's reactive rejoin logic and the scheduler's client-side failover candidate rotation for its own role (no longer needed — a scheduler's lifetime is now scoped to a single controller's primacy, so it never needs to survive and reconnect across a primary change); the need for a scheduler-side lease/renewal mechanism; and the need for standby scheduler processes — host/process death is subsumed into "controller host dies," a failure mode the controller's HA layer already handles robustly, so no second, parallel HA/authority mechanism is being built for schedulers.
• In scope, smaller: a liveness/health check the parent controller performs on its own live child scheduler, to catch the case where the process is alive (so co-location death didn't trigger) but internally wedged (e.g. blocked disk I/O, deadlocked picker/allocator thread) and therefore not doing useful work despite an open connection. This does not require a lease or standby scheduler — it is a local parent-watches-child health signal, and on failure the controller may restart its own child scheduler.
• Explicitly OUT of scope for this spec (raised and deliberately deferred during discussion):
  • Standby/redundant scheduler processes competing for a lease — rejected in favor of piggybacking on the controller's own already-tested election mechanism.
  • A general per-host "role manager" component that starts/stops arbitrary XCE components (worker, controller, scheduler, apiserver) based on declared host roles and cluster state. This was discussed as a possible bigger direction but deliberately deferred: the controller is uniquely positioned to trigger scheduler startup at zero extra latency the instant it becomes primary, whereas a separate agent would need to watch controller election state itself and introduces a new process whose own supervision becomes a new problem. If a genuine, independent need for general host/role management across all XCE components emerges later, it should be its own spec, not inherited scope from this one.
  • Any change to how worker or apiserver processes are deployed/supervised.

Constraint carried into design: scheduler and controller currently value independent deployability (separate hosts, separate scaling/tuning, separate release cadence per .kiro/steering/structure.md's component layout). This spec deliberately trades that independence away for the scheduler specifically, in exchange for eliminating an entire class of availability bugs (reactive rejoin deadlock, host-death handling, wedge detection) by reusing already-hardened controller HA machinery instead of building a parallel one. The design phase must make this tradeoff and its consequences (scheduler can no longer run on a dedicated host, failover now includes scheduler cold-start time, controller gains new child-process-supervision responsibility) explicit rather than incidental.

Related code to ground requirements/design in: src/controller/ha.c (promotion/demotion — e.g. ha_start_primary_heartbeat, raft_cb_become_leader; ha_dispatch_gate already safely rejects scheduler writes against a non-primary, confirmed by reading the code, so no split-brain risk exists today from a stale scheduler connection — this spec is solely about availability, not correctness), src/controller/sched.c (on_scheduler_join(), handle_scheduler_leave() — existing controller-side scheduler-connection lifecycle hooks), src/include/xce/cluster_def.h (XCE_CONTROLLER_HOOK_SCHEDULER, HOOK_DETAIL_SCHEDULER_JOIN/HOOK_DETAIL_SCHEDULER_DEAD hooks already exist and may be relevant plumbing), src/scheduler/src/scheduler.c (sched_needs_rejoin/sched_on_reconnect/sched_publish_check_rejoin — the mechanism being replaced), src/worker/src/process_worker_manager.c (existing precedent in this codebase for a daemon spawning/supervising a child process, though for job execution rather than another XCE daemon), .kiro/steering/ha-async-io.md (controller-never-self-terminate invariant, and the thread-role contract that no liveness-carrying thread — iomgr/delaymgr — may block; any new spawn/monitor/signal mechanism the controller gains must respect this), .claude/skills/xce-deploy-ha/SKILL.md and .claude/skills/xce-test-ha/run-ha-soak.sh (existing tooling that models the scheduler as an independently-started process via start-scheduler.sh — needs to be reconciled with the new model, since multiple existing specs/soak-harness modes depend on today's independent-process assumption). Bug-1760 in .wolf/buglog.json has the full live-repro evidence motivating this spec.

Boundary Context

• In scope: controller-triggered scheduler startup on promotion; graceful scheduler shutdown signaled on demotion; scheduler self-termination via active monitoring of its own originating controller's liveness/primary status; controller detection of, and response to, an unresponsive-but-alive ("wedged") child scheduler; controller resilience so that scheduler-lifecycle management never compromises the controller's own availability or HA correctness; reconciling existing deployment tooling and specs that assume an independently-started scheduler.
• Out of scope: standby/redundant scheduler processes and any lease-election mechanism for schedulers; a general host/role manager governing worker, controller, or apiserver process lifecycle; any change to worker or apiserver deployment/supervision.
• Adjacent expectations: this feature depends on the controller's existing Raft-based primary election (unchanged by this spec) as the sole source of truth for "who may run a scheduler right now." It also depends on the existing ha_dispatch_gate behavior (a non-primary controller already rejects scheduler write operations) continuing to hold, since this spec's safety argument rests on that already-verified behavior rather than re-implementing it. Existing operator-facing deployment tooling that starts a scheduler as an independent process (documented in xce-deploy-ha/xce-test-ha) is expected to change to reflect controller-owned scheduler startup; any spec or tooling that still assumes an independently-started, long-lived scheduler process is an adjacent consumer that must be revisited, not an unstated exception to this feature's behavior.

Requirements

Requirement 1: Scheduler startup on controller promotion

Objective: As a cluster operator, I want a scheduler to start automatically whenever a controller becomes primary, so that the cluster can resume scheduling jobs without any manual intervention after a failover.

Acceptance Criteria

1. When a controller transitions to the primary role, the Controller shall start a Scheduler Process associated with itself.
2. While a controller remains primary, the Controller shall ensure at most one Scheduler Process it started is running.
3. If a controller is already primary and already has a running Scheduler Process associated with itself, the Controller shall not start a second Scheduler Process.
4. When the Controller starts a Scheduler Process, the Scheduler Process shall join the cluster and become able to receive job, queue, and configuration state from that specific controller instance.
5. If the Controller fails to start a Scheduler Process, the Controller shall record an operator-visible error and shall continue to retry starting one while it remains primary.

Requirement 2: Scheduler shutdown on graceful controller demotion

Objective: As a cluster operator, I want the scheduler tied to a controller to stop cleanly when that controller steps down, so that no scheduler keeps running on behalf of a controller that is no longer authoritative.

Acceptance Criteria

1. When a controller that started a Scheduler Process transitions away from the primary role through a graceful demotion (voluntary step-down, or discovering a higher election term while still running), the Controller shall signal that Scheduler Process to terminate.
2. When a Scheduler Process receives a termination signal from its originating controller, the Scheduler Process shall exit within a bounded time.
3. The Controller shall not remain in the primary-to-standby transition indefinitely waiting on its Scheduler Process to exit; a Scheduler Process that does not exit within the bounded time shall not prevent the Controller from completing its own role transition.
4. When a Scheduler Process is terminated for any reason (graceful signal, self-detected loss of its originating primary, or abrupt process/host death), no job shall be left in an inconsistent or lost state as a result — any job the terminated Scheduler Process had not yet caused to be durably recorded shall remain available for a subsequent Scheduler Process to consider.

Requirement 3: Scheduler self-termination on loss of its originating primary controller

Objective: As a cluster operator, I want the scheduler itself to notice when its own controller is no longer a live primary and exit on its own, so that a scheduler can never keep running detached from any authoritative controller, and so that no cross-controller reconnect logic is needed in the scheduler at all.

Acceptance Criteria

1. While a Scheduler Process is running, the Scheduler Process shall continuously verify that the specific controller instance that started it is reachable and still holds the primary role.
2. When a Scheduler Process can no longer confirm that its originating controller instance is reachable, the Scheduler Process shall terminate.
3. When a Scheduler Process learns that its originating controller instance is no longer primary, the Scheduler Process shall terminate, even if that controller instance is still reachable.
4. The Scheduler Process shall not attempt to join, rejoin, or continue operating against any controller instance other than the one that started it.
5. If the host or process running a Scheduler Process terminates abruptly (crash, host failure) before the Scheduler Process can detect the loss of its originating controller on its own, no further scheduler-side action is required — the Scheduler Process is expected to be gone as a direct consequence of that failure.

Requirement 4: Controller resilience to scheduler-lifecycle management

Objective: As a cluster operator, I want a controller's own availability and correctness to never depend on its child scheduler's health, so that scheduler-management responsibilities cannot introduce a new way for the controller to become unavailable.

Acceptance Criteria

1. The Controller shall never block its own heartbeat processing or election-timeout handling while starting, monitoring, or signaling a Scheduler Process.
2. If starting, monitoring, or signaling a Scheduler Process fails for any reason, the Controller shall not crash, hang, or self-terminate as a result.
3. The Controller shall log an operator-visible error whenever it fails to start, loses, or forcibly restarts a Scheduler Process it owns.
4. The absence of a running Scheduler Process shall not prevent a controller from being elected primary, remaining primary, or serving read-only and replication traffic.

Requirement 5: Detecting and recovering a wedged Scheduler Process

Objective: As a cluster operator, I want the controller to notice when its own scheduler is alive but not making progress, so that jobs do not silently stop being scheduled while everything still looks connected.

Acceptance Criteria

1. While a Controller has a running Scheduler Process, the Controller shall monitor whether that Scheduler Process is making scheduling progress, not merely whether its connection remains open.
2. If a Scheduler Process is connected but fails to make scheduling progress for longer than an operator-observable threshold, the Controller shall treat that Scheduler Process as unresponsive.
3. When the Controller treats its Scheduler Process as unresponsive, the Controller shall terminate that Scheduler Process and start a replacement.
4. The Controller shall log an operator-visible event whenever it terminates a Scheduler Process because it was found unresponsive.

Requirement 6: Compatibility with existing scheduler operation and tooling

Objective: As a cluster operator or spec author relying on existing scheduler deployment tooling, I want the change in scheduler ownership to be reflected everywhere the scheduler's lifecycle is currently assumed to be independent, so that existing deployments, tests, and specs do not silently break or become misleading.

Acceptance Criteria

1. Where existing deployment tooling documents starting a scheduler as an independent, manually-managed process, that tooling shall be updated to reflect that a running controller now starts its own scheduler automatically.
2. Where an existing spec or test tooling depends on a scheduler surviving and reconnecting across a controller failover, that dependency shall be identified and reconciled with this feature's behavior rather than left silently incompatible.
3. The Scheduler Process's observable behavior toward jobs, queues, and configuration (join, receive state, dispatch/pend decisions) shall remain unchanged by this feature for the duration that it is running under a live primary controller.