# Implementation Plan - [ ] 1. Foundation: wire protocol schema and scheduler startup contract - [ ] 1.1 Define the controller-to-scheduler graceful-exit wire message - Add a new message the controller can send an already-joined scheduler over their existing connection, carrying no payload beyond identifying it as a graceful-exit request - Observable completion: the schema builds cleanly and the new message is visible in the generated bindings, unused by any caller yet - _Requirements: 2.1, 2.2_ - [ ] 1.2 (P) Give the scheduler a single-controller startup mode - Replace the scheduler's current multi-candidate `--peers`-style startup configuration with a mode that accepts exactly one controller address - Observable completion: the scheduler binary accepts the new single-address startup argument; the existing multi-candidate argument is no longer accepted for this role - _Requirements: 3.4_ - _Boundary: scheduler main.c_ - [ ] 2. Core: controller-side scheduler process manager - [ ] 2.1 Start a scheduler process safely off the Raft drive threads - Implement the start action (spawn via `posix_spawn`, passing the controller's own address using task 1.2's new startup mode) submitted through the existing SlowLane executor with a dedup key, never invoked inline from a Raft callback - Observable completion: requesting a start when one is already in flight or running is deduped (no second process spawned); a start whose spawn fails is logged and does not crash or block the caller - _Requirements: 1.1, 1.2, 1.3, 1.5, 4.1, 4.3_ - _Boundary: scheduler_proc.c_ - _Depends: 1.2_ - [ ] 2.2 Stop a scheduler process gracefully, with a bounded force-terminate fallback - Implement the stop action: send the graceful-exit wire message from task 1.1 over the existing connection, wait up to a bounded timeout for the process to exit, then force-terminate it - Observable completion: a scheduler that exits on its own within the timeout is not force-terminated; one that does not is force-terminated exactly once after the timeout elapses; the caller never blocks waiting for this to finish - _Requirements: 2.1, 2.3, 2.4, 4.2, 4.3_ - _Boundary: scheduler_proc.c_ - _Depends: 1.1_ - [ ] 2.3 (P) Track whether the owned scheduler is making real scheduling progress - Extend the controller's existing scheduler join/leave bookkeeping with a last-progress timestamp, updated whenever a push to the scheduler is acknowledged or a dispatch/pend-reason decision is received from it - Observable completion: the timestamp advances on real scheduler activity, using only existing traffic — no new synthetic ping is introduced - _Requirements: 5.1, 5.2_ - _Boundary: sched.c_ - [ ] 2.4 Detect a wedged scheduler and replace it - Implement a liveness check that reads task 2.3's progress signal; if the scheduler is connected but has made no progress past a configured threshold, submit a restart action (stop via task 2.2, then start via task 2.1) through the same SlowLane path - Observable completion: a connected scheduler with no progress past the threshold is terminated and replaced exactly once per stale period, with an operator-visible log; a scheduler that is merely disconnected (already handled by existing join/leave bookkeeping) is not double-counted as a wedge - _Requirements: 5.3, 5.4, 4.1, 4.3_ - _Boundary: scheduler_proc.c_ - _Depends: 2.1, 2.2, 2.3_ - [ ] 3. (P) Core: pin the scheduler to its one controller and watch it - Using task 1.2's single-address startup mode, connect only to that one controller; periodically query its reachability and role over that same connection - On an unreachable controller, a reported non-primary role, or receipt of task 1.1's graceful-exit message via the existing request handler, exit the process - Observable completion: the scheduler process exits within one check interval of its controller becoming unreachable or non-primary, or immediately on receiving the graceful-exit message; it never attempts any address other than the one it started with - _Requirements: 2.2, 3.1, 3.2, 3.3, 3.4, 3.5_ - _Boundary: sched_owner.c, scheduler request handler_ - _Depends: 1.1, 1.2_ - [ ] 4. Integration: wire promotion, demotion, and liveness/retry checking into controller HA role transitions - [ ] 4.1 Start the scheduler on promotion - Call task 2.1's start action at the point a controller completes promotion, after every existing promotion-refusal gate has already passed - Observable completion: a real promotion results in exactly one scheduler process being started, joining, and receiving job/queue/config state; a refused promotion attempt (any existing gate) never starts one - _Requirements: 1.1, 1.4, 4.4_ - _Boundary: ha.c_ - _Depends: 2.1_ - [ ] 4.2 Stop the scheduler on graceful demotion - Call task 2.2's stop action at the point a controller begins a graceful role transition away from primary, without waiting for it to complete - Observable completion: a real graceful demotion's own role transition completes without waiting on the scheduler's exit; the scheduler process is gone within the bounded timeout - _Requirements: 2.1, 2.3_ - _Boundary: ha.c_ - _Depends: 2.2_ - [ ] 4.3 Wire periodic scheduler presence and liveness checking into the controller's existing cadence - While primary with no scheduler currently owned (e.g. a prior start attempt failed), retry task 2.1's start action on the controller's existing heartbeat-driven cadence — the mechanism Requirement 1.5's retry depends on, with no new timer introduced - While primary with a scheduler owned, run task 2.4's liveness check on that same cadence - Observable completion: a scheduler that failed to start is retried on the next cadence tick without a new timer thread; a wedged scheduler is detected and replaced within one cadence tick of crossing the threshold - _Requirements: 1.5, 5.1, 5.2, 5.3, 5.4_ - _Boundary: ha.c_ - _Depends: 2.1, 2.4_ - [ ] 5. Validation: end-to-end coverage and deployment tooling reconciliation - [ ] 5.1 Full promotion round trip against a real cluster - Exercise a real controller cluster promotion end to end: a scheduler process appears, joins, and receives job/queue/config state with no manual step - Observable completion: the round trip produces a working, job-dispatching scheduler in one live test run - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5_ - _Depends: 4.1_ - [ ] 5.2 Graceful demotion round trip - Force a graceful step-down against a real cluster and confirm the prior primary's scheduler exits within the bounded timeout, verified by process absence - Observable completion: the demoted controller's own role transition completes promptly and the scheduler process is confirmed gone afterward - _Requirements: 2.1, 2.2, 2.3_ - _Depends: 4.2, 3_ - [ ] 5.3 A job survives scheduler termination mid-decision - Dispatch a job to a running scheduler, kill the scheduler process before its dispatch decision is durably recorded, and confirm the job remains available (not lost, not left inconsistent) for a subsequently-started scheduler to consider - Observable completion: the job is present and correctly schedulable after the replacement scheduler starts, with no trace of the killed scheduler's incomplete decision - _Requirements: 2.4_ - _Depends: 2.2_ - [ ] 5.4 Bug-1760 original repro: hard-killed primary recovers automatically - Reproduce this feature's own motivating scenario: hard-kill the primary controller and confirm its orphaned scheduler self-exits, the newly-elected primary starts its own fresh scheduler, and a job submitted after the failover dispatches successfully - Observable completion: the exact bug-1760 symptom (a job stuck PEND after a real controller failover) does not reproduce - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5_ - _Depends: 3, 4.1_ - [ ] 5.5 Wedged scheduler is detected and recovered live - Artificially wedge a running scheduler's progress signal while its connection stays open, against a real cluster, and confirm it is detected and replaced - Observable completion: the wedged scheduler is terminated and a working replacement is dispatching jobs again within the configured threshold - _Requirements: 5.1, 5.2, 5.3, 5.4_ - _Depends: 4.3_ - [ ] 5.6 Rapid promotion and demotion does not thrash scheduler processes - Drive a rapid sequence of promotions and demotions against a real cluster and confirm no more than one scheduler spawn/kill cycle occurs per genuine promotion - Observable completion: no orphaned or duplicate scheduler processes remain after the sequence settles - _Requirements: 4.1, 4.2, 4.3, 4.4_ - _Depends: 5.1, 5.2_ - [ ] 5.7 Update xce-deploy-ha for controller-owned scheduler startup - Update the deployment skill's documentation and generated scripts so they no longer describe or provide a manual, independent scheduler-start step - Observable completion: following the updated deployment steps produces a working scheduler with no manual scheduler-start action, matching task 5.1's proof - _Requirements: 6.1_ - _Depends: 5.1, 5.2_ - [ ] 5.8 Update the HA soak harness and sweep for other dependent tooling - Remove the soak harness's own manual scheduler start/restart logic (added as this feature's own motivating bug's workaround) and replace it with a check that a working scheduler is present after each round, with no harness-side management action - Also search other specs and test tooling in the repository for the same independently-started, reconnecting-scheduler assumption and flag or reconcile any found, not just the soak harness - Observable completion: the soak harness's sequential and chaotic modes pass unmodified in every other respect, with the manual scheduler-management code paths gone; any other dependent spec/tool found is either reconciled or explicitly flagged with a stated reason it is not - _Requirements: 6.2_ - _Depends: 5.4_ - [ ] 6. Cleanup: remove the scheduler's superseded cross-controller rejoin logic - Remove the reactive rejoin mechanism and the general multi-candidate failover-rotation path for the scheduler's own role, now that task 3 fully replaces its purpose and tasks 4-5 have proven and reconciled the new model end to end — matching design.md's own Migration Strategy ordering, where this removal is deliberately the last, hardest-to-reverse phase - Observable completion: the removed functions no longer exist; the existing scheduler test suite passes unmodified with no regression; the scheduler's own job/queue/config handling behavior is unchanged - _Requirements: 3.4, 6.3_ - _Boundary: scheduler.c_ - _Depends: 3, 4.1, 4.2, 5.7, 5.8_