Rongsong Shen
0f7699deba
Requirements and design approved (design fixed the fork()-vs-posix_spawn mutex-inheritance-deadlock risk product.md's own "no fork() for snapshots" decision already rejected); tasks generated but never approved. Superseded by xce-node-manager after the user decided the controller should not directly own spawning/killing its own scheduler process -- a new per-node xce-node-manager component takes over that responsibility instead. Kept in place as a reference (notably the fork()-vs-posix_spawn analysis); this spec's tasks.md must not be implemented (see spec.json's superseded_note). Checkpointing uncommitted spec work found in this shared worktree. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
129 lines
10 KiB
Markdown
129 lines
10 KiB
Markdown
# Implementation Plan
|
|
|
|
- [ ] 1. Foundation: wire protocol schema and scheduler startup contract
|
|
- [ ] 1.1 Define the controller-to-scheduler graceful-exit wire message
|
|
- Add a new message the controller can send an already-joined scheduler over their existing connection, carrying no payload beyond identifying it as a graceful-exit request
|
|
- Observable completion: the schema builds cleanly and the new message is visible in the generated bindings, unused by any caller yet
|
|
- _Requirements: 2.1, 2.2_
|
|
|
|
- [ ] 1.2 (P) Give the scheduler a single-controller startup mode
|
|
- Replace the scheduler's current multi-candidate `--peers`-style startup configuration with a mode that accepts exactly one controller address
|
|
- Observable completion: the scheduler binary accepts the new single-address startup argument; the existing multi-candidate argument is no longer accepted for this role
|
|
- _Requirements: 3.4_
|
|
- _Boundary: scheduler main.c_
|
|
|
|
- [ ] 2. Core: controller-side scheduler process manager
|
|
- [ ] 2.1 Start a scheduler process safely off the Raft drive threads
|
|
- Implement the start action (spawn via `posix_spawn`, passing the controller's own address using task 1.2's new startup mode) submitted through the existing SlowLane executor with a dedup key, never invoked inline from a Raft callback
|
|
- Observable completion: requesting a start when one is already in flight or running is deduped (no second process spawned); a start whose spawn fails is logged and does not crash or block the caller
|
|
- _Requirements: 1.1, 1.2, 1.3, 1.5, 4.1, 4.3_
|
|
- _Boundary: scheduler_proc.c_
|
|
- _Depends: 1.2_
|
|
|
|
- [ ] 2.2 Stop a scheduler process gracefully, with a bounded force-terminate fallback
|
|
- Implement the stop action: send the graceful-exit wire message from task 1.1 over the existing connection, wait up to a bounded timeout for the process to exit, then force-terminate it
|
|
- Observable completion: a scheduler that exits on its own within the timeout is not force-terminated; one that does not is force-terminated exactly once after the timeout elapses; the caller never blocks waiting for this to finish
|
|
- _Requirements: 2.1, 2.3, 2.4, 4.2, 4.3_
|
|
- _Boundary: scheduler_proc.c_
|
|
- _Depends: 1.1_
|
|
|
|
- [ ] 2.3 (P) Track whether the owned scheduler is making real scheduling progress
|
|
- Extend the controller's existing scheduler join/leave bookkeeping with a last-progress timestamp, updated whenever a push to the scheduler is acknowledged or a dispatch/pend-reason decision is received from it
|
|
- Observable completion: the timestamp advances on real scheduler activity, using only existing traffic — no new synthetic ping is introduced
|
|
- _Requirements: 5.1, 5.2_
|
|
- _Boundary: sched.c_
|
|
|
|
- [ ] 2.4 Detect a wedged scheduler and replace it
|
|
- Implement a liveness check that reads task 2.3's progress signal; if the scheduler is connected but has made no progress past a configured threshold, submit a restart action (stop via task 2.2, then start via task 2.1) through the same SlowLane path
|
|
- Observable completion: a connected scheduler with no progress past the threshold is terminated and replaced exactly once per stale period, with an operator-visible log; a scheduler that is merely disconnected (already handled by existing join/leave bookkeeping) is not double-counted as a wedge
|
|
- _Requirements: 5.3, 5.4, 4.1, 4.3_
|
|
- _Boundary: scheduler_proc.c_
|
|
- _Depends: 2.1, 2.2, 2.3_
|
|
|
|
- [ ] 3. (P) Core: pin the scheduler to its one controller and watch it
|
|
- Using task 1.2's single-address startup mode, connect only to that one controller; periodically query its reachability and role over that same connection
|
|
- On an unreachable controller, a reported non-primary role, or receipt of task 1.1's graceful-exit message via the existing request handler, exit the process
|
|
- Observable completion: the scheduler process exits within one check interval of its controller becoming unreachable or non-primary, or immediately on receiving the graceful-exit message; it never attempts any address other than the one it started with
|
|
- _Requirements: 2.2, 3.1, 3.2, 3.3, 3.4, 3.5_
|
|
- _Boundary: sched_owner.c, scheduler request handler_
|
|
- _Depends: 1.1, 1.2_
|
|
|
|
- [ ] 4. Integration: wire promotion, demotion, and liveness/retry checking into controller HA role transitions
|
|
- [ ] 4.1 Start the scheduler on promotion
|
|
- Call task 2.1's start action at the point a controller completes promotion, after every existing promotion-refusal gate has already passed
|
|
- Observable completion: a real promotion results in exactly one scheduler process being started, joining, and receiving job/queue/config state; a refused promotion attempt (any existing gate) never starts one
|
|
- _Requirements: 1.1, 1.4, 4.4_
|
|
- _Boundary: ha.c_
|
|
- _Depends: 2.1_
|
|
|
|
- [ ] 4.2 Stop the scheduler on graceful demotion
|
|
- Call task 2.2's stop action at the point a controller begins a graceful role transition away from primary, without waiting for it to complete
|
|
- Observable completion: a real graceful demotion's own role transition completes without waiting on the scheduler's exit; the scheduler process is gone within the bounded timeout
|
|
- _Requirements: 2.1, 2.3_
|
|
- _Boundary: ha.c_
|
|
- _Depends: 2.2_
|
|
|
|
- [ ] 4.3 Wire periodic scheduler presence and liveness checking into the controller's existing cadence
|
|
- While primary with no scheduler currently owned (e.g. a prior start attempt failed), retry task 2.1's start action on the controller's existing heartbeat-driven cadence — the mechanism Requirement 1.5's retry depends on, with no new timer introduced
|
|
- While primary with a scheduler owned, run task 2.4's liveness check on that same cadence
|
|
- Observable completion: a scheduler that failed to start is retried on the next cadence tick without a new timer thread; a wedged scheduler is detected and replaced within one cadence tick of crossing the threshold
|
|
- _Requirements: 1.5, 5.1, 5.2, 5.3, 5.4_
|
|
- _Boundary: ha.c_
|
|
- _Depends: 2.1, 2.4_
|
|
|
|
- [ ] 5. Validation: end-to-end coverage and deployment tooling reconciliation
|
|
- [ ] 5.1 Full promotion round trip against a real cluster
|
|
- Exercise a real controller cluster promotion end to end: a scheduler process appears, joins, and receives job/queue/config state with no manual step
|
|
- Observable completion: the round trip produces a working, job-dispatching scheduler in one live test run
|
|
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5_
|
|
- _Depends: 4.1_
|
|
|
|
- [ ] 5.2 Graceful demotion round trip
|
|
- Force a graceful step-down against a real cluster and confirm the prior primary's scheduler exits within the bounded timeout, verified by process absence
|
|
- Observable completion: the demoted controller's own role transition completes promptly and the scheduler process is confirmed gone afterward
|
|
- _Requirements: 2.1, 2.2, 2.3_
|
|
- _Depends: 4.2, 3_
|
|
|
|
- [ ] 5.3 A job survives scheduler termination mid-decision
|
|
- Dispatch a job to a running scheduler, kill the scheduler process before its dispatch decision is durably recorded, and confirm the job remains available (not lost, not left inconsistent) for a subsequently-started scheduler to consider
|
|
- Observable completion: the job is present and correctly schedulable after the replacement scheduler starts, with no trace of the killed scheduler's incomplete decision
|
|
- _Requirements: 2.4_
|
|
- _Depends: 2.2_
|
|
|
|
- [ ] 5.4 Bug-1760 original repro: hard-killed primary recovers automatically
|
|
- Reproduce this feature's own motivating scenario: hard-kill the primary controller and confirm its orphaned scheduler self-exits, the newly-elected primary starts its own fresh scheduler, and a job submitted after the failover dispatches successfully
|
|
- Observable completion: the exact bug-1760 symptom (a job stuck PEND after a real controller failover) does not reproduce
|
|
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5_
|
|
- _Depends: 3, 4.1_
|
|
|
|
- [ ] 5.5 Wedged scheduler is detected and recovered live
|
|
- Artificially wedge a running scheduler's progress signal while its connection stays open, against a real cluster, and confirm it is detected and replaced
|
|
- Observable completion: the wedged scheduler is terminated and a working replacement is dispatching jobs again within the configured threshold
|
|
- _Requirements: 5.1, 5.2, 5.3, 5.4_
|
|
- _Depends: 4.3_
|
|
|
|
- [ ] 5.6 Rapid promotion and demotion does not thrash scheduler processes
|
|
- Drive a rapid sequence of promotions and demotions against a real cluster and confirm no more than one scheduler spawn/kill cycle occurs per genuine promotion
|
|
- Observable completion: no orphaned or duplicate scheduler processes remain after the sequence settles
|
|
- _Requirements: 4.1, 4.2, 4.3, 4.4_
|
|
- _Depends: 5.1, 5.2_
|
|
|
|
- [ ] 5.7 Update xce-deploy-ha for controller-owned scheduler startup
|
|
- Update the deployment skill's documentation and generated scripts so they no longer describe or provide a manual, independent scheduler-start step
|
|
- Observable completion: following the updated deployment steps produces a working scheduler with no manual scheduler-start action, matching task 5.1's proof
|
|
- _Requirements: 6.1_
|
|
- _Depends: 5.1, 5.2_
|
|
|
|
- [ ] 5.8 Update the HA soak harness and sweep for other dependent tooling
|
|
- Remove the soak harness's own manual scheduler start/restart logic (added as this feature's own motivating bug's workaround) and replace it with a check that a working scheduler is present after each round, with no harness-side management action
|
|
- Also search other specs and test tooling in the repository for the same independently-started, reconnecting-scheduler assumption and flag or reconcile any found, not just the soak harness
|
|
- Observable completion: the soak harness's sequential and chaotic modes pass unmodified in every other respect, with the manual scheduler-management code paths gone; any other dependent spec/tool found is either reconciled or explicitly flagged with a stated reason it is not
|
|
- _Requirements: 6.2_
|
|
- _Depends: 5.4_
|
|
|
|
- [ ] 6. Cleanup: remove the scheduler's superseded cross-controller rejoin logic
|
|
- Remove the reactive rejoin mechanism and the general multi-candidate failover-rotation path for the scheduler's own role, now that task 3 fully replaces its purpose and tasks 4-5 have proven and reconciled the new model end to end — matching design.md's own Migration Strategy ordering, where this removal is deliberately the last, hardest-to-reverse phase
|
|
- Observable completion: the removed functions no longer exist; the existing scheduler test suite passes unmodified with no regression; the scheduler's own job/queue/config handling behavior is unchanged
|
|
- _Requirements: 3.4, 6.3_
|
|
- _Boundary: scheduler.c_
|
|
- _Depends: 3, 4.1, 4.2, 5.7, 5.8_
|