lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/scheduler-primary-lifecycle/tasks.md
Rongsong Shen 0f7699deba
docs(scheduler-primary-lifecycle): add superseded spec (requirements/design/tasks) 
Requirements and design approved (design fixed the fork()-vs-posix_spawn
mutex-inheritance-deadlock risk product.md's own "no fork() for
snapshots" decision already rejected); tasks generated but never
approved. Superseded by xce-node-manager after the user decided the
controller should not directly own spawning/killing its own scheduler
process -- a new per-node xce-node-manager component takes over that
responsibility instead. Kept in place as a reference (notably the
fork()-vs-posix_spawn analysis); this spec's tasks.md must not be
implemented (see spec.json's superseded_note).

Checkpointing uncommitted spec work found in this shared worktree.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-03 09:13:01 +00:00

10 KiB
Raw Blame History

Implementation Plan

  • 1. Foundation: wire protocol schema and scheduler startup contract

  • 1.1 Define the controller-to-scheduler graceful-exit wire message

    • Add a new message the controller can send an already-joined scheduler over their existing connection, carrying no payload beyond identifying it as a graceful-exit request
    • Observable completion: the schema builds cleanly and the new message is visible in the generated bindings, unused by any caller yet
    • Requirements: 2.1, 2.2

  • 1.2 (P) Give the scheduler a single-controller startup mode

    • Replace the scheduler's current multi-candidate --peers-style startup configuration with a mode that accepts exactly one controller address
    • Observable completion: the scheduler binary accepts the new single-address startup argument; the existing multi-candidate argument is no longer accepted for this role
    • Requirements: 3.4
    • Boundary: scheduler main.c

  • 2. Core: controller-side scheduler process manager

  • 2.1 Start a scheduler process safely off the Raft drive threads

    • Implement the start action (spawn via posix_spawn, passing the controller's own address using task 1.2's new startup mode) submitted through the existing SlowLane executor with a dedup key, never invoked inline from a Raft callback
    • Observable completion: requesting a start when one is already in flight or running is deduped (no second process spawned); a start whose spawn fails is logged and does not crash or block the caller
    • Requirements: 1.1, 1.2, 1.3, 1.5, 4.1, 4.3
    • Boundary: scheduler_proc.c
    • Depends: 1.2

  • 2.2 Stop a scheduler process gracefully, with a bounded force-terminate fallback

    • Implement the stop action: send the graceful-exit wire message from task 1.1 over the existing connection, wait up to a bounded timeout for the process to exit, then force-terminate it
    • Observable completion: a scheduler that exits on its own within the timeout is not force-terminated; one that does not is force-terminated exactly once after the timeout elapses; the caller never blocks waiting for this to finish
    • Requirements: 2.1, 2.3, 2.4, 4.2, 4.3
    • Boundary: scheduler_proc.c
    • Depends: 1.1

  • 2.3 (P) Track whether the owned scheduler is making real scheduling progress

    • Extend the controller's existing scheduler join/leave bookkeeping with a last-progress timestamp, updated whenever a push to the scheduler is acknowledged or a dispatch/pend-reason decision is received from it
    • Observable completion: the timestamp advances on real scheduler activity, using only existing traffic — no new synthetic ping is introduced
    • Requirements: 5.1, 5.2
    • Boundary: sched.c

  • 2.4 Detect a wedged scheduler and replace it

    • Implement a liveness check that reads task 2.3's progress signal; if the scheduler is connected but has made no progress past a configured threshold, submit a restart action (stop via task 2.2, then start via task 2.1) through the same SlowLane path
    • Observable completion: a connected scheduler with no progress past the threshold is terminated and replaced exactly once per stale period, with an operator-visible log; a scheduler that is merely disconnected (already handled by existing join/leave bookkeeping) is not double-counted as a wedge
    • Requirements: 5.3, 5.4, 4.1, 4.3
    • Boundary: scheduler_proc.c
    • Depends: 2.1, 2.2, 2.3

  • 3. (P) Core: pin the scheduler to its one controller and watch it

    • Using task 1.2's single-address startup mode, connect only to that one controller; periodically query its reachability and role over that same connection
    • On an unreachable controller, a reported non-primary role, or receipt of task 1.1's graceful-exit message via the existing request handler, exit the process
    • Observable completion: the scheduler process exits within one check interval of its controller becoming unreachable or non-primary, or immediately on receiving the graceful-exit message; it never attempts any address other than the one it started with
    • Requirements: 2.2, 3.1, 3.2, 3.3, 3.4, 3.5
    • Boundary: sched_owner.c, scheduler request handler
    • Depends: 1.1, 1.2

  • 4. Integration: wire promotion, demotion, and liveness/retry checking into controller HA role transitions

  • 4.1 Start the scheduler on promotion

    • Call task 2.1's start action at the point a controller completes promotion, after every existing promotion-refusal gate has already passed
    • Observable completion: a real promotion results in exactly one scheduler process being started, joining, and receiving job/queue/config state; a refused promotion attempt (any existing gate) never starts one
    • Requirements: 1.1, 1.4, 4.4
    • Boundary: ha.c
    • Depends: 2.1

  • 4.2 Stop the scheduler on graceful demotion

    • Call task 2.2's stop action at the point a controller begins a graceful role transition away from primary, without waiting for it to complete
    • Observable completion: a real graceful demotion's own role transition completes without waiting on the scheduler's exit; the scheduler process is gone within the bounded timeout
    • Requirements: 2.1, 2.3
    • Boundary: ha.c
    • Depends: 2.2

  • 4.3 Wire periodic scheduler presence and liveness checking into the controller's existing cadence

    • While primary with no scheduler currently owned (e.g. a prior start attempt failed), retry task 2.1's start action on the controller's existing heartbeat-driven cadence — the mechanism Requirement 1.5's retry depends on, with no new timer introduced
    • While primary with a scheduler owned, run task 2.4's liveness check on that same cadence
    • Observable completion: a scheduler that failed to start is retried on the next cadence tick without a new timer thread; a wedged scheduler is detected and replaced within one cadence tick of crossing the threshold
    • Requirements: 1.5, 5.1, 5.2, 5.3, 5.4
    • Boundary: ha.c
    • Depends: 2.1, 2.4

  • 5. Validation: end-to-end coverage and deployment tooling reconciliation

  • 5.1 Full promotion round trip against a real cluster

    • Exercise a real controller cluster promotion end to end: a scheduler process appears, joins, and receives job/queue/config state with no manual step
    • Observable completion: the round trip produces a working, job-dispatching scheduler in one live test run
    • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5
    • Depends: 4.1

  • 5.2 Graceful demotion round trip

    • Force a graceful step-down against a real cluster and confirm the prior primary's scheduler exits within the bounded timeout, verified by process absence
    • Observable completion: the demoted controller's own role transition completes promptly and the scheduler process is confirmed gone afterward
    • Requirements: 2.1, 2.2, 2.3
    • Depends: 4.2, 3

  • 5.3 A job survives scheduler termination mid-decision

    • Dispatch a job to a running scheduler, kill the scheduler process before its dispatch decision is durably recorded, and confirm the job remains available (not lost, not left inconsistent) for a subsequently-started scheduler to consider
    • Observable completion: the job is present and correctly schedulable after the replacement scheduler starts, with no trace of the killed scheduler's incomplete decision
    • Requirements: 2.4
    • Depends: 2.2

  • 5.4 Bug-1760 original repro: hard-killed primary recovers automatically

    • Reproduce this feature's own motivating scenario: hard-kill the primary controller and confirm its orphaned scheduler self-exits, the newly-elected primary starts its own fresh scheduler, and a job submitted after the failover dispatches successfully
    • Observable completion: the exact bug-1760 symptom (a job stuck PEND after a real controller failover) does not reproduce
    • Requirements: 3.1, 3.2, 3.3, 3.4, 3.5
    • Depends: 3, 4.1

  • 5.5 Wedged scheduler is detected and recovered live

    • Artificially wedge a running scheduler's progress signal while its connection stays open, against a real cluster, and confirm it is detected and replaced
    • Observable completion: the wedged scheduler is terminated and a working replacement is dispatching jobs again within the configured threshold
    • Requirements: 5.1, 5.2, 5.3, 5.4
    • Depends: 4.3

  • 5.6 Rapid promotion and demotion does not thrash scheduler processes

    • Drive a rapid sequence of promotions and demotions against a real cluster and confirm no more than one scheduler spawn/kill cycle occurs per genuine promotion
    • Observable completion: no orphaned or duplicate scheduler processes remain after the sequence settles
    • Requirements: 4.1, 4.2, 4.3, 4.4
    • Depends: 5.1, 5.2

  • 5.7 Update xce-deploy-ha for controller-owned scheduler startup

    • Update the deployment skill's documentation and generated scripts so they no longer describe or provide a manual, independent scheduler-start step
    • Observable completion: following the updated deployment steps produces a working scheduler with no manual scheduler-start action, matching task 5.1's proof
    • Requirements: 6.1
    • Depends: 5.1, 5.2

  • 5.8 Update the HA soak harness and sweep for other dependent tooling

    • Remove the soak harness's own manual scheduler start/restart logic (added as this feature's own motivating bug's workaround) and replace it with a check that a working scheduler is present after each round, with no harness-side management action
    • Also search other specs and test tooling in the repository for the same independently-started, reconnecting-scheduler assumption and flag or reconcile any found, not just the soak harness
    • Observable completion: the soak harness's sequential and chaotic modes pass unmodified in every other respect, with the manual scheduler-management code paths gone; any other dependent spec/tool found is either reconciled or explicitly flagged with a stated reason it is not
    • Requirements: 6.2
    • Depends: 5.4

  • 6. Cleanup: remove the scheduler's superseded cross-controller rejoin logic

    • Remove the reactive rejoin mechanism and the general multi-candidate failover-rotation path for the scheduler's own role, now that task 3 fully replaces its purpose and tasks 4-5 have proven and reconciled the new model end to end — matching design.md's own Migration Strategy ordering, where this removal is deliberately the last, hardest-to-reverse phase
    • Observable completion: the removed functions no longer exist; the existing scheduler test suite passes unmodified with no regression; the scheduler's own job/queue/config handling behavior is unchanged
    • Requirements: 3.4, 6.3
    • Boundary: scheduler.c
    • Depends: 3, 4.1, 4.2, 5.7, 5.8