myxce/.kiro/specs/scheduler-primary-lifecycle/design.md

Overview

Purpose: This feature binds the scheduler process's lifecycle to the controller cluster's primary role, replacing the scheduler's independent-process-with-reactive-rejoin model with a controller-owned child process, so that a controller failover always results in a working scheduler within one promotion cycle instead of a potentially unbounded stuck-PEND deadlock (bug-1760).

Users: XCE cluster operators, and transitively every job submitted to the cluster, benefit from this feature never having to notice or intervene after a controller failover.

Impact: Changes the current independent, long-lived xce_scheduler process (joining any of a configured candidate list, reconnecting reactively) into a process spawned, owned, and torn down by whichever controller is currently primary, pinned to that one controller for its entire lifetime.

Goals

• A controller starts exactly one scheduler process the moment it becomes primary, and that scheduler is usable (joined, receiving state) without any operator action.
• A controller's own scheduler exits promptly on a graceful demotion, and independently detects and exits on any other loss of that controller's primacy (crash, network partition, wedge), without ever attempting to reconnect elsewhere.
• Scheduler-lifecycle management can never compromise the controller's own liveness, HA correctness, or availability.
• A wedged-but-connected scheduler is distinguished from a healthy one and recovered automatically.
• Existing tooling and specs that assume an independently-started, reconnecting scheduler are identified and updated.

Non-Goals

• No standby/redundant scheduler processes or lease-election mechanism (superseded by piggybacking on the controller's existing Raft election).
• No general per-host role-manager component governing worker/controller/scheduler/apiserver placement.
• No change to worker or apiserver deployment/supervision.
• No change to the scheduler's job/queue/config-handling logic, wire behavior toward jobs, or picker/allocator internals — only its process lifecycle and its relationship to the controller cluster.

Boundary Commitments

This Spec Owns

• The controller-side decision of when to start, stop, or restart its own scheduler child process (tied to promotion, demotion, and a detected wedge).
• The mechanism used to start/stop/monitor that child process without blocking the controller's Raft drive threads.
• The scheduler's own startup contract change (pinned to one controller, no candidate list) and its active liveness/primacy check of that one controller.
• The new controller-to-scheduler wire message used for a graceful-exit notice.
• Updating existing deployment tooling (xce-deploy-ha, xce-test-ha) and flagging any other spec that assumes an independently-started, reconnecting scheduler.

Out of Boundary

• The scheduler's job/queue/config decision logic (picker, allocator, plugins) — untouched.
• The Raft election, replication, and commit-safety mechanisms the controller already implements — this feature is a consumer of "who is primary," never a modifier of how that is decided.
• Worker and apiserver process lifecycle.
• Any lease/renewal protocol or standby-scheduler redundancy (deliberately rejected — see research.md).
• ha_dispatch_gate's existing role-based write rejection — this feature depends on it, does not change it.

Allowed Dependencies

• src/controller/slowlane.c (slowlane_submit) — the sole mechanism for running scheduler-management work off the Raft drive threads. No new async-execution primitive may be introduced.
• src/controller/ha.c's raft_cb_become_leader / raft_cb_become_follower — the sole trigger points for scheduler start/stop. No parallel promotion/demotion detection path may be built.
• src/controller/sched.c's existing on_scheduler_join() / handle_scheduler_leave() / cluster_t.sched.ep bookkeeping — extended, not replaced, for tracking whether the owned child has actually joined.
• The existing controller-scheduler wire connection and Cap'n Proto schema (src/schema/controller.capnp) — the graceful-exit notice is a small addition to this existing channel, not a new transport.

Revalidation Triggers

• Any change to raft_cb_become_leader's or raft_cb_become_follower's promotion/demotion gating order (a new gate inserted before or after the scheduler-start/stop call site changes when a scheduler is (not) started).
• Any change to slowlane_submit's dedup-key semantics or admission-cap behavior.
• Any change to ha_dispatch_gate's role-based opcode allow-lists (the safety argument that a stale scheduler connection cannot mutate state depends on this staying true).
• Any other spec or tool that starts xce_scheduler directly (bypassing controller-owned startup) must be re-checked against this design's single-controller startup contract.

Project Invariants Checked

This design touches the controller's HA/process-lifecycle surface, so it is checked against .kiro/steering/ha-async-io.md's documented invariants directly, not deferred to a later audit:

• Thread-role contract (no blocking work on iomgr/delaymgr): satisfied — every scheduler-management action (posix_spawn, waitpid, signal, status check) is submitted through the existing slowlane_submit() executor, never invoked inline from a Raft callback. See Architecture Pattern & Boundary Map and the SchedulerProcessManager component.
• Controller-never-self-terminate: satisfied — a scheduler-management failure (start failure, stop timeout, wedge) is logged and retried/handled locally; it never crashes, hangs, or exits the controller process, and is never wired as a new promotion-refusal gate (Requirement 4.2, 4.4). See Error Handling.
• ha_dispatch_gate role-based write rejection: depended upon, not modified — this design's safety argument (a stale/orphaned scheduler connection cannot mutate cluster state) rests on this existing, unchanged behavior (see research.md).

Architecture

Existing Architecture Analysis

• The controller today has no concept of owning a child process of any kind — cluster_t has no process-management state. The only precedent for one XCE daemon supervising a child process is the worker daemon's job-execution supervision (process_worker_manager.c, uv_ipc.c), which is libuv-based and job-execution-specific (see research.md) and is not reused directly.
• The scheduler today is a fully general Raft-cluster client: it holds a candidate list (--peers), connects via xcecli_start/xcecli_join, and recovers from any disconnect via the client library's own XCECLI_FAILOVER_CALL rotation plus its own reactive sched_publish_check_rejoin(). This general-client shape is replaced for the scheduler specifically; workers and the xce_test/CLI clients are unaffected and keep the existing candidate-list/failover-rotation model.
• ha_dispatch_gate already enforces that only a primary controller accepts scheduler write opcodes (job dispatch, pend-reason) — this pre-existing behavior is why this feature is purely about availability, not correctness (see research.md).

Architecture Pattern & Boundary Map

graph TB
    subgraph HostA
        CtrlA[Controller Instance A Primary]
        SchedA[Scheduler Process]
        CtrlA -->|spawns and owns| SchedA
    end
    subgraph HostB
        CtrlB[Controller Instance B Standby]
    end
    subgraph HostC
        CtrlC[Controller Instance C Standby]
    end
    CtrlA <-->|raft election and replication| CtrlB
    CtrlA <-->|raft election and replication| CtrlC
    CtrlB <-->|raft election and replication| CtrlC
    SchedA -->|pinned single connection join and status query| CtrlA

Architecture Integration:

• Selected pattern: controller-owned child process, triggered directly from the existing Raft promotion/demotion callbacks, executed through the existing SlowLane off-drive-thread executor.
• Domain/feature boundaries: controller-side process ownership (scheduler_proc.c, new) is separate from controller-side scheduler-connection bookkeeping (sched.c, existing, extended); scheduler-side controller-liveness watching (sched_owner.c, new) is separate from scheduler-side scheduling logic (scheduler.c, unchanged except for removing the superseded rejoin logic).
• Existing patterns preserved: SlowLane for off-drive-thread work; ha_dispatch_gate as the sole write-safety mechanism; the existing Cap'n Proto wire connection and request-handler dispatch shape on both sides.
• New components rationale: scheduler_proc.c is a new domain (process ownership) with no existing owner; sched_owner.c is a new domain (single-controller liveness pinning) that cannot be folded into scheduler.c's existing scheduling-engine responsibility without violating single-responsibility.
• Steering compliance: respects ha-async-io.md's thread-role contract (all blocking work — posix_spawn/waitpid/kill/status-query — runs via SlowLane or the scheduler's own non-drive-thread context, never inline on iomgr/delaymgr) and the controller-never-self-terminate invariant (a scheduler-management failure logs loudly and retries; it never causes the controller to crash, hang, or exit, and never becomes a new promotion-refusal gate).

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Controller process management	POSIX posix_spawn/waitpid/signal (C, src/controller/scheduler_proc.c)	Start, monitor, and stop the scheduler child process	product.md's locked decision "no fork() for snapshots (mutex-inheritance deadlock in multithreaded controller)" applies just as much here: a raw fork() only duplicates the calling thread, so a lock held by any other thread (glibc malloc, XCE's own mylock_t locks) is duplicated in whatever state it was in, and the child can deadlock if it touches malloc or any XCE library call before execve() — being on a dedicated SlowLane worker does not change this, since the hazard comes from other threads in the same process, not the calling one. posix_spawn() is used instead specifically to avoid re-triggering this exact, already-documented risk class, not merely as a style preference
Async execution	slowlane_submit() (existing, src/controller/slowlane.c)	Runs all scheduler-management actions off the Raft drive threads	Reused as-is, no changes
Wire protocol	Cap'n Proto (existing, src/schema/controller.capnp)	Carries the new graceful-exit notice from controller to scheduler over the existing connection	One small schema addition; no new transport
Scheduler startup config	CLI argument (existing mechanism, new single-address mode)	Controller passes its own address to the scheduler it spawns, replacing the multi-candidate --peers list for this specific relationship	Deployment tooling generating scheduler startup commands must change accordingly

File Structure Plan

Directory Structure

src/
├── controller/
│   ├── scheduler_proc.c        # NEW: owns posix_spawn/monitor/signal/reap of this controller's scheduler child
│   ├── include/
│   │   └── scheduler_proc.h    # NEW: SchedulerProcessManager service interface
│   ├── ha.c                    # MODIFIED: call scheduler_proc_start() at the tail of raft_cb_become_leader,
│   │                           #   scheduler_proc_stop() at the top of raft_cb_become_follower
│   └── sched.c                 # MODIFIED: extend join/leave bookkeeping to feed liveness/wedge detection
├── scheduler/
│   └── src/
│       ├── sched_owner.c       # NEW: pins to one controller address, periodic liveness/role check, exits on loss
│       ├── scheduler.c         # MODIFIED: remove sched_needs_rejoin/sched_on_reconnect/sched_publish_check_rejoin
│       │                       #   and the multi-candidate startup path for this role
│       ├── handler.c           # MODIFIED: scheduler_req_handler gains a case for the graceful-exit notice
│       └── main.c              # MODIFIED: new single-controller-address startup argument
├── schema/
│   └── controller.capnp        # MODIFIED: add the controller-to-scheduler graceful-exit message
└── controller/testsuite/, scheduler/testsuite/  # new/modified test files, see Testing Strategy

Modified Files (tooling and specs, outside src/)

• .claude/skills/xce-deploy-ha/SKILL.md — remove/replace Step 4b's manual start-scheduler.sh guidance; document that a running controller now starts its own scheduler automatically once it becomes primary.
• src/tools/deploy-ha.sh — stop generating a standalone start-scheduler.sh intended for manual/independent invocation (or repurpose it to reflect the new single-controller-address contract if a manual escape hatch is still useful for local debugging).
• .claude/skills/xce-test-ha/run-ha-soak.sh — remove the worker-rejoin-resync soak coverage's start_worker_and_scheduler()/restart_scheduler_for_new_primary() manual scheduler management added this session (bug-1760's workaround becomes unnecessary — the controller now manages this itself); the soak harness should instead just verify a scheduler is present after each promotion.

System Flows

Promotion: controller starts its scheduler

sequenceDiagram
    participant Raft as Raft Election
    participant HA as ha.c become leader
    participant Lane as SlowLane Executor
    participant Proc as scheduler_proc.c
    participant Sched as Scheduler Process
    Raft->>HA: on_become_leader
    HA->>HA: run existing promotion eligibility gates
    HA->>Lane: slowlane_submit start action dedup key this controller
    Lane->>Proc: run start action on a lane worker
    Proc->>Sched: posix_spawn passing this controller address
    Sched->>HA: join using pinned single controller address
    HA->>Sched: push jobs queues config existing behavior

Gating conditions: the SlowLane submission only happens after every existing promotion-refusal gate has already passed (Requirement 1.1). A failed or deferred slowlane_submit() (already-in-flight dedup, or admission-cap deferral) is not an error — the work is still queued and will run (Requirement 1.5's retry is satisfied by SlowLane's own queuing semantics plus an explicit retry-on-failure inside the submitted action).

Graceful demotion: controller signals its scheduler to exit

sequenceDiagram
    participant Raft as Raft Election
    participant HA as ha.c become follower
    participant Lane as SlowLane Executor
    participant Proc as scheduler_proc.c
    participant Sched as Scheduler Process
    HA->>Lane: slowlane_submit stop action dedup key this controller
    HA->>HA: continue completing its own role transition immediately
    Lane->>Proc: run stop action on a lane worker
    Proc->>Sched: send graceful exit notice over existing connection
    Sched->>Sched: finish in flight work and exit
    Proc->>Proc: wait bounded time then reap or force terminate

The role transition in raft_cb_become_follower never waits on the SlowLane task's completion (Requirement 2.3) — submission is fire-and-forget from the Raft callback's perspective.

Scheduler self-detects loss of its owning controller

sequenceDiagram
    participant Sched as Scheduler Process
    participant Owner as sched_owner.c
    participant Ctrl as Its Owning Controller
    loop periodic check
        Owner->>Ctrl: status query over the pinned connection
        Ctrl-->>Owner: role and reachability
    end
    alt controller unreachable or reports it is no longer primary
        Owner->>Sched: request exit
        Sched->>Sched: exit
    end

This path is independent of the graceful-exit notice above — it is the general-case detector covering crash, network partition, and any demotion trigger the explicit notice did not reach in time (Requirement 3.1-3.3). No candidate list exists to fall back to (Requirement 3.4) — the only outcome of a failed check is exit.

Wedged-scheduler detection and recovery

sequenceDiagram
    participant HA as ha.c or sched.c
    participant Lane as SlowLane Executor
    participant Proc as scheduler_proc.c
    participant Sched as Scheduler Process
    HA->>Proc: observe scheduler connected but no scheduling progress signal within threshold
    Proc->>Lane: slowlane_submit restart action dedup key this controller
    Lane->>Proc: run restart action
    Proc->>Sched: terminate
    Proc->>Proc: posix_spawn a replacement per the promotion flow

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1, 1.2, 1.3	Scheduler starts on promotion, at most one in flight, no duplicate start	scheduler_proc.c, ha.c	scheduler_proc_start	Promotion flow
1.4	Scheduler joins and receives job/queue/config state after being started	scheduler_proc.c, sched.c	scheduler_proc_start	Promotion flow
1.5	Controller retries on start failure, logs loudly	scheduler_proc.c	scheduler_proc_start	Promotion flow
2.1, 2.2, 2.3	Graceful exit signaled on demotion; scheduler exits within bounded time; controller's own role transition never waits on it	scheduler_proc.c, ha.c, controller.capnp addition, scheduler handler.c	scheduler_proc_stop, graceful-exit wire message	Graceful demotion flow
2.4	No job left inconsistent or lost regardless of how the scheduler was terminated	scheduler_proc.c (bounded stop/force-kill), controller's existing WAL-commit ownership (unchanged)	scheduler_proc_stop	Graceful demotion flow, Data Models
3.1, 3.2, 3.3	Scheduler actively verifies its owning controller is reachable and primary; exits on failure of either check	sched_owner.c	sched_owner_watch_start	Self-detection flow
3.4	Scheduler never joins or operates against any controller other than the one that started it	sched_owner.c, scheduler.c (rejoin logic removed)	sched_owner_watch_start	Self-detection flow
3.5	Abrupt host/process death needs no scheduler-side action	(no component — a structural consequence of co-location, not new logic)	n/a	Self-detection flow (contrast case)
4.1, 4.2, 4.3	Controller never blocks heartbeats/elections, never crashes/hangs, always logs scheduler-management failures	scheduler_proc.c (via SlowLane)	scheduler_proc_start/stop/check_liveness (all SlowLane-submitted)	All flows
4.4	Absence of a running scheduler never blocks a controller's own HA role	scheduler_proc.c, ha.c (unchanged promotion gates)	n/a	Promotion flow
5.1, 5.2	Controller monitors scheduling progress, not just connection state; flags no-progress past a threshold as unresponsive	SchedulerConnectionBookkeeping (sched.c), scheduler_proc.c	scheduler_proc_check_liveness	Wedge detection flow
5.3, 5.4	Unresponsive scheduler is terminated and replaced, with an operator-visible log	scheduler_proc.c	scheduler_proc_check_liveness	Wedge detection flow
6.1	Existing deployment tooling documentation updated to reflect controller-owned scheduler startup	xce-deploy-ha SKILL.md, deploy-ha.sh	n/a (tooling)	Migration Strategy
6.2	Existing specs/tooling depending on scheduler survive-and-reconnect behavior identified and reconciled	xce-test-ha run-ha-soak.sh (this session's own bug-1760 workaround is the concrete example identified and reconciled here)	n/a (tooling)	Migration Strategy, Testing Strategy E2E
6.3	Scheduler's observable job/queue/config behavior unchanged while running under a live primary	scheduler.c (unchanged scheduling logic)	n/a	Data Models

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
SchedulerProcessManager	Controller	Owns posix_spawn/monitor/signal/reap of this controller's scheduler child	1, 2, 4, 5	SlowLane executor (P0), ha.c callbacks (P0)	Service
SchedulerConnectionBookkeeping	Controller	Existing join/leave tracking, extended to feed liveness/progress signal	5	SchedulerProcessManager (P1)	Service
SchedOwnerWatch	Scheduler	Pins to one controller address; periodic liveness/role check; exits on loss or graceful notice	3	Existing wire connection/status query (P0)	Service
Graceful-Exit Wire Message	Wire protocol	Controller-to-scheduler notice carried on the existing connection	2	Cap'n Proto schema (P0)	Event

Controller

SchedulerProcessManager

Field	Detail
Intent	Start, monitor, signal, and reap this controller's own scheduler child process, entirely off the Raft drive threads
Requirements	1.1, 1.2, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3, 4.1, 4.2, 4.3, 4.4, 5.1, 5.2, 5.3, 5.4

Responsibilities & Constraints

• Owns the decision of whether a scheduler child is currently running for this controller instance; ha.c never spawns or signals a process directly.
• Every action (start, stop, restart) is submitted to the SlowLane executor with a dedup key scoped to this controller instance, never invoked inline from a Raft callback.
• Never allowed to block, crash, or cause the controller process to exit — every failure path logs an operator-visible error and either retries (start) or proceeds with the role transition regardless (stop).

Dependencies

• Outbound: SlowLane executor (slowlane_submit) — P0, the only allowed execution path for this component's blocking work.
• Inbound: ha.c's raft_cb_become_leader (calls scheduler_proc_start), raft_cb_become_follower (calls scheduler_proc_stop) — P0.
• Inbound: a periodic liveness check (driven from the existing controller heartbeat cadence, not a new timer thread) — P1, feeds scheduler_proc_check_liveness.

Contracts: Service [x]

Service Interface

/* src/controller/include/scheduler_proc.h */

/* Requests that this controller's scheduler child be started, if one is not
 * already running or already starting. Non-blocking: submits to the SlowLane
 * executor and returns immediately. Safe to call multiple times in quick
 * succession — the SlowLane dedup key ensures at most one start action is
 * ever in flight per controller instance.
 * Returns 0 if the request was accepted (started or already in flight/running),
 * a negative errno if the request itself could not even be submitted (e.g.
 * SlowLane shutting down) — never blocks, never crashes the caller.
 */
int scheduler_proc_start(cluster_t *c);

/* Requests that this controller's scheduler child be stopped: sends the
 * graceful-exit notice, waits up to timeout_ms for the child to exit, then
 * force-terminates if it has not. Non-blocking to the caller: submitted to
 * the SlowLane executor. Does not block the caller's own role transition.
 */
int scheduler_proc_stop(cluster_t *c, uint32_t timeout_ms);

/* Evaluates whether the currently-running scheduler child (if any) is making
 * scheduling progress, using the existing push/acknowledgment traffic
 * (SchedulerConnectionBookkeeping) as the progress signal. If the child is
 * connected but has made no progress for longer than the configured
 * threshold, submits a restart action. A no-op if no scheduler child is
 * currently owned. Non-blocking.
 */
int scheduler_proc_check_liveness(cluster_t *c);

• Preconditions: c is a valid, initialized cluster_t; called only from the controller's own process.
• Postconditions: the requested action is either already complete, in flight on a SlowLane worker, or queued — never silently dropped.
• Invariants: at most one scheduler child process is owned by a given controller instance at any time; a stop-then-start (demotion followed by a fresh promotion of the same instance) always completes the stop's reap before a new start is submitted, since a demoted instance must win a fresh election before it can be promoted again.

Implementation Notes

• Integration: hook call sites are the tail of raft_cb_become_leader (after all existing promotion-refusal gates) and the top of raft_cb_become_follower.
• Validation: an operator-visible log line on every start failure, stop-timeout-forced-kill, and liveness-triggered restart (Requirement 4.3, 5.4).
• Risks: sizing the shared SlowLane instance's max_inflight/n_workers for the additional scheduler-management workload alongside its existing consumers is a follow-up check, not a design change (see research.md).

SchedulerConnectionBookkeeping (extends existing sched.c)

Field	Detail
Intent	Track whether the owned scheduler child has joined and is making scheduling progress, feeding scheduler_proc_check_liveness
Requirements	5.1, 5.2

Responsibilities & Constraints

• Extends the existing on_scheduler_join()/handle_scheduler_leave()/cluster_t.sched.ep bookkeeping with a last-progress timestamp, updated whenever a push to the scheduler is acknowledged or a dispatch/pend-reason decision is received from it.
• Does not itself decide to restart anything — it only exposes the progress signal scheduler_proc_check_liveness reads.

Dependencies

• Inbound: existing scheduler request handling in cluster.c/sched.c — P1.
• Outbound: read by SchedulerProcessManager — P1.

Contracts: Service [x]

Implementation Notes

• Integration: reuses existing traffic; no new synthetic heartbeat payload is introduced (see research.md Risks).

Scheduler

SchedOwnerWatch

Field	Detail
Intent	Pin the scheduler to the single controller instance that started it; periodically confirm that controller is reachable and still primary; exit otherwise
Requirements	3.1, 3.2, 3.3, 3.4, 3.5

Responsibilities & Constraints

• At startup, the scheduler receives exactly one controller address (not a candidate list) and establishes its connection only to that address.
• Periodically issues the existing role-agnostic status query against that one connection; on an unreachable connection or a reported non-primary role, requests the scheduler process to exit.
• Never attempts any other controller address, and never re-derives a candidate list — this component structurally has no such list to fall back to.
• Also receives and acts on the graceful-exit notice (Requirement 2.2) via the existing request-handler dispatch, exiting within a bounded time.

Dependencies

• Outbound: the existing controller-scheduler wire connection/status query — P0.
• Inbound: scheduler_req_handler's new graceful-exit case — P0.

Contracts: Service [x]

Service Interface

/* src/scheduler/include/sched_owner.h */

/* Starts the periodic ownership-watch loop against the single controller
 * address this scheduler process was given at startup. Runs on its own
 * thread (mirroring the worker daemon's existing checking_connection()
 * cadence/shape), independent of the picker/allocator threads.
 */
int sched_owner_watch_start(const char *owner_host, uint16_t owner_port);

/* Called by the request handler when the graceful-exit notice arrives on the
 * existing connection. Triggers the same exit path the periodic watch loop
 * uses on a failed check.
 */
void sched_owner_request_exit(void);

• Preconditions: owner_host/owner_port identify the specific controller instance that spawned this process.
• Postconditions: the process exits once either the periodic check fails or a graceful-exit notice is received — no other outcome resumes normal operation with a different controller.
• Invariants: never issues xcecli_join/any wire call against any address other than owner_host:owner_port.

Implementation Notes

• Integration: replaces sched_needs_rejoin/sched_on_reconnect/sched_publish_check_rejoin in scheduler.c entirely — these are removed, not left dormant, since Requirement 3.4 makes their cross-controller-rejoin purpose permanently unreachable.
• Risks: the periodic check interval trades detection latency against status-query load; finalize the exact interval during implementation, following the worker's existing update_interval-style convention rather than inventing a new tunable pattern.

Wire Protocol

Graceful-Exit Wire Message

Event Contract

• Published by: the controller, from scheduler_proc_stop's SlowLane action, over the existing controller-scheduler connection.
• Subscribed by: the scheduler's existing scheduler_req_handler (src/scheduler/src/handler.c), which invokes sched_owner_request_exit().
• Delivery: best-effort over the existing persistent connection — if delivery fails (connection already gone), scheduler_proc_stop's bounded wait-then-force-terminate path (Requirement 2.3) still guarantees the child does not outlive the timeout, and SchedOwnerWatch's independent periodic check (Requirement 3) is the backstop if the notice is lost.
• Ordering/idempotency: at most one graceful-exit notice is ever sent per stop action (SlowLane dedup); receiving it more than once (e.g. a retry) is a no-op if the process is already exiting.

Data Models

No new persisted entity is introduced. The scheduler remains fully stateless with respect to durable storage — its in-memory job/queue/config state is always rebuilt from the controller's existing push traffic on join, unchanged by this feature (Requirement 6.3). The only new in-memory state is:

• Controller side: which scheduler child (if any) this controller instance currently owns, its process handle, and its last-progress timestamp — process-lifetime only, never persisted, rebuilt from scratch on controller restart (a restarted controller that is primary starts a fresh scheduler exactly as any other promotion would).
• Scheduler side: the single owning-controller address it was started with — process-lifetime only.

Since nothing here is written to or read from a durable log, snapshot, or checkpoint, the project's save/restore field-completeness review does not apply to this design.

Error Handling

Error Strategy

Every failure in scheduler-lifecycle management follows the project's existing fail-loud-stay-alive philosophy (ha-async-io.md, controller/CLAUDE.md): log an operator-visible error, never crash or hang the controller, and retry where retrying is meaningful.

Error Categories and Responses

• Scheduler fails to start (posix_spawn failure, immediate crash): scheduler_proc_start logs an error and retries on the controller's existing heartbeat-driven cadence while it remains primary (Requirement 1.5); the controller itself continues serving every other role unaffected (Requirement 4.4).
• Scheduler does not exit within the graceful-exit timeout: scheduler_proc_stop force-terminates the child after the bounded wait; the controller's own demotion completes regardless (Requirement 2.3).
• Scheduler connected but not progressing (wedged): treated as unresponsive once past the threshold, terminated and replaced (Requirement 5.3), with an operator-visible log (Requirement 5.4).
• Scheduler cannot reach or confirm its owning controller: the scheduler itself exits (Requirement 3.2, 3.3) — no error is reported back to the controller for this case, since the controller side detects the resulting disconnect via existing handle_scheduler_leave() bookkeeping.

Monitoring

All new log lines (start failure, forced-kill-on-timeout, liveness-triggered restart) use the existing controller logging conventions (WARN/ERROR macros already used throughout ha.c) — no new logging mechanism is introduced.

Testing Strategy

Unit Tests

• scheduler_proc_start submits exactly one SlowLane action per promotion and a second concurrent call while one is in flight is deduped (not double-submitted) — directly exercises Requirement 1.2/1.3 via the existing test_slowlane.c dedup behavior.
• raft_cb_become_leader calls scheduler_proc_start only after every existing promotion-refusal gate returns eligible, and does not call it on any refusal path — regression-guards Requirement 1.1 against a future gate reordering.
• raft_cb_become_follower calls scheduler_proc_stop and completes its own role transition without waiting on that call's result — regression-guards Requirement 2.3.
• SchedOwnerWatch's periodic check exits the process on a simulated unreachable controller and on a simulated non-primary role response, and never attempts a second address — directly exercises Requirement 3.1-3.4.
• scheduler_proc_check_liveness triggers a restart only when the last-progress timestamp exceeds the threshold while the connection remains open (not on a disconnect, which is a separate, already-detected case) — exercises Requirement 5.1/5.2.

Integration Tests

• A real 3-node controller cluster promotion results in a scheduler process appearing, joining, and receiving job/queue/config state, with no manual step — exercises Requirement 1.1-1.4 end to end.
• A graceful step-down (forced via the existing test-only demotion trigger) results in the prior primary's scheduler exiting within the bounded timeout, verified by process absence, not just a log line — exercises Requirement 2.1/2.2.
• A hard-killed primary (matching this session's original bug-1760 repro exactly) results in the orphaned scheduler self-exiting within its own periodic-check interval, and the newly-elected primary's fresh scheduler successfully dispatching a job submitted after the failover — exercises Requirement 3.1-3.5 and directly closes bug-1760.
• A scheduler artificially wedged (e.g. its progress-signal thread blocked) while its connection stays open is detected and replaced — exercises Requirement 5.1-5.3.
• A job dispatched by a scheduler that is then killed mid-decision (before its dispatch is durably recorded) remains available for the replacement scheduler to consider — exercises Requirement 2.4.

E2E / Soak Tests

• Extend .claude/skills/xce-test-ha/run-ha-soak.sh's sequential and chaotic modes (already exercising controller failover) to assert a working scheduler is present after every round without any harness-side scheduler start/stop action — this replaces the manual start_worker_and_scheduler()/restart_scheduler_for_new_primary() logic added this session as bug-1760's workaround, which becomes dead code once this feature ships.
• A rapid sequence of promotions/demotions (chaotic mode's overlapping-kill pattern) produces no scheduler spawn/kill thrash beyond one scheduler per genuinely-promoted instance — exercises the SlowLane-dedup mitigation documented in research.md's Risks.

Migration Strategy

flowchart TB
    Phase1[Phase 1 controller and scheduler code changes land behind no behavior change until wired]
    Phase2[Phase 2 wire promotion and demotion hooks and pinned single controller startup]
    Phase3[Phase 3 update xce deploy ha and xce test ha tooling to stop manually starting the scheduler]
    Phase4[Phase 4 remove superseded rejoin logic and multi candidate scheduler startup path]
    Phase1 --> Phase2 --> Phase3 --> Phase4

• Rollback trigger: if a real deployment depends on independently placing the scheduler on a dedicated host for resource-isolation reasons not previously surfaced, that is a genuine requirements gap discovered late — the fix is to pause before Phase 4 (superseded-logic removal) rather than force the migration through, since Phase 4 is the only phase that is hard to reverse cheaply.
• Validation checkpoint: after Phase 2, run the soak-harness scenario in Testing Strategy's E2E section before proceeding to Phase 3/4, since it is the closest thing to a direct regression test for bug-1760 itself.

Open Questions / Risks

• The exact wire opcode/schema shape for the graceful-exit notice is left to implementation (a small, additive controller.capnp change, following the same pattern recent specs have used for new controller-to-daemon messages) — this is appropriately an implementation detail, not a design decision, since no existing opcode fits (confirmed by inspection: OPCode_shutdown is an inbound admin-to-controller command, not a controller-to-scheduler one, and is not reusable here).
• The precise "no scheduling progress" threshold and periodic-check interval (Requirement 5, Requirement 3) are implementation-time tuning decisions, not design decisions — see research.md Risks for the reasoning against inventing a bespoke synthetic heartbeat.
• Confirm during implementation whether any other in-tree consumer starts xce_scheduler with a multi-candidate --peers list for a reason unrelated to this feature (e.g. a test fixture) that would need explicit review rather than silent breakage (carried over from research.md).