docs(scheduler-primary-lifecycle): add superseded spec (requirements/design/tasks)

Requirements and design approved (design fixed the fork()-vs-posix_spawn mutex-inheritance-deadlock risk product.md's own "no fork() for snapshots" decision already rejected); tasks generated but never approved. Superseded by xce-node-manager after the user decided the controller should not directly own spawning/killing its own scheduler process -- a new per-node xce-node-manager component takes over that responsibility instead. Kept in place as a reference (notably the fork()-vs-posix_spawn analysis); this spec's tasks.md must not be implemented (see spec.json's superseded_note). Checkpointing uncommitted spec work found in this shared worktree. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-03 09:13:01 +00:00 · 2026-09-03 09:13:01 +00:00 · 0f7699deba
commit 0f7699deba
parent 651b513850
5 changed files with 743 additions and 0 deletions
--- a/.kiro/specs/scheduler-primary-lifecycle/design.md
+++ b/.kiro/specs/scheduler-primary-lifecycle/design.md
@ -0,0 +1,421 @@
 ## Overview
 **Purpose**: This feature binds the scheduler process's lifecycle to the controller cluster's primary role, replacing the scheduler's independent-process-with-reactive-rejoin model with a controller-owned child process, so that a controller failover always results in a working scheduler within one promotion cycle instead of a potentially unbounded stuck-PEND deadlock (bug-1760).
 **Users**: XCE cluster operators, and transitively every job submitted to the cluster, benefit from this feature never having to notice or intervene after a controller failover.
 **Impact**: Changes the current independent, long-lived `xce_scheduler` process (joining any of a configured candidate list, reconnecting reactively) into a process spawned, owned, and torn down by whichever controller is currently primary, pinned to that one controller for its entire lifetime.
 ### Goals
 - A controller starts exactly one scheduler process the moment it becomes primary, and that scheduler is usable (joined, receiving state) without any operator action.
 - A controller's own scheduler exits promptly on a graceful demotion, and independently detects and exits on any other loss of that controller's primacy (crash, network partition, wedge), without ever attempting to reconnect elsewhere.
 - Scheduler-lifecycle management can never compromise the controller's own liveness, HA correctness, or availability.
 - A wedged-but-connected scheduler is distinguished from a healthy one and recovered automatically.
 - Existing tooling and specs that assume an independently-started, reconnecting scheduler are identified and updated.
 ### Non-Goals
 - No standby/redundant scheduler processes or lease-election mechanism (superseded by piggybacking on the controller's existing Raft election).
 - No general per-host role-manager component governing worker/controller/scheduler/apiserver placement.
 - No change to worker or apiserver deployment/supervision.
 - No change to the scheduler's job/queue/config-handling logic, wire behavior toward jobs, or picker/allocator internals — only its process lifecycle and its relationship to the controller cluster.
 ## Boundary Commitments
 ### This Spec Owns
 - The controller-side decision of *when* to start, stop, or restart its own scheduler child process (tied to promotion, demotion, and a detected wedge).
 - The mechanism used to start/stop/monitor that child process without blocking the controller's Raft drive threads.
 - The scheduler's own startup contract change (pinned to one controller, no candidate list) and its active liveness/primacy check of that one controller.
 - The new controller-to-scheduler wire message used for a graceful-exit notice.
 - Updating existing deployment tooling (`xce-deploy-ha`, `xce-test-ha`) and flagging any other spec that assumes an independently-started, reconnecting scheduler.
 ### Out of Boundary
 - The scheduler's job/queue/config decision logic (picker, allocator, plugins) — untouched.
 - The Raft election, replication, and commit-safety mechanisms the controller already implements — this feature is a *consumer* of "who is primary," never a modifier of how that is decided.
 - Worker and apiserver process lifecycle.
 - Any lease/renewal protocol or standby-scheduler redundancy (deliberately rejected — see research.md).
 - `ha_dispatch_gate`'s existing role-based write rejection — this feature depends on it, does not change it.
 ### Allowed Dependencies
 - `src/controller/slowlane.c` (`slowlane_submit`) — the sole mechanism for running scheduler-management work off the Raft drive threads. No new async-execution primitive may be introduced.
 - `src/controller/ha.c`'s `raft_cb_become_leader` / `raft_cb_become_follower` — the sole trigger points for scheduler start/stop. No parallel promotion/demotion detection path may be built.
 - `src/controller/sched.c`'s existing `on_scheduler_join()` / `handle_scheduler_leave()` / `cluster_t.sched.ep` bookkeeping — extended, not replaced, for tracking whether the owned child has actually joined.
 - The existing controller-scheduler wire connection and Cap'n Proto schema (`src/schema/controller.capnp`) — the graceful-exit notice is a small addition to this existing channel, not a new transport.
 ### Revalidation Triggers
 - Any change to `raft_cb_become_leader`'s or `raft_cb_become_follower`'s promotion/demotion gating order (a new gate inserted before or after the scheduler-start/stop call site changes when a scheduler is (not) started).
 - Any change to `slowlane_submit`'s dedup-key semantics or admission-cap behavior.
 - Any change to `ha_dispatch_gate`'s role-based opcode allow-lists (the safety argument that a stale scheduler connection cannot mutate state depends on this staying true).
 - Any other spec or tool that starts `xce_scheduler` directly (bypassing controller-owned startup) must be re-checked against this design's single-controller startup contract.
 ### Project Invariants Checked
 This design touches the controller's HA/process-lifecycle surface, so it is checked against `.kiro/steering/ha-async-io.md`'s documented invariants directly, not deferred to a later audit:
 - **Thread-role contract** (no blocking work on iomgr/delaymgr): satisfied — every scheduler-management action (posix_spawn, waitpid, signal, status check) is submitted through the existing `slowlane_submit()` executor, never invoked inline from a Raft callback. See Architecture Pattern & Boundary Map and the SchedulerProcessManager component.
 - **Controller-never-self-terminate**: satisfied — a scheduler-management failure (start failure, stop timeout, wedge) is logged and retried/handled locally; it never crashes, hangs, or exits the controller process, and is never wired as a new promotion-refusal gate (Requirement 4.2, 4.4). See Error Handling.
 - **`ha_dispatch_gate` role-based write rejection**: depended upon, not modified — this design's safety argument (a stale/orphaned scheduler connection cannot mutate cluster state) rests on this existing, unchanged behavior (see research.md).
 ## Architecture
 ### Existing Architecture Analysis
 - The controller today has no concept of owning a child process of any kind — `cluster_t` has no process-management state. The only precedent for one XCE daemon supervising a child process is the worker daemon's job-execution supervision (`process_worker_manager.c`, `uv_ipc.c`), which is libuv-based and job-execution-specific (see research.md) and is not reused directly.
 - The scheduler today is a fully general Raft-cluster client: it holds a candidate list (`--peers`), connects via `xcecli_start`/`xcecli_join`, and recovers from any disconnect via the client library's own `XCECLI_FAILOVER_CALL` rotation plus its own reactive `sched_publish_check_rejoin()`. This general-client shape is replaced for the scheduler specifically; workers and the `xce_test`/CLI clients are unaffected and keep the existing candidate-list/failover-rotation model.
 - `ha_dispatch_gate` already enforces that only a primary controller accepts scheduler write opcodes (job dispatch, pend-reason) — this pre-existing behavior is why this feature is purely about availability, not correctness (see research.md).
 ### Architecture Pattern & Boundary Map
 ```mermaid
 graph TB
    subgraph HostA
        CtrlA[Controller Instance A Primary]
        SchedA[Scheduler Process]
        CtrlA -->|spawns and owns| SchedA
    end
    subgraph HostB
        CtrlB[Controller Instance B Standby]
    end
    subgraph HostC
        CtrlC[Controller Instance C Standby]
    end
    CtrlA <-->|raft election and replication| CtrlB
    CtrlA <-->|raft election and replication| CtrlC
    CtrlB <-->|raft election and replication| CtrlC
    SchedA -->|pinned single connection join and status query| CtrlA
 ```
 **Architecture Integration**:
 - Selected pattern: controller-owned child process, triggered directly from the existing Raft promotion/demotion callbacks, executed through the existing SlowLane off-drive-thread executor.
 - Domain/feature boundaries: controller-side process ownership (`scheduler_proc.c`, new) is separate from controller-side scheduler-connection bookkeeping (`sched.c`, existing, extended); scheduler-side controller-liveness watching (`sched_owner.c`, new) is separate from scheduler-side scheduling logic (`scheduler.c`, unchanged except for removing the superseded rejoin logic).
 - Existing patterns preserved: SlowLane for off-drive-thread work; `ha_dispatch_gate` as the sole write-safety mechanism; the existing Cap'n Proto wire connection and request-handler dispatch shape on both sides.
 - New components rationale: `scheduler_proc.c` is a new domain (process ownership) with no existing owner; `sched_owner.c` is a new domain (single-controller liveness pinning) that cannot be folded into `scheduler.c`'s existing scheduling-engine responsibility without violating single-responsibility.
 - Steering compliance: respects `ha-async-io.md`'s thread-role contract (all blocking work — posix_spawn/waitpid/kill/status-query — runs via SlowLane or the scheduler's own non-drive-thread context, never inline on iomgr/delaymgr) and the controller-never-self-terminate invariant (a scheduler-management failure logs loudly and retries; it never causes the controller to crash, hang, or exit, and never becomes a new promotion-refusal gate).
 ### Technology Stack
 | Layer | Choice / Version | Role in Feature | Notes |
 |-------|------------------|-----------------|-------|
 | Controller process management | POSIX `posix_spawn`/`waitpid`/signal (C, `src/controller/scheduler_proc.c`) | Start, monitor, and stop the scheduler child process | `product.md`'s locked decision "no `fork()` for snapshots (mutex-inheritance deadlock in multithreaded controller)" applies just as much here: a raw `fork()` only duplicates the calling thread, so a lock held by any *other* thread (glibc malloc, XCE's own `mylock_t` locks) is duplicated in whatever state it was in, and the child can deadlock if it touches malloc or any XCE library call before `execve()` — being on a dedicated SlowLane worker does not change this, since the hazard comes from *other* threads in the same process, not the calling one. `posix_spawn()` is used instead specifically to avoid re-triggering this exact, already-documented risk class, not merely as a style preference |
 | Async execution | `slowlane_submit()` (existing, `src/controller/slowlane.c`) | Runs all scheduler-management actions off the Raft drive threads | Reused as-is, no changes |
 | Wire protocol | Cap'n Proto (existing, `src/schema/controller.capnp`) | Carries the new graceful-exit notice from controller to scheduler over the existing connection | One small schema addition; no new transport |
 | Scheduler startup config | CLI argument (existing mechanism, new single-address mode) | Controller passes its own address to the scheduler it spawns, replacing the multi-candidate `--peers` list for this specific relationship | Deployment tooling generating scheduler startup commands must change accordingly |
 ## File Structure Plan
 ### Directory Structure
 ```
 src/
 ├── controller/
 │   ├── scheduler_proc.c        # NEW: owns posix_spawn/monitor/signal/reap of this controller's scheduler child
 │   ├── include/
 │   │   └── scheduler_proc.h    # NEW: SchedulerProcessManager service interface
 │   ├── ha.c                    # MODIFIED: call scheduler_proc_start() at the tail of raft_cb_become_leader,
 │   │                           #   scheduler_proc_stop() at the top of raft_cb_become_follower
 │   └── sched.c                 # MODIFIED: extend join/leave bookkeeping to feed liveness/wedge detection
 ├── scheduler/
 │   └── src/
 │       ├── sched_owner.c       # NEW: pins to one controller address, periodic liveness/role check, exits on loss
 │       ├── scheduler.c         # MODIFIED: remove sched_needs_rejoin/sched_on_reconnect/sched_publish_check_rejoin
 │       │                       #   and the multi-candidate startup path for this role
 │       ├── handler.c           # MODIFIED: scheduler_req_handler gains a case for the graceful-exit notice
 │       └── main.c              # MODIFIED: new single-controller-address startup argument
 ├── schema/
 │   └── controller.capnp        # MODIFIED: add the controller-to-scheduler graceful-exit message
 └── controller/testsuite/, scheduler/testsuite/  # new/modified test files, see Testing Strategy
 ```
 ### Modified Files (tooling and specs, outside `src/`)
 - `.claude/skills/xce-deploy-ha/SKILL.md` — remove/replace Step 4b's manual `start-scheduler.sh` guidance; document that a running controller now starts its own scheduler automatically once it becomes primary.
 - `src/tools/deploy-ha.sh` — stop generating a standalone `start-scheduler.sh` intended for manual/independent invocation (or repurpose it to reflect the new single-controller-address contract if a manual escape hatch is still useful for local debugging).
 - `.claude/skills/xce-test-ha/run-ha-soak.sh` — remove the worker-rejoin-resync soak coverage's `start_worker_and_scheduler()`/`restart_scheduler_for_new_primary()` manual scheduler management added this session (bug-1760's workaround becomes unnecessary — the controller now manages this itself); the soak harness should instead just verify a scheduler is present after each promotion.
 ## System Flows
 ### Promotion: controller starts its scheduler
 ```mermaid
 sequenceDiagram
    participant Raft as Raft Election
    participant HA as ha.c become leader
    participant Lane as SlowLane Executor
    participant Proc as scheduler_proc.c
    participant Sched as Scheduler Process
    Raft->>HA: on_become_leader
    HA->>HA: run existing promotion eligibility gates
    HA->>Lane: slowlane_submit start action dedup key this controller
    Lane->>Proc: run start action on a lane worker
    Proc->>Sched: posix_spawn passing this controller address
    Sched->>HA: join using pinned single controller address
    HA->>Sched: push jobs queues config existing behavior
 ```
 Gating conditions: the SlowLane submission only happens after every existing promotion-refusal gate has already passed (Requirement 1.1). A failed or deferred `slowlane_submit()` (already-in-flight dedup, or admission-cap deferral) is not an error — the work is still queued and will run (Requirement 1.5's retry is satisfied by SlowLane's own queuing semantics plus an explicit retry-on-failure inside the submitted action).
 ### Graceful demotion: controller signals its scheduler to exit
 ```mermaid
 sequenceDiagram
    participant Raft as Raft Election
    participant HA as ha.c become follower
    participant Lane as SlowLane Executor
    participant Proc as scheduler_proc.c
    participant Sched as Scheduler Process
    HA->>Lane: slowlane_submit stop action dedup key this controller
    HA->>HA: continue completing its own role transition immediately
    Lane->>Proc: run stop action on a lane worker
    Proc->>Sched: send graceful exit notice over existing connection
    Sched->>Sched: finish in flight work and exit
    Proc->>Proc: wait bounded time then reap or force terminate
 ```
 The role transition in `raft_cb_become_follower` never waits on the SlowLane task's completion (Requirement 2.3) — submission is fire-and-forget from the Raft callback's perspective.
 ### Scheduler self-detects loss of its owning controller
 ```mermaid
 sequenceDiagram
    participant Sched as Scheduler Process
    participant Owner as sched_owner.c
    participant Ctrl as Its Owning Controller
    loop periodic check
        Owner->>Ctrl: status query over the pinned connection
        Ctrl-->>Owner: role and reachability
    end
    alt controller unreachable or reports it is no longer primary
        Owner->>Sched: request exit
        Sched->>Sched: exit
    end
 ```
 This path is independent of the graceful-exit notice above — it is the general-case detector covering crash, network partition, and any demotion trigger the explicit notice did not reach in time (Requirement 3.1-3.3). No candidate list exists to fall back to (Requirement 3.4) — the only outcome of a failed check is exit.
 ### Wedged-scheduler detection and recovery
 ```mermaid
 sequenceDiagram
    participant HA as ha.c or sched.c
    participant Lane as SlowLane Executor
    participant Proc as scheduler_proc.c
    participant Sched as Scheduler Process
    HA->>Proc: observe scheduler connected but no scheduling progress signal within threshold
    Proc->>Lane: slowlane_submit restart action dedup key this controller
    Lane->>Proc: run restart action
    Proc->>Sched: terminate
    Proc->>Proc: posix_spawn a replacement per the promotion flow
 ```
 ## Requirements Traceability
 | Requirement | Summary | Components | Interfaces | Flows |
 |-------------|---------|------------|------------|-------|
 | 1.1, 1.2, 1.3 | Scheduler starts on promotion, at most one in flight, no duplicate start | scheduler_proc.c, ha.c | `scheduler_proc_start` | Promotion flow |
 | 1.4 | Scheduler joins and receives job/queue/config state after being started | scheduler_proc.c, sched.c | `scheduler_proc_start` | Promotion flow |
 | 1.5 | Controller retries on start failure, logs loudly | scheduler_proc.c | `scheduler_proc_start` | Promotion flow |
 | 2.1, 2.2, 2.3 | Graceful exit signaled on demotion; scheduler exits within bounded time; controller's own role transition never waits on it | scheduler_proc.c, ha.c, controller.capnp addition, scheduler handler.c | `scheduler_proc_stop`, graceful-exit wire message | Graceful demotion flow |
 | 2.4 | No job left inconsistent or lost regardless of how the scheduler was terminated | scheduler_proc.c (bounded stop/force-kill), controller's existing WAL-commit ownership (unchanged) | `scheduler_proc_stop` | Graceful demotion flow, Data Models |
 | 3.1, 3.2, 3.3 | Scheduler actively verifies its owning controller is reachable and primary; exits on failure of either check | sched_owner.c | `sched_owner_watch_start` | Self-detection flow |
 | 3.4 | Scheduler never joins or operates against any controller other than the one that started it | sched_owner.c, scheduler.c (rejoin logic removed) | `sched_owner_watch_start` | Self-detection flow |
 | 3.5 | Abrupt host/process death needs no scheduler-side action | (no component — a structural consequence of co-location, not new logic) | n/a | Self-detection flow (contrast case) |
 | 4.1, 4.2, 4.3 | Controller never blocks heartbeats/elections, never crashes/hangs, always logs scheduler-management failures | scheduler_proc.c (via SlowLane) | `scheduler_proc_start`/`stop`/`check_liveness` (all SlowLane-submitted) | All flows |
 | 4.4 | Absence of a running scheduler never blocks a controller's own HA role | scheduler_proc.c, ha.c (unchanged promotion gates) | n/a | Promotion flow |
 | 5.1, 5.2 | Controller monitors scheduling progress, not just connection state; flags no-progress past a threshold as unresponsive | SchedulerConnectionBookkeeping (sched.c), scheduler_proc.c | `scheduler_proc_check_liveness` | Wedge detection flow |
 | 5.3, 5.4 | Unresponsive scheduler is terminated and replaced, with an operator-visible log | scheduler_proc.c | `scheduler_proc_check_liveness` | Wedge detection flow |
 | 6.1 | Existing deployment tooling documentation updated to reflect controller-owned scheduler startup | xce-deploy-ha SKILL.md, deploy-ha.sh | n/a (tooling) | Migration Strategy |
 | 6.2 | Existing specs/tooling depending on scheduler survive-and-reconnect behavior identified and reconciled | xce-test-ha run-ha-soak.sh (this session's own bug-1760 workaround is the concrete example identified and reconciled here) | n/a (tooling) | Migration Strategy, Testing Strategy E2E |
 | 6.3 | Scheduler's observable job/queue/config behavior unchanged while running under a live primary | scheduler.c (unchanged scheduling logic) | n/a | Data Models |
 ## Components and Interfaces
 | Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
 |-----------|--------------|--------|---------------|--------------------------|-----------|
 | SchedulerProcessManager | Controller | Owns posix_spawn/monitor/signal/reap of this controller's scheduler child | 1, 2, 4, 5 | SlowLane executor (P0), ha.c callbacks (P0) | Service |
 | SchedulerConnectionBookkeeping | Controller | Existing join/leave tracking, extended to feed liveness/progress signal | 5 | SchedulerProcessManager (P1) | Service |
 | SchedOwnerWatch | Scheduler | Pins to one controller address; periodic liveness/role check; exits on loss or graceful notice | 3 | Existing wire connection/status query (P0) | Service |
 | Graceful-Exit Wire Message | Wire protocol | Controller-to-scheduler notice carried on the existing connection | 2 | Cap'n Proto schema (P0) | Event |
 ### Controller
 #### SchedulerProcessManager
 | Field | Detail |
 |-------|--------|
 | Intent | Start, monitor, signal, and reap this controller's own scheduler child process, entirely off the Raft drive threads |
 | Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3, 4.1, 4.2, 4.3, 4.4, 5.1, 5.2, 5.3, 5.4 |
 **Responsibilities & Constraints**
 - Owns the decision of whether a scheduler child is currently running for this controller instance; `ha.c` never spawns or signals a process directly.
 - Every action (start, stop, restart) is submitted to the SlowLane executor with a dedup key scoped to this controller instance, never invoked inline from a Raft callback.
 - Never allowed to block, crash, or cause the controller process to exit — every failure path logs an operator-visible error and either retries (start) or proceeds with the role transition regardless (stop).
 **Dependencies**
 - Outbound: SlowLane executor (`slowlane_submit`) — P0, the only allowed execution path for this component's blocking work.
 - Inbound: `ha.c`'s `raft_cb_become_leader` (calls `scheduler_proc_start`), `raft_cb_become_follower` (calls `scheduler_proc_stop`) — P0.
 - Inbound: a periodic liveness check (driven from the existing controller heartbeat cadence, not a new timer thread) — P1, feeds `scheduler_proc_check_liveness`.
 **Contracts**: Service [x]
 ##### Service Interface
 ```c
 /* src/controller/include/scheduler_proc.h */
 /* Requests that this controller's scheduler child be started, if one is not
 * already running or already starting. Non-blocking: submits to the SlowLane
 * executor and returns immediately. Safe to call multiple times in quick
 * succession — the SlowLane dedup key ensures at most one start action is
 * ever in flight per controller instance.
 * Returns 0 if the request was accepted (started or already in flight/running),
 * a negative errno if the request itself could not even be submitted (e.g.
 * SlowLane shutting down) — never blocks, never crashes the caller.
 */
 int scheduler_proc_start(cluster_t *c);
 /* Requests that this controller's scheduler child be stopped: sends the
 * graceful-exit notice, waits up to timeout_ms for the child to exit, then
 * force-terminates if it has not. Non-blocking to the caller: submitted to
 * the SlowLane executor. Does not block the caller's own role transition.
 */
 int scheduler_proc_stop(cluster_t *c, uint32_t timeout_ms);
 /* Evaluates whether the currently-running scheduler child (if any) is making
 * scheduling progress, using the existing push/acknowledgment traffic
 * (SchedulerConnectionBookkeeping) as the progress signal. If the child is
 * connected but has made no progress for longer than the configured
 * threshold, submits a restart action. A no-op if no scheduler child is
 * currently owned. Non-blocking.
 */
 int scheduler_proc_check_liveness(cluster_t *c);
 ```
 - Preconditions: `c` is a valid, initialized `cluster_t`; called only from the controller's own process.
 - Postconditions: the requested action is either already complete, in flight on a SlowLane worker, or queued — never silently dropped.
 - Invariants: at most one scheduler child process is owned by a given controller instance at any time; a stop-then-start (demotion followed by a fresh promotion of the same instance) always completes the stop's reap before a new start is submitted, since a demoted instance must win a fresh election before it can be promoted again.
 **Implementation Notes**
 - Integration: hook call sites are the tail of `raft_cb_become_leader` (after all existing promotion-refusal gates) and the top of `raft_cb_become_follower`.
 - Validation: an operator-visible log line on every start failure, stop-timeout-forced-kill, and liveness-triggered restart (Requirement 4.3, 5.4).
 - Risks: sizing the shared SlowLane instance's `max_inflight`/`n_workers` for the additional scheduler-management workload alongside its existing consumers is a follow-up check, not a design change (see research.md).
 #### SchedulerConnectionBookkeeping (extends existing `sched.c`)
 | Field | Detail |
 |-------|--------|
 | Intent | Track whether the owned scheduler child has joined and is making scheduling progress, feeding `scheduler_proc_check_liveness` |
 | Requirements | 5.1, 5.2 |
 **Responsibilities & Constraints**
 - Extends the existing `on_scheduler_join()`/`handle_scheduler_leave()`/`cluster_t.sched.ep` bookkeeping with a last-progress timestamp, updated whenever a push to the scheduler is acknowledged or a dispatch/pend-reason decision is received from it.
 - Does not itself decide to restart anything — it only exposes the progress signal `scheduler_proc_check_liveness` reads.
 **Dependencies**
 - Inbound: existing scheduler request handling in `cluster.c`/`sched.c` — P1.
 - Outbound: read by SchedulerProcessManager — P1.
 **Contracts**: Service [x]
 **Implementation Notes**
 - Integration: reuses existing traffic; no new synthetic heartbeat payload is introduced (see research.md Risks).
 ### Scheduler
 #### SchedOwnerWatch
 | Field | Detail |
 |-------|--------|
 | Intent | Pin the scheduler to the single controller instance that started it; periodically confirm that controller is reachable and still primary; exit otherwise |
 | Requirements | 3.1, 3.2, 3.3, 3.4, 3.5 |
 **Responsibilities & Constraints**
 - At startup, the scheduler receives exactly one controller address (not a candidate list) and establishes its connection only to that address.
 - Periodically issues the existing role-agnostic status query against that one connection; on an unreachable connection or a reported non-primary role, requests the scheduler process to exit.
 - Never attempts any other controller address, and never re-derives a candidate list — this component structurally has no such list to fall back to.
 - Also receives and acts on the graceful-exit notice (Requirement 2.2) via the existing request-handler dispatch, exiting within a bounded time.
 **Dependencies**
 - Outbound: the existing controller-scheduler wire connection/status query — P0.
 - Inbound: `scheduler_req_handler`'s new graceful-exit case — P0.
 **Contracts**: Service [x]
 ##### Service Interface
 ```c
 /* src/scheduler/include/sched_owner.h */
 /* Starts the periodic ownership-watch loop against the single controller
 * address this scheduler process was given at startup. Runs on its own
 * thread (mirroring the worker daemon's existing checking_connection()
 * cadence/shape), independent of the picker/allocator threads.
 */
 int sched_owner_watch_start(const char *owner_host, uint16_t owner_port);
 /* Called by the request handler when the graceful-exit notice arrives on the
 * existing connection. Triggers the same exit path the periodic watch loop
 * uses on a failed check.
 */
 void sched_owner_request_exit(void);
 ```
 - Preconditions: `owner_host`/`owner_port` identify the specific controller instance that spawned this process.
 - Postconditions: the process exits once either the periodic check fails or a graceful-exit notice is received — no other outcome resumes normal operation with a different controller.
 - Invariants: never issues `xcecli_join`/any wire call against any address other than `owner_host:owner_port`.
 **Implementation Notes**
 - Integration: replaces `sched_needs_rejoin`/`sched_on_reconnect`/`sched_publish_check_rejoin` in `scheduler.c` entirely — these are removed, not left dormant, since Requirement 3.4 makes their cross-controller-rejoin purpose permanently unreachable.
 - Risks: the periodic check interval trades detection latency against status-query load; finalize the exact interval during implementation, following the worker's existing `update_interval`-style convention rather than inventing a new tunable pattern.
 ### Wire Protocol
 #### Graceful-Exit Wire Message
 **Event Contract**
 - Published by: the controller, from `scheduler_proc_stop`'s SlowLane action, over the existing controller-scheduler connection.
 - Subscribed by: the scheduler's existing `scheduler_req_handler` (`src/scheduler/src/handler.c`), which invokes `sched_owner_request_exit()`.
 - Delivery: best-effort over the existing persistent connection — if delivery fails (connection already gone), `scheduler_proc_stop`'s bounded wait-then-force-terminate path (Requirement 2.3) still guarantees the child does not outlive the timeout, and `SchedOwnerWatch`'s independent periodic check (Requirement 3) is the backstop if the notice is lost.
 - Ordering/idempotency: at most one graceful-exit notice is ever sent per stop action (SlowLane dedup); receiving it more than once (e.g. a retry) is a no-op if the process is already exiting.
 ## Data Models
 No new persisted entity is introduced. The scheduler remains fully stateless with respect to durable storage — its in-memory job/queue/config state is always rebuilt from the controller's existing push traffic on join, unchanged by this feature (Requirement 6.3). The only new in-memory state is:
 - Controller side: which scheduler child (if any) this controller instance currently owns, its process handle, and its last-progress timestamp — process-lifetime only, never persisted, rebuilt from scratch on controller restart (a restarted controller that is primary starts a fresh scheduler exactly as any other promotion would).
 - Scheduler side: the single owning-controller address it was started with — process-lifetime only.
 Since nothing here is written to or read from a durable log, snapshot, or checkpoint, the project's save/restore field-completeness review does not apply to this design.
 ## Error Handling
 ### Error Strategy
 Every failure in scheduler-lifecycle management follows the project's existing fail-loud-stay-alive philosophy (`ha-async-io.md`, `controller/CLAUDE.md`): log an operator-visible error, never crash or hang the controller, and retry where retrying is meaningful.
 ### Error Categories and Responses
 - **Scheduler fails to start** (posix_spawn failure, immediate crash): `scheduler_proc_start` logs an error and retries on the controller's existing heartbeat-driven cadence while it remains primary (Requirement 1.5); the controller itself continues serving every other role unaffected (Requirement 4.4).
 - **Scheduler does not exit within the graceful-exit timeout**: `scheduler_proc_stop` force-terminates the child after the bounded wait; the controller's own demotion completes regardless (Requirement 2.3).
 - **Scheduler connected but not progressing (wedged)**: treated as unresponsive once past the threshold, terminated and replaced (Requirement 5.3), with an operator-visible log (Requirement 5.4).
 - **Scheduler cannot reach or confirm its owning controller**: the scheduler itself exits (Requirement 3.2, 3.3) — no error is reported back to the controller for this case, since the controller side detects the resulting disconnect via existing `handle_scheduler_leave()` bookkeeping.
 ### Monitoring
 All new log lines (start failure, forced-kill-on-timeout, liveness-triggered restart) use the existing controller logging conventions (`WARN`/`ERROR` macros already used throughout `ha.c`) — no new logging mechanism is introduced.
 ## Testing Strategy
 ### Unit Tests
 - `scheduler_proc_start` submits exactly one SlowLane action per promotion and a second concurrent call while one is in flight is deduped (not double-submitted) — directly exercises Requirement 1.2/1.3 via the existing `test_slowlane.c` dedup behavior.
 - `raft_cb_become_leader` calls `scheduler_proc_start` only after every existing promotion-refusal gate returns eligible, and does not call it on any refusal path — regression-guards Requirement 1.1 against a future gate reordering.
 - `raft_cb_become_follower` calls `scheduler_proc_stop` and completes its own role transition without waiting on that call's result — regression-guards Requirement 2.3.
 - `SchedOwnerWatch`'s periodic check exits the process on a simulated unreachable controller and on a simulated non-primary role response, and never attempts a second address — directly exercises Requirement 3.1-3.4.
 - `scheduler_proc_check_liveness` triggers a restart only when the last-progress timestamp exceeds the threshold while the connection remains open (not on a disconnect, which is a separate, already-detected case) — exercises Requirement 5.1/5.2.
 ### Integration Tests
 - A real 3-node controller cluster promotion results in a scheduler process appearing, joining, and receiving job/queue/config state, with no manual step — exercises Requirement 1.1-1.4 end to end.
 - A graceful step-down (forced via the existing test-only demotion trigger) results in the prior primary's scheduler exiting within the bounded timeout, verified by process absence, not just a log line — exercises Requirement 2.1/2.2.
 - A hard-killed primary (matching this session's original bug-1760 repro exactly) results in the orphaned scheduler self-exiting within its own periodic-check interval, and the newly-elected primary's fresh scheduler successfully dispatching a job submitted after the failover — exercises Requirement 3.1-3.5 and directly closes bug-1760.
 - A scheduler artificially wedged (e.g. its progress-signal thread blocked) while its connection stays open is detected and replaced — exercises Requirement 5.1-5.3.
 - A job dispatched by a scheduler that is then killed mid-decision (before its dispatch is durably recorded) remains available for the replacement scheduler to consider — exercises Requirement 2.4.
 ### E2E / Soak Tests
 - Extend `.claude/skills/xce-test-ha/run-ha-soak.sh`'s sequential and chaotic modes (already exercising controller failover) to assert a working scheduler is present after every round without any harness-side scheduler start/stop action — this replaces the manual `start_worker_and_scheduler()`/`restart_scheduler_for_new_primary()` logic added this session as bug-1760's workaround, which becomes dead code once this feature ships.
 - A rapid sequence of promotions/demotions (chaotic mode's overlapping-kill pattern) produces no scheduler spawn/kill thrash beyond one scheduler per genuinely-promoted instance — exercises the SlowLane-dedup mitigation documented in research.md's Risks.
 ## Migration Strategy
 ```mermaid
 flowchart TB
    Phase1[Phase 1 controller and scheduler code changes land behind no behavior change until wired]
    Phase2[Phase 2 wire promotion and demotion hooks and pinned single controller startup]
    Phase3[Phase 3 update xce deploy ha and xce test ha tooling to stop manually starting the scheduler]
    Phase4[Phase 4 remove superseded rejoin logic and multi candidate scheduler startup path]
    Phase1 --> Phase2 --> Phase3 --> Phase4
 ```
 - Rollback trigger: if a real deployment depends on independently placing the scheduler on a dedicated host for resource-isolation reasons not previously surfaced, that is a genuine requirements gap discovered late — the fix is to pause before Phase 4 (superseded-logic removal) rather than force the migration through, since Phase 4 is the only phase that is hard to reverse cheaply.
 - Validation checkpoint: after Phase 2, run the soak-harness scenario in Testing Strategy's E2E section before proceeding to Phase 3/4, since it is the closest thing to a direct regression test for bug-1760 itself.
 ## Open Questions / Risks
 - The exact wire opcode/schema shape for the graceful-exit notice is left to implementation (a small, additive `controller.capnp` change, following the same pattern recent specs have used for new controller-to-daemon messages) — this is appropriately an implementation detail, not a design decision, since no existing opcode fits (confirmed by inspection: `OPCode_shutdown` is an inbound admin-to-controller command, not a controller-to-scheduler one, and is not reusable here).
 - The precise "no scheduling progress" threshold and periodic-check interval (Requirement 5, Requirement 3) are implementation-time tuning decisions, not design decisions — see research.md Risks for the reasoning against inventing a bespoke synthetic heartbeat.
 - Confirm during implementation whether any other in-tree consumer starts `xce_scheduler` with a multi-candidate `--peers` list for a reason unrelated to this feature (e.g. a test fixture) that would need explicit review rather than silent breakage (carried over from research.md).
--- a/.kiro/specs/scheduler-primary-lifecycle/requirements.md
+++ b/.kiro/specs/scheduler-primary-lifecycle/requirements.md
@ -0,0 +1,85 @@
 # Requirements Document
 ## Project Description (Input)
 Who has the problem: XCE cluster operators, and by extension every job that depends on the scheduler to be reachable after a controller failover.
 Current situation: the scheduler (`xce_scheduler`) runs today as a fully independent, long-lived process that joins the controller cluster via `xcecli_join(HostRole_scheduler)` and publishes dispatch/pend-reason decisions via `sched_publish_dispatch()`/`sched_publish_pend_reason()`. Its only rejoin mechanism after a controller failover is reactive: `sched_publish_check_rejoin()` re-issues `xcecli_join()` only as a side effect of its own *next successful* publish call. After a real controller failover this creates a chicken-and-egg deadlock — the scheduler has nothing queued to publish (no new job ever reaches its picker, since the controller-side registration of "where the scheduler is" is live-connection-scoped state on whichever controller instance the scheduler last explicitly joined, and does not appear to carry over to a newly-elected primary) — so it can never publish anything to trigger its own rejoin. This was live-confirmed against a real 3-node cluster: after killing the primary, the scheduler showed zero log activity of any kind (no WARN/INFO/ERROR, no reconnect attempt) for 60+ seconds, while a job submitted to the new primary sat stuck PEND. Logged as bug-1760 in `.wolf/buglog.json` with full repro evidence. Separately, there is today no story at all for what happens if the scheduler's own host/process dies, is network-partitioned, or wedges internally (e.g. a disk stall blocking its recording I/O, or a deadlocked allocator thread) — no periodic health check, no standby, no automatic recovery.
 What should change (scope decided across discussion, see rationale below): bind the scheduler process's lifecycle to the controller cluster's primary role, replacing its current independent-process-with-reactive-rejoin model entirely — including replacing its client-side candidate-rotation/rejoin logic, not just supplementing it.
 - When a controller transitions to `CTRL_PRIMARY` (wins election / is promoted), it spawns and owns a scheduler process **co-located on its own host** (a genuine child process, not a signal to some independently-running scheduler).
 - When that controller loses primary status **gracefully** (demoted, discovers a higher term, steps down), it explicitly signals its child scheduler to exit before completing the transition.
 - The scheduler process itself **actively monitors** whether the specific controller instance that started it is still alive and still primary, and terminates itself the moment it can no longer confirm that — this is the primary detection mechanism (not a passive fallback), because it also covers cases OS-level parent-death propagation cannot guarantee (e.g. the controller process dying while its host survives). Co-located death (host crash, etc.) provides a redundant backstop for the same outcome. Critically, the scheduler does **not** attempt to rediscover or reconnect to whichever controller becomes the *next* primary — that job belongs entirely to the next primary's own act of spawning a fresh scheduler. This is a deliberate simplification: a scheduler's operating logic no longer needs any cross-primary candidate-rotation or rejoin behavior at all, because its lifetime is scoped to exactly one controller instance's primacy.
 - The newly-elected primary spawns its own fresh scheduler exactly as it would on any other promotion, via the controller's existing, already-hardened Raft election — no new election/lease mechanism for schedulers.
 - This eliminates: `sched_needs_rejoin`/`sched_publish_check_rejoin`'s reactive rejoin logic and the scheduler's client-side failover candidate rotation for its own role (no longer needed — a scheduler's lifetime is now scoped to a single controller's primacy, so it never needs to survive and reconnect across a primary change); the need for a scheduler-side lease/renewal mechanism; and the need for standby scheduler processes — host/process death is subsumed into "controller host dies," a failure mode the controller's HA layer already handles robustly, so no second, parallel HA/authority mechanism is being built for schedulers.
 - In scope, smaller: a liveness/health check the parent controller performs on its own live child scheduler, to catch the case where the process is alive (so co-location death didn't trigger) but internally wedged (e.g. blocked disk I/O, deadlocked picker/allocator thread) and therefore not doing useful work despite an open connection. This does not require a lease or standby scheduler — it is a local parent-watches-child health signal, and on failure the controller may restart its own child scheduler.
 - Explicitly OUT of scope for this spec (raised and deliberately deferred during discussion):
  - Standby/redundant scheduler processes competing for a lease — rejected in favor of piggybacking on the controller's own already-tested election mechanism.
  - A general per-host "role manager" component that starts/stops arbitrary XCE components (worker, controller, scheduler, apiserver) based on declared host roles and cluster state. This was discussed as a possible bigger direction but deliberately deferred: the controller is uniquely positioned to trigger scheduler startup at zero extra latency the instant it becomes primary, whereas a separate agent would need to watch controller election state itself and introduces a new process whose own supervision becomes a new problem. If a genuine, independent need for general host/role management across all XCE components emerges later, it should be its own spec, not inherited scope from this one.
  - Any change to how `worker` or `apiserver` processes are deployed/supervised.
 Constraint carried into design: scheduler and controller currently value independent deployability (separate hosts, separate scaling/tuning, separate release cadence per `.kiro/steering/structure.md`'s component layout). This spec deliberately trades that independence away for the scheduler specifically, in exchange for eliminating an entire class of availability bugs (reactive rejoin deadlock, host-death handling, wedge detection) by reusing already-hardened controller HA machinery instead of building a parallel one. The design phase must make this tradeoff and its consequences (scheduler can no longer run on a dedicated host, failover now includes scheduler cold-start time, controller gains new child-process-supervision responsibility) explicit rather than incidental.
 Related code to ground requirements/design in: `src/controller/ha.c` (promotion/demotion — e.g. `ha_start_primary_heartbeat`, `raft_cb_become_leader`; `ha_dispatch_gate` already safely rejects scheduler writes against a non-primary, confirmed by reading the code, so no split-brain risk exists today from a stale scheduler connection — this spec is solely about availability, not correctness), `src/controller/sched.c` (`on_scheduler_join()`, `handle_scheduler_leave()` — existing controller-side scheduler-connection lifecycle hooks), `src/include/xce/cluster_def.h` (`XCE_CONTROLLER_HOOK_SCHEDULER`, `HOOK_DETAIL_SCHEDULER_JOIN`/`HOOK_DETAIL_SCHEDULER_DEAD` hooks already exist and may be relevant plumbing), `src/scheduler/src/scheduler.c` (`sched_needs_rejoin`/`sched_on_reconnect`/`sched_publish_check_rejoin` — the mechanism being replaced), `src/worker/src/process_worker_manager.c` (existing precedent in this codebase for a daemon spawning/supervising a child process, though for job execution rather than another XCE daemon), `.kiro/steering/ha-async-io.md` (controller-never-self-terminate invariant, and the thread-role contract that no liveness-carrying thread — iomgr/delaymgr — may block; any new spawn/monitor/signal mechanism the controller gains must respect this), `.claude/skills/xce-deploy-ha/SKILL.md` and `.claude/skills/xce-test-ha/run-ha-soak.sh` (existing tooling that models the scheduler as an independently-started process via `start-scheduler.sh` — needs to be reconciled with the new model, since multiple existing specs/soak-harness modes depend on today's independent-process assumption). Bug-1760 in `.wolf/buglog.json` has the full live-repro evidence motivating this spec.
 ## Boundary Context
 - **In scope**: controller-triggered scheduler startup on promotion; graceful scheduler shutdown signaled on demotion; scheduler self-termination via active monitoring of its own originating controller's liveness/primary status; controller detection of, and response to, an unresponsive-but-alive ("wedged") child scheduler; controller resilience so that scheduler-lifecycle management never compromises the controller's own availability or HA correctness; reconciling existing deployment tooling and specs that assume an independently-started scheduler.
 - **Out of scope**: standby/redundant scheduler processes and any lease-election mechanism for schedulers; a general host/role manager governing worker, controller, or apiserver process lifecycle; any change to worker or apiserver deployment/supervision.
 - **Adjacent expectations**: this feature depends on the controller's existing Raft-based primary election (unchanged by this spec) as the sole source of truth for "who may run a scheduler right now." It also depends on the existing `ha_dispatch_gate` behavior (a non-primary controller already rejects scheduler write operations) continuing to hold, since this spec's safety argument rests on that already-verified behavior rather than re-implementing it. Existing operator-facing deployment tooling that starts a scheduler as an independent process (documented in `xce-deploy-ha`/`xce-test-ha`) is expected to change to reflect controller-owned scheduler startup; any spec or tooling that still assumes an independently-started, long-lived scheduler process is an adjacent consumer that must be revisited, not an unstated exception to this feature's behavior.
 ## Requirements
 ### Requirement 1: Scheduler startup on controller promotion
 **Objective:** As a cluster operator, I want a scheduler to start automatically whenever a controller becomes primary, so that the cluster can resume scheduling jobs without any manual intervention after a failover.
 #### Acceptance Criteria
 1. When a controller transitions to the primary role, the Controller shall start a Scheduler Process associated with itself.
 2. While a controller remains primary, the Controller shall ensure at most one Scheduler Process it started is running.
 3. If a controller is already primary and already has a running Scheduler Process associated with itself, the Controller shall not start a second Scheduler Process.
 4. When the Controller starts a Scheduler Process, the Scheduler Process shall join the cluster and become able to receive job, queue, and configuration state from that specific controller instance.
 5. If the Controller fails to start a Scheduler Process, the Controller shall record an operator-visible error and shall continue to retry starting one while it remains primary.
 ### Requirement 2: Scheduler shutdown on graceful controller demotion
 **Objective:** As a cluster operator, I want the scheduler tied to a controller to stop cleanly when that controller steps down, so that no scheduler keeps running on behalf of a controller that is no longer authoritative.
 #### Acceptance Criteria
 1. When a controller that started a Scheduler Process transitions away from the primary role through a graceful demotion (voluntary step-down, or discovering a higher election term while still running), the Controller shall signal that Scheduler Process to terminate.
 2. When a Scheduler Process receives a termination signal from its originating controller, the Scheduler Process shall exit within a bounded time.
 3. The Controller shall not remain in the primary-to-standby transition indefinitely waiting on its Scheduler Process to exit; a Scheduler Process that does not exit within the bounded time shall not prevent the Controller from completing its own role transition.
 4. When a Scheduler Process is terminated for any reason (graceful signal, self-detected loss of its originating primary, or abrupt process/host death), no job shall be left in an inconsistent or lost state as a result — any job the terminated Scheduler Process had not yet caused to be durably recorded shall remain available for a subsequent Scheduler Process to consider.
 ### Requirement 3: Scheduler self-termination on loss of its originating primary controller
 **Objective:** As a cluster operator, I want the scheduler itself to notice when its own controller is no longer a live primary and exit on its own, so that a scheduler can never keep running detached from any authoritative controller, and so that no cross-controller reconnect logic is needed in the scheduler at all.
 #### Acceptance Criteria
 1. While a Scheduler Process is running, the Scheduler Process shall continuously verify that the specific controller instance that started it is reachable and still holds the primary role.
 2. When a Scheduler Process can no longer confirm that its originating controller instance is reachable, the Scheduler Process shall terminate.
 3. When a Scheduler Process learns that its originating controller instance is no longer primary, the Scheduler Process shall terminate, even if that controller instance is still reachable.
 4. The Scheduler Process shall not attempt to join, rejoin, or continue operating against any controller instance other than the one that started it.
 5. If the host or process running a Scheduler Process terminates abruptly (crash, host failure) before the Scheduler Process can detect the loss of its originating controller on its own, no further scheduler-side action is required — the Scheduler Process is expected to be gone as a direct consequence of that failure.
 ### Requirement 4: Controller resilience to scheduler-lifecycle management
 **Objective:** As a cluster operator, I want a controller's own availability and correctness to never depend on its child scheduler's health, so that scheduler-management responsibilities cannot introduce a new way for the controller to become unavailable.
 #### Acceptance Criteria
 1. The Controller shall never block its own heartbeat processing or election-timeout handling while starting, monitoring, or signaling a Scheduler Process.
 2. If starting, monitoring, or signaling a Scheduler Process fails for any reason, the Controller shall not crash, hang, or self-terminate as a result.
 3. The Controller shall log an operator-visible error whenever it fails to start, loses, or forcibly restarts a Scheduler Process it owns.
 4. The absence of a running Scheduler Process shall not prevent a controller from being elected primary, remaining primary, or serving read-only and replication traffic.
 ### Requirement 5: Detecting and recovering a wedged Scheduler Process
 **Objective:** As a cluster operator, I want the controller to notice when its own scheduler is alive but not making progress, so that jobs do not silently stop being scheduled while everything still looks connected.
 #### Acceptance Criteria
 1. While a Controller has a running Scheduler Process, the Controller shall monitor whether that Scheduler Process is making scheduling progress, not merely whether its connection remains open.
 2. If a Scheduler Process is connected but fails to make scheduling progress for longer than an operator-observable threshold, the Controller shall treat that Scheduler Process as unresponsive.
 3. When the Controller treats its Scheduler Process as unresponsive, the Controller shall terminate that Scheduler Process and start a replacement.
 4. The Controller shall log an operator-visible event whenever it terminates a Scheduler Process because it was found unresponsive.
 ### Requirement 6: Compatibility with existing scheduler operation and tooling
 **Objective:** As a cluster operator or spec author relying on existing scheduler deployment tooling, I want the change in scheduler ownership to be reflected everywhere the scheduler's lifecycle is currently assumed to be independent, so that existing deployments, tests, and specs do not silently break or become misleading.
 #### Acceptance Criteria
 1. Where existing deployment tooling documents starting a scheduler as an independent, manually-managed process, that tooling shall be updated to reflect that a running controller now starts its own scheduler automatically.
 2. Where an existing spec or test tooling depends on a scheduler surviving and reconnecting across a controller failover, that dependency shall be identified and reconciled with this feature's behavior rather than left silently incompatible.
 3. The Scheduler Process's observable behavior toward jobs, queues, and configuration (join, receive state, dispatch/pend decisions) shall remain unchanged by this feature for the duration that it is running under a live primary controller.
--- a/.kiro/specs/scheduler-primary-lifecycle/research.md
+++ b/.kiro/specs/scheduler-primary-lifecycle/research.md
@ -0,0 +1,82 @@
 ## Summary
 - **Feature**: `scheduler-primary-lifecycle`
 - **Discovery Scope**: Extension (existing controller + scheduler components; no new daemon type, no new external dependency)
 - **Key Findings**:
  - `slowlane_submit()` (`src/controller/slowlane.c`) already exists, is tested, and is the exact primitive this feature needs: a bounded, admission-capped executor with per-key in-flight dedup that runs submitted work off the Raft drive threads (iomgr/delaymgr). No new async-execution mechanism needs to be built.
  - `XCE_CONTROLLER_HOOK_SCHEDULER` / `HOOK_DETAIL_SCHEDULER_JOIN` / `HOOK_DETAIL_SCHEDULER_DEAD` (`src/include/xce/cluster_def.h`) are declared but never fired or consumed anywhere in `src/controller/*.c` — confirmed by exhaustive grep. They are dead enum values, not existing plumbing to build on.
  - The controller already tracks scheduler connection lifecycle via `on_scheduler_join()` / `handle_scheduler_leave()` (`src/controller/sched.c`) and a single `cluster_t.sched.ep` slot — this is reusable bookkeeping for "is my child scheduler currently joined," but it has no concept today of *owning* the scheduler process itself.
  - The scheduler's current rejoin logic (`sched_needs_rejoin`/`sched_on_reconnect`/`sched_publish_check_rejoin`, `src/scheduler/src/scheduler.c`) and its use of the general-purpose, candidate-rotating client library (`xcecli_join`, `XCECLI_FAILOVER_CALL`) both become dead weight under the new model and should be removed, not left dormant — the requirements (3.4) explicitly forbid the scheduler from ever trying another controller.
  - `ha_dispatch_gate()` (`src/controller/ha.c:4573`) already rejects scheduler write opcodes against a non-primary controller (`CTRL_STANDBY`'s allow-list excludes job dispatch / pend-reason). Confirmed by reading the gate's full switch statement. This means the safety argument for this feature is already true today — the feature is solely about *availability*, not correctness.
 ## Research Log
 ### Where does controller promotion actually complete?
 - **Context**: Need the concrete integration point for "start a scheduler on promotion."
 - **Sources Consulted**: `src/controller/ha.c`, `raft_cb_become_leader` (line 1615), read via codegraph.
 - **Findings**: `raft_cb_become_leader` is long and gated: election-timeout cancellation, safe-committed-LSN computation, first-boot `cluster_init_phase2`, WAL truncation-floor handling, then a sequence of promotion-refusal gates (recovery-truncated, apply-divergent, self-restart-pending, apply-resource-fenced, config-serve-fenced) that each `return` without completing promotion, staying alive per the project's never-self-terminate invariant. Only after every gate passes does the function proceed to authoritative promotion (role flip to `CTRL_PRIMARY`, `ha_start_primary_heartbeat`, `repl_add_standby` for reachable voters).
 - **Implications**: The scheduler-start action must be triggered only at the tail of this function, after all refusal gates have passed — never speculatively earlier. A promotion attempt that gets refused by any gate must not start a scheduler.
 ### Where does demotion happen?
 - **Context**: Need the integration point for "signal my scheduler to exit."
 - **Sources Consulted**: `src/controller/ha.c`, `raft_cb_become_follower` (line 2120), read via codegraph.
 - **Findings**: `raft_cb_become_follower` is the Raft library's `on_become_follower` callback, invoked from multiple paths (losing an election, observing a higher term, an append-entries request from a legitimate newer leader). This is the single funnel for every graceful role-loss case named in Requirement 2.1 (voluntary step-down and higher-term discovery both route through it).
 - **Implications**: One hook site covers all of Requirement 2's graceful-demotion triggers; no separate handling needed per trigger type.
 ### Is there an existing async-safe way to do blocking work from the HA path?
 - **Context**: `.kiro/steering/ha-async-io.md`'s thread-role contract forbids blocking work on iomgr/delaymgr, and spawning/signaling/monitoring a child process is unavoidably blocking-ish (fork/exec, waitpid, kill).
 - **Sources Consulted**: `src/controller/slowlane.c`, `src/controller/include/slowlane.h`, existing callers in `ha_convergence.c`, `config_repl.c`, `persist.c`.
 - **Findings**: `slowlane_submit(sl, key, fn, arg)` already exists for exactly this purpose ("long blocking I/O off the Raft drive threads"), is non-blocking to the caller, runs `fn` on a dedicated worker thread with no lock held, and provides per-key in-flight dedup (`-EEXIST` on a duplicate submit of the same key while one is still running/queued). A `test_slowlane.c` suite already covers non-blocking submission, dedup, and admission-cap behavior.
 - **Implications**: Build vs. Adopt — **adopt**. Scheduler start/stop/monitor actions are ordinary `slowlane_submit()` work items. The dedup key (scoped to "this controller's scheduler-management action") mechanically satisfies Requirement 1.2/1.3 ("at most one Scheduler Process," "shall not start a second") as a side effect of reusing an already-tested primitive, with no new synchronization to design or verify.
 ### Is there an existing pattern for one XCE daemon supervising a child process?
 - **Context**: Wanted to know if `process_worker_manager.c`/`uv_ipc.c` (the worker daemon's job-process supervision) is directly reusable.
 - **Sources Consulted**: `src/worker/src/process_worker_manager.c`, `src/worker/src/uv_ipc.c` (libuv-based named-pipe IPC with a heartbeat callback between the worker daemon and its forked job-executor children).
 - **Findings**: The pattern (fork a child, communicate over a local IPC channel, monitor a heartbeat) is directly analogous to what this feature needs between a controller and its scheduler child. The concrete code, however, is tightly coupled to job-execution concerns (cgroups, dbus, job exit-status plumbing) and to libuv, a dependency the controller does not currently carry.
 - **Implications**: Build vs. Adopt — **adopt the pattern, not the code**. The controller gets its own small, purpose-built scheduler-process-manager component (fork/exec + reap + a liveness signal), following the same shape (parent owns child lifecycle, detects child exit, watches liveness) without pulling in libuv or the worker's job-execution-specific machinery. Simplification lens: a shared abstraction with the worker's job-process manager was considered and rejected — the two "child processes" (a job executable vs. a companion XCE daemon) differ enough (job exit codes/signals/cgroups vs. a single well-known daemon binary with its own wire protocol) that forcing a shared interface would add an abstraction layer with only one real implementation on each side.
 ### How does the scheduler detect its own controller is dead or no longer primary?
 - **Context**: Requirement 3 needs an active detection mechanism, not just reliance on OS-level parent-death propagation (discussed and explicitly not relied upon as the primary mechanism, since it doesn't cover "controller process dies while the host lives").
 - **Sources Consulted**: `src/lib/client.c` (`xcecli_join`, `XCECLI_FAILOVER_CALL`), existing `ctrlstatus`/`OPCode_ctrlstatus` read-only status query (confirmed answerable in every controller role per `ha_dispatch_gate`), `src/controller/sched.c`.
 - **Findings**: The scheduler already opens a persistent, Cap'n-Proto-framed TCP connection to a controller and already has a role-agnostic status query (`ctrlstatus`) available in every role. Connection loss (RDHUP) is already detected by the existing transport layer everywhere else in this codebase (workers, clients) — nothing new needs to be built to notice "my controller is unreachable." What is missing is (a) pinning the scheduler to exactly one controller address with no candidate list/rotation, and (b) a periodic check of that one controller's reported role, so a controller that demotes without the connection ever dropping (the common case — TCP stays up across a graceful step-down) is still detected.
 - **Implications**: Build vs. Adopt — adopt the existing wire protocol and status query; build only the pin-to-one-controller startup mode and the periodic self-check loop (a much smaller version of the worker's `checking_connection()` shape, minus any rejoin/rotation behavior). The explicit graceful-exit signal (Requirement 2) rides the same existing connection as a new, small message case in the scheduler's existing request handler (`scheduler_req_handler`, `src/scheduler/src/handler.c`) rather than a separate channel.
 ## Architecture Pattern Evaluation
 | Option | Description | Strengths | Risks / Limitations | Notes |
 |--------|-------------|-----------|---------------------|-------|
 | Controller-owned child process (selected) | Primary controller forks/execs a scheduler bound to itself; scheduler self-exits on losing that specific controller | Reuses controller's already-hardened Raft election for "who's authoritative"; host death handled for free; no new election/lease protocol | Controller gains a new, bounded process-supervision responsibility; scheduler loses independent host placement | Matches decision reached in requirements discussion |
 | Independent scheduler + periodic reconnect (rejected) | Keep scheduler as today, add an unconditional periodic `xcecli_join()` retry (worker-style `checking_connection()`) | Smaller code change; preserves independent deployment | Does not solve scheduler-host-death or wedged-scheduler detection; still needs a second mechanism for those | Considered and superseded once host-death handling became an explicit requirement |
 | Standby schedulers + controller-granted lease (rejected) | Multiple scheduler processes compete for a renewable lease from the primary controller | Handles wedge/host-death without host co-location; scheduler keeps independent placement | A second, parallel authority/liveness protocol to design, build, and prove correct, duplicating the controller's own Raft election in spirit | Rejected in favor of piggybacking on existing Raft election (simplification lens) |
 | General per-host role-manager agent (deferred) | A new component per host starts/stops worker/controller/scheduler/apiserver by declared role and cluster state | Generalizes beyond just scheduler | New always-on process whose own supervision is unsolved; would need to watch controller election state itself, adding latency and an indirection layer | Explicitly deferred in requirements as a separate, independently-motivated initiative, not scope for this spec |
 ## Design Decisions
 ### Decision: Use `slowlane_submit()` for all scheduler process management
 - **Context**: Starting, monitoring, and signaling a child process from `raft_cb_become_leader`/`raft_cb_become_follower` must not block the Raft drive threads.
 - **Alternatives Considered**:
  1. Spawn a brand-new dedicated pthread per promotion — works, but reinvents admission control and dedup that already exist.
  2. Do the fork/exec inline in the Raft callback — violates the thread-role contract outright (fork/exec/waitpid are blocking).
 - **Selected Approach**: Submit scheduler-management actions (start on promotion, signal-and-reap on demotion, restart on detected wedge) to the existing SlowLane executor with a per-controller-instance dedup key.
 - **Rationale**: Already built, already tested, already the established pattern for exactly this class of problem in this codebase (`raft-drive-async` spec).
 - **Trade-offs**: None material — this is a pure reuse with no new abstraction.
 - **Follow-up**: Confirm the SlowLane instance's `max_inflight`/`n_workers` sizing (set at `slowlane_init()` call sites in `cluster.c`) has headroom for scheduler-management tasks alongside its existing consumers (peer connect, snapshot transfer, config-push); this is a sizing check, not a new design.
 ### Decision: Scheduler pins to exactly one controller address at startup, no candidate list
 - **Context**: Requirement 3.4 forbids the scheduler from ever joining or continuing to operate against any controller other than the one that started it.
 - **Alternatives Considered**:
  1. Keep the existing `--peers` full-candidate-list startup mode and just suppress rotation in the failover-call path — leaves a footgun where the shape of the config still implies "any of these are fine."
  2. Give the scheduler a single-controller startup mode entirely, with no candidate list construct at all.
 - **Selected Approach**: Option 2. The controller passes its own address (not the full peer list) when it starts its scheduler.
 - **Rationale**: Makes the "one controller owns one scheduler" invariant structurally true from the scheduler's own configuration, not just an unenforced behavioral convention.
 - **Trade-offs**: The scheduler's startup contract changes (breaking change to its CLI/config surface); existing deployment tooling that passes a full peer list to `xce_scheduler` must be updated (see Requirement 6 / Integration & Migration Notes in design.md).
 - **Follow-up**: Confirm during implementation whether any other in-tree consumer starts `xce_scheduler` with a multi-candidate list for a reason unrelated to this feature (e.g. a test fixture) — such usages need explicit review, not silent breakage.
 ## Risks & Mitigations
 - Risk: A rapid succession of promotions/demotions (chaotic failover) could thrash scheduler process spawn/kill cycles. — Mitigation: the SlowLane dedup key naturally coalesces overlapping start requests for the same controller instance into at most one in-flight action; a demotion always signals-and-reaps before that controller instance could ever legitimately be promoted again (a demoted node must win a fresh election first).
 - Risk: The wedged-scheduler health check (Requirement 5) needs a "making progress" signal, not just an open connection — inventing a bespoke internal probe risks becoming its own source of false positives. — Mitigation: reuse the existing job/queue/config push traffic the controller already sends its scheduler as the progress signal (e.g., confirmation that a recent push was acted on), rather than adding a new synthetic heartbeat payload; finalize the exact signal during implementation against real traffic patterns.
 - Risk: Existing tooling and specs (xce-deploy-ha, xce-test-ha's soak harness, any other spec assuming an independent, reconnecting scheduler) silently drift out of sync with the new model. — Mitigation: explicit File Structure Plan entries in design.md for every piece of tooling identified as depending on today's model (see design.md Integration & Migration Notes).
 ## References
 - `.wolf/buglog.json` bug-1760 — original live repro motivating this spec.
 - `.kiro/specs/raft-drive-async/design.md` — original SlowLane design (cited verbatim in `slowlane.h`'s own header comment).
 - `.kiro/steering/ha-async-io.md` — thread-role contract and never-self-terminate invariant this design must respect.
--- a/.kiro/specs/scheduler-primary-lifecycle/spec.json
+++ b/.kiro/specs/scheduler-primary-lifecycle/spec.json
@ -0,0 +1,26 @@
 {
  "feature_name": "scheduler-primary-lifecycle",
  "created_at": "2026-09-02T04:39:22Z",
  "updated_at": "2026-09-02T07:25:03Z",
  "language": "en",
  "phase": "superseded",
  "superseded_by": "xce-node-manager",
  "superseded_note": "2026-09-02: after further discussion, the user decided the controller should not directly own spawning/killing its own scheduler process. Instead, a new general per-node xce-node-manager component (one per host, role-assigned: manager/worker/apiserver/reverse-proxy) will own starting XCE components for its node's role, including starting/stopping the scheduler on a manager node in reaction to that node's controller reporting itself primary vs. standby over a local heartbeat (Unix domain socket, client logic shared via libxce). This supersedes this spec's core mechanism (controller-internal posix_spawn of its own scheduler via scheduler_proc.c/ha.c) though the underlying requirements this spec captured -- bug-1760's reactive-rejoin deadlock, wedge detection, graceful shutdown, controller-resilience invariants -- carry forward into xce-node-manager's requirements. This spec's approved design.md and its review findings (notably the fork()-vs-posix_spawn mutex-inheritance-deadlock analysis) remain a useful reference and are kept in place, not deleted. Do not implement this spec's tasks.md.",
  "approvals": {
    "requirements": {
      "generated": true,
      "approved": true
    },
    "design": {
      "generated": true,
      "approved": true,
      "validation_note": "kiro-validate-design 2026-09-02: pass 1 found 1 critical issue -- Technology Stack's raw fork()+execve() justification for scheduler_proc.c incorrectly claimed no mutex-inheritance-deadlock risk because the call happens on a dedicated SlowLane worker thread; this doesn't hold, since the hazard comes from OTHER threads in the same multithreaded controller process holding a lock at fork time (glibc malloc, XCE mylock_t), directly re-triggering the risk class product.md's own locked decision (\"no fork() for snapshots\") already rejected. Fixed by switching to posix_spawn() throughout design.md (Technology Stack, both sequence diagrams, File Structure Plan, Components summary, architecture diagram, Error Handling) with the corrected reasoning stated explicitly. Pass 2 (fresh re-read + verified posix_spawn's argv/envp compatibility with execve, and confirmed via grep that the controller has zero existing SIGCHLD/signal handling to conflict with the new reaping logic) found no additional issues. GO."
    },
    "tasks": {
      "generated": true,
      "approved": false,
      "validation_note": "kiro-spec-tasks 2026-09-02: 6 major tasks / 15 sub-tasks (plus 2 collapsed single-item major tasks) drafted covering all 25 requirement IDs (1.1-6.3) and every design.md component (SchedulerProcessManager, SchedulerConnectionBookkeeping, SchedOwnerWatch, the graceful-exit wire message). Independent task-graph sanity review (fresh subagent, pass 1): NEEDS_FIXES -- (1) task 1.2's (P) marker missing a _Boundary:_ tag, (2) task 2.4's _Depends:_ omitted 2.2 despite needing it, (3) Requirement 1.5's heartbeat-driven start-retry had no task wiring an actual periodic call site, (4) the old-rejoin-logic removal task was sequenced in Core, before ha.c wiring and tooling reconciliation, contradicting design.md's own Migration Strategy phase order (removal must be last), (5) design.md's own Testing Strategy lists a dedicated \"job survives scheduler killed mid-decision\" integration test with no owning task, (6) the tooling-reconciliation task covered only the known run-ha-soak.sh case, not design.md's broader \"flag any other dependent spec\" commitment. All 6 fixed: boundary tag added; dependency completed; new task 4.3 combines the heartbeat-driven start-retry and wedge-check wiring; the removal task became standalone Task 6 (last in the file, depending on 3, 4.1, 4.2, 5.7, 5.8, matching the Migration Strategy phases); new task 5.3 added for the job-durability scenario; task 5.8 broadened to sweep other specs/tooling. Sanity review pass 2 (fresh subagent): PASS -- all 6 fixes verified correct, plus fresh independent checks (full requirement-ID coverage, all _Depends:_ references resolve to real tasks including the two collapsed single-item major tasks, no (P)-marker boundary conflicts, Task 6's dependency list matches the Migration Strategy phases). Awaiting user approval before implementation."
    }
  },
  "ready_for_implementation": false
 }
--- a/.kiro/specs/scheduler-primary-lifecycle/tasks.md
+++ b/.kiro/specs/scheduler-primary-lifecycle/tasks.md
@ -0,0 +1,129 @@
 # Implementation Plan
 - [ ] 1. Foundation: wire protocol schema and scheduler startup contract
 - [ ] 1.1 Define the controller-to-scheduler graceful-exit wire message
  - Add a new message the controller can send an already-joined scheduler over their existing connection, carrying no payload beyond identifying it as a graceful-exit request
  - Observable completion: the schema builds cleanly and the new message is visible in the generated bindings, unused by any caller yet
  - _Requirements: 2.1, 2.2_
 - [ ] 1.2 (P) Give the scheduler a single-controller startup mode
  - Replace the scheduler's current multi-candidate `--peers`-style startup configuration with a mode that accepts exactly one controller address
  - Observable completion: the scheduler binary accepts the new single-address startup argument; the existing multi-candidate argument is no longer accepted for this role
  - _Requirements: 3.4_
  - _Boundary: scheduler main.c_
 - [ ] 2. Core: controller-side scheduler process manager
 - [ ] 2.1 Start a scheduler process safely off the Raft drive threads
  - Implement the start action (spawn via `posix_spawn`, passing the controller's own address using task 1.2's new startup mode) submitted through the existing SlowLane executor with a dedup key, never invoked inline from a Raft callback
  - Observable completion: requesting a start when one is already in flight or running is deduped (no second process spawned); a start whose spawn fails is logged and does not crash or block the caller
  - _Requirements: 1.1, 1.2, 1.3, 1.5, 4.1, 4.3_
  - _Boundary: scheduler_proc.c_
  - _Depends: 1.2_
 - [ ] 2.2 Stop a scheduler process gracefully, with a bounded force-terminate fallback
  - Implement the stop action: send the graceful-exit wire message from task 1.1 over the existing connection, wait up to a bounded timeout for the process to exit, then force-terminate it
  - Observable completion: a scheduler that exits on its own within the timeout is not force-terminated; one that does not is force-terminated exactly once after the timeout elapses; the caller never blocks waiting for this to finish
  - _Requirements: 2.1, 2.3, 2.4, 4.2, 4.3_
  - _Boundary: scheduler_proc.c_
  - _Depends: 1.1_
 - [ ] 2.3 (P) Track whether the owned scheduler is making real scheduling progress
  - Extend the controller's existing scheduler join/leave bookkeeping with a last-progress timestamp, updated whenever a push to the scheduler is acknowledged or a dispatch/pend-reason decision is received from it
  - Observable completion: the timestamp advances on real scheduler activity, using only existing traffic — no new synthetic ping is introduced
  - _Requirements: 5.1, 5.2_
  - _Boundary: sched.c_
 - [ ] 2.4 Detect a wedged scheduler and replace it
  - Implement a liveness check that reads task 2.3's progress signal; if the scheduler is connected but has made no progress past a configured threshold, submit a restart action (stop via task 2.2, then start via task 2.1) through the same SlowLane path
  - Observable completion: a connected scheduler with no progress past the threshold is terminated and replaced exactly once per stale period, with an operator-visible log; a scheduler that is merely disconnected (already handled by existing join/leave bookkeeping) is not double-counted as a wedge
  - _Requirements: 5.3, 5.4, 4.1, 4.3_
  - _Boundary: scheduler_proc.c_
  - _Depends: 2.1, 2.2, 2.3_
 - [ ] 3. (P) Core: pin the scheduler to its one controller and watch it
  - Using task 1.2's single-address startup mode, connect only to that one controller; periodically query its reachability and role over that same connection
  - On an unreachable controller, a reported non-primary role, or receipt of task 1.1's graceful-exit message via the existing request handler, exit the process
  - Observable completion: the scheduler process exits within one check interval of its controller becoming unreachable or non-primary, or immediately on receiving the graceful-exit message; it never attempts any address other than the one it started with
  - _Requirements: 2.2, 3.1, 3.2, 3.3, 3.4, 3.5_
  - _Boundary: sched_owner.c, scheduler request handler_
  - _Depends: 1.1, 1.2_
 - [ ] 4. Integration: wire promotion, demotion, and liveness/retry checking into controller HA role transitions
 - [ ] 4.1 Start the scheduler on promotion
  - Call task 2.1's start action at the point a controller completes promotion, after every existing promotion-refusal gate has already passed
  - Observable completion: a real promotion results in exactly one scheduler process being started, joining, and receiving job/queue/config state; a refused promotion attempt (any existing gate) never starts one
  - _Requirements: 1.1, 1.4, 4.4_
  - _Boundary: ha.c_
  - _Depends: 2.1_
 - [ ] 4.2 Stop the scheduler on graceful demotion
  - Call task 2.2's stop action at the point a controller begins a graceful role transition away from primary, without waiting for it to complete
  - Observable completion: a real graceful demotion's own role transition completes without waiting on the scheduler's exit; the scheduler process is gone within the bounded timeout
  - _Requirements: 2.1, 2.3_
  - _Boundary: ha.c_
  - _Depends: 2.2_
 - [ ] 4.3 Wire periodic scheduler presence and liveness checking into the controller's existing cadence
  - While primary with no scheduler currently owned (e.g. a prior start attempt failed), retry task 2.1's start action on the controller's existing heartbeat-driven cadence — the mechanism Requirement 1.5's retry depends on, with no new timer introduced
  - While primary with a scheduler owned, run task 2.4's liveness check on that same cadence
  - Observable completion: a scheduler that failed to start is retried on the next cadence tick without a new timer thread; a wedged scheduler is detected and replaced within one cadence tick of crossing the threshold
  - _Requirements: 1.5, 5.1, 5.2, 5.3, 5.4_
  - _Boundary: ha.c_
  - _Depends: 2.1, 2.4_
 - [ ] 5. Validation: end-to-end coverage and deployment tooling reconciliation
 - [ ] 5.1 Full promotion round trip against a real cluster
  - Exercise a real controller cluster promotion end to end: a scheduler process appears, joins, and receives job/queue/config state with no manual step
  - Observable completion: the round trip produces a working, job-dispatching scheduler in one live test run
  - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5_
  - _Depends: 4.1_
 - [ ] 5.2 Graceful demotion round trip
  - Force a graceful step-down against a real cluster and confirm the prior primary's scheduler exits within the bounded timeout, verified by process absence
  - Observable completion: the demoted controller's own role transition completes promptly and the scheduler process is confirmed gone afterward
  - _Requirements: 2.1, 2.2, 2.3_
  - _Depends: 4.2, 3_
 - [ ] 5.3 A job survives scheduler termination mid-decision
  - Dispatch a job to a running scheduler, kill the scheduler process before its dispatch decision is durably recorded, and confirm the job remains available (not lost, not left inconsistent) for a subsequently-started scheduler to consider
  - Observable completion: the job is present and correctly schedulable after the replacement scheduler starts, with no trace of the killed scheduler's incomplete decision
  - _Requirements: 2.4_
  - _Depends: 2.2_
 - [ ] 5.4 Bug-1760 original repro: hard-killed primary recovers automatically
  - Reproduce this feature's own motivating scenario: hard-kill the primary controller and confirm its orphaned scheduler self-exits, the newly-elected primary starts its own fresh scheduler, and a job submitted after the failover dispatches successfully
  - Observable completion: the exact bug-1760 symptom (a job stuck PEND after a real controller failover) does not reproduce
  - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5_
  - _Depends: 3, 4.1_
 - [ ] 5.5 Wedged scheduler is detected and recovered live
  - Artificially wedge a running scheduler's progress signal while its connection stays open, against a real cluster, and confirm it is detected and replaced
  - Observable completion: the wedged scheduler is terminated and a working replacement is dispatching jobs again within the configured threshold
  - _Requirements: 5.1, 5.2, 5.3, 5.4_
  - _Depends: 4.3_
 - [ ] 5.6 Rapid promotion and demotion does not thrash scheduler processes
  - Drive a rapid sequence of promotions and demotions against a real cluster and confirm no more than one scheduler spawn/kill cycle occurs per genuine promotion
  - Observable completion: no orphaned or duplicate scheduler processes remain after the sequence settles
  - _Requirements: 4.1, 4.2, 4.3, 4.4_
  - _Depends: 5.1, 5.2_
 - [ ] 5.7 Update xce-deploy-ha for controller-owned scheduler startup
  - Update the deployment skill's documentation and generated scripts so they no longer describe or provide a manual, independent scheduler-start step
  - Observable completion: following the updated deployment steps produces a working scheduler with no manual scheduler-start action, matching task 5.1's proof
  - _Requirements: 6.1_
  - _Depends: 5.1, 5.2_
 - [ ] 5.8 Update the HA soak harness and sweep for other dependent tooling
  - Remove the soak harness's own manual scheduler start/restart logic (added as this feature's own motivating bug's workaround) and replace it with a check that a working scheduler is present after each round, with no harness-side management action
  - Also search other specs and test tooling in the repository for the same independently-started, reconnecting-scheduler assumption and flag or reconcile any found, not just the soak harness
  - Observable completion: the soak harness's sequential and chaotic modes pass unmodified in every other respect, with the manual scheduler-management code paths gone; any other dependent spec/tool found is either reconciled or explicitly flagged with a stated reason it is not
  - _Requirements: 6.2_
  - _Depends: 5.4_
 - [ ] 6. Cleanup: remove the scheduler's superseded cross-controller rejoin logic
  - Remove the reactive rejoin mechanism and the general multi-candidate failover-rotation path for the scheduler's own role, now that task 3 fully replaces its purpose and tasks 4-5 have proven and reconciled the new model end to end — matching design.md's own Migration Strategy ordering, where this removal is deliberately the last, hardest-to-reverse phase
  - Observable completion: the removed functions no longer exist; the existing scheduler test suite passes unmodified with no regression; the scheduler's own job/queue/config handling behavior is unchanged
  - _Requirements: 3.4, 6.3_
  - _Boundary: scheduler.c_
  - _Depends: 3, 4.1, 4.2, 5.7, 5.8_