myxce/.kiro/specs/scheduler-primary-lifecycle/design.md

## Overview

**Purpose**: This feature binds the scheduler process's lifecycle to the controller cluster's primary role, replacing the scheduler's independent-process-with-reactive-rejoin model with a controller-owned child process, so that a controller failover always results in a working scheduler within one promotion cycle instead of a potentially unbounded stuck-PEND deadlock (bug-1760).

**Users**: XCE cluster operators, and transitively every job submitted to the cluster, benefit from this feature never having to notice or intervene after a controller failover.

**Impact**: Changes the current independent, long-lived `xce_scheduler` process (joining any of a configured candidate list, reconnecting reactively) into a process spawned, owned, and torn down by whichever controller is currently primary, pinned to that one controller for its entire lifetime.

### Goals
- A controller starts exactly one scheduler process the moment it becomes primary, and that scheduler is usable (joined, receiving state) without any operator action.
- A controller's own scheduler exits promptly on a graceful demotion, and independently detects and exits on any other loss of that controller's primacy (crash, network partition, wedge), without ever attempting to reconnect elsewhere.
- Scheduler-lifecycle management can never compromise the controller's own liveness, HA correctness, or availability.
- A wedged-but-connected scheduler is distinguished from a healthy one and recovered automatically.
- Existing tooling and specs that assume an independently-started, reconnecting scheduler are identified and updated.

### Non-Goals
- No standby/redundant scheduler processes or lease-election mechanism (superseded by piggybacking on the controller's existing Raft election).
- No general per-host role-manager component governing worker/controller/scheduler/apiserver placement.
- No change to worker or apiserver deployment/supervision.
- No change to the scheduler's job/queue/config-handling logic, wire behavior toward jobs, or picker/allocator internals — only its process lifecycle and its relationship to the controller cluster.

## Boundary Commitments

### This Spec Owns
- The controller-side decision of *when* to start, stop, or restart its own scheduler child process (tied to promotion, demotion, and a detected wedge).
- The mechanism used to start/stop/monitor that child process without blocking the controller's Raft drive threads.
- The scheduler's own startup contract change (pinned to one controller, no candidate list) and its active liveness/primacy check of that one controller.
- The new controller-to-scheduler wire message used for a graceful-exit notice.
- Updating existing deployment tooling (`xce-deploy-ha`, `xce-test-ha`) and flagging any other spec that assumes an independently-started, reconnecting scheduler.

### Out of Boundary
- The scheduler's job/queue/config decision logic (picker, allocator, plugins) — untouched.
- The Raft election, replication, and commit-safety mechanisms the controller already implements — this feature is a *consumer* of "who is primary," never a modifier of how that is decided.
- Worker and apiserver process lifecycle.
- Any lease/renewal protocol or standby-scheduler redundancy (deliberately rejected — see research.md).
- `ha_dispatch_gate`'s existing role-based write rejection — this feature depends on it, does not change it.

### Allowed Dependencies
- `src/controller/slowlane.c` (`slowlane_submit`) — the sole mechanism for running scheduler-management work off the Raft drive threads. No new async-execution primitive may be introduced.
- `src/controller/ha.c`'s `raft_cb_become_leader` / `raft_cb_become_follower` — the sole trigger points for scheduler start/stop. No parallel promotion/demotion detection path may be built.
- `src/controller/sched.c`'s existing `on_scheduler_join()` / `handle_scheduler_leave()` / `cluster_t.sched.ep` bookkeeping — extended, not replaced, for tracking whether the owned child has actually joined.
- The existing controller-scheduler wire connection and Cap'n Proto schema (`src/schema/controller.capnp`) — the graceful-exit notice is a small addition to this existing channel, not a new transport.

### Revalidation Triggers
- Any change to `raft_cb_become_leader`'s or `raft_cb_become_follower`'s promotion/demotion gating order (a new gate inserted before or after the scheduler-start/stop call site changes when a scheduler is (not) started).
- Any change to `slowlane_submit`'s dedup-key semantics or admission-cap behavior.
- Any change to `ha_dispatch_gate`'s role-based opcode allow-lists (the safety argument that a stale scheduler connection cannot mutate state depends on this staying true).
- Any other spec or tool that starts `xce_scheduler` directly (bypassing controller-owned startup) must be re-checked against this design's single-controller startup contract.

### Project Invariants Checked
This design touches the controller's HA/process-lifecycle surface, so it is checked against `.kiro/steering/ha-async-io.md`'s documented invariants directly, not deferred to a later audit:
- **Thread-role contract** (no blocking work on iomgr/delaymgr): satisfied — every scheduler-management action (posix_spawn, waitpid, signal, status check) is submitted through the existing `slowlane_submit()` executor, never invoked inline from a Raft callback. See Architecture Pattern & Boundary Map and the SchedulerProcessManager component.
- **Controller-never-self-terminate**: satisfied — a scheduler-management failure (start failure, stop timeout, wedge) is logged and retried/handled locally; it never crashes, hangs, or exits the controller process, and is never wired as a new promotion-refusal gate (Requirement 4.2, 4.4). See Error Handling.
- **`ha_dispatch_gate` role-based write rejection**: depended upon, not modified — this design's safety argument (a stale/orphaned scheduler connection cannot mutate cluster state) rests on this existing, unchanged behavior (see research.md).

## Architecture

### Existing Architecture Analysis
- The controller today has no concept of owning a child process of any kind — `cluster_t` has no process-management state. The only precedent for one XCE daemon supervising a child process is the worker daemon's job-execution supervision (`process_worker_manager.c`, `uv_ipc.c`), which is libuv-based and job-execution-specific (see research.md) and is not reused directly.
- The scheduler today is a fully general Raft-cluster client: it holds a candidate list (`--peers`), connects via `xcecli_start`/`xcecli_join`, and recovers from any disconnect via the client library's own `XCECLI_FAILOVER_CALL` rotation plus its own reactive `sched_publish_check_rejoin()`. This general-client shape is replaced for the scheduler specifically; workers and the `xce_test`/CLI clients are unaffected and keep the existing candidate-list/failover-rotation model.
- `ha_dispatch_gate` already enforces that only a primary controller accepts scheduler write opcodes (job dispatch, pend-reason) — this pre-existing behavior is why this feature is purely about availability, not correctness (see research.md).

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    subgraph HostA
        CtrlA[Controller Instance A Primary]
        SchedA[Scheduler Process]
        CtrlA -->|spawns and owns| SchedA
    end
    subgraph HostB
        CtrlB[Controller Instance B Standby]
    end
    subgraph HostC
        CtrlC[Controller Instance C Standby]
    end
    CtrlA <-->|raft election and replication| CtrlB
    CtrlA <-->|raft election and replication| CtrlC
    CtrlB <-->|raft election and replication| CtrlC
    SchedA -->|pinned single connection join and status query| CtrlA
```

**Architecture Integration**:
- Selected pattern: controller-owned child process, triggered directly from the existing Raft promotion/demotion callbacks, executed through the existing SlowLane off-drive-thread executor.
- Domain/feature boundaries: controller-side process ownership (`scheduler_proc.c`, new) is separate from controller-side scheduler-connection bookkeeping (`sched.c`, existing, extended); scheduler-side controller-liveness watching (`sched_owner.c`, new) is separate from scheduler-side scheduling logic (`scheduler.c`, unchanged except for removing the superseded rejoin logic).
- Existing patterns preserved: SlowLane for off-drive-thread work; `ha_dispatch_gate` as the sole write-safety mechanism; the existing Cap'n Proto wire connection and request-handler dispatch shape on both sides.
- New components rationale: `scheduler_proc.c` is a new domain (process ownership) with no existing owner; `sched_owner.c` is a new domain (single-controller liveness pinning) that cannot be folded into `scheduler.c`'s existing scheduling-engine responsibility without violating single-responsibility.
- Steering compliance: respects `ha-async-io.md`'s thread-role contract (all blocking work — posix_spawn/waitpid/kill/status-query — runs via SlowLane or the scheduler's own non-drive-thread context, never inline on iomgr/delaymgr) and the controller-never-self-terminate invariant (a scheduler-management failure logs loudly and retries; it never causes the controller to crash, hang, or exit, and never becomes a new promotion-refusal gate).

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Controller process management | POSIX `posix_spawn`/`waitpid`/signal (C, `src/controller/scheduler_proc.c`) | Start, monitor, and stop the scheduler child process | `product.md`'s locked decision "no `fork()` for snapshots (mutex-inheritance deadlock in multithreaded controller)" applies just as much here: a raw `fork()` only duplicates the calling thread, so a lock held by any *other* thread (glibc malloc, XCE's own `mylock_t` locks) is duplicated in whatever state it was in, and the child can deadlock if it touches malloc or any XCE library call before `execve()` — being on a dedicated SlowLane worker does not change this, since the hazard comes from *other* threads in the same process, not the calling one. `posix_spawn()` is used instead specifically to avoid re-triggering this exact, already-documented risk class, not merely as a style preference |
| Async execution | `slowlane_submit()` (existing, `src/controller/slowlane.c`) | Runs all scheduler-management actions off the Raft drive threads | Reused as-is, no changes |
| Wire protocol | Cap'n Proto (existing, `src/schema/controller.capnp`) | Carries the new graceful-exit notice from controller to scheduler over the existing connection | One small schema addition; no new transport |
| Scheduler startup config | CLI argument (existing mechanism, new single-address mode) | Controller passes its own address to the scheduler it spawns, replacing the multi-candidate `--peers` list for this specific relationship | Deployment tooling generating scheduler startup commands must change accordingly |

## File Structure Plan

### Directory Structure
```
src/
├── controller/
│   ├── scheduler_proc.c        # NEW: owns posix_spawn/monitor/signal/reap of this controller's scheduler child
│   ├── include/
│   │   └── scheduler_proc.h    # NEW: SchedulerProcessManager service interface
│   ├── ha.c                    # MODIFIED: call scheduler_proc_start() at the tail of raft_cb_become_leader,
│   │                           #   scheduler_proc_stop() at the top of raft_cb_become_follower
│   └── sched.c                 # MODIFIED: extend join/leave bookkeeping to feed liveness/wedge detection
├── scheduler/
│   └── src/
│       ├── sched_owner.c       # NEW: pins to one controller address, periodic liveness/role check, exits on loss
│       ├── scheduler.c         # MODIFIED: remove sched_needs_rejoin/sched_on_reconnect/sched_publish_check_rejoin
│       │                       #   and the multi-candidate startup path for this role
│       ├── handler.c           # MODIFIED: scheduler_req_handler gains a case for the graceful-exit notice
│       └── main.c              # MODIFIED: new single-controller-address startup argument
├── schema/
│   └── controller.capnp        # MODIFIED: add the controller-to-scheduler graceful-exit message
└── controller/testsuite/, scheduler/testsuite/  # new/modified test files, see Testing Strategy
```

### Modified Files (tooling and specs, outside `src/`)
- `.claude/skills/xce-deploy-ha/SKILL.md` — remove/replace Step 4b's manual `start-scheduler.sh` guidance; document that a running controller now starts its own scheduler automatically once it becomes primary.
- `src/tools/deploy-ha.sh` — stop generating a standalone `start-scheduler.sh` intended for manual/independent invocation (or repurpose it to reflect the new single-controller-address contract if a manual escape hatch is still useful for local debugging).
- `.claude/skills/xce-test-ha/run-ha-soak.sh` — remove the worker-rejoin-resync soak coverage's `start_worker_and_scheduler()`/`restart_scheduler_for_new_primary()` manual scheduler management added this session (bug-1760's workaround becomes unnecessary — the controller now manages this itself); the soak harness should instead just verify a scheduler is present after each promotion.

## System Flows

### Promotion: controller starts its scheduler
```mermaid
sequenceDiagram
    participant Raft as Raft Election
    participant HA as ha.c become leader
    participant Lane as SlowLane Executor
    participant Proc as scheduler_proc.c
    participant Sched as Scheduler Process
    Raft->>HA: on_become_leader
    HA->>HA: run existing promotion eligibility gates
    HA->>Lane: slowlane_submit start action dedup key this controller
    Lane->>Proc: run start action on a lane worker
    Proc->>Sched: posix_spawn passing this controller address
    Sched->>HA: join using pinned single controller address
    HA->>Sched: push jobs queues config existing behavior
```
Gating conditions: the SlowLane submission only happens after every existing promotion-refusal gate has already passed (Requirement 1.1). A failed or deferred `slowlane_submit()` (already-in-flight dedup, or admission-cap deferral) is not an error — the work is still queued and will run (Requirement 1.5's retry is satisfied by SlowLane's own queuing semantics plus an explicit retry-on-failure inside the submitted action).

### Graceful demotion: controller signals its scheduler to exit
```mermaid
sequenceDiagram
    participant Raft as Raft Election
    participant HA as ha.c become follower
    participant Lane as SlowLane Executor
    participant Proc as scheduler_proc.c
    participant Sched as Scheduler Process
    HA->>Lane: slowlane_submit stop action dedup key this controller
    HA->>HA: continue completing its own role transition immediately
    Lane->>Proc: run stop action on a lane worker
    Proc->>Sched: send graceful exit notice over existing connection
    Sched->>Sched: finish in flight work and exit
    Proc->>Proc: wait bounded time then reap or force terminate
```
The role transition in `raft_cb_become_follower` never waits on the SlowLane task's completion (Requirement 2.3) — submission is fire-and-forget from the Raft callback's perspective.

### Scheduler self-detects loss of its owning controller
```mermaid
sequenceDiagram
    participant Sched as Scheduler Process
    participant Owner as sched_owner.c
    participant Ctrl as Its Owning Controller
    loop periodic check
        Owner->>Ctrl: status query over the pinned connection
        Ctrl-->>Owner: role and reachability
    end
    alt controller unreachable or reports it is no longer primary
        Owner->>Sched: request exit
        Sched->>Sched: exit
    end
```
This path is independent of the graceful-exit notice above — it is the general-case detector covering crash, network partition, and any demotion trigger the explicit notice did not reach in time (Requirement 3.1-3.3). No candidate list exists to fall back to (Requirement 3.4) — the only outcome of a failed check is exit.

### Wedged-scheduler detection and recovery
```mermaid
sequenceDiagram
    participant HA as ha.c or sched.c
    participant Lane as SlowLane Executor
    participant Proc as scheduler_proc.c
    participant Sched as Scheduler Process
    HA->>Proc: observe scheduler connected but no scheduling progress signal within threshold
    Proc->>Lane: slowlane_submit restart action dedup key this controller
    Lane->>Proc: run restart action
    Proc->>Sched: terminate
    Proc->>Proc: posix_spawn a replacement per the promotion flow
```

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1, 1.2, 1.3 | Scheduler starts on promotion, at most one in flight, no duplicate start | scheduler_proc.c, ha.c | `scheduler_proc_start` | Promotion flow |
| 1.4 | Scheduler joins and receives job/queue/config state after being started | scheduler_proc.c, sched.c | `scheduler_proc_start` | Promotion flow |
| 1.5 | Controller retries on start failure, logs loudly | scheduler_proc.c | `scheduler_proc_start` | Promotion flow |
| 2.1, 2.2, 2.3 | Graceful exit signaled on demotion; scheduler exits within bounded time; controller's own role transition never waits on it | scheduler_proc.c, ha.c, controller.capnp addition, scheduler handler.c | `scheduler_proc_stop`, graceful-exit wire message | Graceful demotion flow |
| 2.4 | No job left inconsistent or lost regardless of how the scheduler was terminated | scheduler_proc.c (bounded stop/force-kill), controller's existing WAL-commit ownership (unchanged) | `scheduler_proc_stop` | Graceful demotion flow, Data Models |
| 3.1, 3.2, 3.3 | Scheduler actively verifies its owning controller is reachable and primary; exits on failure of either check | sched_owner.c | `sched_owner_watch_start` | Self-detection flow |
| 3.4 | Scheduler never joins or operates against any controller other than the one that started it | sched_owner.c, scheduler.c (rejoin logic removed) | `sched_owner_watch_start` | Self-detection flow |
| 3.5 | Abrupt host/process death needs no scheduler-side action | (no component — a structural consequence of co-location, not new logic) | n/a | Self-detection flow (contrast case) |
| 4.1, 4.2, 4.3 | Controller never blocks heartbeats/elections, never crashes/hangs, always logs scheduler-management failures | scheduler_proc.c (via SlowLane) | `scheduler_proc_start`/`stop`/`check_liveness` (all SlowLane-submitted) | All flows |
| 4.4 | Absence of a running scheduler never blocks a controller's own HA role | scheduler_proc.c, ha.c (unchanged promotion gates) | n/a | Promotion flow |
| 5.1, 5.2 | Controller monitors scheduling progress, not just connection state; flags no-progress past a threshold as unresponsive | SchedulerConnectionBookkeeping (sched.c), scheduler_proc.c | `scheduler_proc_check_liveness` | Wedge detection flow |
| 5.3, 5.4 | Unresponsive scheduler is terminated and replaced, with an operator-visible log | scheduler_proc.c | `scheduler_proc_check_liveness` | Wedge detection flow |
| 6.1 | Existing deployment tooling documentation updated to reflect controller-owned scheduler startup | xce-deploy-ha SKILL.md, deploy-ha.sh | n/a (tooling) | Migration Strategy |
| 6.2 | Existing specs/tooling depending on scheduler survive-and-reconnect behavior identified and reconciled | xce-test-ha run-ha-soak.sh (this session's own bug-1760 workaround is the concrete example identified and reconciled here) | n/a (tooling) | Migration Strategy, Testing Strategy E2E |
| 6.3 | Scheduler's observable job/queue/config behavior unchanged while running under a live primary | scheduler.c (unchanged scheduling logic) | n/a | Data Models |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|---------------|--------------------------|-----------|
| SchedulerProcessManager | Controller | Owns posix_spawn/monitor/signal/reap of this controller's scheduler child | 1, 2, 4, 5 | SlowLane executor (P0), ha.c callbacks (P0) | Service |
| SchedulerConnectionBookkeeping | Controller | Existing join/leave tracking, extended to feed liveness/progress signal | 5 | SchedulerProcessManager (P1) | Service |
| SchedOwnerWatch | Scheduler | Pins to one controller address; periodic liveness/role check; exits on loss or graceful notice | 3 | Existing wire connection/status query (P0) | Service |
| Graceful-Exit Wire Message | Wire protocol | Controller-to-scheduler notice carried on the existing connection | 2 | Cap'n Proto schema (P0) | Event |

### Controller

#### SchedulerProcessManager

| Field | Detail |
|-------|--------|
| Intent | Start, monitor, signal, and reap this controller's own scheduler child process, entirely off the Raft drive threads |
| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3, 4.1, 4.2, 4.3, 4.4, 5.1, 5.2, 5.3, 5.4 |

**Responsibilities & Constraints**
- Owns the decision of whether a scheduler child is currently running for this controller instance; `ha.c` never spawns or signals a process directly.
- Every action (start, stop, restart) is submitted to the SlowLane executor with a dedup key scoped to this controller instance, never invoked inline from a Raft callback.
- Never allowed to block, crash, or cause the controller process to exit — every failure path logs an operator-visible error and either retries (start) or proceeds with the role transition regardless (stop).

**Dependencies**
- Outbound: SlowLane executor (`slowlane_submit`) — P0, the only allowed execution path for this component's blocking work.
- Inbound: `ha.c`'s `raft_cb_become_leader` (calls `scheduler_proc_start`), `raft_cb_become_follower` (calls `scheduler_proc_stop`) — P0.
- Inbound: a periodic liveness check (driven from the existing controller heartbeat cadence, not a new timer thread) — P1, feeds `scheduler_proc_check_liveness`.

**Contracts**: Service [x]

##### Service Interface
```c
/* src/controller/include/scheduler_proc.h */

/* Requests that this controller's scheduler child be started, if one is not
 * already running or already starting. Non-blocking: submits to the SlowLane
 * executor and returns immediately. Safe to call multiple times in quick
 * succession — the SlowLane dedup key ensures at most one start action is
 * ever in flight per controller instance.
 * Returns 0 if the request was accepted (started or already in flight/running),
 * a negative errno if the request itself could not even be submitted (e.g.
 * SlowLane shutting down) — never blocks, never crashes the caller.
 */
int scheduler_proc_start(cluster_t *c);

/* Requests that this controller's scheduler child be stopped: sends the
 * graceful-exit notice, waits up to timeout_ms for the child to exit, then
 * force-terminates if it has not. Non-blocking to the caller: submitted to
 * the SlowLane executor. Does not block the caller's own role transition.
 */
int scheduler_proc_stop(cluster_t *c, uint32_t timeout_ms);

/* Evaluates whether the currently-running scheduler child (if any) is making
 * scheduling progress, using the existing push/acknowledgment traffic
 * (SchedulerConnectionBookkeeping) as the progress signal. If the child is
 * connected but has made no progress for longer than the configured
 * threshold, submits a restart action. A no-op if no scheduler child is
 * currently owned. Non-blocking.
 */
int scheduler_proc_check_liveness(cluster_t *c);
```
- Preconditions: `c` is a valid, initialized `cluster_t`; called only from the controller's own process.
- Postconditions: the requested action is either already complete, in flight on a SlowLane worker, or queued — never silently dropped.
- Invariants: at most one scheduler child process is owned by a given controller instance at any time; a stop-then-start (demotion followed by a fresh promotion of the same instance) always completes the stop's reap before a new start is submitted, since a demoted instance must win a fresh election before it can be promoted again.

**Implementation Notes**
- Integration: hook call sites are the tail of `raft_cb_become_leader` (after all existing promotion-refusal gates) and the top of `raft_cb_become_follower`.
- Validation: an operator-visible log line on every start failure, stop-timeout-forced-kill, and liveness-triggered restart (Requirement 4.3, 5.4).
- Risks: sizing the shared SlowLane instance's `max_inflight`/`n_workers` for the additional scheduler-management workload alongside its existing consumers is a follow-up check, not a design change (see research.md).

#### SchedulerConnectionBookkeeping (extends existing `sched.c`)

| Field | Detail |
|-------|--------|
| Intent | Track whether the owned scheduler child has joined and is making scheduling progress, feeding `scheduler_proc_check_liveness` |
| Requirements | 5.1, 5.2 |

**Responsibilities & Constraints**
- Extends the existing `on_scheduler_join()`/`handle_scheduler_leave()`/`cluster_t.sched.ep` bookkeeping with a last-progress timestamp, updated whenever a push to the scheduler is acknowledged or a dispatch/pend-reason decision is received from it.
- Does not itself decide to restart anything — it only exposes the progress signal `scheduler_proc_check_liveness` reads.

**Dependencies**
- Inbound: existing scheduler request handling in `cluster.c`/`sched.c` — P1.
- Outbound: read by SchedulerProcessManager — P1.

**Contracts**: Service [x]

**Implementation Notes**
- Integration: reuses existing traffic; no new synthetic heartbeat payload is introduced (see research.md Risks).

### Scheduler

#### SchedOwnerWatch

| Field | Detail |
|-------|--------|
| Intent | Pin the scheduler to the single controller instance that started it; periodically confirm that controller is reachable and still primary; exit otherwise |
| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5 |

**Responsibilities & Constraints**
- At startup, the scheduler receives exactly one controller address (not a candidate list) and establishes its connection only to that address.
- Periodically issues the existing role-agnostic status query against that one connection; on an unreachable connection or a reported non-primary role, requests the scheduler process to exit.
- Never attempts any other controller address, and never re-derives a candidate list — this component structurally has no such list to fall back to.
- Also receives and acts on the graceful-exit notice (Requirement 2.2) via the existing request-handler dispatch, exiting within a bounded time.

**Dependencies**
- Outbound: the existing controller-scheduler wire connection/status query — P0.
- Inbound: `scheduler_req_handler`'s new graceful-exit case — P0.

**Contracts**: Service [x]

##### Service Interface
```c
/* src/scheduler/include/sched_owner.h */

/* Starts the periodic ownership-watch loop against the single controller
 * address this scheduler process was given at startup. Runs on its own
 * thread (mirroring the worker daemon's existing checking_connection()
 * cadence/shape), independent of the picker/allocator threads.
 */
int sched_owner_watch_start(const char *owner_host, uint16_t owner_port);

/* Called by the request handler when the graceful-exit notice arrives on the
 * existing connection. Triggers the same exit path the periodic watch loop
 * uses on a failed check.
 */
void sched_owner_request_exit(void);
```
- Preconditions: `owner_host`/`owner_port` identify the specific controller instance that spawned this process.
- Postconditions: the process exits once either the periodic check fails or a graceful-exit notice is received — no other outcome resumes normal operation with a different controller.
- Invariants: never issues `xcecli_join`/any wire call against any address other than `owner_host:owner_port`.

**Implementation Notes**
- Integration: replaces `sched_needs_rejoin`/`sched_on_reconnect`/`sched_publish_check_rejoin` in `scheduler.c` entirely — these are removed, not left dormant, since Requirement 3.4 makes their cross-controller-rejoin purpose permanently unreachable.
- Risks: the periodic check interval trades detection latency against status-query load; finalize the exact interval during implementation, following the worker's existing `update_interval`-style convention rather than inventing a new tunable pattern.

### Wire Protocol

#### Graceful-Exit Wire Message

**Event Contract**
- Published by: the controller, from `scheduler_proc_stop`'s SlowLane action, over the existing controller-scheduler connection.
- Subscribed by: the scheduler's existing `scheduler_req_handler` (`src/scheduler/src/handler.c`), which invokes `sched_owner_request_exit()`.
- Delivery: best-effort over the existing persistent connection — if delivery fails (connection already gone), `scheduler_proc_stop`'s bounded wait-then-force-terminate path (Requirement 2.3) still guarantees the child does not outlive the timeout, and `SchedOwnerWatch`'s independent periodic check (Requirement 3) is the backstop if the notice is lost.
- Ordering/idempotency: at most one graceful-exit notice is ever sent per stop action (SlowLane dedup); receiving it more than once (e.g. a retry) is a no-op if the process is already exiting.

## Data Models

No new persisted entity is introduced. The scheduler remains fully stateless with respect to durable storage — its in-memory job/queue/config state is always rebuilt from the controller's existing push traffic on join, unchanged by this feature (Requirement 6.3). The only new in-memory state is:
- Controller side: which scheduler child (if any) this controller instance currently owns, its process handle, and its last-progress timestamp — process-lifetime only, never persisted, rebuilt from scratch on controller restart (a restarted controller that is primary starts a fresh scheduler exactly as any other promotion would).
- Scheduler side: the single owning-controller address it was started with — process-lifetime only.

Since nothing here is written to or read from a durable log, snapshot, or checkpoint, the project's save/restore field-completeness review does not apply to this design.

## Error Handling

### Error Strategy
Every failure in scheduler-lifecycle management follows the project's existing fail-loud-stay-alive philosophy (`ha-async-io.md`, `controller/CLAUDE.md`): log an operator-visible error, never crash or hang the controller, and retry where retrying is meaningful.

### Error Categories and Responses
- **Scheduler fails to start** (posix_spawn failure, immediate crash): `scheduler_proc_start` logs an error and retries on the controller's existing heartbeat-driven cadence while it remains primary (Requirement 1.5); the controller itself continues serving every other role unaffected (Requirement 4.4).
- **Scheduler does not exit within the graceful-exit timeout**: `scheduler_proc_stop` force-terminates the child after the bounded wait; the controller's own demotion completes regardless (Requirement 2.3).
- **Scheduler connected but not progressing (wedged)**: treated as unresponsive once past the threshold, terminated and replaced (Requirement 5.3), with an operator-visible log (Requirement 5.4).
- **Scheduler cannot reach or confirm its owning controller**: the scheduler itself exits (Requirement 3.2, 3.3) — no error is reported back to the controller for this case, since the controller side detects the resulting disconnect via existing `handle_scheduler_leave()` bookkeeping.

### Monitoring
All new log lines (start failure, forced-kill-on-timeout, liveness-triggered restart) use the existing controller logging conventions (`WARN`/`ERROR` macros already used throughout `ha.c`) — no new logging mechanism is introduced.

## Testing Strategy

### Unit Tests
- `scheduler_proc_start` submits exactly one SlowLane action per promotion and a second concurrent call while one is in flight is deduped (not double-submitted) — directly exercises Requirement 1.2/1.3 via the existing `test_slowlane.c` dedup behavior.
- `raft_cb_become_leader` calls `scheduler_proc_start` only after every existing promotion-refusal gate returns eligible, and does not call it on any refusal path — regression-guards Requirement 1.1 against a future gate reordering.
- `raft_cb_become_follower` calls `scheduler_proc_stop` and completes its own role transition without waiting on that call's result — regression-guards Requirement 2.3.
- `SchedOwnerWatch`'s periodic check exits the process on a simulated unreachable controller and on a simulated non-primary role response, and never attempts a second address — directly exercises Requirement 3.1-3.4.
- `scheduler_proc_check_liveness` triggers a restart only when the last-progress timestamp exceeds the threshold while the connection remains open (not on a disconnect, which is a separate, already-detected case) — exercises Requirement 5.1/5.2.

### Integration Tests
- A real 3-node controller cluster promotion results in a scheduler process appearing, joining, and receiving job/queue/config state, with no manual step — exercises Requirement 1.1-1.4 end to end.
- A graceful step-down (forced via the existing test-only demotion trigger) results in the prior primary's scheduler exiting within the bounded timeout, verified by process absence, not just a log line — exercises Requirement 2.1/2.2.
- A hard-killed primary (matching this session's original bug-1760 repro exactly) results in the orphaned scheduler self-exiting within its own periodic-check interval, and the newly-elected primary's fresh scheduler successfully dispatching a job submitted after the failover — exercises Requirement 3.1-3.5 and directly closes bug-1760.
- A scheduler artificially wedged (e.g. its progress-signal thread blocked) while its connection stays open is detected and replaced — exercises Requirement 5.1-5.3.
- A job dispatched by a scheduler that is then killed mid-decision (before its dispatch is durably recorded) remains available for the replacement scheduler to consider — exercises Requirement 2.4.

### E2E / Soak Tests
- Extend `.claude/skills/xce-test-ha/run-ha-soak.sh`'s sequential and chaotic modes (already exercising controller failover) to assert a working scheduler is present after every round without any harness-side scheduler start/stop action — this replaces the manual `start_worker_and_scheduler()`/`restart_scheduler_for_new_primary()` logic added this session as bug-1760's workaround, which becomes dead code once this feature ships.
- A rapid sequence of promotions/demotions (chaotic mode's overlapping-kill pattern) produces no scheduler spawn/kill thrash beyond one scheduler per genuinely-promoted instance — exercises the SlowLane-dedup mitigation documented in research.md's Risks.

## Migration Strategy

```mermaid
flowchart TB
    Phase1[Phase 1 controller and scheduler code changes land behind no behavior change until wired]
    Phase2[Phase 2 wire promotion and demotion hooks and pinned single controller startup]
    Phase3[Phase 3 update xce deploy ha and xce test ha tooling to stop manually starting the scheduler]
    Phase4[Phase 4 remove superseded rejoin logic and multi candidate scheduler startup path]
    Phase1 --> Phase2 --> Phase3 --> Phase4
```
- Rollback trigger: if a real deployment depends on independently placing the scheduler on a dedicated host for resource-isolation reasons not previously surfaced, that is a genuine requirements gap discovered late — the fix is to pause before Phase 4 (superseded-logic removal) rather than force the migration through, since Phase 4 is the only phase that is hard to reverse cheaply.
- Validation checkpoint: after Phase 2, run the soak-harness scenario in Testing Strategy's E2E section before proceeding to Phase 3/4, since it is the closest thing to a direct regression test for bug-1760 itself.

## Open Questions / Risks
- The exact wire opcode/schema shape for the graceful-exit notice is left to implementation (a small, additive `controller.capnp` change, following the same pattern recent specs have used for new controller-to-daemon messages) — this is appropriately an implementation detail, not a design decision, since no existing opcode fits (confirmed by inspection: `OPCode_shutdown` is an inbound admin-to-controller command, not a controller-to-scheduler one, and is not reusable here).
- The precise "no scheduling progress" threshold and periodic-check interval (Requirement 5, Requirement 3) are implementation-time tuning decisions, not design decisions — see research.md Risks for the reasoning against inventing a bespoke synthetic heartbeat.
- Confirm during implementation whether any other in-tree consumer starts `xce_scheduler` with a multi-candidate `--peers` list for a reason unrelated to this feature (e.g. a test fixture) that would need explicit review rather than silent breakage (carried over from research.md).