myxce/.kiro/specs/scheduler-primary-lifecycle/research.md

## Summary
- **Feature**: `scheduler-primary-lifecycle`
- **Discovery Scope**: Extension (existing controller + scheduler components; no new daemon type, no new external dependency)
- **Key Findings**:
  - `slowlane_submit()` (`src/controller/slowlane.c`) already exists, is tested, and is the exact primitive this feature needs: a bounded, admission-capped executor with per-key in-flight dedup that runs submitted work off the Raft drive threads (iomgr/delaymgr). No new async-execution mechanism needs to be built.
  - `XCE_CONTROLLER_HOOK_SCHEDULER` / `HOOK_DETAIL_SCHEDULER_JOIN` / `HOOK_DETAIL_SCHEDULER_DEAD` (`src/include/xce/cluster_def.h`) are declared but never fired or consumed anywhere in `src/controller/*.c` — confirmed by exhaustive grep. They are dead enum values, not existing plumbing to build on.
  - The controller already tracks scheduler connection lifecycle via `on_scheduler_join()` / `handle_scheduler_leave()` (`src/controller/sched.c`) and a single `cluster_t.sched.ep` slot — this is reusable bookkeeping for "is my child scheduler currently joined," but it has no concept today of *owning* the scheduler process itself.
  - The scheduler's current rejoin logic (`sched_needs_rejoin`/`sched_on_reconnect`/`sched_publish_check_rejoin`, `src/scheduler/src/scheduler.c`) and its use of the general-purpose, candidate-rotating client library (`xcecli_join`, `XCECLI_FAILOVER_CALL`) both become dead weight under the new model and should be removed, not left dormant — the requirements (3.4) explicitly forbid the scheduler from ever trying another controller.
  - `ha_dispatch_gate()` (`src/controller/ha.c:4573`) already rejects scheduler write opcodes against a non-primary controller (`CTRL_STANDBY`'s allow-list excludes job dispatch / pend-reason). Confirmed by reading the gate's full switch statement. This means the safety argument for this feature is already true today — the feature is solely about *availability*, not correctness.

## Research Log

### Where does controller promotion actually complete?
- **Context**: Need the concrete integration point for "start a scheduler on promotion."
- **Sources Consulted**: `src/controller/ha.c`, `raft_cb_become_leader` (line 1615), read via codegraph.
- **Findings**: `raft_cb_become_leader` is long and gated: election-timeout cancellation, safe-committed-LSN computation, first-boot `cluster_init_phase2`, WAL truncation-floor handling, then a sequence of promotion-refusal gates (recovery-truncated, apply-divergent, self-restart-pending, apply-resource-fenced, config-serve-fenced) that each `return` without completing promotion, staying alive per the project's never-self-terminate invariant. Only after every gate passes does the function proceed to authoritative promotion (role flip to `CTRL_PRIMARY`, `ha_start_primary_heartbeat`, `repl_add_standby` for reachable voters).
- **Implications**: The scheduler-start action must be triggered only at the tail of this function, after all refusal gates have passed — never speculatively earlier. A promotion attempt that gets refused by any gate must not start a scheduler.

### Where does demotion happen?
- **Context**: Need the integration point for "signal my scheduler to exit."
- **Sources Consulted**: `src/controller/ha.c`, `raft_cb_become_follower` (line 2120), read via codegraph.
- **Findings**: `raft_cb_become_follower` is the Raft library's `on_become_follower` callback, invoked from multiple paths (losing an election, observing a higher term, an append-entries request from a legitimate newer leader). This is the single funnel for every graceful role-loss case named in Requirement 2.1 (voluntary step-down and higher-term discovery both route through it).
- **Implications**: One hook site covers all of Requirement 2's graceful-demotion triggers; no separate handling needed per trigger type.

### Is there an existing async-safe way to do blocking work from the HA path?
- **Context**: `.kiro/steering/ha-async-io.md`'s thread-role contract forbids blocking work on iomgr/delaymgr, and spawning/signaling/monitoring a child process is unavoidably blocking-ish (fork/exec, waitpid, kill).
- **Sources Consulted**: `src/controller/slowlane.c`, `src/controller/include/slowlane.h`, existing callers in `ha_convergence.c`, `config_repl.c`, `persist.c`.
- **Findings**: `slowlane_submit(sl, key, fn, arg)` already exists for exactly this purpose ("long blocking I/O off the Raft drive threads"), is non-blocking to the caller, runs `fn` on a dedicated worker thread with no lock held, and provides per-key in-flight dedup (`-EEXIST` on a duplicate submit of the same key while one is still running/queued). A `test_slowlane.c` suite already covers non-blocking submission, dedup, and admission-cap behavior.
- **Implications**: Build vs. Adopt — **adopt**. Scheduler start/stop/monitor actions are ordinary `slowlane_submit()` work items. The dedup key (scoped to "this controller's scheduler-management action") mechanically satisfies Requirement 1.2/1.3 ("at most one Scheduler Process," "shall not start a second") as a side effect of reusing an already-tested primitive, with no new synchronization to design or verify.

### Is there an existing pattern for one XCE daemon supervising a child process?
- **Context**: Wanted to know if `process_worker_manager.c`/`uv_ipc.c` (the worker daemon's job-process supervision) is directly reusable.
- **Sources Consulted**: `src/worker/src/process_worker_manager.c`, `src/worker/src/uv_ipc.c` (libuv-based named-pipe IPC with a heartbeat callback between the worker daemon and its forked job-executor children).
- **Findings**: The pattern (fork a child, communicate over a local IPC channel, monitor a heartbeat) is directly analogous to what this feature needs between a controller and its scheduler child. The concrete code, however, is tightly coupled to job-execution concerns (cgroups, dbus, job exit-status plumbing) and to libuv, a dependency the controller does not currently carry.
- **Implications**: Build vs. Adopt — **adopt the pattern, not the code**. The controller gets its own small, purpose-built scheduler-process-manager component (fork/exec + reap + a liveness signal), following the same shape (parent owns child lifecycle, detects child exit, watches liveness) without pulling in libuv or the worker's job-execution-specific machinery. Simplification lens: a shared abstraction with the worker's job-process manager was considered and rejected — the two "child processes" (a job executable vs. a companion XCE daemon) differ enough (job exit codes/signals/cgroups vs. a single well-known daemon binary with its own wire protocol) that forcing a shared interface would add an abstraction layer with only one real implementation on each side.

### How does the scheduler detect its own controller is dead or no longer primary?
- **Context**: Requirement 3 needs an active detection mechanism, not just reliance on OS-level parent-death propagation (discussed and explicitly not relied upon as the primary mechanism, since it doesn't cover "controller process dies while the host lives").
- **Sources Consulted**: `src/lib/client.c` (`xcecli_join`, `XCECLI_FAILOVER_CALL`), existing `ctrlstatus`/`OPCode_ctrlstatus` read-only status query (confirmed answerable in every controller role per `ha_dispatch_gate`), `src/controller/sched.c`.
- **Findings**: The scheduler already opens a persistent, Cap'n-Proto-framed TCP connection to a controller and already has a role-agnostic status query (`ctrlstatus`) available in every role. Connection loss (RDHUP) is already detected by the existing transport layer everywhere else in this codebase (workers, clients) — nothing new needs to be built to notice "my controller is unreachable." What is missing is (a) pinning the scheduler to exactly one controller address with no candidate list/rotation, and (b) a periodic check of that one controller's reported role, so a controller that demotes without the connection ever dropping (the common case — TCP stays up across a graceful step-down) is still detected.
- **Implications**: Build vs. Adopt — adopt the existing wire protocol and status query; build only the pin-to-one-controller startup mode and the periodic self-check loop (a much smaller version of the worker's `checking_connection()` shape, minus any rejoin/rotation behavior). The explicit graceful-exit signal (Requirement 2) rides the same existing connection as a new, small message case in the scheduler's existing request handler (`scheduler_req_handler`, `src/scheduler/src/handler.c`) rather than a separate channel.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Controller-owned child process (selected) | Primary controller forks/execs a scheduler bound to itself; scheduler self-exits on losing that specific controller | Reuses controller's already-hardened Raft election for "who's authoritative"; host death handled for free; no new election/lease protocol | Controller gains a new, bounded process-supervision responsibility; scheduler loses independent host placement | Matches decision reached in requirements discussion |
| Independent scheduler + periodic reconnect (rejected) | Keep scheduler as today, add an unconditional periodic `xcecli_join()` retry (worker-style `checking_connection()`) | Smaller code change; preserves independent deployment | Does not solve scheduler-host-death or wedged-scheduler detection; still needs a second mechanism for those | Considered and superseded once host-death handling became an explicit requirement |
| Standby schedulers + controller-granted lease (rejected) | Multiple scheduler processes compete for a renewable lease from the primary controller | Handles wedge/host-death without host co-location; scheduler keeps independent placement | A second, parallel authority/liveness protocol to design, build, and prove correct, duplicating the controller's own Raft election in spirit | Rejected in favor of piggybacking on existing Raft election (simplification lens) |
| General per-host role-manager agent (deferred) | A new component per host starts/stops worker/controller/scheduler/apiserver by declared role and cluster state | Generalizes beyond just scheduler | New always-on process whose own supervision is unsolved; would need to watch controller election state itself, adding latency and an indirection layer | Explicitly deferred in requirements as a separate, independently-motivated initiative, not scope for this spec |

## Design Decisions

### Decision: Use `slowlane_submit()` for all scheduler process management
- **Context**: Starting, monitoring, and signaling a child process from `raft_cb_become_leader`/`raft_cb_become_follower` must not block the Raft drive threads.
- **Alternatives Considered**:
  1. Spawn a brand-new dedicated pthread per promotion — works, but reinvents admission control and dedup that already exist.
  2. Do the fork/exec inline in the Raft callback — violates the thread-role contract outright (fork/exec/waitpid are blocking).
- **Selected Approach**: Submit scheduler-management actions (start on promotion, signal-and-reap on demotion, restart on detected wedge) to the existing SlowLane executor with a per-controller-instance dedup key.
- **Rationale**: Already built, already tested, already the established pattern for exactly this class of problem in this codebase (`raft-drive-async` spec).
- **Trade-offs**: None material — this is a pure reuse with no new abstraction.
- **Follow-up**: Confirm the SlowLane instance's `max_inflight`/`n_workers` sizing (set at `slowlane_init()` call sites in `cluster.c`) has headroom for scheduler-management tasks alongside its existing consumers (peer connect, snapshot transfer, config-push); this is a sizing check, not a new design.

### Decision: Scheduler pins to exactly one controller address at startup, no candidate list
- **Context**: Requirement 3.4 forbids the scheduler from ever joining or continuing to operate against any controller other than the one that started it.
- **Alternatives Considered**:
  1. Keep the existing `--peers` full-candidate-list startup mode and just suppress rotation in the failover-call path — leaves a footgun where the shape of the config still implies "any of these are fine."
  2. Give the scheduler a single-controller startup mode entirely, with no candidate list construct at all.
- **Selected Approach**: Option 2. The controller passes its own address (not the full peer list) when it starts its scheduler.
- **Rationale**: Makes the "one controller owns one scheduler" invariant structurally true from the scheduler's own configuration, not just an unenforced behavioral convention.
- **Trade-offs**: The scheduler's startup contract changes (breaking change to its CLI/config surface); existing deployment tooling that passes a full peer list to `xce_scheduler` must be updated (see Requirement 6 / Integration & Migration Notes in design.md).
- **Follow-up**: Confirm during implementation whether any other in-tree consumer starts `xce_scheduler` with a multi-candidate list for a reason unrelated to this feature (e.g. a test fixture) — such usages need explicit review, not silent breakage.

## Risks & Mitigations
- Risk: A rapid succession of promotions/demotions (chaotic failover) could thrash scheduler process spawn/kill cycles. — Mitigation: the SlowLane dedup key naturally coalesces overlapping start requests for the same controller instance into at most one in-flight action; a demotion always signals-and-reaps before that controller instance could ever legitimately be promoted again (a demoted node must win a fresh election first).
- Risk: The wedged-scheduler health check (Requirement 5) needs a "making progress" signal, not just an open connection — inventing a bespoke internal probe risks becoming its own source of false positives. — Mitigation: reuse the existing job/queue/config push traffic the controller already sends its scheduler as the progress signal (e.g., confirmation that a recent push was acted on), rather than adding a new synthetic heartbeat payload; finalize the exact signal during implementation against real traffic patterns.
- Risk: Existing tooling and specs (xce-deploy-ha, xce-test-ha's soak harness, any other spec assuming an independent, reconnecting scheduler) silently drift out of sync with the new model. — Mitigation: explicit File Structure Plan entries in design.md for every piece of tooling identified as depending on today's model (see design.md Integration & Migration Notes).

## References
- `.wolf/buglog.json` bug-1760 — original live repro motivating this spec.
- `.kiro/specs/raft-drive-async/design.md` — original SlowLane design (cited verbatim in `slowlane.h`'s own header comment).
- `.kiro/steering/ha-async-io.md` — thread-role contract and never-self-terminate invariant this design must respect.