docs(xce-node-manager): add spec (requirements/design approved)

Successor to scheduler-primary-lifecycle: a new per-node xce-node-manager component (one per host, role-assigned: manager/worker/apiserver/ reverse-proxy) owns starting/stopping XCE components for its node's role -- including starting/stopping the scheduler on a manager node in reaction to that node's controller reporting itself primary vs. standby over a local heartbeat (Unix domain socket, client logic shared via libxce) -- rather than the controller directly spawning/killing its own scheduler process. Carries forward scheduler-primary-lifecycle's underlying requirements (bug-1760's reactive-rejoin deadlock, wedge detection, graceful shutdown, controller-resilience invariants). Design review (kiro-validate-design) fixed 3 issues before GO: a uniform-single-OS-user assumption that was factually wrong given xce-deploy-ha's real worker daemon runs as root (resolved: root only on worker-role nodes); Requirement 4.3's wedge-detection mechanism verified directly against the actual scheduler code (sched_job_picker's cbuf_get() does an unbounded wait when idle, confirming the naive design would false-positive -- fixed by swapping to the existing cbuf_timedget() bounded-deadline variant); and PortCheckMonitor's TCP connect made explicitly non-blocking to match HeartbeatListener's own discipline. Tasks not yet generated. Checkpointing uncommitted spec work found in this shared worktree. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-03 09:13:38 +00:00 · 2026-09-03 09:13:38 +00:00 · 177e9b292a
commit 177e9b292a
parent 6bb37cf470
4 changed files with 693 additions and 0 deletions
--- a/.kiro/specs/xce-node-manager/design.md
+++ b/.kiro/specs/xce-node-manager/design.md
@ -0,0 +1,488 @@
+## Overview
+
+**Purpose**: This feature introduces `xce-node-manager`, a per-node daemon that starts and supervises the XCE component(s) appropriate to a node's assigned role — replacing today's model where each of the Controller, Scheduler, Worker, and Apiserver is started and supervised as an independent, manually-managed process. On a manager-role node it additionally binds the Scheduler's own lifetime to its co-located Controller's primary/standby status, closing bug-1760 (a hard-killed primary's Scheduler never rejoined the newly-elected primary, leaving jobs stuck PEND indefinitely).
+
+**Users**: XCE cluster operators, and transitively every job submitted to the cluster, benefit from never having to manually start a component after a deploy or notice/intervene after a Controller failover or a component crash.
+
+**Impact**: Introduces one new top-level daemon (`xce_nodemgr`) and one new local, same-host wire protocol (a heartbeat/liveness report) consumed by the Controller, Scheduler, Worker, and Apiserver. Supersedes `.kiro/specs/scheduler-primary-lifecycle/`'s mechanism (a Controller directly `posix_spawn`-ing and killing its own Scheduler child) — that responsibility moves to `xce-node-manager`, driven by the Controller's own reported primary/standby state rather than by the Controller managing a child process itself.
+
+### Goals
+- A node's assigned role(s) determine which XCE component(s) start on it, with no manual per-component start step.
+- A component that should always be running (Controller, Worker, Apiserver, Reverse Proxy) is restarted automatically if it exits unexpectedly.
+- The Scheduler on a manager node starts the moment its co-located Controller becomes primary and stops promptly when that Controller stops being primary or becomes unreachable — with no cross-Controller reconnect logic in the Scheduler.
+- A component that is alive but not doing useful work (a wedged Scheduler round loop, in particular) is distinguished from a healthy one and replaced.
+- `xce-node-manager`'s own presence, absence, or failure can never compromise a Controller's own availability or HA correctness.
+- Existing deployment tooling and specs that assume independently-started components are identified and reconciled.
+
+### Non-Goals
+- Supervising `xce-node-manager` itself (delegated to the host's existing process-supervision facilities).
+- Any change to the Reverse Proxy's own configuration, routing rules, or request handling.
+- Any change to how a Worker's or Apiserver's own client-side connection discovers/reconnects to whichever Controller is currently primary (owned by the existing/adjacent client-failover mechanism).
+- Any change to RBAC, authentication, or job/queue/scheduling decision logic.
+- Standby/redundant Scheduler processes or a lease-election mechanism for Schedulers (superseded design's rejection still holds — piggybacking on the Controller's own already-hardened Raft election remains the model, just triggered one layer up).
+- A new operator-facing remote management API for `xce-node-manager` beyond existing logging conventions.
+- The Scheduler's simulation mode (`sim.c`'s driver, distinct from the live driver's `xcecli`/Controller connection) — a simulation-mode Scheduler is a standalone, manually-invoked tool with no real Controller or primary/standby concept, and continues to run entirely independently of `xce-node-manager`, exactly as it does today.
+
+## Boundary Commitments
+
+### This Spec Owns
+- Reading a node's local role assignment and starting the corresponding XCE component(s).
+- Continuous supervision (unexpected-exit detection, automatic restart) of every component it starts.
+- The decision of *when* the Scheduler on a manager node should be running, derived from its co-located Controller's self-reported primary/standby status.
+- The heartbeat/liveness wire format and the `libxce` client primitive components use to report it.
+- Distinguishing "alive" from "making real progress" for components where that distinction matters (the Scheduler).
+- A distinct liveness mechanism (network port reachability) for the off-the-shelf Reverse Proxy role.
+- Recognizing already-running components on its own restart instead of double-spawning them.
+- Updating existing deployment tooling (`xce-deploy-ha`, `xce-test-ha`) and the AppImage entry point, and flagging any other spec/tool that assumes an independently-started component.
+
+### Out of Boundary
+- The Controller's Raft election, replication, and commit-safety mechanisms — this feature only reads the Controller's already-decided primary/standby role, never influences how that role is decided.
+- `ha_dispatch_gate`'s existing role-based write rejection — depended upon (unchanged), not modified.
+- The Scheduler's own job/queue/config decision logic (picker, allocator, plugins) — untouched.
+- Worker and Apiserver job-execution/API-serving logic — untouched.
+- Any lease/renewal protocol or standby-component redundancy.
+- Supervising `xce-node-manager`'s own process availability.
+- The Scheduler's simulation mode — untouched, unmanaged, and unaware of `xce-node-manager`'s existence; only the live driver (the one that joins a real Controller cluster) is in scope.
+
+### Allowed Dependencies
+- `posix_spawn`/`waitpid`/signal (POSIX) — the sole mechanism `xce-node-manager` uses to start/monitor/stop a managed child process, matching the already-reviewed `scheduler-primary-lifecycle` decision (never raw `fork()` — see Technology Stack).
+- Unix domain datagram sockets, local to the host — the sole transport for the heartbeat protocol; no cross-host traffic is introduced by this feature.
+- The existing controller-scheduler wire connection (unchanged) — the Scheduler's own backstop controller-liveness check (Requirement 6) reuses it exactly as the superseded design specified; this feature adds no new field to that connection's schema.
+- `libxce` (`src/lib/`) — the heartbeat client primitive is added here so the Controller, Scheduler, and Worker (all of which already link it) get it for free.
+
+### Revalidation Triggers
+- Any change to how the Controller exposes its own primary/standby role (today the atomic `ha->role` field) — the heartbeat report's Controller-side source of truth.
+- Any change to the Scheduler's round/picker loop structure that removes or alters its bounded-cadence-even-when-idle property — the property Requirement 4.3's progress signal depends on.
+- Any other spec or tool that starts the Controller, Scheduler, Worker, or Apiserver directly (bypassing `xce-node-manager`) must be re-checked against this design's role-driven startup contract.
+- Introduction of a genuine cross-host or cross-version consumer of the heartbeat protocol would invalidate the "same-host, same-build-version" reasoning behind skipping Cap'n Proto for it (see Technology Stack).
+
+### Project Invariants Checked
+This design adds a small periodic reporting point to the Controller and touches the Scheduler's process-lifecycle surface, so it is checked directly against `ha-async-io.md` and `product.md`'s locked decisions:
+- **Thread-role contract** (no blocking work on iomgr/delaymgr): satisfied — the Controller's heartbeat report is a single non-blocking, connectionless (`SOCK_DGRAM`) local send with no retry/backoff, issued from the same periodic point on every tick; a full/unreachable receiver drops the datagram rather than blocking the sender (see Error Handling). The equivalent hazard on the Node Manager's own side — its single management loop must not block on `HeartbeatListener`'s socket reads or `PortCheckMonitor`'s TCP connects, since either would stall supervision of every role on that node, not just the one being checked — is satisfied the same way: non-blocking socket handling throughout (see PortCheckMonitor's Implementation Note).
+- **Controller-never-self-terminate**: satisfied — a heartbeat-send failure is not logged as an error path that can escalate to a crash/hang/exit; it is a silent, expected condition when no `xce-node-manager` is present (Requirement 5.2/5.3).
+- **No `fork()` for snapshots" (generalized precedent)**: `xce-node-manager` spawns every managed component via `posix_spawn`, never raw `fork()+exec()` — the same reasoning `scheduler-primary-lifecycle`'s design review already established (a lock held by another thread at fork time can deadlock a child that touches malloc/libxce before `execve()`) applies identically here, and is called out explicitly because `src/worker/src/process_worker_manager.c`'s existing `process_manager_start_worker()` is a real, already-shipped counter-example using raw `fork()+execl()` — that pattern is not to be imitated.
+- **`ha_dispatch_gate` role-based write rejection**: depended upon, unchanged — this design does not touch it.
+- **Scheduler core/driver separation** (`scheduler-core` spec's established invariant: `xce_sched_core_init()`'s core — `core.c`/`core_events.c`/`core_session.c`/`core_clock.c` — performs zero controller I/O): satisfied — the heartbeat call site is `sched_job_picker` in `scheduler.c`, the **live** driver layer that already owns the picker/allocator thread lifecycle and the `xcecli` connection, exactly where the `cbuf_timedget()` change also lands. The pure decision engine in `core.c` and siblings is untouched by this feature; the Scheduler's outside-facing I/O surface (Controller connection, and now the local heartbeat) remains confined to the live driver, matching the existing architecture rather than introducing a second I/O path into the core. The simulation driver (`sim.c`) is a distinct code path from `scheduler.c`'s live driver and is not touched — a simulation-mode Scheduler has no heartbeat call site added and is never started, stopped, or monitored by `xce-node-manager` (see Non-Goals).
+
+## Architecture
+
+### Existing Architecture Analysis
+- No component in this codebase today starts or supervises another XCE daemon based on a declared role. `tools/appimage/entry.sh` is the closest thing to an orchestrator, and it only execs one named binary in the foreground per invocation (`xce controller`, `xce worker`, ...) — no supervision, no multi-process coordination, and no case for the already-bundled `haproxy` at all.
+- The only existing precedent for one XCE daemon supervising a child process is the Worker's job-execution supervision (`process_manager.c`, spawning per-job `xcecd_worker` processes via raw `fork()+execl()`) — job-execution-specific, and using a spawn mechanism this design deliberately does not reuse (see Project Invariants Checked).
+- The Controller already exposes its own primary/standby state as an atomic field (`ha->role`) and already runs a non-blocking, delaymgr-driven periodic task pattern (`primary_heartbeat_fn`) proving this class of "cheap periodic local report" is safe and idiomatic here — though that specific function is unsuitable to extend directly (see `research.md`).
+- The Scheduler's `sched_job_picker` thread (`scheduler.c`) calls `cbuf_get()` (`src/lib/mylock.c:144`) to pull the next job, which does an **unbounded** `pthread_cond_wait` when the queue is empty — confirmed by reading the code, not assumed. A heartbeat call placed naively at the top of this loop would never fire during a genuinely idle cluster, since the thread simply never wakes. `cbuf_timedget()` (same file), an already-existing bounded-deadline variant used elsewhere in the codebase, is the fix this design adopts (see Scheduler integration in File Structure Plan).
+- `ha_dispatch_gate` already enforces that only a primary Controller accepts Scheduler write opcodes — unchanged, and still the reason this feature is purely about availability, not correctness.
+
+### Architecture Pattern & Boundary Map
+
+```mermaid
+graph TB
+    subgraph ManagerNode
+        NM1[Node Manager]
+        Ctrl[Controller]
+        Sched[Scheduler Process]
+        NM1 -->|starts and supervises| Ctrl
+        NM1 -->|starts stops on Controller role| Sched
+        Ctrl -.->|heartbeat role status| NM1
+        Sched -.->|heartbeat round progress| NM1
+        Sched -->|pinned backstop liveness check| Ctrl
+    end
+    subgraph WorkerNode
+        NM2[Node Manager]
+        Wrk[Worker Process]
+        NM2 -->|starts and supervises| Wrk
+        Wrk -.->|heartbeat| NM2
+    end
+    subgraph ApiserverNode
+        NM3[Node Manager]
+        Api[Apiserver Process]
+        NM3 -->|starts and supervises| Api
+        Api -.->|heartbeat| NM3
+    end
+    subgraph ProxyNode
+        NM4[Node Manager]
+        Proxy[Reverse Proxy Process]
+        NM4 -->|starts and supervises| Proxy
+        NM4 -.->|port reachability check| Proxy
+    end
+    Ctrl <-->|raft election and replication existing| OtherControllers[Other Controller Instances]
+```
+
+**Architecture Integration**:
+- Selected pattern: a per-node supervisor process (`xce-node-manager`) that owns starting, restarting, and — for the Scheduler specifically — stopping its managed children, using a generic desired-state-plus-liveness engine with one role-specific policy adapter (Scheduler-bound-to-Controller-role).
+- Domain/feature boundaries: the generic supervision engine (`ComponentSupervisor`) is separate from the liveness-signal sources (`LivenessMonitor`, two implementations: heartbeat-based and port-check-based), which is separate from the one XCE-domain-specific policy (`SchedulerDesiredStatePolicy`) — the only piece of `xce-node-manager` that knows anything about Controller primary/standby semantics.
+- Existing patterns preserved: `posix_spawn`-based child process ownership (the already-reviewed decision from `scheduler-primary-lifecycle`); the Scheduler's own independent controller-liveness backstop check (`SchedOwnerWatch`, carried forward from that same superseded design, unchanged in behavior); the existing controller-scheduler wire connection and `ha_dispatch_gate` safety argument, both untouched.
+- New components rationale: `xce-node-manager` is a wholly new domain (cross-component process lifecycle) with no existing owner; the heartbeat primitive is new but deliberately minimal (see Technology Stack) and lives in `libxce` alongside the other cross-daemon primitives it already provides.
+- Steering compliance: respects `ha-async-io.md`'s thread-role contract (the Controller's heartbeat report is a single non-blocking local send, never inline blocking work on iomgr/delaymgr) and the controller-never-self-terminate invariant (a heartbeat failure is never escalated to a crash/hang/exit); respects `structure.md`'s component-per-directory convention for the new `src/nodemgr/` daemon.
+
+### Technology Stack
+
+| Layer | Choice / Version | Role in Feature | Notes |
+|-------|------------------|-----------------|-------|
+| Node Manager process management | POSIX `posix_spawn`/`waitpid`/signal (C, `src/nodemgr/src/component_supervisor.c`) | Start, monitor, restart, and stop every managed component | Same `posix_spawn`-over-`fork()` rationale already established by `scheduler-primary-lifecycle`'s design review; see Project Invariants Checked |
+| Heartbeat transport | Unix domain datagram socket (`SOCK_DGRAM`), local host only | Carries the liveness/role-status report from each managed component to its node's Node Manager | Deliberately not Cap'n Proto — see Technology Stack Deviation below |
+| Heartbeat wire format | Fixed-layout binary struct (magic, version, role, status, pid, sequence), `src/include/xce_heartbeat.h` | Self-identifying payload (receiver does not depend on socket peer identity) | One sequence counter serves as both liveness proof (all roles) and real-progress proof (Scheduler specifically) — see Data Models |
+| Heartbeat client (C) | `libxce`, `src/lib/heartbeat.c` | Used by the Controller, Scheduler, and Worker, each already linking `libxce` | Reused as-is by all three, zero new link-time cost |
+| Heartbeat client (Go) | Native Go, `src/frontend/apiserver/heartbeat/` | Used by the Apiserver via a small ticker goroutine | Deliberately not routed through the existing CGO bridge — see Technology Stack Deviation below |
+| Reverse Proxy liveness | TCP connect against the proxy's configured port | Liveness check for the off-the-shelf Reverse Proxy role | The Reverse Proxy cannot participate in the heartbeat protocol (not XCE-authored code) |
+| Node role assignment | Local file, read at Node Manager startup (format is an implementation detail; see Data Models for its logical shape) | Tells `xce-node-manager` which component(s) to start on this node | Deliberately not cluster-distributed config — a node must know its role before any Controller is reachable |
+
+**Technology Stack Deviation**: `structure.md` states new inter-daemon protocols use Cap'n Proto schemas. This protocol is a deliberate, narrow exception: it is same-host-only (never crosses a network boundary) and always exchanged between binaries from the same build (the Node Manager and the components it manages ship together), so Cap'n Proto's cross-version/cross-language schema-evolution guarantees — the reason `controller.capnp`/`c.capnp` earn their cost for the real cluster wire protocol — buy nothing here. Adopting it for the Apiserver side specifically would also mean a brand-new Go dependency (`go.mod`/`go.sum` confirmed the Apiserver has no existing capnp dependency) purely to encode four fixed-size fields. See `research.md`'s Decision log for the full reasoning.
+
+## File Structure Plan
+
+### Directory Structure
+```
+src/
+├── nodemgr/                        # NEW component — the Node Manager daemon
+│   ├── meson.build
+│   ├── CLAUDE.md
+│   ├── include/
+│   │   ├── role_manifest.h         # Node Role Manifest types + loader interface
+│   │   ├── component_supervisor.h  # ComponentSupervisor service interface
+│   │   ├── liveness_monitor.h      # LivenessMonitor interface (heartbeat-based, port-check-based)
+│   │   └── scheduler_policy.h      # SchedulerDesiredStatePolicy interface
+│   ├── src/
+│   │   ├── main.c                  # entry point: load manifest, wire supervisor + monitors, run
+│   │   ├── role_manifest.c
+│   │   ├── component_supervisor.c  # posix_spawn/waitpid/kill, restart-on-exit, adopt-on-restart
+│   │   ├── heartbeat_listener.c    # LivenessMonitor impl: UDS SOCK_DGRAM server, per-role tracking
+│   │   ├── port_check_monitor.c    # LivenessMonitor impl: TCP connect against a configured port
+│   │   └── scheduler_policy.c      # reads Controller status from heartbeat_listener, drives Scheduler desired-state
+│   └── testsuite/                  # test_component_supervisor.c, test_heartbeat_listener.c, test_scheduler_policy.c, ...
+├── lib/
+│   └── heartbeat.c                 # NEW: xce_heartbeat_report() client primitive, wire encode/decode
+├── include/
+│   └── xce_heartbeat.h             # NEW: public heartbeat wire format + client API (see Data Models)
+├── controller/
+│   └── ha.c                        # MODIFIED: one new small periodic task, armed unconditionally at controller startup, calls xce_heartbeat_report() with the current atomic role
+├── scheduler/
+│   └── src/
+│       ├── scheduler.c             # MODIFIED (driver layer only, core.c untouched): sched_job_picker's cbuf_get() -> cbuf_timedget() with a bounded idle-tick deadline; call xce_heartbeat_report() once per loop iteration on BOTH the real-job path and the timeout path, so the sequence advances on a bounded cadence even when idle
+│       └── sched_owner.c           # NEW: backstop controller-liveness self-check, carried forward from scheduler-primary-lifecycle's design (Requirement 6)
+├── worker/
+│   └── src/xcecd_worker.c          # MODIFIED: call xce_heartbeat_report() once per main-loop iteration
+└── frontend/apiserver/
+    ├── heartbeat/
+    │   └── heartbeat.go            # NEW: native wire-format encoder + ticker goroutine, no CGO
+    └── cmd/main.go                 # MODIFIED: starts the heartbeat ticker at apiserver startup
+```
+
+### Modified Files (tooling, outside `src/`)
+- `tools/appimage/entry.sh` — add a `manager` case that execs `xce_nodemgr`; no other case changes (the Reverse Proxy role is launched by `xce_nodemgr` invoking the already-bundled `haproxy` binary directly, not through `entry.sh`).
+- `.claude/skills/xce-deploy-ha/SKILL.md` and `src/tools/deploy-ha.sh` — stop generating independent `start-worker.sh`/`start-scheduler.sh` intended for manual invocation; generate a Node Role Manifest per node instead and launch `xce_nodemgr`.
+- `.claude/skills/xce-test-ha/run-ha-soak.sh` — remove this session's own manual `start_worker_and_scheduler()`/`restart_scheduler_for_new_primary()` logic (bug-1760's workaround, now unnecessary); assert a working Scheduler/Worker/Apiserver is present after each round instead.
+
+## System Flows
+
+### Scheduler start/stop bound to Controller role
+```mermaid
+sequenceDiagram
+    participant Ctrl as Controller
+    participant HB as Node Manager HeartbeatListener
+    participant Pol as SchedulerDesiredStatePolicy
+    participant Sup as ComponentSupervisor
+    participant Sched as Scheduler Process
+    loop every Controller loop iteration
+        Ctrl->>HB: heartbeat role=controller status=primary or standby seq++
+    end
+    HB->>Pol: latest Controller status
+    alt status is primary and no Scheduler running
+        Pol->>Sup: desired state Scheduler=running
+        Sup->>Sched: posix_spawn passing this Controller address
+        Sched->>Ctrl: join using pinned single controller address existing behavior
+    else status is standby or Controller heartbeat overdue
+        Pol->>Sup: desired state Scheduler=stopped
+        Sup->>Sched: terminate
+    end
+    Sched->>Ctrl: independent backstop liveness check existing connection
+```
+The Node Manager's decision (top path) and the Scheduler's own backstop check (bottom, Requirement 6) are independent — either one alone is sufficient to stop a Scheduler whose Controller is no longer primary, so a Node Manager outage does not strand a Scheduler indefinitely.
+
+### Wedge detection and replacement (generic — shown for the Scheduler)
+```mermaid
+sequenceDiagram
+    participant Sched as Scheduler Process
+    participant HB as Node Manager HeartbeatListener
+    participant Sup as ComponentSupervisor
+    loop every round-loop iteration whether or not work is found
+        Sched->>HB: heartbeat role=scheduler seq++
+    end
+    HB->>HB: observe seq has not advanced past threshold
+    HB->>Sup: component unresponsive
+    Sup->>Sched: terminate
+    Sup->>Sched: posix_spawn a replacement
+```
+The same path applies to the Worker and Apiserver using their own heartbeat sequence; the Reverse Proxy uses the `port_check_monitor` liveness source instead of `heartbeat_listener` but feeds the same `ComponentSupervisor` unresponsive-replace action. For the Scheduler specifically, "every round-loop iteration" means both the real-job path and the bounded-idle-timeout path of `cbuf_timedget()` (see Architecture and File Structure Plan) — the sequence still advances on a healthy idle Scheduler, so only a genuine mid-decision wedge (stuck after a job is already dequeued, not merely waiting for one) fails to advance it.
+
+### Node Manager restart: adopt, don't duplicate
+```mermaid
+sequenceDiagram
+    participant NM as Node Manager on restart
+    participant Rec as Process Record file
+    participant OS as Host OS
+    participant Sup as ComponentSupervisor
+    NM->>Rec: read recorded role to pid mapping
+    NM->>OS: is that pid alive and running the expected binary
+    alt recorded process still alive and matches
+        NM->>Sup: adopt existing process for this role
+    else record missing stale or mismatched
+        NM->>Sup: start a fresh process for this role
+    end
+```
+
+## Requirements Traceability
+
+| Requirement | Summary | Components | Interfaces | Flows |
+|-------------|---------|------------|------------|-------|
+| 1.1, 1.2, 1.3, 1.4, 1.5, 1.6 | Node reads its role(s) locally and starts the matching component(s); errors logged and retried | RoleManifest, ComponentSupervisor | `component_supervisor_start` | Node Manager startup (Architecture) |
+| 2.1, 2.2, 2.3, 2.4 | Always-on components restarted on unexpected exit, not on an intentional stop | ComponentSupervisor | `component_supervisor_start`/`_stop` | Wedge/unexpected-exit flow |
+| 3.1, 3.2, 3.3, 3.4, 3.5, 3.6 | Scheduler start/stop bound to Controller primary/standby; bounded stop time; no job loss; keep retrying start while primary | ComponentSupervisor, SchedulerDesiredStatePolicy, HeartbeatListener, Controller's existing WAL-commit ownership (unchanged) | `scheduler_policy_evaluate`, `component_supervisor_start`/`_stop` | Scheduler start/stop flow |
+| 4.1, 4.2, 4.3, 4.4, 4.5, 4.6 | Heartbeat-based liveness; Scheduler liveness reflects real progress; Reverse Proxy uses port check (4.5); unresponsive components replaced and logged | HeartbeatListener, PortCheckMonitor, `libxce` heartbeat client | `xce_heartbeat_report`, `liveness_monitor_check` | Wedge detection flow |
+| 5.1, 5.2, 5.3 | Reporting never blocks a component's own responsibilities; failure to report doesn't affect the component; absent Node Manager doesn't affect Controller HA | `libxce` heartbeat client (non-blocking send), Controller/Scheduler/Worker/Apiserver integration points | `xce_heartbeat_report` | Scheduler start/stop flow, Wedge detection flow |
+| 6.1, 6.2, 6.3 | Scheduler independently verifies its Controller and exits on its own if it cannot | SchedOwnerWatch (Scheduler) | `sched_owner_watch_start` | Scheduler start/stop flow |
+| 7.1, 7.2 | Node Manager adopts already-running components on its own restart instead of duplicating | ComponentSupervisor, Process Record | `component_supervisor_start` (adoption path) | Node Manager restart flow |
+| 8.1, 8.2, 8.3 | Deployment tooling and adjacent specs reconciled; observable component behavior unchanged | `xce-deploy-ha`, `xce-test-ha`, `entry.sh` | n/a (tooling) | Migration Strategy |
+
+## Components and Interfaces
+
+| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
+|-----------|--------------|--------|---------------|--------------------------|-----------|
+| ComponentSupervisor | Node Manager | Start/stop/restart/adopt every managed component via `posix_spawn` | 1, 2, 3, 7 | RoleManifest (P0), LivenessMonitor (P1), SchedulerDesiredStatePolicy (P1) | Service |
+| HeartbeatListener | Node Manager | UDS `SOCK_DGRAM` server tracking per-role liveness/sequence/status | 4, 5 | `libxce` heartbeat wire format (P0) | Service, State |
+| PortCheckMonitor | Node Manager | TCP-reachability liveness for the Reverse Proxy role | 4.5, 4.6 | none | Service |
+| SchedulerDesiredStatePolicy | Node Manager | Translates the Controller's reported status into a Scheduler desired-state decision | 3 | HeartbeatListener (P0) | Service |
+| `libxce` heartbeat client | Shared library | Non-blocking heartbeat send, used from each component's own loop | 4, 5 | none | Service |
+| SchedOwnerWatch | Scheduler | Backstop: independently verifies the owning Controller and exits on loss | 6 | Existing controller-scheduler connection (P0) | Service |
+
+### Node Manager
+
+#### ComponentSupervisor
+
+| Field | Detail |
+|-------|--------|
+| Intent | Own the decision of whether each managed component is currently running, and start/stop/restart it accordingly |
+| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 2.1, 2.2, 2.3, 2.4, 3.1, 3.4, 3.6, 7.1, 7.2 |
+
+**Responsibilities & Constraints**
+- For each role the local Node Role Manifest assigns, holds a desired-state flag: always-running (Worker, Apiserver, Reverse Proxy, Controller) or policy-driven (Scheduler, via `SchedulerDesiredStatePolicy`).
+- On its own startup, consults the Process Record for each managed role before spawning — adopts an already-running, matching process instead of starting a duplicate (Requirement 7).
+- Detects an unexpected child exit (`waitpid`) and distinguishes it from a stop it requested itself; only the former triggers an automatic restart.
+- Never blocks; every `posix_spawn`/`waitpid`/signal call happens on the Node Manager's own management loop. That loop is itself liveness-critical for every role the node manages, not exempt from the thread-role contract — a stall here delays supervision of every other role on the same node, which is exactly why `HeartbeatListener` and `PortCheckMonitor` are both required to be non-blocking (see Project Invariants Checked).
+
+**Dependencies**
+- Inbound: `main.c`'s startup sequence (P0), `LivenessMonitor` unresponsive-component signal (P1), `SchedulerDesiredStatePolicy`'s desired-state signal (P1).
+- Outbound: the host OS process API (`posix_spawn`/`waitpid`/`kill`) (P0).
+
+**Contracts**: Service [x] / State [x]
+
+##### Service Interface
+```c
+/* src/nodemgr/include/component_supervisor.h */
+
+typedef enum {
+    XCE_ROLE_CONTROLLER,
+    XCE_ROLE_SCHEDULER,
+    XCE_ROLE_WORKER,
+    XCE_ROLE_APISERVER,
+    XCE_ROLE_REVERSE_PROXY,
+} xce_component_role_t;
+
+/* Ensures a component for the given role is running (spawns if not, adopts
+ * an already-running matching process on first call after Node Manager
+ * startup). Idempotent: a role already running or already starting is a
+ * no-op. Non-blocking. */
+int component_supervisor_start(xce_component_role_t role);
+
+/* Requests the running component for the given role be stopped. For the
+ * Scheduler, sends the existing graceful-exit wire notice first and waits up
+ * to timeout_ms before force-terminating (mirrors scheduler-primary-
+ * lifecycle's already-reviewed stop sequence); for other roles, sends a
+ * termination signal directly. Non-blocking to the caller. */
+int component_supervisor_stop(xce_component_role_t role, uint32_t timeout_ms);
+
+/* Marks the currently-running component for the given role as unresponsive:
+ * stops it and starts a replacement, following the same path as an
+ * unexpected exit. A no-op if no component is currently owned for that
+ * role. */
+int component_supervisor_replace(xce_component_role_t role);
+```
+- Preconditions: called only from the Node Manager's own single management loop (no concurrent callers).
+- Postconditions: the requested action is either already complete or in progress — never silently dropped.
+- Invariants: at most one process is owned per role at any time; a stop-then-start for the same role always completes the stop's reap before a new start begins.
+
+**Implementation Notes**
+- Integration: `SchedulerDesiredStatePolicy`'s output and `LivenessMonitor`'s unresponsive signal are the only two inbound triggers besides initial startup — both non-blocking calls into this component's own loop.
+- Validation: an operator-visible log line on every start failure, restart, adoption, and forced-kill-on-timeout (Requirements 1.6, 2.4, 4.6, 7.2).
+- Risks: sizing the graceful-stop timeout for the Scheduler is an implementation-time tuning decision (same posture as the superseded design).
+
+#### HeartbeatListener
+
+| Field | Detail |
+|-------|--------|
+| Intent | Receive heartbeat datagrams from every locally-running managed component and track per-role liveness/status/progress |
+| Requirements | 4.1, 4.2, 4.3, 4.4, 4.6, 5.1 |
+
+**Responsibilities & Constraints**
+- Binds one `SOCK_DGRAM` Unix domain socket per node; every datagram is self-identifying (role + pid fields in the payload — see Data Models), so no socket-level peer identity is required.
+- Tracks, per role: last-seen wall-clock time, last-seen sequence number, and (for the Controller) the last-reported status byte (primary/standby).
+- Treats a role as unresponsive when its sequence number has not advanced within the operator-configurable threshold — this single check covers both "no datagrams arriving at all" (process dead) and "datagrams arriving but the sequence is frozen" (a heartbeat call site that keeps executing on a thread independent of the wedged work, which is exactly why component integration points are constrained to each component's own real work loop — see Architecture).
+- Never blocks: reads from the socket are handled on the Node Manager's own loop; a full or absent socket buffer on the sending side is the sender's problem (see `libxce` heartbeat client), never this component's.
+
+**Dependencies**
+- Inbound: `libxce` heartbeat client sends from the Controller, Scheduler, Worker (P0) and the Apiserver's native Go client (P0).
+- Outbound: read by `ComponentSupervisor` (unresponsive signal, P1) and `SchedulerDesiredStatePolicy` (Controller status, P0).
+
+**Contracts**: Service [x] / State [x]
+
+##### Service Interface
+```c
+/* src/nodemgr/include/liveness_monitor.h */
+
+typedef struct {
+    bool     seen_at_least_once;
+    bool     unresponsive;
+    uint8_t  last_status;     /* meaningful for XCE_ROLE_CONTROLLER only */
+    uint64_t last_seq;
+    uint32_t last_pid;
+} xce_liveness_t;
+
+/* Returns the current liveness snapshot for the given role. Non-blocking,
+ * reads in-memory state only. */
+xce_liveness_t liveness_monitor_check(xce_component_role_t role);
+```
+- Preconditions: none — safe to call before any heartbeat has ever been received (`seen_at_least_once=false`).
+- Postconditions: reflects the most recently received datagram for that role, or the port-check result for `XCE_ROLE_REVERSE_PROXY` (see PortCheckMonitor).
+- Invariants: `unresponsive` is only ever set by elapsed-time comparison against the configured threshold, never by a single missed tick.
+
+**Implementation Notes**
+- Integration: `PortCheckMonitor` implements this same `liveness_monitor_check` contract for `XCE_ROLE_REVERSE_PROXY`, so `ComponentSupervisor` reads liveness through one uniform interface regardless of role.
+- Risks: threshold sizing is implementation-time tuning, not a design decision (same posture as the superseded spec's wedge-detection threshold).
+
+#### PortCheckMonitor (summary)
+
+Implements the same `liveness_monitor_check` contract as `HeartbeatListener`, for `XCE_ROLE_REVERSE_PROXY` only: a periodic TCP connect attempt against the port the Node Role Manifest configures for that role, with no heartbeat payload involved (the Reverse Proxy is off-the-shelf and cannot participate in the heartbeat protocol). A failed or refused connect attempt within the threshold sets `unresponsive=true`, feeding `ComponentSupervisor`'s replace path exactly as a stalled heartbeat sequence does for other roles (Requirements 4.5, 4.6). The connect is non-blocking (`O_NONBLOCK` plus a bounded `poll`/`select`, not a plain blocking `connect()`) — a silently-dropping firewall must not stall this check for an OS-level TCP timeout (which can run to minutes), since the Node Manager's single management loop also services every other role on that node; this is the same non-blocking discipline `HeartbeatListener`'s socket handling already applies, made explicit here so it is not accidentally implemented as a blocking call.
+
+#### SchedulerDesiredStatePolicy
+
+| Field | Detail |
+|-------|--------|
+| Intent | The one XCE-domain-specific policy in the Node Manager: turn the Controller's reported role into a Scheduler desired-state decision |
+| Requirements | 3.1, 3.2, 3.3, 3.5, 3.6 |
+
+**Responsibilities & Constraints**
+- Only active on a node whose role includes manager.
+- Evaluated on every Node Manager loop tick: desired state is "running" iff `liveness_monitor_check(XCE_ROLE_CONTROLLER)` reports `last_status == PRIMARY` and not `unresponsive`; "stopped" otherwise (standby, or Controller heartbeat overdue).
+- Contains no Raft/election logic of its own — it is a pure function of the Controller's already-decided, already-reported state (Out of Boundary: this feature never decides who is primary).
+
+**Dependencies**
+- Inbound: `HeartbeatListener` (P0).
+- Outbound: `ComponentSupervisor` (P0).
+
+**Contracts**: Service [x]
+
+**Implementation Notes**
+- Integration: a plain evaluate-and-call function on the Node Manager's own loop; no separate thread.
+- Validation: unit-testable in isolation by feeding synthetic `xce_liveness_t` values.
+
+### Scheduler
+
+#### SchedOwnerWatch
+
+| Field | Detail |
+|-------|--------|
+| Intent | Backstop: independently confirm the owning Controller is reachable and primary; exit otherwise, without depending on the Node Manager |
+| Requirements | 6.1, 6.2, 6.3 |
+
+Carried forward unchanged from `scheduler-primary-lifecycle`'s design (same component, same contract, same rationale — a Node Manager outage must not strand an orphaned Scheduler). See that design's `SchedOwnerWatch` section for the full service interface (`sched_owner_watch_start(owner_host, owner_port)`); this spec restates it as Requirement 6 rather than re-deriving it, since the mechanism is unchanged.
+
+**Implementation Notes**
+- Integration: replaces the Scheduler's superseded reactive rejoin logic exactly as `scheduler-primary-lifecycle` already specified (`sched_needs_rejoin`/`sched_on_reconnect`/`sched_publish_check_rejoin` removed, not left dormant).
+- This component's existence is independent of who *starts* the Scheduler — it verifies the Scheduler's *continued* right to run against its one pinned Controller regardless of whether `xce-node-manager` or (in the superseded design) the Controller itself did the spawning.
+
+## Data Models
+
+No durable cluster state (WAL, checkpoint, replicated config) is introduced or modified by this feature — the heartbeat protocol and the Node Manager's own bookkeeping are both process-lifetime or node-local-file concerns, not cluster state, so the project's cluster-wide save/restore field-completeness review does not apply. Two local persistence artifacts are introduced, and each is enumerated field-by-field below per the project's persistence-completeness review discipline:
+
+### Heartbeat wire message (in-memory only, never persisted)
+| Field | Type | Meaning |
+|-------|------|---------|
+| magic | uint32 | Protocol identifier + version, rejects mismatched/garbage datagrams |
+| role | uint8 (enum) | `XCE_ROLE_CONTROLLER` / `SCHEDULER` / `WORKER` / `APISERVER` |
+| status | uint8 | Meaningful for `XCE_ROLE_CONTROLLER` only: primary or standby; reserved (zero) for other roles |
+| pid | uint32 | Sender's process id — used by `ComponentSupervisor` to cross-check against its own spawn/adoption records |
+| seq | uint64 | Monotonically increasing counter, incremented once per call from the component's own loop iteration; doubles as the real-progress signal for the Scheduler (Requirement 4.3) |
+
+### Process Record (one small local file per managed role, read/written by `xce-node-manager` only)
+| Field | Captured on write | Restored on read | Notes |
+|-------|--------------------|-------------------|-------|
+| role | yes | yes | Which managed role this record describes |
+| pid | yes | yes | Checked against the OS on Node Manager startup before adoption |
+| binary_path | yes | yes | Checked against the running process's executable to reject a coincidentally-reused pid |
+| start_time | yes | yes | Diagnostic only — not used for any adoption decision |
+
+A record that is missing, unreadable, or fails the pid/binary_path cross-check is treated as "no component currently running for this role," which leads `ComponentSupervisor` to start a fresh one (Requirement 7) — never treated as an error requiring operator intervention, since a fresh start is always safe (Out of Boundary items above establish that at most one process per role is a `ComponentSupervisor` invariant, and each managed component's own listen-port bind-failure is a natural backstop against a true accidental duplicate).
+
+### Node Role Manifest (local, static, read at Node Manager startup only)
+Logical shape only (concrete file format is an implementation detail): a list of one or more roles (`manager`, `worker`, `apiserver`, `reverse-proxy`) assigned to this node, and, per role, the pass-through startup parameters that component's own existing configuration mechanism already expects (the Controller's own `--peers`/cert arguments, the Worker's own arguments, the Apiserver's own config file path, the Reverse Proxy's own config file path). This feature does not invent new per-component configuration — it only decides *whether and when* to invoke each component with the configuration that component already understands. Deliberately not cluster-distributed: available to `xce-node-manager` without depending on connectivity to any Controller (Requirement 1.1).
+
+## Error Handling
+
+### Error Strategy
+Every failure in component-lifecycle management follows the project's existing fail-loud-stay-alive philosophy (`ha-async-io.md`, `controller/CLAUDE.md`): log an operator-visible error, never crash or hang the reporting component or the Node Manager, and retry where retrying is meaningful.
+
+### Error Categories and Responses
+- **A managed component fails to start** (`posix_spawn` failure, immediate crash): `ComponentSupervisor` logs an error and retries — continuously for always-on roles, and only while the Controller remains primary for the Scheduler (Requirement 1.5, 3.6).
+- **A managed component does not exit within the Scheduler's graceful-stop timeout**: force-terminated after the bounded wait; the requesting action (a Controller demotion, in the Scheduler's case) never waits on this to finish (Requirement 3.4).
+- **A managed component is connected/reporting but not progressing (wedged)**: treated as unresponsive once its sequence number stalls past the threshold, replaced, with an operator-visible log (Requirement 4.2, 4.4, 4.6).
+- **A component cannot reach `xce-node-manager`'s heartbeat socket** (absent, full buffer, permission issue): the sending component logs nothing beyond its own existing conventions and continues operating normally — this is an expected condition (no Node Manager present, e.g. during isolated testing) and never affects the component's own responsibilities (Requirement 5.2, 5.3).
+- **The Node Manager itself cannot determine whether a recorded process is genuinely still the one it manages** (Process Record pid reused by an unrelated process): treated as "not running," a fresh component is started (Requirement 7.1).
+
+### Monitoring
+All new log lines (start failure, forced-kill-on-timeout, unresponsive-replacement, adoption-on-restart) use each component's own existing logging conventions (the Controller's `WARN`/`ERROR` macros, the Node Manager's equivalent) — no new logging mechanism is introduced.
+
+## Testing Strategy
+
+### Unit Tests
+- `component_supervisor_start` adopts an already-running, matching process instead of spawning a duplicate when a valid Process Record exists, and starts fresh when the record is missing/stale/mismatched — directly exercises Requirement 7.1.
+- `component_supervisor_start`/`_replace` distinguishes an intentional stop from an unexpected exit, restarting only the latter — exercises Requirement 2.1-2.3.
+- `SchedulerDesiredStatePolicy` evaluates "running" only when the Controller's last-reported status is primary and not unresponsive, "stopped" otherwise (standby, overdue, or never-seen) — exercises Requirement 3.1-3.3.
+- `liveness_monitor_check` (heartbeat-based) marks a role unresponsive only once its sequence number has stalled past the threshold, not on a single delayed tick — exercises Requirement 4.2.
+- `xce_heartbeat_report`'s send path never blocks when the receiving socket is absent or its buffer is full — exercises Requirement 5.1, 5.2.
+
+### Integration Tests
+- A real 3-node Controller cluster promotion results in `xce-node-manager` starting a Scheduler process on the newly-primary node with no manual step, and the prior primary's `xce-node-manager` stopping its Scheduler within the bounded timeout — exercises Requirement 3.1-3.4 end to end.
+- A hard-killed primary Controller (bug-1760's original repro) results in the orphaned Scheduler self-exiting via `SchedOwnerWatch` even before `xce-node-manager` would otherwise notice, and the newly-elected primary's `xce-node-manager` starting a fresh Scheduler that successfully dispatches a job submitted after the failover — exercises Requirement 6.1-6.3 and directly closes bug-1760.
+- A Scheduler artificially wedged after dequeuing a job (e.g. blocked inside the allocator, past the `cbuf_timedget()` call) is detected as unresponsive and replaced; a Scheduler left genuinely idle (no jobs submitted) for longer than the threshold is NOT falsely flagged, confirming both halves of Requirement 4.3 — the wedge case and the false-positive-avoidance case the naive design would have failed.
+- A `xce-node-manager` process killed and restarted while its Controller and Scheduler keep running adopts both without starting duplicates — exercises Requirement 7.1.
+- A job dispatched by a Scheduler that is then killed mid-decision (before its dispatch is durably recorded) remains available for the replacement Scheduler to consider — exercises Requirement 3.5.
+
+### E2E / Soak Tests
+- Extend `.claude/skills/xce-test-ha/run-ha-soak.sh`'s sequential and chaotic modes to assert a working Scheduler, Worker, and Apiserver are present after every round with no harness-side start/stop action for any of them — replaces this session's own bug-1760 workaround, which becomes dead code once this feature ships (Requirement 8.2).
+- A rapid sequence of Controller promotions/demotions produces no more than one Scheduler spawn/stop cycle per genuine promotion, with no orphaned or duplicate processes remaining after the sequence settles (Requirement 3.6, 7.1).
+
+## Security Considerations
+
+**Privilege model is role-conditional, not uniform.** `xce-deploy-ha`'s own tooling documents that the real (non-virtual) Worker daemon (`xcecd`, as opposed to the `xcecd_virtual` test double) runs as root — "plain kill can't touch it," requiring `sudo`. `xce-node-manager` must therefore run as root on any node whose assigned role includes `worker`, so `ComponentSupervisor`'s `posix_spawn`/`kill`/`waitpid` calls against `xcecd` need no special-cased privilege escalation; on a node whose roles are limited to manager/apiserver/reverse-proxy, `xce-node-manager` runs as an ordinary service account. This is a per-node deployment decision (how the host's own service manager launches `xce_nodemgr`), not a per-role runtime toggle inside the binary itself — the Node Role Manifest already fixes a node's roles at Node Manager startup, so the required privilege level is knowable at that same point.
+
+The heartbeat socket is a new local IPC surface whose payload (in particular the Controller's reported primary/standby status) influences whether `xce-node-manager` starts or stops a component — a local process able to write to it could send spoofed reports. Consistent with `product.md`'s already-locked decision to trust the OS boundary rather than add application-layer encryption (`chmod 0700`, no new crypto), the heartbeat socket is created with permissions restricting it to the OS user `xce-node-manager` itself runs as on that node (root on a worker-role node, matching `xcecd`'s own existing requirement; an ordinary service account otherwise) — the same trust boundary the WAL/checkpoint directories already rely on. No new authentication mechanism is introduced; a host already compromised at that privilege level can already directly signal/kill the same processes this feature manages, so the socket adds no new privilege-escalation surface beyond what local code execution at that level already grants. On a worker-role node specifically, running `xce-node-manager` (and therefore the socket) as root is a direct consequence of `xcecd`'s own pre-existing requirement, not a new exposure this feature introduces.
+
+## Migration Strategy
+
+```mermaid
+flowchart TB
+    Phase1[Phase 1 build xce nodemgr and the libxce heartbeat primitive behind no behavior change]
+    Phase2[Phase 2 wire heartbeat calls into controller scheduler worker apiserver loops]
+    Phase3[Phase 3 add the entry.sh manager case and update xce deploy ha and xce test ha tooling]
+    Phase4[Phase 4 cut over real deployments to node manager driven startup]
+    Phase1 --> Phase2 --> Phase3 --> Phase4
+```
+- Rollback trigger: if a real deployment depends on placing a component on a host with no Node Manager present for reasons not previously surfaced, that is a genuine requirements gap discovered late — the fix is to pause before Phase 4 (cutover) rather than force it through, since Phase 4 is the first phase that changes how a real deployment is operated.
+- Validation checkpoint: after Phase 2, run the soak-harness scenario in Testing Strategy's E2E section before proceeding to Phase 3/4, since it is the closest thing to a direct regression test for bug-1760 itself.
+
+## Open Questions / Risks
+- The exact Node Role Manifest file format (YAML to match `src/configs/*.yaml`'s existing convention, vs. a simpler key-value form) is left to implementation — an additive, appropriately implementation-level decision, not a design decision, since Data Models already fixes its logical shape.
+- The precise heartbeat interval and "no progress" threshold (Requirements 4.2, 4.3) are implementation-time tuning decisions, following the same posture the superseded spec already took for its own liveness thresholds.
+- `tools/appimage/entry.sh` bundles `haproxy` today with no dispatch case reaching it at all — this feature is what first wires it up (via `xce-node-manager` invoking it directly for the reverse-proxy role), which is a pre-existing gap this design closes as a side effect rather than a pre-existing behavior it must preserve.
+
+Resolved during design review (kept here for traceability, not left as open items): the Scheduler round-loop idle-cadence question was confirmed against the actual code (`cbuf_get()`'s unbounded wait) and resolved via `cbuf_timedget()` — see Architecture and File Structure Plan; it does not need re-verification at implementation time. The Worker-node root-privilege gap was resolved as a role-conditional privilege model — see Security Considerations.