myxce/.kiro/specs/xce-node-manager/design.md

Overview

Purpose: This feature introduces xce-node-manager, a per-node daemon that starts and supervises the XCE component(s) appropriate to a node's assigned role — replacing today's model where each of the Controller, Scheduler, Worker, and Apiserver is started and supervised as an independent, manually-managed process. On a manager-role node it additionally binds the Scheduler's own lifetime to its co-located Controller's primary/standby status, closing bug-1760 (a hard-killed primary's Scheduler never rejoined the newly-elected primary, leaving jobs stuck PEND indefinitely).

Users: XCE cluster operators, and transitively every job submitted to the cluster, benefit from never having to manually start a component after a deploy or notice/intervene after a Controller failover or a component crash.

Impact: Introduces one new top-level daemon (xce_nodemgr) and one new local, same-host wire protocol (a heartbeat/liveness report) consumed by the Controller, Scheduler, Worker, and Apiserver. Supersedes .kiro/specs/scheduler-primary-lifecycle/'s mechanism (a Controller directly posix_spawn-ing and killing its own Scheduler child) — that responsibility moves to xce-node-manager, driven by the Controller's own reported primary/standby state rather than by the Controller managing a child process itself.

Goals

• A node's assigned role(s) determine which XCE component(s) start on it, with no manual per-component start step.
• A component that should always be running (Controller, Worker, Apiserver, Reverse Proxy) is restarted automatically if it exits unexpectedly.
• The Scheduler on a manager node starts the moment its co-located Controller becomes primary and stops promptly when that Controller stops being primary or becomes unreachable — with no cross-Controller reconnect logic in the Scheduler.
• A component that is alive but not doing useful work (a wedged Scheduler round loop, in particular) is distinguished from a healthy one and replaced.
• xce-node-manager's own presence, absence, or failure can never compromise a Controller's own availability or HA correctness.
• Existing deployment tooling and specs that assume independently-started components are identified and reconciled.

Non-Goals

• Supervising xce-node-manager itself (delegated to the host's existing process-supervision facilities).
• Any change to the Reverse Proxy's own configuration, routing rules, or request handling.
• Any change to how a Worker's or Apiserver's own client-side connection discovers/reconnects to whichever Controller is currently primary (owned by the existing/adjacent client-failover mechanism).
• Any change to RBAC, authentication, or job/queue/scheduling decision logic.
• Standby/redundant Scheduler processes or a lease-election mechanism for Schedulers (superseded design's rejection still holds — piggybacking on the Controller's own already-hardened Raft election remains the model, just triggered one layer up).
• A new operator-facing remote management API for xce-node-manager beyond existing logging conventions.
• The Scheduler's simulation mode (sim.c's driver, distinct from the live driver's xcecli/Controller connection) — a simulation-mode Scheduler is a standalone, manually-invoked tool with no real Controller or primary/standby concept, and continues to run entirely independently of xce-node-manager, exactly as it does today.

Boundary Commitments

This Spec Owns

• Reading a node's local role assignment and starting the corresponding XCE component(s).
• Continuous supervision (unexpected-exit detection, automatic restart) of every component it starts.
• The decision of when the Scheduler on a manager node should be running, derived from its co-located Controller's self-reported primary/standby status.
• The heartbeat/liveness wire format and the libxce client primitive components use to report it.
• Distinguishing "alive" from "making real progress" for components where that distinction matters (the Scheduler).
• A distinct liveness mechanism (network port reachability) for the off-the-shelf Reverse Proxy role.
• Recognizing already-running components on its own restart instead of double-spawning them.
• Updating existing deployment tooling (xce-deploy-ha, xce-test-ha) and the AppImage entry point, and flagging any other spec/tool that assumes an independently-started component.

Out of Boundary

• The Controller's Raft election, replication, and commit-safety mechanisms — this feature only reads the Controller's already-decided primary/standby role, never influences how that role is decided.
• ha_dispatch_gate's existing role-based write rejection — depended upon (unchanged), not modified.
• The Scheduler's own job/queue/config decision logic (picker, allocator, plugins) — untouched.
• Worker and Apiserver job-execution/API-serving logic — untouched.
• Any lease/renewal protocol or standby-component redundancy.
• Supervising xce-node-manager's own process availability.
• The Scheduler's simulation mode — untouched, unmanaged, and unaware of xce-node-manager's existence; only the live driver (the one that joins a real Controller cluster) is in scope.

Allowed Dependencies

• posix_spawn/waitpid/signal (POSIX) — the sole mechanism xce-node-manager uses to start/monitor/stop a managed child process, matching the already-reviewed scheduler-primary-lifecycle decision (never raw fork() — see Technology Stack).
• Unix domain datagram sockets, local to the host — the sole transport for the heartbeat protocol; no cross-host traffic is introduced by this feature.
• The existing controller-scheduler wire connection (unchanged) — the Scheduler's own backstop controller-liveness check (Requirement 6) reuses it exactly as the superseded design specified; this feature adds no new field to that connection's schema.
• libxce (src/lib/) — the heartbeat client primitive is added here so the Controller, Scheduler, and Worker (all of which already link it) get it for free.

Revalidation Triggers

• Any change to how the Controller exposes its own primary/standby role (today the atomic ha->role field) — the heartbeat report's Controller-side source of truth.
• Any change to the Scheduler's round/picker loop structure that removes or alters its bounded-cadence-even-when-idle property — the property Requirement 4.3's progress signal depends on.
• Any other spec or tool that starts the Controller, Scheduler, Worker, or Apiserver directly (bypassing xce-node-manager) must be re-checked against this design's role-driven startup contract.
• Introduction of a genuine cross-host or cross-version consumer of the heartbeat protocol would invalidate the "same-host, same-build-version" reasoning behind skipping Cap'n Proto for it (see Technology Stack).

Project Invariants Checked

This design adds a small periodic reporting point to the Controller and touches the Scheduler's process-lifecycle surface, so it is checked directly against ha-async-io.md and product.md's locked decisions:

• Thread-role contract (no blocking work on iomgr/delaymgr): satisfied — the Controller's heartbeat report is a single non-blocking, connectionless (SOCK_DGRAM) local send with no retry/backoff, issued from the same periodic point on every tick; a full/unreachable receiver drops the datagram rather than blocking the sender (see Error Handling). The equivalent hazard on the Node Manager's own side — its single management loop must not block on HeartbeatListener's socket reads or PortCheckMonitor's TCP connects, since either would stall supervision of every role on that node, not just the one being checked — is satisfied the same way: non-blocking socket handling throughout (see PortCheckMonitor's Implementation Note).
• Controller-never-self-terminate: satisfied — a heartbeat-send failure is not logged as an error path that can escalate to a crash/hang/exit; it is a silent, expected condition when no xce-node-manager is present (Requirement 5.2/5.3).
• No fork() for snapshots" (generalized precedent): xce-node-manager spawns every managed component via posix_spawn, never raw fork()+exec() — the same reasoning scheduler-primary-lifecycle's design review already established (a lock held by another thread at fork time can deadlock a child that touches malloc/libxce before execve()) applies identically here, and is called out explicitly because src/worker/src/process_worker_manager.c's existing process_manager_start_worker() is a real, already-shipped counter-example using raw fork()+execl() — that pattern is not to be imitated.
• ha_dispatch_gate role-based write rejection: depended upon, unchanged — this design does not touch it.
• Scheduler core/driver separation (scheduler-core spec's established invariant: xce_sched_core_init()'s core — core.c/core_events.c/core_session.c/core_clock.c — performs zero controller I/O): satisfied — the heartbeat call site is sched_job_picker in scheduler.c, the live driver layer that already owns the picker/allocator thread lifecycle and the xcecli connection, exactly where the cbuf_timedget() change also lands. The pure decision engine in core.c and siblings is untouched by this feature; the Scheduler's outside-facing I/O surface (Controller connection, and now the local heartbeat) remains confined to the live driver, matching the existing architecture rather than introducing a second I/O path into the core. The simulation driver (sim.c) is a distinct code path from scheduler.c's live driver and is not touched — a simulation-mode Scheduler has no heartbeat call site added and is never started, stopped, or monitored by xce-node-manager (see Non-Goals).

Architecture

Existing Architecture Analysis

• No component in this codebase today starts or supervises another XCE daemon based on a declared role. tools/appimage/entry.sh is the closest thing to an orchestrator, and it only execs one named binary in the foreground per invocation (xce controller, xce worker, ...) — no supervision, no multi-process coordination, and no case for the already-bundled haproxy at all.
• The only existing precedent for one XCE daemon supervising a child process is the Worker's job-execution supervision (process_manager.c, spawning per-job xcecd_worker processes via raw fork()+execl()) — job-execution-specific, and using a spawn mechanism this design deliberately does not reuse (see Project Invariants Checked).
• The Controller already exposes its own primary/standby state as an atomic field (ha->role) and already runs a non-blocking, delaymgr-driven periodic task pattern (primary_heartbeat_fn) proving this class of "cheap periodic local report" is safe and idiomatic here — though that specific function is unsuitable to extend directly (see research.md).
• The Scheduler's sched_job_picker thread (scheduler.c) calls cbuf_get() (src/lib/mylock.c:144) to pull the next job, which does an unbounded pthread_cond_wait when the queue is empty — confirmed by reading the code, not assumed. A heartbeat call placed naively at the top of this loop would never fire during a genuinely idle cluster, since the thread simply never wakes. cbuf_timedget() (same file), an already-existing bounded-deadline variant used elsewhere in the codebase, is the fix this design adopts (see Scheduler integration in File Structure Plan).
• ha_dispatch_gate already enforces that only a primary Controller accepts Scheduler write opcodes — unchanged, and still the reason this feature is purely about availability, not correctness.

Architecture Pattern & Boundary Map

graph TB
    subgraph ManagerNode
        NM1[Node Manager]
        Ctrl[Controller]
        Sched[Scheduler Process]
        NM1 -->|starts and supervises| Ctrl
        NM1 -->|starts stops on Controller role| Sched
        Ctrl -.->|heartbeat role status| NM1
        Sched -.->|heartbeat round progress| NM1
        Sched -->|pinned backstop liveness check| Ctrl
    end
    subgraph WorkerNode
        NM2[Node Manager]
        Wrk[Worker Process]
        NM2 -->|starts and supervises| Wrk
        Wrk -.->|heartbeat| NM2
    end
    subgraph ApiserverNode
        NM3[Node Manager]
        Api[Apiserver Process]
        NM3 -->|starts and supervises| Api
        Api -.->|heartbeat| NM3
    end
    subgraph ProxyNode
        NM4[Node Manager]
        Proxy[Reverse Proxy Process]
        NM4 -->|starts and supervises| Proxy
        NM4 -.->|port reachability check| Proxy
    end
    Ctrl <-->|raft election and replication existing| OtherControllers[Other Controller Instances]

Architecture Integration:

• Selected pattern: a per-node supervisor process (xce-node-manager) that owns starting, restarting, and — for the Scheduler specifically — stopping its managed children, using a generic desired-state-plus-liveness engine with one role-specific policy adapter (Scheduler-bound-to-Controller-role).
• Domain/feature boundaries: the generic supervision engine (ComponentSupervisor) is separate from the liveness-signal sources (LivenessMonitor, two implementations: heartbeat-based and port-check-based), which is separate from the one XCE-domain-specific policy (SchedulerDesiredStatePolicy) — the only piece of xce-node-manager that knows anything about Controller primary/standby semantics.
• Existing patterns preserved: posix_spawn-based child process ownership (the already-reviewed decision from scheduler-primary-lifecycle); the Scheduler's own independent controller-liveness backstop check (SchedOwnerWatch, carried forward from that same superseded design, unchanged in behavior); the existing controller-scheduler wire connection and ha_dispatch_gate safety argument, both untouched.
• New components rationale: xce-node-manager is a wholly new domain (cross-component process lifecycle) with no existing owner; the heartbeat primitive is new but deliberately minimal (see Technology Stack) and lives in libxce alongside the other cross-daemon primitives it already provides.
• Steering compliance: respects ha-async-io.md's thread-role contract (the Controller's heartbeat report is a single non-blocking local send, never inline blocking work on iomgr/delaymgr) and the controller-never-self-terminate invariant (a heartbeat failure is never escalated to a crash/hang/exit); respects structure.md's component-per-directory convention for the new src/nodemgr/ daemon.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Node Manager process management	POSIX posix_spawn/waitpid/signal (C, src/nodemgr/src/component_supervisor.c)	Start, monitor, restart, and stop every managed component	Same posix_spawn-over-fork() rationale already established by scheduler-primary-lifecycle's design review; see Project Invariants Checked
Heartbeat transport	Unix domain datagram socket (SOCK_DGRAM), local host only	Carries the liveness/role-status report from each managed component to its node's Node Manager	Deliberately not Cap'n Proto — see Technology Stack Deviation below
Heartbeat wire format	Fixed-layout binary struct (magic, version, role, status, pid, sequence), src/include/xce_heartbeat.h	Self-identifying payload (receiver does not depend on socket peer identity)	One sequence counter serves as both liveness proof (all roles) and real-progress proof (Scheduler specifically) — see Data Models
Heartbeat client (C)	libxce, src/lib/heartbeat.c	Used by the Controller, Scheduler, and Worker, each already linking libxce	Reused as-is by all three, zero new link-time cost
Heartbeat client (Go)	Native Go, src/frontend/apiserver/heartbeat/	Used by the Apiserver via a small ticker goroutine	Deliberately not routed through the existing CGO bridge — see Technology Stack Deviation below
Reverse Proxy liveness	TCP connect against the proxy's configured port	Liveness check for the off-the-shelf Reverse Proxy role	The Reverse Proxy cannot participate in the heartbeat protocol (not XCE-authored code)
Node role assignment	Local file, read at Node Manager startup (format is an implementation detail; see Data Models for its logical shape)	Tells xce-node-manager which component(s) to start on this node	Deliberately not cluster-distributed config — a node must know its role before any Controller is reachable

Technology Stack Deviation: structure.md states new inter-daemon protocols use Cap'n Proto schemas. This protocol is a deliberate, narrow exception: it is same-host-only (never crosses a network boundary) and always exchanged between binaries from the same build (the Node Manager and the components it manages ship together), so Cap'n Proto's cross-version/cross-language schema-evolution guarantees — the reason controller.capnp/c.capnp earn their cost for the real cluster wire protocol — buy nothing here. Adopting it for the Apiserver side specifically would also mean a brand-new Go dependency (go.mod/go.sum confirmed the Apiserver has no existing capnp dependency) purely to encode four fixed-size fields. See research.md's Decision log for the full reasoning.

File Structure Plan

Directory Structure

src/
├── nodemgr/                        # NEW component — the Node Manager daemon
│   ├── meson.build
│   ├── CLAUDE.md
│   ├── include/
│   │   ├── role_manifest.h         # Node Role Manifest types + loader interface
│   │   ├── component_supervisor.h  # ComponentSupervisor service interface
│   │   ├── liveness_monitor.h      # LivenessMonitor interface (heartbeat-based, port-check-based)
│   │   └── scheduler_policy.h      # SchedulerDesiredStatePolicy interface
│   ├── src/
│   │   ├── main.c                  # entry point: load manifest, wire supervisor + monitors, run
│   │   ├── role_manifest.c
│   │   ├── component_supervisor.c  # posix_spawn/waitpid/kill, restart-on-exit, adopt-on-restart
│   │   ├── heartbeat_listener.c    # LivenessMonitor impl: UDS SOCK_DGRAM server, per-role tracking
│   │   ├── port_check_monitor.c    # LivenessMonitor impl: TCP connect against a configured port
│   │   └── scheduler_policy.c      # reads Controller status from heartbeat_listener, drives Scheduler desired-state
│   └── testsuite/                  # test_component_supervisor.c, test_heartbeat_listener.c, test_scheduler_policy.c, ...
├── lib/
│   └── heartbeat.c                 # NEW: xce_heartbeat_report() client primitive, wire encode/decode
├── include/
│   └── xce_heartbeat.h             # NEW: public heartbeat wire format + client API (see Data Models)
├── controller/
│   └── ha.c                        # MODIFIED: one new small periodic task, armed unconditionally at controller startup, calls xce_heartbeat_report() with the current atomic role
├── scheduler/
│   └── src/
│       ├── scheduler.c             # MODIFIED (driver layer only, core.c untouched): sched_job_picker's cbuf_get() -> cbuf_timedget() with a bounded idle-tick deadline; call xce_heartbeat_report() once per loop iteration on BOTH the real-job path and the timeout path, so the sequence advances on a bounded cadence even when idle
│       └── sched_owner.c           # NEW: backstop controller-liveness self-check, carried forward from scheduler-primary-lifecycle's design (Requirement 6)
├── worker/
│   └── src/xcecd_worker.c          # MODIFIED: call xce_heartbeat_report() once per main-loop iteration
└── frontend/apiserver/
    ├── heartbeat/
    │   └── heartbeat.go            # NEW: native wire-format encoder + ticker goroutine, no CGO
    └── cmd/main.go                 # MODIFIED: starts the heartbeat ticker at apiserver startup

Modified Files (tooling, outside src/)

• tools/appimage/entry.sh — add a manager case that execs xce_nodemgr; no other case changes (the Reverse Proxy role is launched by xce_nodemgr invoking the already-bundled haproxy binary directly, not through entry.sh).
• .claude/skills/xce-deploy-ha/SKILL.md and src/tools/deploy-ha.sh — stop generating independent start-worker.sh/start-scheduler.sh intended for manual invocation; generate a Node Role Manifest per node instead and launch xce_nodemgr.
• .claude/skills/xce-test-ha/run-ha-soak.sh — remove this session's own manual start_worker_and_scheduler()/restart_scheduler_for_new_primary() logic (bug-1760's workaround, now unnecessary); assert a working Scheduler/Worker/Apiserver is present after each round instead.

System Flows

Scheduler start/stop bound to Controller role

sequenceDiagram
    participant Ctrl as Controller
    participant HB as Node Manager HeartbeatListener
    participant Pol as SchedulerDesiredStatePolicy
    participant Sup as ComponentSupervisor
    participant Sched as Scheduler Process
    loop every Controller loop iteration
        Ctrl->>HB: heartbeat role=controller status=primary or standby seq++
    end
    HB->>Pol: latest Controller status
    alt status is primary and no Scheduler running
        Pol->>Sup: desired state Scheduler=running
        Sup->>Sched: posix_spawn passing this Controller address
        Sched->>Ctrl: join using pinned single controller address existing behavior
    else status is standby or Controller heartbeat overdue
        Pol->>Sup: desired state Scheduler=stopped
        Sup->>Sched: terminate
    end
    Sched->>Ctrl: independent backstop liveness check existing connection

The Node Manager's decision (top path) and the Scheduler's own backstop check (bottom, Requirement 6) are independent — either one alone is sufficient to stop a Scheduler whose Controller is no longer primary, so a Node Manager outage does not strand a Scheduler indefinitely.

Wedge detection and replacement (generic — shown for the Scheduler)

sequenceDiagram
    participant Sched as Scheduler Process
    participant HB as Node Manager HeartbeatListener
    participant Sup as ComponentSupervisor
    loop every round-loop iteration whether or not work is found
        Sched->>HB: heartbeat role=scheduler seq++
    end
    HB->>HB: observe seq has not advanced past threshold
    HB->>Sup: component unresponsive
    Sup->>Sched: terminate
    Sup->>Sched: posix_spawn a replacement

The same path applies to the Worker and Apiserver using their own heartbeat sequence; the Reverse Proxy uses the port_check_monitor liveness source instead of heartbeat_listener but feeds the same ComponentSupervisor unresponsive-replace action. For the Scheduler specifically, "every round-loop iteration" means both the real-job path and the bounded-idle-timeout path of cbuf_timedget() (see Architecture and File Structure Plan) — the sequence still advances on a healthy idle Scheduler, so only a genuine mid-decision wedge (stuck after a job is already dequeued, not merely waiting for one) fails to advance it.

Node Manager restart: adopt, don't duplicate

sequenceDiagram
    participant NM as Node Manager on restart
    participant Rec as Process Record file
    participant OS as Host OS
    participant Sup as ComponentSupervisor
    NM->>Rec: read recorded role to pid mapping
    NM->>OS: is that pid alive and running the expected binary
    alt recorded process still alive and matches
        NM->>Sup: adopt existing process for this role
    else record missing stale or mismatched
        NM->>Sup: start a fresh process for this role
    end

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1, 1.2, 1.3, 1.4, 1.5, 1.6	Node reads its role(s) locally and starts the matching component(s); errors logged and retried	RoleManifest, ComponentSupervisor	component_supervisor_start	Node Manager startup (Architecture)
2.1, 2.2, 2.3, 2.4	Always-on components restarted on unexpected exit, not on an intentional stop	ComponentSupervisor	component_supervisor_start/_stop	Wedge/unexpected-exit flow
3.1, 3.2, 3.3, 3.4, 3.5, 3.6	Scheduler start/stop bound to Controller primary/standby; bounded stop time; no job loss; keep retrying start while primary	ComponentSupervisor, SchedulerDesiredStatePolicy, HeartbeatListener, Controller's existing WAL-commit ownership (unchanged)	scheduler_policy_evaluate, component_supervisor_start/_stop	Scheduler start/stop flow
4.1, 4.2, 4.3, 4.4, 4.5, 4.6	Heartbeat-based liveness; Scheduler liveness reflects real progress; Reverse Proxy uses port check (4.5); unresponsive components replaced and logged	HeartbeatListener, PortCheckMonitor, libxce heartbeat client	xce_heartbeat_report, liveness_monitor_check	Wedge detection flow
5.1, 5.2, 5.3	Reporting never blocks a component's own responsibilities; failure to report doesn't affect the component; absent Node Manager doesn't affect Controller HA	libxce heartbeat client (non-blocking send), Controller/Scheduler/Worker/Apiserver integration points	xce_heartbeat_report	Scheduler start/stop flow, Wedge detection flow
6.1, 6.2, 6.3	Scheduler independently verifies its Controller and exits on its own if it cannot	SchedOwnerWatch (Scheduler)	sched_owner_watch_start	Scheduler start/stop flow
7.1, 7.2	Node Manager adopts already-running components on its own restart instead of duplicating	ComponentSupervisor, Process Record	component_supervisor_start (adoption path)	Node Manager restart flow
8.1, 8.2, 8.3	Deployment tooling and adjacent specs reconciled; observable component behavior unchanged	xce-deploy-ha, xce-test-ha, entry.sh	n/a (tooling)	Migration Strategy

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
ComponentSupervisor	Node Manager	Start/stop/restart/adopt every managed component via posix_spawn	1, 2, 3, 7	RoleManifest (P0), LivenessMonitor (P1), SchedulerDesiredStatePolicy (P1)	Service
HeartbeatListener	Node Manager	UDS SOCK_DGRAM server tracking per-role liveness/sequence/status	4, 5	libxce heartbeat wire format (P0)	Service, State
PortCheckMonitor	Node Manager	TCP-reachability liveness for the Reverse Proxy role	4.5, 4.6	none	Service
SchedulerDesiredStatePolicy	Node Manager	Translates the Controller's reported status into a Scheduler desired-state decision	3	HeartbeatListener (P0)	Service
libxce heartbeat client	Shared library	Non-blocking heartbeat send, used from each component's own loop	4, 5	none	Service
SchedOwnerWatch	Scheduler	Backstop: independently verifies the owning Controller and exits on loss	6	Existing controller-scheduler connection (P0)	Service

Node Manager

ComponentSupervisor

Field	Detail
Intent	Own the decision of whether each managed component is currently running, and start/stop/restart it accordingly
Requirements	1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 2.1, 2.2, 2.3, 2.4, 3.1, 3.4, 3.6, 7.1, 7.2

Responsibilities & Constraints

• For each role the local Node Role Manifest assigns, holds a desired-state flag: always-running (Worker, Apiserver, Reverse Proxy, Controller) or policy-driven (Scheduler, via SchedulerDesiredStatePolicy).
• On its own startup, consults the Process Record for each managed role before spawning — adopts an already-running, matching process instead of starting a duplicate (Requirement 7).
• Detects an unexpected child exit (waitpid) and distinguishes it from a stop it requested itself; only the former triggers an automatic restart.
• Never blocks; every posix_spawn/waitpid/signal call happens on the Node Manager's own management loop. That loop is itself liveness-critical for every role the node manages, not exempt from the thread-role contract — a stall here delays supervision of every other role on the same node, which is exactly why HeartbeatListener and PortCheckMonitor are both required to be non-blocking (see Project Invariants Checked).

Dependencies

• Inbound: main.c's startup sequence (P0), LivenessMonitor unresponsive-component signal (P1), SchedulerDesiredStatePolicy's desired-state signal (P1).
• Outbound: the host OS process API (posix_spawn/waitpid/kill) (P0).

Contracts: Service [x] / State [x]

Service Interface

/* src/nodemgr/include/component_supervisor.h */

typedef enum {
    XCE_ROLE_CONTROLLER,
    XCE_ROLE_SCHEDULER,
    XCE_ROLE_WORKER,
    XCE_ROLE_APISERVER,
    XCE_ROLE_REVERSE_PROXY,
} xce_component_role_t;

/* Ensures a component for the given role is running (spawns if not, adopts
 * an already-running matching process on first call after Node Manager
 * startup). Idempotent: a role already running or already starting is a
 * no-op. Non-blocking. */
int component_supervisor_start(xce_component_role_t role);

/* Requests the running component for the given role be stopped. For the
 * Scheduler, sends the existing graceful-exit wire notice first and waits up
 * to timeout_ms before force-terminating (mirrors scheduler-primary-
 * lifecycle's already-reviewed stop sequence); for other roles, sends a
 * termination signal directly. Non-blocking to the caller. */
int component_supervisor_stop(xce_component_role_t role, uint32_t timeout_ms);

/* Marks the currently-running component for the given role as unresponsive:
 * stops it and starts a replacement, following the same path as an
 * unexpected exit. A no-op if no component is currently owned for that
 * role. */
int component_supervisor_replace(xce_component_role_t role);

• Preconditions: called only from the Node Manager's own single management loop (no concurrent callers).
• Postconditions: the requested action is either already complete or in progress — never silently dropped.
• Invariants: at most one process is owned per role at any time; a stop-then-start for the same role always completes the stop's reap before a new start begins.

Implementation Notes

• Integration: SchedulerDesiredStatePolicy's output and LivenessMonitor's unresponsive signal are the only two inbound triggers besides initial startup — both non-blocking calls into this component's own loop.
• Validation: an operator-visible log line on every start failure, restart, adoption, and forced-kill-on-timeout (Requirements 1.6, 2.4, 4.6, 7.2).
• Risks: sizing the graceful-stop timeout for the Scheduler is an implementation-time tuning decision (same posture as the superseded design).

HeartbeatListener

Field	Detail
Intent	Receive heartbeat datagrams from every locally-running managed component and track per-role liveness/status/progress
Requirements	4.1, 4.2, 4.3, 4.4, 4.6, 5.1

Responsibilities & Constraints

• Binds one SOCK_DGRAM Unix domain socket per node; every datagram is self-identifying (role + pid fields in the payload — see Data Models), so no socket-level peer identity is required.
• Tracks, per role: last-seen wall-clock time, last-seen sequence number, and (for the Controller) the last-reported status byte (primary/standby).
• Treats a role as unresponsive when its sequence number has not advanced within the operator-configurable threshold — this single check covers both "no datagrams arriving at all" (process dead) and "datagrams arriving but the sequence is frozen" (a heartbeat call site that keeps executing on a thread independent of the wedged work, which is exactly why component integration points are constrained to each component's own real work loop — see Architecture).
• Never blocks: reads from the socket are handled on the Node Manager's own loop; a full or absent socket buffer on the sending side is the sender's problem (see libxce heartbeat client), never this component's.

Dependencies

• Inbound: libxce heartbeat client sends from the Controller, Scheduler, Worker (P0) and the Apiserver's native Go client (P0).
• Outbound: read by ComponentSupervisor (unresponsive signal, P1) and SchedulerDesiredStatePolicy (Controller status, P0).

Contracts: Service [x] / State [x]

Service Interface

/* src/nodemgr/include/liveness_monitor.h */

typedef struct {
    bool     seen_at_least_once;
    bool     unresponsive;
    uint8_t  last_status;     /* meaningful for XCE_ROLE_CONTROLLER only */
    uint64_t last_seq;
    uint32_t last_pid;
} xce_liveness_t;

/* Returns the current liveness snapshot for the given role. Non-blocking,
 * reads in-memory state only. */
xce_liveness_t liveness_monitor_check(xce_component_role_t role);

• Preconditions: none — safe to call before any heartbeat has ever been received (seen_at_least_once=false).
• Postconditions: reflects the most recently received datagram for that role, or the port-check result for XCE_ROLE_REVERSE_PROXY (see PortCheckMonitor).
• Invariants: unresponsive is only ever set by elapsed-time comparison against the configured threshold, never by a single missed tick.

Implementation Notes

• Integration: PortCheckMonitor implements this same liveness_monitor_check contract for XCE_ROLE_REVERSE_PROXY, so ComponentSupervisor reads liveness through one uniform interface regardless of role.
• Risks: threshold sizing is implementation-time tuning, not a design decision (same posture as the superseded spec's wedge-detection threshold).

PortCheckMonitor (summary)

Implements the same liveness_monitor_check contract as HeartbeatListener, for XCE_ROLE_REVERSE_PROXY only: a periodic TCP connect attempt against the port the Node Role Manifest configures for that role, with no heartbeat payload involved (the Reverse Proxy is off-the-shelf and cannot participate in the heartbeat protocol). A failed or refused connect attempt within the threshold sets unresponsive=true, feeding ComponentSupervisor's replace path exactly as a stalled heartbeat sequence does for other roles (Requirements 4.5, 4.6). The connect is non-blocking (O_NONBLOCK plus a bounded poll/select, not a plain blocking connect()) — a silently-dropping firewall must not stall this check for an OS-level TCP timeout (which can run to minutes), since the Node Manager's single management loop also services every other role on that node; this is the same non-blocking discipline HeartbeatListener's socket handling already applies, made explicit here so it is not accidentally implemented as a blocking call.

SchedulerDesiredStatePolicy

Field	Detail
Intent	The one XCE-domain-specific policy in the Node Manager: turn the Controller's reported role into a Scheduler desired-state decision
Requirements	3.1, 3.2, 3.3, 3.5, 3.6

Responsibilities & Constraints

• Only active on a node whose role includes manager.
• Evaluated on every Node Manager loop tick: desired state is "running" iff liveness_monitor_check(XCE_ROLE_CONTROLLER) reports last_status == PRIMARY and not unresponsive; "stopped" otherwise (standby, or Controller heartbeat overdue).
• Contains no Raft/election logic of its own — it is a pure function of the Controller's already-decided, already-reported state (Out of Boundary: this feature never decides who is primary).

Dependencies

• Inbound: HeartbeatListener (P0).
• Outbound: ComponentSupervisor (P0).

Contracts: Service [x]

Implementation Notes

• Integration: a plain evaluate-and-call function on the Node Manager's own loop; no separate thread.
• Validation: unit-testable in isolation by feeding synthetic xce_liveness_t values.

Scheduler

SchedOwnerWatch

Field	Detail
Intent	Backstop: independently confirm the owning Controller is reachable and primary; exit otherwise, without depending on the Node Manager
Requirements	6.1, 6.2, 6.3

Carried forward unchanged from scheduler-primary-lifecycle's design (same component, same contract, same rationale — a Node Manager outage must not strand an orphaned Scheduler). See that design's SchedOwnerWatch section for the full service interface (sched_owner_watch_start(owner_host, owner_port)); this spec restates it as Requirement 6 rather than re-deriving it, since the mechanism is unchanged.

Implementation Notes

• Integration: replaces the Scheduler's superseded reactive rejoin logic exactly as scheduler-primary-lifecycle already specified (sched_needs_rejoin/sched_on_reconnect/sched_publish_check_rejoin removed, not left dormant).
• This component's existence is independent of who starts the Scheduler — it verifies the Scheduler's continued right to run against its one pinned Controller regardless of whether xce-node-manager or (in the superseded design) the Controller itself did the spawning.

Data Models

No durable cluster state (WAL, checkpoint, replicated config) is introduced or modified by this feature — the heartbeat protocol and the Node Manager's own bookkeeping are both process-lifetime or node-local-file concerns, not cluster state, so the project's cluster-wide save/restore field-completeness review does not apply. Two local persistence artifacts are introduced, and each is enumerated field-by-field below per the project's persistence-completeness review discipline:

Heartbeat wire message (in-memory only, never persisted)

Field	Type	Meaning
magic	uint32	Protocol identifier + version, rejects mismatched/garbage datagrams
role	uint8 (enum)	XCE_ROLE_CONTROLLER / SCHEDULER / WORKER / APISERVER
status	uint8	Meaningful for XCE_ROLE_CONTROLLER only: primary or standby; reserved (zero) for other roles
pid	uint32	Sender's process id — used by ComponentSupervisor to cross-check against its own spawn/adoption records
seq	uint64	Monotonically increasing counter, incremented once per call from the component's own loop iteration; doubles as the real-progress signal for the Scheduler (Requirement 4.3)

Process Record (one small local file per managed role, read/written by xce-node-manager only)

Field	Captured on write	Restored on read	Notes
role	yes	yes	Which managed role this record describes
pid	yes	yes	Checked against the OS on Node Manager startup before adoption
binary_path	yes	yes	Checked against the running process's executable to reject a coincidentally-reused pid
start_time	yes	yes	Diagnostic only — not used for any adoption decision

A record that is missing, unreadable, or fails the pid/binary_path cross-check is treated as "no component currently running for this role," which leads ComponentSupervisor to start a fresh one (Requirement 7) — never treated as an error requiring operator intervention, since a fresh start is always safe (Out of Boundary items above establish that at most one process per role is a ComponentSupervisor invariant, and each managed component's own listen-port bind-failure is a natural backstop against a true accidental duplicate).

Node Role Manifest (local, static, read at Node Manager startup only)

Logical shape only (concrete file format is an implementation detail): a list of one or more roles (manager, worker, apiserver, reverse-proxy) assigned to this node, and, per role, the pass-through startup parameters that component's own existing configuration mechanism already expects (the Controller's own --peers/cert arguments, the Worker's own arguments, the Apiserver's own config file path, the Reverse Proxy's own config file path). This feature does not invent new per-component configuration — it only decides whether and when to invoke each component with the configuration that component already understands. Deliberately not cluster-distributed: available to xce-node-manager without depending on connectivity to any Controller (Requirement 1.1).

Error Handling

Error Strategy

Every failure in component-lifecycle management follows the project's existing fail-loud-stay-alive philosophy (ha-async-io.md, controller/CLAUDE.md): log an operator-visible error, never crash or hang the reporting component or the Node Manager, and retry where retrying is meaningful.

Error Categories and Responses

• A managed component fails to start (posix_spawn failure, immediate crash): ComponentSupervisor logs an error and retries — continuously for always-on roles, and only while the Controller remains primary for the Scheduler (Requirement 1.5, 3.6).
• A managed component does not exit within the Scheduler's graceful-stop timeout: force-terminated after the bounded wait; the requesting action (a Controller demotion, in the Scheduler's case) never waits on this to finish (Requirement 3.4).
• A managed component is connected/reporting but not progressing (wedged): treated as unresponsive once its sequence number stalls past the threshold, replaced, with an operator-visible log (Requirement 4.2, 4.4, 4.6).
• A component cannot reach xce-node-manager's heartbeat socket (absent, full buffer, permission issue): the sending component logs nothing beyond its own existing conventions and continues operating normally — this is an expected condition (no Node Manager present, e.g. during isolated testing) and never affects the component's own responsibilities (Requirement 5.2, 5.3).
• The Node Manager itself cannot determine whether a recorded process is genuinely still the one it manages (Process Record pid reused by an unrelated process): treated as "not running," a fresh component is started (Requirement 7.1).

Monitoring

All new log lines (start failure, forced-kill-on-timeout, unresponsive-replacement, adoption-on-restart) use each component's own existing logging conventions (the Controller's WARN/ERROR macros, the Node Manager's equivalent) — no new logging mechanism is introduced.

Testing Strategy

Unit Tests

• component_supervisor_start adopts an already-running, matching process instead of spawning a duplicate when a valid Process Record exists, and starts fresh when the record is missing/stale/mismatched — directly exercises Requirement 7.1.
• component_supervisor_start/_replace distinguishes an intentional stop from an unexpected exit, restarting only the latter — exercises Requirement 2.1-2.3.
• SchedulerDesiredStatePolicy evaluates "running" only when the Controller's last-reported status is primary and not unresponsive, "stopped" otherwise (standby, overdue, or never-seen) — exercises Requirement 3.1-3.3.
• liveness_monitor_check (heartbeat-based) marks a role unresponsive only once its sequence number has stalled past the threshold, not on a single delayed tick — exercises Requirement 4.2.
• xce_heartbeat_report's send path never blocks when the receiving socket is absent or its buffer is full — exercises Requirement 5.1, 5.2.

Integration Tests

• A real 3-node Controller cluster promotion results in xce-node-manager starting a Scheduler process on the newly-primary node with no manual step, and the prior primary's xce-node-manager stopping its Scheduler within the bounded timeout — exercises Requirement 3.1-3.4 end to end.
• A hard-killed primary Controller (bug-1760's original repro) results in the orphaned Scheduler self-exiting via SchedOwnerWatch even before xce-node-manager would otherwise notice, and the newly-elected primary's xce-node-manager starting a fresh Scheduler that successfully dispatches a job submitted after the failover — exercises Requirement 6.1-6.3 and directly closes bug-1760.
• A Scheduler artificially wedged after dequeuing a job (e.g. blocked inside the allocator, past the cbuf_timedget() call) is detected as unresponsive and replaced; a Scheduler left genuinely idle (no jobs submitted) for longer than the threshold is NOT falsely flagged, confirming both halves of Requirement 4.3 — the wedge case and the false-positive-avoidance case the naive design would have failed.
• A xce-node-manager process killed and restarted while its Controller and Scheduler keep running adopts both without starting duplicates — exercises Requirement 7.1.
• A job dispatched by a Scheduler that is then killed mid-decision (before its dispatch is durably recorded) remains available for the replacement Scheduler to consider — exercises Requirement 3.5.

E2E / Soak Tests

• Extend .claude/skills/xce-test-ha/run-ha-soak.sh's sequential and chaotic modes to assert a working Scheduler, Worker, and Apiserver are present after every round with no harness-side start/stop action for any of them — replaces this session's own bug-1760 workaround, which becomes dead code once this feature ships (Requirement 8.2).
• A rapid sequence of Controller promotions/demotions produces no more than one Scheduler spawn/stop cycle per genuine promotion, with no orphaned or duplicate processes remaining after the sequence settles (Requirement 3.6, 7.1).

Security Considerations

Privilege model is role-conditional, not uniform. xce-deploy-ha's own tooling documents that the real (non-virtual) Worker daemon (xcecd, as opposed to the xcecd_virtual test double) runs as root — "plain kill can't touch it," requiring sudo. xce-node-manager must therefore run as root on any node whose assigned role includes worker, so ComponentSupervisor's posix_spawn/kill/waitpid calls against xcecd need no special-cased privilege escalation; on a node whose roles are limited to manager/apiserver/reverse-proxy, xce-node-manager runs as an ordinary service account. This is a per-node deployment decision (how the host's own service manager launches xce_nodemgr), not a per-role runtime toggle inside the binary itself — the Node Role Manifest already fixes a node's roles at Node Manager startup, so the required privilege level is knowable at that same point.

The heartbeat socket is a new local IPC surface whose payload (in particular the Controller's reported primary/standby status) influences whether xce-node-manager starts or stops a component — a local process able to write to it could send spoofed reports. Consistent with product.md's already-locked decision to trust the OS boundary rather than add application-layer encryption (chmod 0700, no new crypto), the heartbeat socket is created with permissions restricting it to the OS user xce-node-manager itself runs as on that node (root on a worker-role node, matching xcecd's own existing requirement; an ordinary service account otherwise) — the same trust boundary the WAL/checkpoint directories already rely on. No new authentication mechanism is introduced; a host already compromised at that privilege level can already directly signal/kill the same processes this feature manages, so the socket adds no new privilege-escalation surface beyond what local code execution at that level already grants. On a worker-role node specifically, running xce-node-manager (and therefore the socket) as root is a direct consequence of xcecd's own pre-existing requirement, not a new exposure this feature introduces.

Migration Strategy

flowchart TB
    Phase1[Phase 1 build xce nodemgr and the libxce heartbeat primitive behind no behavior change]
    Phase2[Phase 2 wire heartbeat calls into controller scheduler worker apiserver loops]
    Phase3[Phase 3 add the entry.sh manager case and update xce deploy ha and xce test ha tooling]
    Phase4[Phase 4 cut over real deployments to node manager driven startup]
    Phase1 --> Phase2 --> Phase3 --> Phase4

• Rollback trigger: if a real deployment depends on placing a component on a host with no Node Manager present for reasons not previously surfaced, that is a genuine requirements gap discovered late — the fix is to pause before Phase 4 (cutover) rather than force it through, since Phase 4 is the first phase that changes how a real deployment is operated.
• Validation checkpoint: after Phase 2, run the soak-harness scenario in Testing Strategy's E2E section before proceeding to Phase 3/4, since it is the closest thing to a direct regression test for bug-1760 itself.

Open Questions / Risks

• The exact Node Role Manifest file format (YAML to match src/configs/*.yaml's existing convention, vs. a simpler key-value form) is left to implementation — an additive, appropriately implementation-level decision, not a design decision, since Data Models already fixes its logical shape.
• The precise heartbeat interval and "no progress" threshold (Requirements 4.2, 4.3) are implementation-time tuning decisions, following the same posture the superseded spec already took for its own liveness thresholds.
• tools/appimage/entry.sh bundles haproxy today with no dispatch case reaching it at all — this feature is what first wires it up (via xce-node-manager invoking it directly for the reverse-proxy role), which is a pre-existing gap this design closes as a side effect rather than a pre-existing behavior it must preserve.

Resolved during design review (kept here for traceability, not left as open items): the Scheduler round-loop idle-cadence question was confirmed against the actual code (cbuf_get()'s unbounded wait) and resolved via cbuf_timedget() — see Architecture and File Structure Plan; it does not need re-verification at implementation time. The Worker-node root-privilege gap was resolved as a role-conditional privilege model — see Security Considerations.