lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(xce-node-manager): add spec (requirements/design approved)

Browse source 
Successor to scheduler-primary-lifecycle: a new per-node xce-node-manager
component (one per host, role-assigned: manager/worker/apiserver/
reverse-proxy) owns starting/stopping XCE components for its node's
role -- including starting/stopping the scheduler on a manager node in
reaction to that node's controller reporting itself primary vs.
standby over a local heartbeat (Unix domain socket, client logic
shared via libxce) -- rather than the controller directly
spawning/killing its own scheduler process. Carries forward
scheduler-primary-lifecycle's underlying requirements (bug-1760's
reactive-rejoin deadlock, wedge detection, graceful shutdown,
controller-resilience invariants).

Design review (kiro-validate-design) fixed 3 issues before GO: a
uniform-single-OS-user assumption that was factually wrong given
xce-deploy-ha's real worker daemon runs as root (resolved: root only
on worker-role nodes); Requirement 4.3's wedge-detection mechanism
verified directly against the actual scheduler code
(sched_job_picker's cbuf_get() does an unbounded wait when idle,
confirming the naive design would false-positive -- fixed by swapping
to the existing cbuf_timedget() bounded-deadline variant); and
PortCheckMonitor's TCP connect made explicitly non-blocking to match
HeartbeatListener's own discipline. Tasks not yet generated.

Checkpointing uncommitted spec work found in this shared worktree.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
This commit is contained in:
Rongsong Shen 2026-09-03 09:13:38 +00:00
parent 6bb37cf470
commit 177e9b292a
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
4 changed files with 693 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

488
.kiro/specs/xce-node-manager/design.md Normal file
View file

@ -0,0 +1,488 @@
## Overview
**Purpose**: This feature introduces `xce-node-manager`, a per-node daemon that starts and supervises the XCE component(s) appropriate to a node's assigned role — replacing today's model where each of the Controller, Scheduler, Worker, and Apiserver is started and supervised as an independent, manually-managed process. On a manager-role node it additionally binds the Scheduler's own lifetime to its co-located Controller's primary/standby status, closing bug-1760 (a hard-killed primary's Scheduler never rejoined the newly-elected primary, leaving jobs stuck PEND indefinitely).
**Users**: XCE cluster operators, and transitively every job submitted to the cluster, benefit from never having to manually start a component after a deploy or notice/intervene after a Controller failover or a component crash.
**Impact**: Introduces one new top-level daemon (`xce_nodemgr`) and one new local, same-host wire protocol (a heartbeat/liveness report) consumed by the Controller, Scheduler, Worker, and Apiserver. Supersedes `.kiro/specs/scheduler-primary-lifecycle/`'s mechanism (a Controller directly `posix_spawn`-ing and killing its own Scheduler child) — that responsibility moves to `xce-node-manager`, driven by the Controller's own reported primary/standby state rather than by the Controller managing a child process itself.
### Goals
- A node's assigned role(s) determine which XCE component(s) start on it, with no manual per-component start step.
- A component that should always be running (Controller, Worker, Apiserver, Reverse Proxy) is restarted automatically if it exits unexpectedly.
- The Scheduler on a manager node starts the moment its co-located Controller becomes primary and stops promptly when that Controller stops being primary or becomes unreachable — with no cross-Controller reconnect logic in the Scheduler.
- A component that is alive but not doing useful work (a wedged Scheduler round loop, in particular) is distinguished from a healthy one and replaced.
- `xce-node-manager`'s own presence, absence, or failure can never compromise a Controller's own availability or HA correctness.
- Existing deployment tooling and specs that assume independently-started components are identified and reconciled.
### Non-Goals
- Supervising `xce-node-manager` itself (delegated to the host's existing process-supervision facilities).
- Any change to the Reverse Proxy's own configuration, routing rules, or request handling.
- Any change to how a Worker's or Apiserver's own client-side connection discovers/reconnects to whichever Controller is currently primary (owned by the existing/adjacent client-failover mechanism).
- Any change to RBAC, authentication, or job/queue/scheduling decision logic.
- Standby/redundant Scheduler processes or a lease-election mechanism for Schedulers (superseded design's rejection still holds — piggybacking on the Controller's own already-hardened Raft election remains the model, just triggered one layer up).
- A new operator-facing remote management API for `xce-node-manager` beyond existing logging conventions.
- The Scheduler's simulation mode (`sim.c`'s driver, distinct from the live driver's `xcecli`/Controller connection) — a simulation-mode Scheduler is a standalone, manually-invoked tool with no real Controller or primary/standby concept, and continues to run entirely independently of `xce-node-manager`, exactly as it does today.
## Boundary Commitments
### This Spec Owns
- Reading a node's local role assignment and starting the corresponding XCE component(s).
- Continuous supervision (unexpected-exit detection, automatic restart) of every component it starts.
- The decision of *when* the Scheduler on a manager node should be running, derived from its co-located Controller's self-reported primary/standby status.
- The heartbeat/liveness wire format and the `libxce` client primitive components use to report it.
- Distinguishing "alive" from "making real progress" for components where that distinction matters (the Scheduler).
- A distinct liveness mechanism (network port reachability) for the off-the-shelf Reverse Proxy role.
- Recognizing already-running components on its own restart instead of double-spawning them.
- Updating existing deployment tooling (`xce-deploy-ha`, `xce-test-ha`) and the AppImage entry point, and flagging any other spec/tool that assumes an independently-started component.
### Out of Boundary
- The Controller's Raft election, replication, and commit-safety mechanisms — this feature only reads the Controller's already-decided primary/standby role, never influences how that role is decided.
- `ha_dispatch_gate`'s existing role-based write rejection — depended upon (unchanged), not modified.
- The Scheduler's own job/queue/config decision logic (picker, allocator, plugins) — untouched.
- Worker and Apiserver job-execution/API-serving logic — untouched.
- Any lease/renewal protocol or standby-component redundancy.
- Supervising `xce-node-manager`'s own process availability.
- The Scheduler's simulation mode — untouched, unmanaged, and unaware of `xce-node-manager`'s existence; only the live driver (the one that joins a real Controller cluster) is in scope.
### Allowed Dependencies
- `posix_spawn`/`waitpid`/signal (POSIX) — the sole mechanism `xce-node-manager` uses to start/monitor/stop a managed child process, matching the already-reviewed `scheduler-primary-lifecycle` decision (never raw `fork()` — see Technology Stack).
- Unix domain datagram sockets, local to the host — the sole transport for the heartbeat protocol; no cross-host traffic is introduced by this feature.
- The existing controller-scheduler wire connection (unchanged) — the Scheduler's own backstop controller-liveness check (Requirement 6) reuses it exactly as the superseded design specified; this feature adds no new field to that connection's schema.
- `libxce` (`src/lib/`) — the heartbeat client primitive is added here so the Controller, Scheduler, and Worker (all of which already link it) get it for free.
### Revalidation Triggers
- Any change to how the Controller exposes its own primary/standby role (today the atomic `ha->role` field) — the heartbeat report's Controller-side source of truth.
- Any change to the Scheduler's round/picker loop structure that removes or alters its bounded-cadence-even-when-idle property — the property Requirement 4.3's progress signal depends on.
- Any other spec or tool that starts the Controller, Scheduler, Worker, or Apiserver directly (bypassing `xce-node-manager`) must be re-checked against this design's role-driven startup contract.
- Introduction of a genuine cross-host or cross-version consumer of the heartbeat protocol would invalidate the "same-host, same-build-version" reasoning behind skipping Cap'n Proto for it (see Technology Stack).
### Project Invariants Checked
This design adds a small periodic reporting point to the Controller and touches the Scheduler's process-lifecycle surface, so it is checked directly against `ha-async-io.md` and `product.md`'s locked decisions:
- **Thread-role contract** (no blocking work on iomgr/delaymgr): satisfied — the Controller's heartbeat report is a single non-blocking, connectionless (`SOCK_DGRAM`) local send with no retry/backoff, issued from the same periodic point on every tick; a full/unreachable receiver drops the datagram rather than blocking the sender (see Error Handling). The equivalent hazard on the Node Manager's own side — its single management loop must not block on `HeartbeatListener`'s socket reads or `PortCheckMonitor`'s TCP connects, since either would stall supervision of every role on that node, not just the one being checked — is satisfied the same way: non-blocking socket handling throughout (see PortCheckMonitor's Implementation Note).
- **Controller-never-self-terminate**: satisfied — a heartbeat-send failure is not logged as an error path that can escalate to a crash/hang/exit; it is a silent, expected condition when no `xce-node-manager` is present (Requirement 5.2/5.3).
- **No `fork()` for snapshots" (generalized precedent)**: `xce-node-manager` spawns every managed component via `posix_spawn`, never raw `fork()+exec()` — the same reasoning `scheduler-primary-lifecycle`'s design review already established (a lock held by another thread at fork time can deadlock a child that touches malloc/libxce before `execve()`) applies identically here, and is called out explicitly because `src/worker/src/process_worker_manager.c`'s existing `process_manager_start_worker()` is a real, already-shipped counter-example using raw `fork()+execl()` — that pattern is not to be imitated.
- **`ha_dispatch_gate` role-based write rejection**: depended upon, unchanged — this design does not touch it.
- **Scheduler core/driver separation** (`scheduler-core` spec's established invariant: `xce_sched_core_init()`'s core — `core.c`/`core_events.c`/`core_session.c`/`core_clock.c` — performs zero controller I/O): satisfied — the heartbeat call site is `sched_job_picker` in `scheduler.c`, the **live** driver layer that already owns the picker/allocator thread lifecycle and the `xcecli` connection, exactly where the `cbuf_timedget()` change also lands. The pure decision engine in `core.c` and siblings is untouched by this feature; the Scheduler's outside-facing I/O surface (Controller connection, and now the local heartbeat) remains confined to the live driver, matching the existing architecture rather than introducing a second I/O path into the core. The simulation driver (`sim.c`) is a distinct code path from `scheduler.c`'s live driver and is not touched — a simulation-mode Scheduler has no heartbeat call site added and is never started, stopped, or monitored by `xce-node-manager` (see Non-Goals).
## Architecture
### Existing Architecture Analysis
- No component in this codebase today starts or supervises another XCE daemon based on a declared role. `tools/appimage/entry.sh` is the closest thing to an orchestrator, and it only execs one named binary in the foreground per invocation (`xce controller`, `xce worker`, ...) — no supervision, no multi-process coordination, and no case for the already-bundled `haproxy` at all.
- The only existing precedent for one XCE daemon supervising a child process is the Worker's job-execution supervision (`process_manager.c`, spawning per-job `xcecd_worker` processes via raw `fork()+execl()`) — job-execution-specific, and using a spawn mechanism this design deliberately does not reuse (see Project Invariants Checked).
- The Controller already exposes its own primary/standby state as an atomic field (`ha->role`) and already runs a non-blocking, delaymgr-driven periodic task pattern (`primary_heartbeat_fn`) proving this class of "cheap periodic local report" is safe and idiomatic here — though that specific function is unsuitable to extend directly (see `research.md`).
- The Scheduler's `sched_job_picker` thread (`scheduler.c`) calls `cbuf_get()` (`src/lib/mylock.c:144`) to pull the next job, which does an **unbounded** `pthread_cond_wait` when the queue is empty — confirmed by reading the code, not assumed. A heartbeat call placed naively at the top of this loop would never fire during a genuinely idle cluster, since the thread simply never wakes. `cbuf_timedget()` (same file), an already-existing bounded-deadline variant used elsewhere in the codebase, is the fix this design adopts (see Scheduler integration in File Structure Plan).
- `ha_dispatch_gate` already enforces that only a primary Controller accepts Scheduler write opcodes — unchanged, and still the reason this feature is purely about availability, not correctness.
### Architecture Pattern & Boundary Map
```mermaid
graph TB
subgraph ManagerNode
NM1[Node Manager]
Ctrl[Controller]
Sched[Scheduler Process]
NM1 -->|starts and supervises| Ctrl
NM1 -->|starts stops on Controller role| Sched
Ctrl -.->|heartbeat role status| NM1
Sched -.->|heartbeat round progress| NM1
Sched -->|pinned backstop liveness check| Ctrl
end
subgraph WorkerNode
NM2[Node Manager]
Wrk[Worker Process]
NM2 -->|starts and supervises| Wrk
Wrk -.->|heartbeat| NM2
end
subgraph ApiserverNode
NM3[Node Manager]
Api[Apiserver Process]
NM3 -->|starts and supervises| Api
Api -.->|heartbeat| NM3
end
subgraph ProxyNode
NM4[Node Manager]
Proxy[Reverse Proxy Process]
NM4 -->|starts and supervises| Proxy
NM4 -.->|port reachability check| Proxy
end
Ctrl <-->|raft election and replication existing| OtherControllers[Other Controller Instances]
```
**Architecture Integration**:
- Selected pattern: a per-node supervisor process (`xce-node-manager`) that owns starting, restarting, and — for the Scheduler specifically — stopping its managed children, using a generic desired-state-plus-liveness engine with one role-specific policy adapter (Scheduler-bound-to-Controller-role).
- Domain/feature boundaries: the generic supervision engine (`ComponentSupervisor`) is separate from the liveness-signal sources (`LivenessMonitor`, two implementations: heartbeat-based and port-check-based), which is separate from the one XCE-domain-specific policy (`SchedulerDesiredStatePolicy`) — the only piece of `xce-node-manager` that knows anything about Controller primary/standby semantics.
- Existing patterns preserved: `posix_spawn`-based child process ownership (the already-reviewed decision from `scheduler-primary-lifecycle`); the Scheduler's own independent controller-liveness backstop check (`SchedOwnerWatch`, carried forward from that same superseded design, unchanged in behavior); the existing controller-scheduler wire connection and `ha_dispatch_gate` safety argument, both untouched.
- New components rationale: `xce-node-manager` is a wholly new domain (cross-component process lifecycle) with no existing owner; the heartbeat primitive is new but deliberately minimal (see Technology Stack) and lives in `libxce` alongside the other cross-daemon primitives it already provides.
- Steering compliance: respects `ha-async-io.md`'s thread-role contract (the Controller's heartbeat report is a single non-blocking local send, never inline blocking work on iomgr/delaymgr) and the controller-never-self-terminate invariant (a heartbeat failure is never escalated to a crash/hang/exit); respects `structure.md`'s component-per-directory convention for the new `src/nodemgr/` daemon.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Node Manager process management | POSIX `posix_spawn`/`waitpid`/signal (C, `src/nodemgr/src/component_supervisor.c`) | Start, monitor, restart, and stop every managed component | Same `posix_spawn`-over-`fork()` rationale already established by `scheduler-primary-lifecycle`'s design review; see Project Invariants Checked |
| Heartbeat transport | Unix domain datagram socket (`SOCK_DGRAM`), local host only | Carries the liveness/role-status report from each managed component to its node's Node Manager | Deliberately not Cap'n Proto — see Technology Stack Deviation below |
| Heartbeat wire format | Fixed-layout binary struct (magic, version, role, status, pid, sequence), `src/include/xce_heartbeat.h` | Self-identifying payload (receiver does not depend on socket peer identity) | One sequence counter serves as both liveness proof (all roles) and real-progress proof (Scheduler specifically) — see Data Models |
| Heartbeat client (C) | `libxce`, `src/lib/heartbeat.c` | Used by the Controller, Scheduler, and Worker, each already linking `libxce` | Reused as-is by all three, zero new link-time cost |
| Heartbeat client (Go) | Native Go, `src/frontend/apiserver/heartbeat/` | Used by the Apiserver via a small ticker goroutine | Deliberately not routed through the existing CGO bridge — see Technology Stack Deviation below |
| Reverse Proxy liveness | TCP connect against the proxy's configured port | Liveness check for the off-the-shelf Reverse Proxy role | The Reverse Proxy cannot participate in the heartbeat protocol (not XCE-authored code) |
| Node role assignment | Local file, read at Node Manager startup (format is an implementation detail; see Data Models for its logical shape) | Tells `xce-node-manager` which component(s) to start on this node | Deliberately not cluster-distributed config — a node must know its role before any Controller is reachable |
**Technology Stack Deviation**: `structure.md` states new inter-daemon protocols use Cap'n Proto schemas. This protocol is a deliberate, narrow exception: it is same-host-only (never crosses a network boundary) and always exchanged between binaries from the same build (the Node Manager and the components it manages ship together), so Cap'n Proto's cross-version/cross-language schema-evolution guarantees — the reason `controller.capnp`/`c.capnp` earn their cost for the real cluster wire protocol — buy nothing here. Adopting it for the Apiserver side specifically would also mean a brand-new Go dependency (`go.mod`/`go.sum` confirmed the Apiserver has no existing capnp dependency) purely to encode four fixed-size fields. See `research.md`'s Decision log for the full reasoning.
## File Structure Plan
### Directory Structure
```
src/
├── nodemgr/ # NEW component — the Node Manager daemon
│ ├── meson.build
│ ├── CLAUDE.md
│ ├── include/
│ │ ├── role_manifest.h # Node Role Manifest types + loader interface
│ │ ├── component_supervisor.h # ComponentSupervisor service interface
│ │ ├── liveness_monitor.h # LivenessMonitor interface (heartbeat-based, port-check-based)
│ │ └── scheduler_policy.h # SchedulerDesiredStatePolicy interface
│ ├── src/
│ │ ├── main.c # entry point: load manifest, wire supervisor + monitors, run
│ │ ├── role_manifest.c
│ │ ├── component_supervisor.c # posix_spawn/waitpid/kill, restart-on-exit, adopt-on-restart
│ │ ├── heartbeat_listener.c # LivenessMonitor impl: UDS SOCK_DGRAM server, per-role tracking
│ │ ├── port_check_monitor.c # LivenessMonitor impl: TCP connect against a configured port
│ │ └── scheduler_policy.c # reads Controller status from heartbeat_listener, drives Scheduler desired-state
│ └── testsuite/ # test_component_supervisor.c, test_heartbeat_listener.c, test_scheduler_policy.c, ...
├── lib/
│ └── heartbeat.c # NEW: xce_heartbeat_report() client primitive, wire encode/decode
├── include/
│ └── xce_heartbeat.h # NEW: public heartbeat wire format + client API (see Data Models)
├── controller/
│ └── ha.c # MODIFIED: one new small periodic task, armed unconditionally at controller startup, calls xce_heartbeat_report() with the current atomic role
├── scheduler/
│ └── src/
│ ├── scheduler.c # MODIFIED (driver layer only, core.c untouched): sched_job_picker's cbuf_get() -> cbuf_timedget() with a bounded idle-tick deadline; call xce_heartbeat_report() once per loop iteration on BOTH the real-job path and the timeout path, so the sequence advances on a bounded cadence even when idle
│ └── sched_owner.c # NEW: backstop controller-liveness self-check, carried forward from scheduler-primary-lifecycle's design (Requirement 6)
├── worker/
│ └── src/xcecd_worker.c # MODIFIED: call xce_heartbeat_report() once per main-loop iteration
└── frontend/apiserver/
├── heartbeat/
│ └── heartbeat.go # NEW: native wire-format encoder + ticker goroutine, no CGO
└── cmd/main.go # MODIFIED: starts the heartbeat ticker at apiserver startup
```
### Modified Files (tooling, outside `src/`)
- `tools/appimage/entry.sh` — add a `manager` case that execs `xce_nodemgr`; no other case changes (the Reverse Proxy role is launched by `xce_nodemgr` invoking the already-bundled `haproxy` binary directly, not through `entry.sh`).
- `.claude/skills/xce-deploy-ha/SKILL.md` and `src/tools/deploy-ha.sh` — stop generating independent `start-worker.sh`/`start-scheduler.sh` intended for manual invocation; generate a Node Role Manifest per node instead and launch `xce_nodemgr`.
- `.claude/skills/xce-test-ha/run-ha-soak.sh` — remove this session's own manual `start_worker_and_scheduler()`/`restart_scheduler_for_new_primary()` logic (bug-1760's workaround, now unnecessary); assert a working Scheduler/Worker/Apiserver is present after each round instead.
## System Flows
### Scheduler start/stop bound to Controller role
```mermaid
sequenceDiagram
participant Ctrl as Controller
participant HB as Node Manager HeartbeatListener
participant Pol as SchedulerDesiredStatePolicy
participant Sup as ComponentSupervisor
participant Sched as Scheduler Process
loop every Controller loop iteration
Ctrl->>HB: heartbeat role=controller status=primary or standby seq++
end
HB->>Pol: latest Controller status
alt status is primary and no Scheduler running
Pol->>Sup: desired state Scheduler=running
Sup->>Sched: posix_spawn passing this Controller address
Sched->>Ctrl: join using pinned single controller address existing behavior
else status is standby or Controller heartbeat overdue
Pol->>Sup: desired state Scheduler=stopped
Sup->>Sched: terminate
end
Sched->>Ctrl: independent backstop liveness check existing connection
```
The Node Manager's decision (top path) and the Scheduler's own backstop check (bottom, Requirement 6) are independent — either one alone is sufficient to stop a Scheduler whose Controller is no longer primary, so a Node Manager outage does not strand a Scheduler indefinitely.
### Wedge detection and replacement (generic — shown for the Scheduler)
```mermaid
sequenceDiagram
participant Sched as Scheduler Process
participant HB as Node Manager HeartbeatListener
participant Sup as ComponentSupervisor
loop every round-loop iteration whether or not work is found
Sched->>HB: heartbeat role=scheduler seq++
end
HB->>HB: observe seq has not advanced past threshold
HB->>Sup: component unresponsive
Sup->>Sched: terminate
Sup->>Sched: posix_spawn a replacement
```
The same path applies to the Worker and Apiserver using their own heartbeat sequence; the Reverse Proxy uses the `port_check_monitor` liveness source instead of `heartbeat_listener` but feeds the same `ComponentSupervisor` unresponsive-replace action. For the Scheduler specifically, "every round-loop iteration" means both the real-job path and the bounded-idle-timeout path of `cbuf_timedget()` (see Architecture and File Structure Plan) — the sequence still advances on a healthy idle Scheduler, so only a genuine mid-decision wedge (stuck after a job is already dequeued, not merely waiting for one) fails to advance it.
### Node Manager restart: adopt, don't duplicate
```mermaid
sequenceDiagram
participant NM as Node Manager on restart
participant Rec as Process Record file
participant OS as Host OS
participant Sup as ComponentSupervisor
NM->>Rec: read recorded role to pid mapping
NM->>OS: is that pid alive and running the expected binary
alt recorded process still alive and matches
NM->>Sup: adopt existing process for this role
else record missing stale or mismatched
NM->>Sup: start a fresh process for this role
end
```
## Requirements Traceability
| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1, 1.2, 1.3, 1.4, 1.5, 1.6 | Node reads its role(s) locally and starts the matching component(s); errors logged and retried | RoleManifest, ComponentSupervisor | `component_supervisor_start` | Node Manager startup (Architecture) |
| 2.1, 2.2, 2.3, 2.4 | Always-on components restarted on unexpected exit, not on an intentional stop | ComponentSupervisor | `component_supervisor_start`/`_stop` | Wedge/unexpected-exit flow |
| 3.1, 3.2, 3.3, 3.4, 3.5, 3.6 | Scheduler start/stop bound to Controller primary/standby; bounded stop time; no job loss; keep retrying start while primary | ComponentSupervisor, SchedulerDesiredStatePolicy, HeartbeatListener, Controller's existing WAL-commit ownership (unchanged) | `scheduler_policy_evaluate`, `component_supervisor_start`/`_stop` | Scheduler start/stop flow |
| 4.1, 4.2, 4.3, 4.4, 4.5, 4.6 | Heartbeat-based liveness; Scheduler liveness reflects real progress; Reverse Proxy uses port check (4.5); unresponsive components replaced and logged | HeartbeatListener, PortCheckMonitor, `libxce` heartbeat client | `xce_heartbeat_report`, `liveness_monitor_check` | Wedge detection flow |
| 5.1, 5.2, 5.3 | Reporting never blocks a component's own responsibilities; failure to report doesn't affect the component; absent Node Manager doesn't affect Controller HA | `libxce` heartbeat client (non-blocking send), Controller/Scheduler/Worker/Apiserver integration points | `xce_heartbeat_report` | Scheduler start/stop flow, Wedge detection flow |
| 6.1, 6.2, 6.3 | Scheduler independently verifies its Controller and exits on its own if it cannot | SchedOwnerWatch (Scheduler) | `sched_owner_watch_start` | Scheduler start/stop flow |
| 7.1, 7.2 | Node Manager adopts already-running components on its own restart instead of duplicating | ComponentSupervisor, Process Record | `component_supervisor_start` (adoption path) | Node Manager restart flow |
| 8.1, 8.2, 8.3 | Deployment tooling and adjacent specs reconciled; observable component behavior unchanged | `xce-deploy-ha`, `xce-test-ha`, `entry.sh` | n/a (tooling) | Migration Strategy |
## Components and Interfaces
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|---------------|--------------------------|-----------|
| ComponentSupervisor | Node Manager | Start/stop/restart/adopt every managed component via `posix_spawn` | 1, 2, 3, 7 | RoleManifest (P0), LivenessMonitor (P1), SchedulerDesiredStatePolicy (P1) | Service |
| HeartbeatListener | Node Manager | UDS `SOCK_DGRAM` server tracking per-role liveness/sequence/status | 4, 5 | `libxce` heartbeat wire format (P0) | Service, State |
| PortCheckMonitor | Node Manager | TCP-reachability liveness for the Reverse Proxy role | 4.5, 4.6 | none | Service |
| SchedulerDesiredStatePolicy | Node Manager | Translates the Controller's reported status into a Scheduler desired-state decision | 3 | HeartbeatListener (P0) | Service |
| `libxce` heartbeat client | Shared library | Non-blocking heartbeat send, used from each component's own loop | 4, 5 | none | Service |
| SchedOwnerWatch | Scheduler | Backstop: independently verifies the owning Controller and exits on loss | 6 | Existing controller-scheduler connection (P0) | Service |
### Node Manager
#### ComponentSupervisor
| Field | Detail |
|-------|--------|
| Intent | Own the decision of whether each managed component is currently running, and start/stop/restart it accordingly |
| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 2.1, 2.2, 2.3, 2.4, 3.1, 3.4, 3.6, 7.1, 7.2 |
**Responsibilities & Constraints**
- For each role the local Node Role Manifest assigns, holds a desired-state flag: always-running (Worker, Apiserver, Reverse Proxy, Controller) or policy-driven (Scheduler, via `SchedulerDesiredStatePolicy`).
- On its own startup, consults the Process Record for each managed role before spawning — adopts an already-running, matching process instead of starting a duplicate (Requirement 7).
- Detects an unexpected child exit (`waitpid`) and distinguishes it from a stop it requested itself; only the former triggers an automatic restart.
- Never blocks; every `posix_spawn`/`waitpid`/signal call happens on the Node Manager's own management loop. That loop is itself liveness-critical for every role the node manages, not exempt from the thread-role contract — a stall here delays supervision of every other role on the same node, which is exactly why `HeartbeatListener` and `PortCheckMonitor` are both required to be non-blocking (see Project Invariants Checked).
**Dependencies**
- Inbound: `main.c`'s startup sequence (P0), `LivenessMonitor` unresponsive-component signal (P1), `SchedulerDesiredStatePolicy`'s desired-state signal (P1).
- Outbound: the host OS process API (`posix_spawn`/`waitpid`/`kill`) (P0).
**Contracts**: Service [x] / State [x]
##### Service Interface
```c
/* src/nodemgr/include/component_supervisor.h */
typedef enum {
XCE_ROLE_CONTROLLER,
XCE_ROLE_SCHEDULER,
XCE_ROLE_WORKER,
XCE_ROLE_APISERVER,
XCE_ROLE_REVERSE_PROXY,
} xce_component_role_t;
/* Ensures a component for the given role is running (spawns if not, adopts
* an already-running matching process on first call after Node Manager
* startup). Idempotent: a role already running or already starting is a
* no-op. Non-blocking. */
int component_supervisor_start(xce_component_role_t role);
/* Requests the running component for the given role be stopped. For the
* Scheduler, sends the existing graceful-exit wire notice first and waits up
* to timeout_ms before force-terminating (mirrors scheduler-primary-
* lifecycle's already-reviewed stop sequence); for other roles, sends a
* termination signal directly. Non-blocking to the caller. */
int component_supervisor_stop(xce_component_role_t role, uint32_t timeout_ms);
/* Marks the currently-running component for the given role as unresponsive:
* stops it and starts a replacement, following the same path as an
* unexpected exit. A no-op if no component is currently owned for that
* role. */
int component_supervisor_replace(xce_component_role_t role);
```
- Preconditions: called only from the Node Manager's own single management loop (no concurrent callers).
- Postconditions: the requested action is either already complete or in progress — never silently dropped.
- Invariants: at most one process is owned per role at any time; a stop-then-start for the same role always completes the stop's reap before a new start begins.
**Implementation Notes**
- Integration: `SchedulerDesiredStatePolicy`'s output and `LivenessMonitor`'s unresponsive signal are the only two inbound triggers besides initial startup — both non-blocking calls into this component's own loop.
- Validation: an operator-visible log line on every start failure, restart, adoption, and forced-kill-on-timeout (Requirements 1.6, 2.4, 4.6, 7.2).
- Risks: sizing the graceful-stop timeout for the Scheduler is an implementation-time tuning decision (same posture as the superseded design).
#### HeartbeatListener
| Field | Detail |
|-------|--------|
| Intent | Receive heartbeat datagrams from every locally-running managed component and track per-role liveness/status/progress |
| Requirements | 4.1, 4.2, 4.3, 4.4, 4.6, 5.1 |
**Responsibilities & Constraints**
- Binds one `SOCK_DGRAM` Unix domain socket per node; every datagram is self-identifying (role + pid fields in the payload — see Data Models), so no socket-level peer identity is required.
- Tracks, per role: last-seen wall-clock time, last-seen sequence number, and (for the Controller) the last-reported status byte (primary/standby).
- Treats a role as unresponsive when its sequence number has not advanced within the operator-configurable threshold — this single check covers both "no datagrams arriving at all" (process dead) and "datagrams arriving but the sequence is frozen" (a heartbeat call site that keeps executing on a thread independent of the wedged work, which is exactly why component integration points are constrained to each component's own real work loop — see Architecture).
- Never blocks: reads from the socket are handled on the Node Manager's own loop; a full or absent socket buffer on the sending side is the sender's problem (see `libxce` heartbeat client), never this component's.
**Dependencies**
- Inbound: `libxce` heartbeat client sends from the Controller, Scheduler, Worker (P0) and the Apiserver's native Go client (P0).
- Outbound: read by `ComponentSupervisor` (unresponsive signal, P1) and `SchedulerDesiredStatePolicy` (Controller status, P0).
**Contracts**: Service [x] / State [x]
##### Service Interface
```c
/* src/nodemgr/include/liveness_monitor.h */
typedef struct {
bool seen_at_least_once;
bool unresponsive;
uint8_t last_status; /* meaningful for XCE_ROLE_CONTROLLER only */
uint64_t last_seq;
uint32_t last_pid;
} xce_liveness_t;
/* Returns the current liveness snapshot for the given role. Non-blocking,
* reads in-memory state only. */
xce_liveness_t liveness_monitor_check(xce_component_role_t role);
```
- Preconditions: none — safe to call before any heartbeat has ever been received (`seen_at_least_once=false`).
- Postconditions: reflects the most recently received datagram for that role, or the port-check result for `XCE_ROLE_REVERSE_PROXY` (see PortCheckMonitor).
- Invariants: `unresponsive` is only ever set by elapsed-time comparison against the configured threshold, never by a single missed tick.
**Implementation Notes**
- Integration: `PortCheckMonitor` implements this same `liveness_monitor_check` contract for `XCE_ROLE_REVERSE_PROXY`, so `ComponentSupervisor` reads liveness through one uniform interface regardless of role.
- Risks: threshold sizing is implementation-time tuning, not a design decision (same posture as the superseded spec's wedge-detection threshold).
#### PortCheckMonitor (summary)
Implements the same `liveness_monitor_check` contract as `HeartbeatListener`, for `XCE_ROLE_REVERSE_PROXY` only: a periodic TCP connect attempt against the port the Node Role Manifest configures for that role, with no heartbeat payload involved (the Reverse Proxy is off-the-shelf and cannot participate in the heartbeat protocol). A failed or refused connect attempt within the threshold sets `unresponsive=true`, feeding `ComponentSupervisor`'s replace path exactly as a stalled heartbeat sequence does for other roles (Requirements 4.5, 4.6). The connect is non-blocking (`O_NONBLOCK` plus a bounded `poll`/`select`, not a plain blocking `connect()`) — a silently-dropping firewall must not stall this check for an OS-level TCP timeout (which can run to minutes), since the Node Manager's single management loop also services every other role on that node; this is the same non-blocking discipline `HeartbeatListener`'s socket handling already applies, made explicit here so it is not accidentally implemented as a blocking call.
#### SchedulerDesiredStatePolicy
| Field | Detail |
|-------|--------|
| Intent | The one XCE-domain-specific policy in the Node Manager: turn the Controller's reported role into a Scheduler desired-state decision |
| Requirements | 3.1, 3.2, 3.3, 3.5, 3.6 |
**Responsibilities & Constraints**
- Only active on a node whose role includes manager.
- Evaluated on every Node Manager loop tick: desired state is "running" iff `liveness_monitor_check(XCE_ROLE_CONTROLLER)` reports `last_status == PRIMARY` and not `unresponsive`; "stopped" otherwise (standby, or Controller heartbeat overdue).
- Contains no Raft/election logic of its own — it is a pure function of the Controller's already-decided, already-reported state (Out of Boundary: this feature never decides who is primary).
**Dependencies**
- Inbound: `HeartbeatListener` (P0).
- Outbound: `ComponentSupervisor` (P0).
**Contracts**: Service [x]
**Implementation Notes**
- Integration: a plain evaluate-and-call function on the Node Manager's own loop; no separate thread.
- Validation: unit-testable in isolation by feeding synthetic `xce_liveness_t` values.
### Scheduler
#### SchedOwnerWatch
| Field | Detail |
|-------|--------|
| Intent | Backstop: independently confirm the owning Controller is reachable and primary; exit otherwise, without depending on the Node Manager |
| Requirements | 6.1, 6.2, 6.3 |
Carried forward unchanged from `scheduler-primary-lifecycle`'s design (same component, same contract, same rationale — a Node Manager outage must not strand an orphaned Scheduler). See that design's `SchedOwnerWatch` section for the full service interface (`sched_owner_watch_start(owner_host, owner_port)`); this spec restates it as Requirement 6 rather than re-deriving it, since the mechanism is unchanged.
**Implementation Notes**
- Integration: replaces the Scheduler's superseded reactive rejoin logic exactly as `scheduler-primary-lifecycle` already specified (`sched_needs_rejoin`/`sched_on_reconnect`/`sched_publish_check_rejoin` removed, not left dormant).
- This component's existence is independent of who *starts* the Scheduler — it verifies the Scheduler's *continued* right to run against its one pinned Controller regardless of whether `xce-node-manager` or (in the superseded design) the Controller itself did the spawning.
## Data Models
No durable cluster state (WAL, checkpoint, replicated config) is introduced or modified by this feature — the heartbeat protocol and the Node Manager's own bookkeeping are both process-lifetime or node-local-file concerns, not cluster state, so the project's cluster-wide save/restore field-completeness review does not apply. Two local persistence artifacts are introduced, and each is enumerated field-by-field below per the project's persistence-completeness review discipline:
### Heartbeat wire message (in-memory only, never persisted)
| Field | Type | Meaning |
|-------|------|---------|
| magic | uint32 | Protocol identifier + version, rejects mismatched/garbage datagrams |
| role | uint8 (enum) | `XCE_ROLE_CONTROLLER` / `SCHEDULER` / `WORKER` / `APISERVER` |
| status | uint8 | Meaningful for `XCE_ROLE_CONTROLLER` only: primary or standby; reserved (zero) for other roles |
| pid | uint32 | Sender's process id — used by `ComponentSupervisor` to cross-check against its own spawn/adoption records |
| seq | uint64 | Monotonically increasing counter, incremented once per call from the component's own loop iteration; doubles as the real-progress signal for the Scheduler (Requirement 4.3) |
### Process Record (one small local file per managed role, read/written by `xce-node-manager` only)
| Field | Captured on write | Restored on read | Notes |
|-------|--------------------|-------------------|-------|
| role | yes | yes | Which managed role this record describes |
| pid | yes | yes | Checked against the OS on Node Manager startup before adoption |
| binary_path | yes | yes | Checked against the running process's executable to reject a coincidentally-reused pid |
| start_time | yes | yes | Diagnostic only — not used for any adoption decision |
A record that is missing, unreadable, or fails the pid/binary_path cross-check is treated as "no component currently running for this role," which leads `ComponentSupervisor` to start a fresh one (Requirement 7) — never treated as an error requiring operator intervention, since a fresh start is always safe (Out of Boundary items above establish that at most one process per role is a `ComponentSupervisor` invariant, and each managed component's own listen-port bind-failure is a natural backstop against a true accidental duplicate).
### Node Role Manifest (local, static, read at Node Manager startup only)
Logical shape only (concrete file format is an implementation detail): a list of one or more roles (`manager`, `worker`, `apiserver`, `reverse-proxy`) assigned to this node, and, per role, the pass-through startup parameters that component's own existing configuration mechanism already expects (the Controller's own `--peers`/cert arguments, the Worker's own arguments, the Apiserver's own config file path, the Reverse Proxy's own config file path). This feature does not invent new per-component configuration — it only decides *whether and when* to invoke each component with the configuration that component already understands. Deliberately not cluster-distributed: available to `xce-node-manager` without depending on connectivity to any Controller (Requirement 1.1).
## Error Handling
### Error Strategy
Every failure in component-lifecycle management follows the project's existing fail-loud-stay-alive philosophy (`ha-async-io.md`, `controller/CLAUDE.md`): log an operator-visible error, never crash or hang the reporting component or the Node Manager, and retry where retrying is meaningful.
### Error Categories and Responses
- **A managed component fails to start** (`posix_spawn` failure, immediate crash): `ComponentSupervisor` logs an error and retries — continuously for always-on roles, and only while the Controller remains primary for the Scheduler (Requirement 1.5, 3.6).
- **A managed component does not exit within the Scheduler's graceful-stop timeout**: force-terminated after the bounded wait; the requesting action (a Controller demotion, in the Scheduler's case) never waits on this to finish (Requirement 3.4).
- **A managed component is connected/reporting but not progressing (wedged)**: treated as unresponsive once its sequence number stalls past the threshold, replaced, with an operator-visible log (Requirement 4.2, 4.4, 4.6).
- **A component cannot reach `xce-node-manager`'s heartbeat socket** (absent, full buffer, permission issue): the sending component logs nothing beyond its own existing conventions and continues operating normally — this is an expected condition (no Node Manager present, e.g. during isolated testing) and never affects the component's own responsibilities (Requirement 5.2, 5.3).
- **The Node Manager itself cannot determine whether a recorded process is genuinely still the one it manages** (Process Record pid reused by an unrelated process): treated as "not running," a fresh component is started (Requirement 7.1).
### Monitoring
All new log lines (start failure, forced-kill-on-timeout, unresponsive-replacement, adoption-on-restart) use each component's own existing logging conventions (the Controller's `WARN`/`ERROR` macros, the Node Manager's equivalent) — no new logging mechanism is introduced.
## Testing Strategy
### Unit Tests
- `component_supervisor_start` adopts an already-running, matching process instead of spawning a duplicate when a valid Process Record exists, and starts fresh when the record is missing/stale/mismatched — directly exercises Requirement 7.1.
- `component_supervisor_start`/`_replace` distinguishes an intentional stop from an unexpected exit, restarting only the latter — exercises Requirement 2.1-2.3.
- `SchedulerDesiredStatePolicy` evaluates "running" only when the Controller's last-reported status is primary and not unresponsive, "stopped" otherwise (standby, overdue, or never-seen) — exercises Requirement 3.1-3.3.
- `liveness_monitor_check` (heartbeat-based) marks a role unresponsive only once its sequence number has stalled past the threshold, not on a single delayed tick — exercises Requirement 4.2.
- `xce_heartbeat_report`'s send path never blocks when the receiving socket is absent or its buffer is full — exercises Requirement 5.1, 5.2.
### Integration Tests
- A real 3-node Controller cluster promotion results in `xce-node-manager` starting a Scheduler process on the newly-primary node with no manual step, and the prior primary's `xce-node-manager` stopping its Scheduler within the bounded timeout — exercises Requirement 3.1-3.4 end to end.
- A hard-killed primary Controller (bug-1760's original repro) results in the orphaned Scheduler self-exiting via `SchedOwnerWatch` even before `xce-node-manager` would otherwise notice, and the newly-elected primary's `xce-node-manager` starting a fresh Scheduler that successfully dispatches a job submitted after the failover — exercises Requirement 6.1-6.3 and directly closes bug-1760.
- A Scheduler artificially wedged after dequeuing a job (e.g. blocked inside the allocator, past the `cbuf_timedget()` call) is detected as unresponsive and replaced; a Scheduler left genuinely idle (no jobs submitted) for longer than the threshold is NOT falsely flagged, confirming both halves of Requirement 4.3 — the wedge case and the false-positive-avoidance case the naive design would have failed.
- A `xce-node-manager` process killed and restarted while its Controller and Scheduler keep running adopts both without starting duplicates — exercises Requirement 7.1.
- A job dispatched by a Scheduler that is then killed mid-decision (before its dispatch is durably recorded) remains available for the replacement Scheduler to consider — exercises Requirement 3.5.
### E2E / Soak Tests
- Extend `.claude/skills/xce-test-ha/run-ha-soak.sh`'s sequential and chaotic modes to assert a working Scheduler, Worker, and Apiserver are present after every round with no harness-side start/stop action for any of them — replaces this session's own bug-1760 workaround, which becomes dead code once this feature ships (Requirement 8.2).
- A rapid sequence of Controller promotions/demotions produces no more than one Scheduler spawn/stop cycle per genuine promotion, with no orphaned or duplicate processes remaining after the sequence settles (Requirement 3.6, 7.1).
## Security Considerations
**Privilege model is role-conditional, not uniform.** `xce-deploy-ha`'s own tooling documents that the real (non-virtual) Worker daemon (`xcecd`, as opposed to the `xcecd_virtual` test double) runs as root — "plain kill can't touch it," requiring `sudo`. `xce-node-manager` must therefore run as root on any node whose assigned role includes `worker`, so `ComponentSupervisor`'s `posix_spawn`/`kill`/`waitpid` calls against `xcecd` need no special-cased privilege escalation; on a node whose roles are limited to manager/apiserver/reverse-proxy, `xce-node-manager` runs as an ordinary service account. This is a per-node deployment decision (how the host's own service manager launches `xce_nodemgr`), not a per-role runtime toggle inside the binary itself — the Node Role Manifest already fixes a node's roles at Node Manager startup, so the required privilege level is knowable at that same point.
The heartbeat socket is a new local IPC surface whose payload (in particular the Controller's reported primary/standby status) influences whether `xce-node-manager` starts or stops a component — a local process able to write to it could send spoofed reports. Consistent with `product.md`'s already-locked decision to trust the OS boundary rather than add application-layer encryption (`chmod 0700`, no new crypto), the heartbeat socket is created with permissions restricting it to the OS user `xce-node-manager` itself runs as on that node (root on a worker-role node, matching `xcecd`'s own existing requirement; an ordinary service account otherwise) — the same trust boundary the WAL/checkpoint directories already rely on. No new authentication mechanism is introduced; a host already compromised at that privilege level can already directly signal/kill the same processes this feature manages, so the socket adds no new privilege-escalation surface beyond what local code execution at that level already grants. On a worker-role node specifically, running `xce-node-manager` (and therefore the socket) as root is a direct consequence of `xcecd`'s own pre-existing requirement, not a new exposure this feature introduces.
## Migration Strategy
```mermaid
flowchart TB
Phase1[Phase 1 build xce nodemgr and the libxce heartbeat primitive behind no behavior change]
Phase2[Phase 2 wire heartbeat calls into controller scheduler worker apiserver loops]
Phase3[Phase 3 add the entry.sh manager case and update xce deploy ha and xce test ha tooling]
Phase4[Phase 4 cut over real deployments to node manager driven startup]
Phase1 --> Phase2 --> Phase3 --> Phase4
```
- Rollback trigger: if a real deployment depends on placing a component on a host with no Node Manager present for reasons not previously surfaced, that is a genuine requirements gap discovered late — the fix is to pause before Phase 4 (cutover) rather than force it through, since Phase 4 is the first phase that changes how a real deployment is operated.
- Validation checkpoint: after Phase 2, run the soak-harness scenario in Testing Strategy's E2E section before proceeding to Phase 3/4, since it is the closest thing to a direct regression test for bug-1760 itself.
## Open Questions / Risks
- The exact Node Role Manifest file format (YAML to match `src/configs/*.yaml`'s existing convention, vs. a simpler key-value form) is left to implementation — an additive, appropriately implementation-level decision, not a design decision, since Data Models already fixes its logical shape.
- The precise heartbeat interval and "no progress" threshold (Requirements 4.2, 4.3) are implementation-time tuning decisions, following the same posture the superseded spec already took for its own liveness thresholds.
- `tools/appimage/entry.sh` bundles `haproxy` today with no dispatch case reaching it at all — this feature is what first wires it up (via `xce-node-manager` invoking it directly for the reverse-proxy role), which is a pre-existing gap this design closes as a side effect rather than a pre-existing behavior it must preserve.
Resolved during design review (kept here for traceability, not left as open items): the Scheduler round-loop idle-cadence question was confirmed against the actual code (`cbuf_get()`'s unbounded wait) and resolved via `cbuf_timedget()` — see Architecture and File Structure Plan; it does not need re-verification at implementation time. The Worker-node root-privilege gap was resolved as a role-conditional privilege model — see Security Considerations.

101
.kiro/specs/xce-node-manager/requirements.md Normal file
View file

@ -0,0 +1,101 @@
# Requirements Document
## Project Description (Input)
xce-node-manager: a new per-node XCE daemon that supervises the lifecycle of the other XCE components on its host, replacing the now-superseded scheduler-primary-lifecycle spec's controller-owns-its-scheduler mechanism.
Who has the problem: XCE cluster operators, and by extension every job that depends on the scheduler being reachable after a controller failover (this carries forward bug-1760 from .wolf/buglog.json -- a hard-killed primary left its scheduler orphaned with no rejoin path, and a job submitted to the new primary sat stuck PEND). More generally, there is today no unified process-lifecycle/health-supervision layer for any XCE component (controller, scheduler, apiserver, worker, reverse proxy) -- each is started independently with no shared liveness protocol.
What should change: introduce xce-node-manager, one instance deployed per host, that reads a declared role assignment for its own node and starts/supervises the XCE component(s) that role requires:
- manager node role: starts the controller unconditionally; starts the scheduler only while that node's own controller instance is currently reporting itself primary, and stops the scheduler when that controller reports standby (or stops heartbeating).
- worker node role: starts the worker.
- apiserver node role: starts the apiserver.
- reverse-proxy node role: the reverse proxy is an off-the-shelf process (e.g. nginx/haproxy), not an XCE-authored binary -- xce-node-manager health-checks it via a TCP port check rather than a heartbeat, but still starts/restarts it per this node's role.
Health/liveness protocol: each XCE-authored managed component (controller, scheduler, worker, apiserver) heartbeats to its own node's local xce-node-manager over a Unix domain socket (same-host IPC only, no cross-host communication in this protocol). Most components send a simple liveness heartbeat ('I'm alive'). The controller's heartbeat additionally carries its current HA role state ('I'm primary' / 'I'm standby') -- this is the sole signal xce-node-manager uses to decide whether to spawn or kill the scheduler on a manager node; no separate election-awareness or Raft-state polling is introduced in xce-node-manager itself. The heartbeat client-side logic should live in libxce (src/lib/), which is already linked by controller/worker/scheduler in C and by the Go apiserver via CGO, so every managed component can share one implementation rather than reimplementing the protocol per-language.
This directly supersedes .kiro/specs/scheduler-primary-lifecycle/ (see that spec's requirements.md and design.md, now marked superseded in its spec.json): that spec gave the controller itself the job of directly posix_spawn-ing and killing its own scheduler child process (scheduler_proc.c/ha.c). In the new model, that responsibility moves entirely to xce-node-manager, driven by the controller's primary/standby heartbeat rather than the controller managing a child process itself. The underlying requirements that spec captured should carry forward and be re-expressed against this new mechanism: the scheduler must start automatically on promotion with no manual step; a scheduler tied to a controller that gracefully demotes must be stopped cleanly without blocking the controller's own role transition; a scheduler that is alive-but-wedged (connected, making no scheduling progress) must be detected and replaced; no job may be lost or left inconsistent when a scheduler is terminated for any reason; and the controller's own availability/correctness must never depend on xce-node-manager or on its scheduler child's health. Additionally, xce-node-manager's own liveness/health-check responsibility now generalizes to worker and apiserver, not just the scheduler.
Constraints and prior art to ground design in: .kiro/specs/scheduler-primary-lifecycle/design.md's Technology Stack section already resolved a real fork()-vs-posix_spawn mutex-inheritance-deadlock hazard for spawning a child process from a multithreaded daemon -- that finding (use posix_spawn, not fork()+exec()) applies equally to xce-node-manager spawning the scheduler and should not be re-litigated from scratch. src/lib/CLAUDE.md describes libxce's existing role as the shared IPC/utility library across controller, worker, scheduler, and (via CGO) the Go apiserver. src/worker/src/process_worker_manager.c is existing precedent in this codebase for a daemon spawning/supervising child processes (for job execution) and should be examined for reusable patterns even though it manages job processes rather than other XCE daemons. .kiro/steering/ha-async-io.md's controller-never-self-terminate invariant and iomgr/delaymgr non-blocking thread-role contract still apply to however the controller reports its primary/standby state to xce-node-manager -- that reporting path must never introduce blocking into a liveness-carrying controller thread. Existing deployment tooling (.claude/skills/xce-deploy-ha/SKILL.md, .claude/skills/xce-test-ha/run-ha-soak.sh) currently starts each XCE component independently and will need reconciling with node-role-driven startup via xce-node-manager, the same tooling-reconciliation concern scheduler-primary-lifecycle's requirements.md already flagged.
Open design questions to resolve during requirements/design phases (do not assume answers): how a node's role assignment is declared/distributed (static per-node config file vs. some cluster-wide source of truth); what happens if xce-node-manager itself crashes or is not running (who restarts it -- likely out of scope, delegated to the OS service manager, but must be stated explicitly); the exact heartbeat cadence and unresponsiveness threshold for each component type; whether xce-node-manager is a new src/ top-level component or lives under an existing directory per src/structure.md conventions; and how much of scheduler-primary-lifecycle's already-designed SchedOwnerWatch self-monitoring behavior (scheduler watching its own controller directly) is still needed as a redundant backstop versus fully replaced by xce-node-manager-driven termination.
## Boundary Context
- **In scope**: node-role-driven startup and continuous supervision of the Controller, Scheduler Process, Worker Process, and Apiserver Process; binding the Scheduler Process's start/stop to its Controller's primary/standby status; heartbeat-based liveness detection that distinguishes real progress from bare connectivity where that distinction matters; a distinct liveness check for the off-the-shelf Reverse Proxy Process; reconciling existing deployment tooling and specs that assume independently-managed component processes.
- **Out of scope**: supervising the Node Manager process itself (delegated to the host's existing process-supervision facilities, e.g. systemd or an equivalent service manager — this feature does not build that); the Reverse Proxy Process's own configuration, routing rules, or request handling; any change to how a Worker Process's or Apiserver Process's own client-side connection discovers or reconnects to whichever Controller is currently primary (owned by the existing/adjacent client-failover mechanism, unchanged by this feature); any change to RBAC, authentication, or job/queue/scheduling decision logic; any new operator-facing remote management API for the Node Manager beyond existing logging conventions.
- **Adjacent expectations**: this feature depends on the Controller's existing Raft-based primary/standby determination and the existing `ha_dispatch_gate` non-primary write-rejection behavior, both unchanged by this feature. This feature supersedes `.kiro/specs/scheduler-primary-lifecycle/`'s approach to scheduler-lifecycle management (a Controller directly owning its own scheduler child process) — the scheduler-availability requirements that spec captured (no manual scheduler startup, prompt shutdown on demotion, wedge detection, no job loss on scheduler termination) are restated here against the Node Manager-based mechanism instead. Existing deployment tooling (`xce-deploy-ha`, `xce-test-ha`) and any spec assuming an independently-started, reconnecting component process is an adjacent consumer that must be revisited, not an unstated exception to this feature's behavior.
## Requirements
### Requirement 1: Node role assignment and component startup
**Objective:** As a cluster operator, I want each node to automatically start the right XCE component(s) for its assigned role, so that deploying a node consists of assigning it a role rather than manually starting and managing individual XCE processes.
#### Acceptance Criteria
1. When the Node Manager starts on a node, the Node Manager shall determine which role or roles are assigned to that node without depending on connectivity to any Controller or other cluster component.
2. The Node Manager shall support, at minimum, the following node roles: manager (runs the Controller, and the Scheduler Process while that Controller is primary), worker (runs the Worker Process), apiserver (runs the Apiserver Process), and reverse-proxy (runs an externally-provided Reverse Proxy Process).
3. When the Node Manager starts on a node assigned the manager role, the Node Manager shall start the Controller for that node.
4. When the Node Manager starts on a node assigned the worker, apiserver, or reverse-proxy role, the Node Manager shall start the corresponding component for that node.
5. If a node is assigned more than one role, the Node Manager shall start the component for each assigned role.
6. If the Node Manager cannot start a component required by the node's assigned role, the Node Manager shall record an operator-visible error and continue attempting to start it.
### Requirement 2: Continuous supervision and restart of always-on components
**Objective:** As a cluster operator, I want a component that is supposed to always be running on a node to be restarted automatically if it stops unexpectedly, so that a crashed Worker Process, Apiserver Process, Controller, or Reverse Proxy Process does not require manual intervention to recover.
#### Acceptance Criteria
1. While a node's assigned role requires a component to be continuously running (Controller, Worker Process, Apiserver Process, or Reverse Proxy Process), the Node Manager shall treat an unexpected exit of that component's process as a failure.
2. When the Node Manager detects that a continuously-required component's process has exited unexpectedly, the Node Manager shall start a replacement process for it.
3. The Node Manager shall not restart a component process that it stopped itself as part of an intentional, requested action.
4. The Node Manager shall log an operator-visible event whenever it restarts a component due to an unexpected exit.
### Requirement 3: Scheduler lifecycle bound to Controller primary/standby status
**Objective:** As a cluster operator, I want the scheduler on a manager node to run only while that node's Controller is primary, so that a Scheduler Process never keeps running on behalf of a Controller that is no longer authoritative, and the cluster always has a working scheduler after a failover without manual intervention.
#### Acceptance Criteria
1. While a node's Controller reports itself as primary, the Node Manager shall ensure exactly one Scheduler Process associated with that Controller is running.
2. When a node's Controller transitions from primary to standby, the Node Manager shall stop the Scheduler Process associated with that Controller.
3. When a node's Controller becomes unreachable to the Node Manager, the Node Manager shall stop the Scheduler Process associated with that Controller.
4. The Node Manager shall stop a Scheduler Process associated with a Controller within a bounded time of that Controller ceasing to be primary or becoming unreachable, without requiring the Controller's own role transition to wait for the Scheduler Process to exit.
5. The Node Manager shall ensure that no job a stopped Scheduler Process had not yet durably recorded is lost or left in an inconsistent state, regardless of why that Scheduler Process was stopped (normal stop request, Controller loss, unresponsiveness, or crash).
6. If a node's Controller is primary but has no Scheduler Process currently running for it, the Node Manager shall keep attempting to start one while that Controller remains primary.
### Requirement 4: Liveness and health detection of managed components
**Objective:** As a cluster operator, I want the Node Manager to detect when a managed component has stopped doing useful work, not merely whether its process still exists, so that a hung or wedged component is recovered instead of silently leaving jobs unserved.
#### Acceptance Criteria
1. While a managed XCE component's process is running, that component shall periodically report its liveness to its node's Node Manager.
2. If a managed component does not report its liveness within an operator-configurable threshold, the Node Manager shall treat that component as unresponsive.
3. While the Scheduler Process is running, the liveness it reports to the Node Manager shall reflect whether it is making real scheduling progress, not merely that its process is still running.
4. When the Node Manager treats a managed component as unresponsive, the Node Manager shall stop that component and start a replacement, following the same startup behavior as if it had exited unexpectedly.
5. Where the managed component is the Reverse Proxy Process, the Node Manager shall determine its liveness by checking reachability of its configured network port instead of a liveness report from the process itself.
6. The Node Manager shall log an operator-visible event whenever it treats a managed component as unresponsive and replaces it.
### Requirement 5: Non-interference with a managed component's own responsibilities
**Objective:** As a cluster operator, I want reporting liveness/role status to the Node Manager to never come at the cost of a component's primary responsibilities, so that adopting Node Manager supervision cannot itself become a new source of cluster unavailability.
#### Acceptance Criteria
1. A managed component's reporting of its liveness or role status to the Node Manager shall never block or delay that component's own primary processing responsibilities (for example, a Controller's handling of heartbeats or election timeouts).
2. If reporting liveness or role status to the Node Manager fails for any reason, the reporting component shall continue operating normally.
3. The absence of a running or reachable Node Manager shall not prevent a Controller from being elected primary, remaining primary, or serving its existing responsibilities.
### Requirement 6: Independent verification of controlling authority (backstop)
**Objective:** As a cluster operator, I want the scheduler to independently notice when its own controller is no longer a live primary, so that a Scheduler Process can never keep running detached from any authoritative Controller even if the Node Manager that started it becomes unavailable.
#### Acceptance Criteria
1. While a Scheduler Process is running, the Scheduler Process shall independently verify that the specific Controller instance that started it is reachable and still holds the primary role.
2. When a Scheduler Process can no longer confirm that its originating Controller instance is reachable and primary, the Scheduler Process shall terminate on its own, independent of any action by the Node Manager.
3. The Scheduler Process shall not attempt to join, rejoin, or continue operating against any Controller instance other than the one that started it.
### Requirement 7: Node Manager recovery without duplicate supervision
**Objective:** As a cluster operator, I want a restarted Node Manager to recognize the components already running on its node instead of starting duplicates, so that the Node Manager's own restart does not cause process thrashing or resource conflicts.
#### Acceptance Criteria
1. When the Node Manager starts and a component required by the node's assigned role is already running, the Node Manager shall recognize it as already running rather than starting a duplicate process for it.
2. The Node Manager shall log an operator-visible event whenever it adopts an already-running component on startup instead of starting a new one.
### Requirement 8: Compatibility with existing deployment tooling and specs
**Objective:** As a cluster operator or spec author relying on existing deployment tooling, I want the introduction of node-role-driven component startup to be reflected everywhere a component's lifecycle is currently assumed to be independently and manually managed, so that existing deployments, tests, and specs do not silently break or become misleading.
#### Acceptance Criteria
1. Where existing deployment tooling documents starting the Controller, Scheduler Process, Worker Process, or Apiserver Process as an independent, manually-managed process, that tooling shall be updated to reflect that the Node Manager now starts these components according to each node's assigned role.
2. Where an existing spec or test tooling depends on a component being started or managed independently of the Node Manager, that dependency shall be identified and reconciled with this feature's behavior rather than left silently incompatible.
3. The observable behavior of the Controller, Scheduler Process, Worker Process, and Apiserver Process toward jobs, queues, configuration, and cluster membership shall remain unchanged by this feature while each is running.

79
.kiro/specs/xce-node-manager/research.md Normal file
View file

@ -0,0 +1,79 @@
# Research & Design Decisions
## Summary
- **Feature**: `xce-node-manager`
- **Discovery Scope**: Complex Integration — new top-level daemon, no direct precedent, but heavily grounded in existing patterns across controller/worker/scheduler/apiserver and the AppImage packaging layer.
- **Key Findings**:
- `tools/appimage/entry.sh` is a thin per-role dispatcher (`xce <role> args...`) that execs exactly one of `xce_controller`/`xcecd`/`xce_scheduler`/`xce_apiserver` in the foreground — there is no existing orchestration layer to extend or conflict with, and `haproxy` (already bundled in the AppImage per `build-deployment.md`) has no dispatch case at all today, i.e. it is currently unreachable through `entry.sh`.
- `src/worker/src/process_worker_manager.c`'s existing `process_manager_start_worker()` spawns via raw `fork()+execl()`, not `posix_spawn()` — an already-shipped instance of the exact mutex-inheritance-deadlock hazard class `product.md`'s locked decision ("no `fork()` for snapshots") and the superseded `scheduler-primary-lifecycle` design already ruled out. Not fixed here (out of scope, unrelated component) but explicitly NOT a pattern to imitate.
- The controller already has a non-blocking, delaymgr-driven, role-aware periodic task (`ha_start_primary_heartbeat`/`primary_heartbeat_fn`, `ha.c:9949`) reading the atomic `ha->role` every tick — proof that a cheap, always-safe periodic local report is a well-trodden pattern in this codebase, though this specific function is gated on first-promotion (`s_armed`) and therefore unsuitable to piggyback on directly (see Decision: Heartbeat Call Site).
- `src/configs/hosts.yaml` (the cluster's existing per-host config, distributed via `OPCode_replConfigPush`) describes scheduler-visible compute-host attributes, not deployment/process-placement roles — confirms no existing config surface for "which node runs which XCE component," and that any such config cannot be cluster-distributed (a node's role must be knowable before any controller is reachable).
## Research Log
### Existing multi-component launch/packaging surface
- **Context**: Before designing a new orchestrator, check whether one already exists in disguise (AppImage bundling, systemd units, an existing launcher script).
- **Sources Consulted**: `tools/appimage/entry.sh`, `build-deployment.md` (AppImage/Production Artifacts sections), `xce-deploy-ha` SKILL.md, `xce-test-ha/run-ha-soak.sh`.
- **Findings**: The AppImage bundles all daemons plus `haproxy` behind one `xce <role>` entry point, but `entry.sh` only launches a single named process in the foreground per invocation — no supervision, no multi-process orchestration, no config-driven role resolution. The local dev/test harness (`xce-deploy-ha`) generates one shell script per component (`start.sh`, `start-worker.sh`, `start-scheduler.sh`), started manually/independently — the same "independently-started component" assumption `scheduler-primary-lifecycle`'s requirements already flagged, now generalized to worker/apiserver too.
- **Implications**: `xce-node-manager` is additive, not competing with an existing mechanism. The natural integration point is one new `entry.sh` case (`manager`) that execs the new `xce_nodemgr` binary; `xce_nodemgr` itself then resolves and spawns the other bundled binaries directly (same `which xce_controller`-style PATH resolution `entry.sh` already uses), including finally wiring up the previously-unreachable bundled `haproxy` for the reverse-proxy role.
### Controller-side periodic reporting hook
- **Context**: Where should the Controller's "I'm alive / I'm primary or standby" report be triggered from, without touching the promotion/demotion callback functions those steering docs flag as the riskiest, least-covered code in the controller?
- **Sources Consulted**: `ha.c` (`primary_heartbeat_fn`, `ha_start_primary_heartbeat`, `raft_cb_become_leader`/`raft_cb_become_follower` — via `codegraph_explore`), `ha-async-io.md`.
- **Findings**: `primary_heartbeat_fn` is armed only once, on a controller instance's *first* promotion (`s_armed` guard) — a controller that has never won an election never runs it, which would leave `xce-node-manager` unable to distinguish "controller alive, still standby" from "controller process is down" on any node that hasn't yet been promoted. `raft_cb_become_leader`/`raft_cb_become_follower` themselves are flagged with no direct test coverage and are the single most call-dense, safety-critical functions in the controller (122+ calls from `raft_cb_become_leader` alone).
- **Implications**: The report must come from a small, independent, unconditionally-armed periodic point tied to the Controller's own process lifetime (not gated on promotion), reading the same atomic `ha->role` field `primary_heartbeat_fn` already reads. This adds zero new logic to `raft_cb_become_leader`/`raft_cb_become_follower` — a strict reduction in touched surface area compared to the superseded spec, which called full spawn/monitor orchestration from inside `raft_cb_become_leader` directly.
### Scheduler wedge-detection signal shape
- **Context**: The superseded spec's Requirement 5 (carried forward as 4.3 here) requires detecting a scheduler that is connected but not making real scheduling progress — a bare "I'm alive" ping cannot distinguish a wedged decision-loop from a healthy one if the two run on different threads.
- **Sources Consulted**: In-conversation review of `scheduler-primary-lifecycle`'s `design.md` (`SchedulerConnectionBookkeeping`'s last-progress-timestamp approach, fed from real dispatch/pend-reason traffic, not a synthetic heartbeat); bug-1760's own root-cause text describing the scheduler's *outbound wire traffic* as reactive/idle-silent ("the scheduler has nothing queued to publish").
- **Findings**: Reusing the scheduler's reactive wire-publish call as the heartbeat trigger would reintroduce the exact idle-silence failure mode bug-1760 already exposed. The correct trigger is the scheduler's own internal round/picker loop iteration — which must tick on a bounded cadence regardless of whether a round finds any job to act on — not the wire-publish call.
- **Implications**: One wire-format field (a monotonically increasing sequence number) serves double duty: for the Controller/Worker/Apiserver it is a bare liveness counter; for the Scheduler, since it is incremented once per round-loop iteration (not once per successful publish), its advancement *is* the real-progress signal Requirement 4.3 needs, with no separate progress field required. See Decision: Unified Heartbeat Wire Format.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Adopt a generic external supervisor (systemd user units, s6, runit, supervisord) for all four roles | Delegate process start/restart entirely to an existing, battle-tested tool | Zero new code for the generic "keep running" case (Requirement 2) | Cannot express Requirement 3's Controller-role-driven conditional scheduler start/stop or Requirement 4.3's progress-aware wedge detection — those are XCE-domain-specific decisions no generic supervisor can make; also conflicts with the AppImage's self-contained, zero-external-dependency portability goal (`product.md`) across arbitrary air-gapped target hosts that may not run systemd | Rejected as the primary mechanism; the generic host service manager is still the right tool for the one thing left over — restarting `xce-node-manager` itself, already scoped out of this feature (Requirement 7 boundary) |
| Custom `xce-node-manager` daemon using only POSIX primitives (`posix_spawn`, Unix domain datagram sockets), consistent with the already-reviewed `scheduler-primary-lifecycle` precedent | Build a small, purpose-specific supervisor | Can express both the generic and Controller-role-driven policies; no new external runtime dependency; reuses the `posix_spawn` decision already vetted against the mutex-inheritance-deadlock hazard | New component to build and test | **Selected** |
## Design Decisions
### Decision: Heartbeat call site is each component's own natural loop iteration, not a new thread
- **Context**: Requirements 4.1 and 4.3 require every managed component to report liveness, and require the Scheduler's report specifically to reflect real progress.
- **Alternatives Considered**:
1. A dedicated heartbeat-sender thread per component, independent of its main work loop — simplest to bolt on, but structurally cannot distinguish a wedged work thread from a healthy one (the sender thread keeps reporting regardless).
2. Piggyback the Controller's report on the existing `primary_heartbeat_fn` — rejected, see Research Log (promotion-gated arming leaves never-yet-primary controllers silent).
- **Selected Approach**: Each component calls a shared `libxce` heartbeat primitive once per iteration of its own existing loop (the Controller's iomgr/delaymgr-driven event loop, the Scheduler's round loop, the Worker's `ipc_worker_run`/`usleep` loop); the Apiserver, having no persistent loop of its own (a request-driven Gin server), uses a small dedicated ticker goroutine instead — the one place a new "thread" is unavoidable, since there is no existing loop to hook into.
- **Rationale**: Guarantees the report can only advance when the component's own real work-loop is actually turning, which is what makes the same signal double as both liveness and (for the Scheduler) progress. The Scheduler's live driver (`scheduler.c`, `sched_job_picker`) is the correct call site, not the pure decision engine in `core.c`/`core_events.c`/`core_session.c`/`core_clock.c` — the `scheduler-core` spec's own core/driver separation already keeps the core free of controller I/O, and this feature's local heartbeat is new outside-facing I/O that belongs on the same side of that boundary as the existing `xcecli` connection. The Scheduler's separate simulation-mode driver (`sim.c`) gets no heartbeat call site at all and stays entirely outside `xce-node-manager`'s awareness (see Non-Goals in `design.md`) — it is a standalone, manually-invoked tool with no Controller or primary/standby concept.
- **Trade-offs**: Slightly different integration shape per component (loop hook vs. ticker) instead of one uniform mechanism; acceptable since the underlying wire format and `libxce` primitive are identical either way.
- **Follow-up resolved during design review** (not deferred to implementation): `codegraph_explore` confirmed `sched_job_picker` calls `cbuf_get()` (`src/lib/mylock.c:144`), an **unbounded** `pthread_cond_wait` when the job queue is empty — a heartbeat placed naively at the top of that loop would never fire during genuine idle periods, causing `xce-node-manager` to falsely treat a healthy idle Scheduler as wedged and kill it. Fix: swap to `cbuf_timedget()` (same file, an existing bounded-deadline variant already used elsewhere in the codebase) with a bounded idle-tick deadline, and call the heartbeat primitive on both the real-job path and the timeout path.
### Decision: Unified heartbeat wire format, self-identifying payload, no CGO for the Apiserver client
- **Context**: The heartbeat protocol must be usable by four component types across two languages (C daemons, Go apiserver) over one shared local socket, with the receiving `xce-node-manager` able to attribute each datagram without relying on socket-level peer identity (a `SOCK_DGRAM` Unix socket exposes no reliable peer address unless the sender also binds, which adds needless coupling).
- **Alternatives Considered**:
1. Route the Apiserver's heartbeat through `libxce` via the existing CGO bridge, matching how `apiserver`'s C-tier calls already work.
2. A small, independently-implemented Go client speaking the identical wire format natively, with no CGO involved.
- **Selected Approach**: One canonical fixed-layout binary wire format (magic/version, role, status, pid, sequence number — no Cap'n Proto), implemented once in `libxce` for the three C daemons (which already link it natively, zero extra cost), and once as a small native Go implementation for the Apiserver.
- **Rationale**: The payload is a fixed, trivial, few-field structure — reimplementing it natively in Go is a handful of lines with negligible drift risk, and avoids pulling a CGO dependency into a code path (a periodic status ticker) that has no other reason to touch the C tier; `tech.md` already documents that Apiserver's own unit tests deliberately run `CGO_ENABLED=0`, and only the existing backend bridge work requires the CGO build. The payload self-identifies its sender (role + pid fields), so the receiver never depends on socket-level peer identity. This is a deliberate, narrow exception to `structure.md`'s "Cap'n Proto for wire formats" convention: that convention targets the real cross-host, cross-version network protocol (`controller.capnp`/`c.capnp`), where schema evolution and cross-language generated bindings earn their cost; this protocol is same-host-only, always emitted and consumed by binaries from the same build (Node Manager and the components it manages ship together), and confirmed (`go.mod`/`go.sum` inspection) that the Go apiserver carries no existing capnp dependency at all — adopting Cap'n Proto here would mean a new Go dependency purely for a fixed 4-field local datagram, which the Simplification lens rejects as disproportionate.
- **Trade-offs**: Two implementations of the same format instead of one; mitigated by the format's small size and by both being validated against the same contract in this design.
- **Follow-up**: none — the format is fixed at design time (see Data Models in `design.md`).
### Decision: Node Manager privilege is role-conditional (root only on worker-role nodes)
- **Context**: `xce-deploy-ha`'s own tooling documents that the real (non-virtual) Worker daemon `xcecd` runs as root ("plain kill can't touch it," needs `sudo`) — discovered during design review, not anticipated during requirements. A uniform single-OS-user assumption for `xce-node-manager` (as first drafted) would leave it unable to start/stop/signal `xcecd` on a real worker-role node.
- **Alternatives Considered**:
1. `xce-node-manager` always runs as root everywhere, regardless of role.
2. `xce-node-manager` stays unprivileged everywhere; grant it a narrow Linux capability or scoped `sudo` rule for the worker case specifically.
3. Treat real (root-requiring) worker management as out of scope for this spec; only support `xcecd_virtual`.
- **Selected Approach** (user decision): `xce-node-manager` runs as root only on a node whose assigned role includes `worker`; on manager/apiserver/reverse-proxy-only nodes it runs as an ordinary service account. This is fixed by the Node Role Manifest at Node Manager startup, not a runtime toggle.
- **Rationale**: Matches `xcecd`'s own pre-existing requirement exactly, with no new privilege-escalation surface beyond what that daemon already needs; simpler than a capability-based scheme, and avoids narrowing this spec's worker-role support to the virtual test double only.
- **Trade-offs**: The heartbeat socket's permission model is now per-node-role rather than one uniform story (see `design.md` Security Considerations) — an implementer must not assume a single OS user across all node roles.
- **Follow-up**: none — fixed at design time.
## Risks & Mitigations
- A node-manager crash on a manager node strands its Controller's Scheduler with nothing left to stop it on a subsequent demotion — mitigated by retaining a scheduler-side independent controller-liveness check as a backstop (Requirement 6), so the Scheduler never depends solely on `xce-node-manager` staying alive.
- `xce-node-manager` restarting could double-spawn a component that is already running — mitigated by a small persisted per-role record (role, pid, start time, binary path) it consults on startup before spawning (see `design.md` Data Models); a stale/mismatched record falls back to a fresh start rather than blindly trusting the file, and each component's own listen-port bind-failure is a natural backstop against an accidental true duplicate.
- Sizing the heartbeat interval/timeout thresholds is an implementation-time tuning decision (same posture the superseded spec already took for its own liveness thresholds), not a design decision.
## References
- `.kiro/specs/scheduler-primary-lifecycle/design.md` — superseded design; source of the `posix_spawn` mutex-inheritance-deadlock finding and the wedge-detection real-progress principle, both carried forward here.
- `.wolf/buglog.json` bug-1760 — motivating incident for the scheduler-lifecycle requirements this spec restates.

25
.kiro/specs/xce-node-manager/spec.json Normal file
View file

@ -0,0 +1,25 @@
{
"feature_name": "xce-node-manager",
"created_at": "2026-09-03T03:45:42Z",
"updated_at": "2026-09-03T06:38:35Z",
"language": "en",
"phase": "design-generated",
"supersedes": "scheduler-primary-lifecycle",
"approvals": {
"requirements": {
"generated": true,
"approved": true,
"validation_note": "kiro-spec-requirements 2026-09-02: 8 requirement areas / 32 acceptance criteria drafted, grounded in in-conversation codebase investigation (ha.c primary_heartbeat_fn/ha->role, process_worker_manager.c's existing fork()-hazard precedent, process_manager.c/h supervision shape, controller_connector.c candidate-list pattern, hosts.yaml config format, xce-deploy-ha tooling, product.md/structure.md/tech.md/roadmap.md steering) plus the fully-read superseded scheduler-primary-lifecycle design.md. Requirements Review Gate run inline during drafting (not as a separate pass): mechanical checks (numeric IDs, EARS-format ACs, no implementation-language leakage) passed; two draft issues caught and fixed before writing -- Requirement 3's original job-durability AC used a non-EARS 'Regardless of...' opening (rewritten to a ubiquitous 'The Node Manager shall ensure...' form), and Requirement 7 originally included a non-AC scope-exclusion bullet (moved to Boundary Context's Out of scope list, replaced with a proper adoption-logging AC). Auto-approved 2026-09-03 via `/kiro-spec-design xce-node-manager -y`."
},
"design": {
"generated": true,
"approved": true,
"validation_note": "kiro-spec-design 2026-09-03: light-to-moderate discovery, see prior note for discovery findings and synthesis (build-vs-adopt rejection of a generic external supervisor; ComponentSupervisor/LivenessMonitor generalization; justified Cap'n Proto deviation). kiro-validate-design 2026-09-03: independent critical review (not the same pass as authoring) found 3 real issues, GO withheld pending fixes -- (1) uniform single-OS-user privilege assumption was factually wrong: xce-deploy-ha's own tooling documents the real (non-virtual) xcecd worker daemon runs as root, unaddressed by the original design; resolved per user decision -- xce-node-manager runs as root only on a node whose assigned role includes worker, ordinary service account otherwise, Security Considerations and research.md's Decision log rewritten accordingly. (2) Requirement 4.3's wedge-detection mechanism depended on the Scheduler's round loop ticking on a bounded cadence even when idle, left as an unverified 'confirm during implementation' open question despite research.md's own cited evidence (bug-1760's reactive/idle-silent root cause) pointing the other way; verified directly against the actual code via codegraph_explore -- sched_job_picker's cbuf_get() (src/lib/mylock.c:144) does an UNBOUNDED pthread_cond_wait when idle, confirming the naive design would cause false-positive wedge detection on every idle Scheduler. Fixed: swap to the already-existing cbuf_timedget() bounded-deadline variant with heartbeat calls on both the real-job and timeout paths; documented as a resolved design decision, not deferred. (3) PortCheckMonitor's TCP connect was never specified as non-blocking despite HeartbeatListener explicitly being so, and ComponentSupervisor's own text incorrectly claimed the Node Manager loop 'has no other liveness-critical responsibility to protect' -- both corrected: PortCheckMonitor's connect is now explicitly non-blocking (O_NONBLOCK + bounded poll/select), and the misleading framing was rewritten to state the loop IS liveness-critical for every role it supervises. Two further scope clarifications volunteered by the user during the fix cycle, applied directly: the heartbeat call site is explicitly the Scheduler's live driver (scheduler.c/sched_job_picker) only, never the pure core (core.c/core_events.c/core_session.c/core_clock.c, scheduler-core spec's own zero-controller-I/O invariant, now cited as a fourth cross-checked project invariant) or the simulation driver (sim.c, added as an explicit Non-Goal/Out-of-Boundary item -- simulation-mode Scheduler runs standalone, entirely outside xce-node-manager's awareness). All fixes applied directly to design.md/research.md in the same session; re-reviewed inline, no further issues found. GO."
},
"tasks": {
"generated": false,
"approved": false
}
},
"ready_for_implementation": false
}