myxce/.kiro/specs/xce-node-manager/requirements.md

Requirements Document

Project Description (Input)

xce-node-manager: a new per-node XCE daemon that supervises the lifecycle of the other XCE components on its host, replacing the now-superseded scheduler-primary-lifecycle spec's controller-owns-its-scheduler mechanism.

Who has the problem: XCE cluster operators, and by extension every job that depends on the scheduler being reachable after a controller failover (this carries forward bug-1760 from .wolf/buglog.json -- a hard-killed primary left its scheduler orphaned with no rejoin path, and a job submitted to the new primary sat stuck PEND). More generally, there is today no unified process-lifecycle/health-supervision layer for any XCE component (controller, scheduler, apiserver, worker, reverse proxy) -- each is started independently with no shared liveness protocol.

What should change: introduce xce-node-manager, one instance deployed per host, that reads a declared role assignment for its own node and starts/supervises the XCE component(s) that role requires:

• manager node role: starts the controller unconditionally; starts the scheduler only while that node's own controller instance is currently reporting itself primary, and stops the scheduler when that controller reports standby (or stops heartbeating).
• worker node role: starts the worker.
• apiserver node role: starts the apiserver.
• reverse-proxy node role: the reverse proxy is an off-the-shelf process (e.g. nginx/haproxy), not an XCE-authored binary -- xce-node-manager health-checks it via a TCP port check rather than a heartbeat, but still starts/restarts it per this node's role.

Health/liveness protocol: each XCE-authored managed component (controller, scheduler, worker, apiserver) heartbeats to its own node's local xce-node-manager over a Unix domain socket (same-host IPC only, no cross-host communication in this protocol). Most components send a simple liveness heartbeat ('I'm alive'). The controller's heartbeat additionally carries its current HA role state ('I'm primary' / 'I'm standby') -- this is the sole signal xce-node-manager uses to decide whether to spawn or kill the scheduler on a manager node; no separate election-awareness or Raft-state polling is introduced in xce-node-manager itself. The heartbeat client-side logic should live in libxce (src/lib/), which is already linked by controller/worker/scheduler in C and by the Go apiserver via CGO, so every managed component can share one implementation rather than reimplementing the protocol per-language.

This directly supersedes .kiro/specs/scheduler-primary-lifecycle/ (see that spec's requirements.md and design.md, now marked superseded in its spec.json): that spec gave the controller itself the job of directly posix_spawn-ing and killing its own scheduler child process (scheduler_proc.c/ha.c). In the new model, that responsibility moves entirely to xce-node-manager, driven by the controller's primary/standby heartbeat rather than the controller managing a child process itself. The underlying requirements that spec captured should carry forward and be re-expressed against this new mechanism: the scheduler must start automatically on promotion with no manual step; a scheduler tied to a controller that gracefully demotes must be stopped cleanly without blocking the controller's own role transition; a scheduler that is alive-but-wedged (connected, making no scheduling progress) must be detected and replaced; no job may be lost or left inconsistent when a scheduler is terminated for any reason; and the controller's own availability/correctness must never depend on xce-node-manager or on its scheduler child's health. Additionally, xce-node-manager's own liveness/health-check responsibility now generalizes to worker and apiserver, not just the scheduler.

Constraints and prior art to ground design in: .kiro/specs/scheduler-primary-lifecycle/design.md's Technology Stack section already resolved a real fork()-vs-posix_spawn mutex-inheritance-deadlock hazard for spawning a child process from a multithreaded daemon -- that finding (use posix_spawn, not fork()+exec()) applies equally to xce-node-manager spawning the scheduler and should not be re-litigated from scratch. src/lib/CLAUDE.md describes libxce's existing role as the shared IPC/utility library across controller, worker, scheduler, and (via CGO) the Go apiserver. src/worker/src/process_worker_manager.c is existing precedent in this codebase for a daemon spawning/supervising child processes (for job execution) and should be examined for reusable patterns even though it manages job processes rather than other XCE daemons. .kiro/steering/ha-async-io.md's controller-never-self-terminate invariant and iomgr/delaymgr non-blocking thread-role contract still apply to however the controller reports its primary/standby state to xce-node-manager -- that reporting path must never introduce blocking into a liveness-carrying controller thread. Existing deployment tooling (.claude/skills/xce-deploy-ha/SKILL.md, .claude/skills/xce-test-ha/run-ha-soak.sh) currently starts each XCE component independently and will need reconciling with node-role-driven startup via xce-node-manager, the same tooling-reconciliation concern scheduler-primary-lifecycle's requirements.md already flagged.

Open design questions to resolve during requirements/design phases (do not assume answers): how a node's role assignment is declared/distributed (static per-node config file vs. some cluster-wide source of truth); what happens if xce-node-manager itself crashes or is not running (who restarts it -- likely out of scope, delegated to the OS service manager, but must be stated explicitly); the exact heartbeat cadence and unresponsiveness threshold for each component type; whether xce-node-manager is a new src/ top-level component or lives under an existing directory per src/structure.md conventions; and how much of scheduler-primary-lifecycle's already-designed SchedOwnerWatch self-monitoring behavior (scheduler watching its own controller directly) is still needed as a redundant backstop versus fully replaced by xce-node-manager-driven termination.

Boundary Context

• In scope: node-role-driven startup and continuous supervision of the Controller, Scheduler Process, Worker Process, and Apiserver Process; binding the Scheduler Process's start/stop to its Controller's primary/standby status; heartbeat-based liveness detection that distinguishes real progress from bare connectivity where that distinction matters; a distinct liveness check for the off-the-shelf Reverse Proxy Process; reconciling existing deployment tooling and specs that assume independently-managed component processes.
• Out of scope: supervising the Node Manager process itself (delegated to the host's existing process-supervision facilities, e.g. systemd or an equivalent service manager — this feature does not build that); the Reverse Proxy Process's own configuration, routing rules, or request handling; any change to how a Worker Process's or Apiserver Process's own client-side connection discovers or reconnects to whichever Controller is currently primary (owned by the existing/adjacent client-failover mechanism, unchanged by this feature); any change to RBAC, authentication, or job/queue/scheduling decision logic; any new operator-facing remote management API for the Node Manager beyond existing logging conventions.
• Adjacent expectations: this feature depends on the Controller's existing Raft-based primary/standby determination and the existing ha_dispatch_gate non-primary write-rejection behavior, both unchanged by this feature. This feature supersedes .kiro/specs/scheduler-primary-lifecycle/'s approach to scheduler-lifecycle management (a Controller directly owning its own scheduler child process) — the scheduler-availability requirements that spec captured (no manual scheduler startup, prompt shutdown on demotion, wedge detection, no job loss on scheduler termination) are restated here against the Node Manager-based mechanism instead. Existing deployment tooling (xce-deploy-ha, xce-test-ha) and any spec assuming an independently-started, reconnecting component process is an adjacent consumer that must be revisited, not an unstated exception to this feature's behavior.

Requirements

Requirement 1: Node role assignment and component startup

Objective: As a cluster operator, I want each node to automatically start the right XCE component(s) for its assigned role, so that deploying a node consists of assigning it a role rather than manually starting and managing individual XCE processes.

Acceptance Criteria

1. When the Node Manager starts on a node, the Node Manager shall determine which role or roles are assigned to that node without depending on connectivity to any Controller or other cluster component.
2. The Node Manager shall support, at minimum, the following node roles: manager (runs the Controller, and the Scheduler Process while that Controller is primary), worker (runs the Worker Process), apiserver (runs the Apiserver Process), and reverse-proxy (runs an externally-provided Reverse Proxy Process).
3. When the Node Manager starts on a node assigned the manager role, the Node Manager shall start the Controller for that node.
4. When the Node Manager starts on a node assigned the worker, apiserver, or reverse-proxy role, the Node Manager shall start the corresponding component for that node.
5. If a node is assigned more than one role, the Node Manager shall start the component for each assigned role.
6. If the Node Manager cannot start a component required by the node's assigned role, the Node Manager shall record an operator-visible error and continue attempting to start it.

Requirement 2: Continuous supervision and restart of always-on components

Objective: As a cluster operator, I want a component that is supposed to always be running on a node to be restarted automatically if it stops unexpectedly, so that a crashed Worker Process, Apiserver Process, Controller, or Reverse Proxy Process does not require manual intervention to recover.

Acceptance Criteria

1. While a node's assigned role requires a component to be continuously running (Controller, Worker Process, Apiserver Process, or Reverse Proxy Process), the Node Manager shall treat an unexpected exit of that component's process as a failure.
2. When the Node Manager detects that a continuously-required component's process has exited unexpectedly, the Node Manager shall start a replacement process for it.
3. The Node Manager shall not restart a component process that it stopped itself as part of an intentional, requested action.
4. The Node Manager shall log an operator-visible event whenever it restarts a component due to an unexpected exit.

Requirement 3: Scheduler lifecycle bound to Controller primary/standby status

Objective: As a cluster operator, I want the scheduler on a manager node to run only while that node's Controller is primary, so that a Scheduler Process never keeps running on behalf of a Controller that is no longer authoritative, and the cluster always has a working scheduler after a failover without manual intervention.

Acceptance Criteria

1. While a node's Controller reports itself as primary, the Node Manager shall ensure exactly one Scheduler Process associated with that Controller is running.
2. When a node's Controller transitions from primary to standby, the Node Manager shall stop the Scheduler Process associated with that Controller.
3. When a node's Controller becomes unreachable to the Node Manager, the Node Manager shall stop the Scheduler Process associated with that Controller.
4. The Node Manager shall stop a Scheduler Process associated with a Controller within a bounded time of that Controller ceasing to be primary or becoming unreachable, without requiring the Controller's own role transition to wait for the Scheduler Process to exit.
5. The Node Manager shall ensure that no job a stopped Scheduler Process had not yet durably recorded is lost or left in an inconsistent state, regardless of why that Scheduler Process was stopped (normal stop request, Controller loss, unresponsiveness, or crash).
6. If a node's Controller is primary but has no Scheduler Process currently running for it, the Node Manager shall keep attempting to start one while that Controller remains primary.

Requirement 4: Liveness and health detection of managed components

Objective: As a cluster operator, I want the Node Manager to detect when a managed component has stopped doing useful work, not merely whether its process still exists, so that a hung or wedged component is recovered instead of silently leaving jobs unserved.

Acceptance Criteria

1. While a managed XCE component's process is running, that component shall periodically report its liveness to its node's Node Manager.
2. If a managed component does not report its liveness within an operator-configurable threshold, the Node Manager shall treat that component as unresponsive.
3. While the Scheduler Process is running, the liveness it reports to the Node Manager shall reflect whether it is making real scheduling progress, not merely that its process is still running.
4. When the Node Manager treats a managed component as unresponsive, the Node Manager shall stop that component and start a replacement, following the same startup behavior as if it had exited unexpectedly.
5. Where the managed component is the Reverse Proxy Process, the Node Manager shall determine its liveness by checking reachability of its configured network port instead of a liveness report from the process itself.
6. The Node Manager shall log an operator-visible event whenever it treats a managed component as unresponsive and replaces it.

Requirement 5: Non-interference with a managed component's own responsibilities

Objective: As a cluster operator, I want reporting liveness/role status to the Node Manager to never come at the cost of a component's primary responsibilities, so that adopting Node Manager supervision cannot itself become a new source of cluster unavailability.

Acceptance Criteria

1. A managed component's reporting of its liveness or role status to the Node Manager shall never block or delay that component's own primary processing responsibilities (for example, a Controller's handling of heartbeats or election timeouts).
2. If reporting liveness or role status to the Node Manager fails for any reason, the reporting component shall continue operating normally.
3. The absence of a running or reachable Node Manager shall not prevent a Controller from being elected primary, remaining primary, or serving its existing responsibilities.

Requirement 6: Independent verification of controlling authority (backstop)

Objective: As a cluster operator, I want the scheduler to independently notice when its own controller is no longer a live primary, so that a Scheduler Process can never keep running detached from any authoritative Controller even if the Node Manager that started it becomes unavailable.

Acceptance Criteria

1. While a Scheduler Process is running, the Scheduler Process shall independently verify that the specific Controller instance that started it is reachable and still holds the primary role.
2. When a Scheduler Process can no longer confirm that its originating Controller instance is reachable and primary, the Scheduler Process shall terminate on its own, independent of any action by the Node Manager.
3. The Scheduler Process shall not attempt to join, rejoin, or continue operating against any Controller instance other than the one that started it.

Requirement 7: Node Manager recovery without duplicate supervision

Objective: As a cluster operator, I want a restarted Node Manager to recognize the components already running on its node instead of starting duplicates, so that the Node Manager's own restart does not cause process thrashing or resource conflicts.

Acceptance Criteria

1. When the Node Manager starts and a component required by the node's assigned role is already running, the Node Manager shall recognize it as already running rather than starting a duplicate process for it.
2. The Node Manager shall log an operator-visible event whenever it adopts an already-running component on startup instead of starting a new one.

Requirement 8: Compatibility with existing deployment tooling and specs

Objective: As a cluster operator or spec author relying on existing deployment tooling, I want the introduction of node-role-driven component startup to be reflected everywhere a component's lifecycle is currently assumed to be independently and manually managed, so that existing deployments, tests, and specs do not silently break or become misleading.

Acceptance Criteria

1. Where existing deployment tooling documents starting the Controller, Scheduler Process, Worker Process, or Apiserver Process as an independent, manually-managed process, that tooling shall be updated to reflect that the Node Manager now starts these components according to each node's assigned role.
2. Where an existing spec or test tooling depends on a component being started or managed independently of the Node Manager, that dependency shall be identified and reconciled with this feature's behavior rather than left silently incompatible.
3. The observable behavior of the Controller, Scheduler Process, Worker Process, and Apiserver Process toward jobs, queues, configuration, and cluster membership shall remain unchanged by this feature while each is running.

Cross-Spec Interactions (2026-10-10, pending requirements revision)

Recorded during the long-running services and auto resource provisioning discoveries (.kiro/steering/roadmap.md). These are not yet acceptance criteria: the approved requirements and design above are unchanged until a revision (/kiro-spec-requirements xce-node-manager) folds them in or rejects them.

1. Two supervision layers, shared mechanics. The Node Manager supervises XCE's own components; the worker supervises user jobs, including service replicas (service-supervision). The Node Manager never supervises job processes (consistent with Requirement 5). Restart policy with backoff, give-up rule and probes (the Node Manager's TCP port check, the worker's command/TCP/HTTP probes) should come from one shared supervision/probe module so their semantics match.
2. Infrastructure services boundary. The Node Manager runs what XCE itself needs, or what must run before or without a Controller (XCE components, the reverse proxy). Everything else, including admin infrastructure services such as license servers or monitoring agents, runs as XCE service jobs (service-replica-set), scheduled with quotas, RBAC and accounting.
3. Roles for lease-based components. resource-provisioner adds a daemon: the Node Manager needs a provisioner role that only keeps the process running; which instance is active is decided by a Controller-granted lease, not by the Node Manager (unlike the Scheduler's primary-bound start/stop). Future XCE daemons should follow the same "role + lease" pattern.
4. Provisioned hosts and decommission (elastic-hosts). A cloud-burst host boots with the Node Manager and the worker role, declared by its host template and written into the Node Manager's configuration by the instance's boot script (one concrete answer to "how a node's role assignment is declared"). On decommission, the Node Manager must stop the worker on that node (or drop its role) instead of restarting it, and the Controller refuses a rejoin from a decommissioned host.
5. Worker restart vs. host loss (host-failure-job-recovery). Jobs run in systemd units and survive a worker daemon restart; the restarted worker re-reports them on rejoin. The host-loss grace period must exceed the Node Manager's worker restart plus rejoin time, so a worker crash never requeues or fails jobs (service replicas included). Optional: the Node Manager reports "worker restarting" so the Controller does not start the grace period.
6. Future service proxy (service-endpoint). A service proxy or load balancer, out of scope for service-endpoint v1, would naturally run under the Node Manager's reverse-proxy role, its routing fed by the endpoint table.