myxce/.kiro/specs/xce-node-manager/research.md

# Research & Design Decisions

## Summary
- **Feature**: `xce-node-manager`
- **Discovery Scope**: Complex Integration — new top-level daemon, no direct precedent, but heavily grounded in existing patterns across controller/worker/scheduler/apiserver and the AppImage packaging layer.
- **Key Findings**:
  - `tools/appimage/entry.sh` is a thin per-role dispatcher (`xce <role> args...`) that execs exactly one of `xce_controller`/`xcecd`/`xce_scheduler`/`xce_apiserver` in the foreground — there is no existing orchestration layer to extend or conflict with, and `haproxy` (already bundled in the AppImage per `build-deployment.md`) has no dispatch case at all today, i.e. it is currently unreachable through `entry.sh`.
  - `src/worker/src/process_worker_manager.c`'s existing `process_manager_start_worker()` spawns via raw `fork()+execl()`, not `posix_spawn()` — an already-shipped instance of the exact mutex-inheritance-deadlock hazard class `product.md`'s locked decision ("no `fork()` for snapshots") and the superseded `scheduler-primary-lifecycle` design already ruled out. Not fixed here (out of scope, unrelated component) but explicitly NOT a pattern to imitate.
  - The controller already has a non-blocking, delaymgr-driven, role-aware periodic task (`ha_start_primary_heartbeat`/`primary_heartbeat_fn`, `ha.c:9949`) reading the atomic `ha->role` every tick — proof that a cheap, always-safe periodic local report is a well-trodden pattern in this codebase, though this specific function is gated on first-promotion (`s_armed`) and therefore unsuitable to piggyback on directly (see Decision: Heartbeat Call Site).
  - `src/configs/hosts.yaml` (the cluster's existing per-host config, distributed via `OPCode_replConfigPush`) describes scheduler-visible compute-host attributes, not deployment/process-placement roles — confirms no existing config surface for "which node runs which XCE component," and that any such config cannot be cluster-distributed (a node's role must be knowable before any controller is reachable).

## Research Log

### Existing multi-component launch/packaging surface
- **Context**: Before designing a new orchestrator, check whether one already exists in disguise (AppImage bundling, systemd units, an existing launcher script).
- **Sources Consulted**: `tools/appimage/entry.sh`, `build-deployment.md` (AppImage/Production Artifacts sections), `xce-deploy-ha` SKILL.md, `xce-test-ha/run-ha-soak.sh`.
- **Findings**: The AppImage bundles all daemons plus `haproxy` behind one `xce <role>` entry point, but `entry.sh` only launches a single named process in the foreground per invocation — no supervision, no multi-process orchestration, no config-driven role resolution. The local dev/test harness (`xce-deploy-ha`) generates one shell script per component (`start.sh`, `start-worker.sh`, `start-scheduler.sh`), started manually/independently — the same "independently-started component" assumption `scheduler-primary-lifecycle`'s requirements already flagged, now generalized to worker/apiserver too.
- **Implications**: `xce-node-manager` is additive, not competing with an existing mechanism. The natural integration point is one new `entry.sh` case (`manager`) that execs the new `xce_nodemgr` binary; `xce_nodemgr` itself then resolves and spawns the other bundled binaries directly (same `which xce_controller`-style PATH resolution `entry.sh` already uses), including finally wiring up the previously-unreachable bundled `haproxy` for the reverse-proxy role.

### Controller-side periodic reporting hook
- **Context**: Where should the Controller's "I'm alive / I'm primary or standby" report be triggered from, without touching the promotion/demotion callback functions those steering docs flag as the riskiest, least-covered code in the controller?
- **Sources Consulted**: `ha.c` (`primary_heartbeat_fn`, `ha_start_primary_heartbeat`, `raft_cb_become_leader`/`raft_cb_become_follower` — via `codegraph_explore`), `ha-async-io.md`.
- **Findings**: `primary_heartbeat_fn` is armed only once, on a controller instance's *first* promotion (`s_armed` guard) — a controller that has never won an election never runs it, which would leave `xce-node-manager` unable to distinguish "controller alive, still standby" from "controller process is down" on any node that hasn't yet been promoted. `raft_cb_become_leader`/`raft_cb_become_follower` themselves are flagged with no direct test coverage and are the single most call-dense, safety-critical functions in the controller (122+ calls from `raft_cb_become_leader` alone).
- **Implications**: The report must come from a small, independent, unconditionally-armed periodic point tied to the Controller's own process lifetime (not gated on promotion), reading the same atomic `ha->role` field `primary_heartbeat_fn` already reads. This adds zero new logic to `raft_cb_become_leader`/`raft_cb_become_follower` — a strict reduction in touched surface area compared to the superseded spec, which called full spawn/monitor orchestration from inside `raft_cb_become_leader` directly.

### Scheduler wedge-detection signal shape
- **Context**: The superseded spec's Requirement 5 (carried forward as 4.3 here) requires detecting a scheduler that is connected but not making real scheduling progress — a bare "I'm alive" ping cannot distinguish a wedged decision-loop from a healthy one if the two run on different threads.
- **Sources Consulted**: In-conversation review of `scheduler-primary-lifecycle`'s `design.md` (`SchedulerConnectionBookkeeping`'s last-progress-timestamp approach, fed from real dispatch/pend-reason traffic, not a synthetic heartbeat); bug-1760's own root-cause text describing the scheduler's *outbound wire traffic* as reactive/idle-silent ("the scheduler has nothing queued to publish").
- **Findings**: Reusing the scheduler's reactive wire-publish call as the heartbeat trigger would reintroduce the exact idle-silence failure mode bug-1760 already exposed. The correct trigger is the scheduler's own internal round/picker loop iteration — which must tick on a bounded cadence regardless of whether a round finds any job to act on — not the wire-publish call.
- **Implications**: One wire-format field (a monotonically increasing sequence number) serves double duty: for the Controller/Worker/Apiserver it is a bare liveness counter; for the Scheduler, since it is incremented once per round-loop iteration (not once per successful publish), its advancement *is* the real-progress signal Requirement 4.3 needs, with no separate progress field required. See Decision: Unified Heartbeat Wire Format.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Adopt a generic external supervisor (systemd user units, s6, runit, supervisord) for all four roles | Delegate process start/restart entirely to an existing, battle-tested tool | Zero new code for the generic "keep running" case (Requirement 2) | Cannot express Requirement 3's Controller-role-driven conditional scheduler start/stop or Requirement 4.3's progress-aware wedge detection — those are XCE-domain-specific decisions no generic supervisor can make; also conflicts with the AppImage's self-contained, zero-external-dependency portability goal (`product.md`) across arbitrary air-gapped target hosts that may not run systemd | Rejected as the primary mechanism; the generic host service manager is still the right tool for the one thing left over — restarting `xce-node-manager` itself, already scoped out of this feature (Requirement 7 boundary) |
| Custom `xce-node-manager` daemon using only POSIX primitives (`posix_spawn`, Unix domain datagram sockets), consistent with the already-reviewed `scheduler-primary-lifecycle` precedent | Build a small, purpose-specific supervisor | Can express both the generic and Controller-role-driven policies; no new external runtime dependency; reuses the `posix_spawn` decision already vetted against the mutex-inheritance-deadlock hazard | New component to build and test | **Selected** |

## Design Decisions

### Decision: Heartbeat call site is each component's own natural loop iteration, not a new thread
- **Context**: Requirements 4.1 and 4.3 require every managed component to report liveness, and require the Scheduler's report specifically to reflect real progress.
- **Alternatives Considered**:
  1. A dedicated heartbeat-sender thread per component, independent of its main work loop — simplest to bolt on, but structurally cannot distinguish a wedged work thread from a healthy one (the sender thread keeps reporting regardless).
  2. Piggyback the Controller's report on the existing `primary_heartbeat_fn` — rejected, see Research Log (promotion-gated arming leaves never-yet-primary controllers silent).
- **Selected Approach**: Each component calls a shared `libxce` heartbeat primitive once per iteration of its own existing loop (the Controller's iomgr/delaymgr-driven event loop, the Scheduler's round loop, the Worker's `ipc_worker_run`/`usleep` loop); the Apiserver, having no persistent loop of its own (a request-driven Gin server), uses a small dedicated ticker goroutine instead — the one place a new "thread" is unavoidable, since there is no existing loop to hook into.
- **Rationale**: Guarantees the report can only advance when the component's own real work-loop is actually turning, which is what makes the same signal double as both liveness and (for the Scheduler) progress. The Scheduler's live driver (`scheduler.c`, `sched_job_picker`) is the correct call site, not the pure decision engine in `core.c`/`core_events.c`/`core_session.c`/`core_clock.c` — the `scheduler-core` spec's own core/driver separation already keeps the core free of controller I/O, and this feature's local heartbeat is new outside-facing I/O that belongs on the same side of that boundary as the existing `xcecli` connection. The Scheduler's separate simulation-mode driver (`sim.c`) gets no heartbeat call site at all and stays entirely outside `xce-node-manager`'s awareness (see Non-Goals in `design.md`) — it is a standalone, manually-invoked tool with no Controller or primary/standby concept.
- **Trade-offs**: Slightly different integration shape per component (loop hook vs. ticker) instead of one uniform mechanism; acceptable since the underlying wire format and `libxce` primitive are identical either way.
- **Follow-up resolved during design review** (not deferred to implementation): `codegraph_explore` confirmed `sched_job_picker` calls `cbuf_get()` (`src/lib/mylock.c:144`), an **unbounded** `pthread_cond_wait` when the job queue is empty — a heartbeat placed naively at the top of that loop would never fire during genuine idle periods, causing `xce-node-manager` to falsely treat a healthy idle Scheduler as wedged and kill it. Fix: swap to `cbuf_timedget()` (same file, an existing bounded-deadline variant already used elsewhere in the codebase) with a bounded idle-tick deadline, and call the heartbeat primitive on both the real-job path and the timeout path.

### Decision: Unified heartbeat wire format, self-identifying payload, no CGO for the Apiserver client
- **Context**: The heartbeat protocol must be usable by four component types across two languages (C daemons, Go apiserver) over one shared local socket, with the receiving `xce-node-manager` able to attribute each datagram without relying on socket-level peer identity (a `SOCK_DGRAM` Unix socket exposes no reliable peer address unless the sender also binds, which adds needless coupling).
- **Alternatives Considered**:
  1. Route the Apiserver's heartbeat through `libxce` via the existing CGO bridge, matching how `apiserver`'s C-tier calls already work.
  2. A small, independently-implemented Go client speaking the identical wire format natively, with no CGO involved.
- **Selected Approach**: One canonical fixed-layout binary wire format (magic/version, role, status, pid, sequence number — no Cap'n Proto), implemented once in `libxce` for the three C daemons (which already link it natively, zero extra cost), and once as a small native Go implementation for the Apiserver.
- **Rationale**: The payload is a fixed, trivial, few-field structure — reimplementing it natively in Go is a handful of lines with negligible drift risk, and avoids pulling a CGO dependency into a code path (a periodic status ticker) that has no other reason to touch the C tier; `tech.md` already documents that Apiserver's own unit tests deliberately run `CGO_ENABLED=0`, and only the existing backend bridge work requires the CGO build. The payload self-identifies its sender (role + pid fields), so the receiver never depends on socket-level peer identity. This is a deliberate, narrow exception to `structure.md`'s "Cap'n Proto for wire formats" convention: that convention targets the real cross-host, cross-version network protocol (`controller.capnp`/`c.capnp`), where schema evolution and cross-language generated bindings earn their cost; this protocol is same-host-only, always emitted and consumed by binaries from the same build (Node Manager and the components it manages ship together), and confirmed (`go.mod`/`go.sum` inspection) that the Go apiserver carries no existing capnp dependency at all — adopting Cap'n Proto here would mean a new Go dependency purely for a fixed 4-field local datagram, which the Simplification lens rejects as disproportionate.
- **Trade-offs**: Two implementations of the same format instead of one; mitigated by the format's small size and by both being validated against the same contract in this design.
- **Follow-up**: none — the format is fixed at design time (see Data Models in `design.md`).

### Decision: Node Manager privilege is role-conditional (root only on worker-role nodes)
- **Context**: `xce-deploy-ha`'s own tooling documents that the real (non-virtual) Worker daemon `xcecd` runs as root ("plain kill can't touch it," needs `sudo`) — discovered during design review, not anticipated during requirements. A uniform single-OS-user assumption for `xce-node-manager` (as first drafted) would leave it unable to start/stop/signal `xcecd` on a real worker-role node.
- **Alternatives Considered**:
  1. `xce-node-manager` always runs as root everywhere, regardless of role.
  2. `xce-node-manager` stays unprivileged everywhere; grant it a narrow Linux capability or scoped `sudo` rule for the worker case specifically.
  3. Treat real (root-requiring) worker management as out of scope for this spec; only support `xcecd_virtual`.
- **Selected Approach** (user decision): `xce-node-manager` runs as root only on a node whose assigned role includes `worker`; on manager/apiserver/reverse-proxy-only nodes it runs as an ordinary service account. This is fixed by the Node Role Manifest at Node Manager startup, not a runtime toggle.
- **Rationale**: Matches `xcecd`'s own pre-existing requirement exactly, with no new privilege-escalation surface beyond what that daemon already needs; simpler than a capability-based scheme, and avoids narrowing this spec's worker-role support to the virtual test double only.
- **Trade-offs**: The heartbeat socket's permission model is now per-node-role rather than one uniform story (see `design.md` Security Considerations) — an implementer must not assume a single OS user across all node roles.
- **Follow-up**: none — fixed at design time.

## Risks & Mitigations
- A node-manager crash on a manager node strands its Controller's Scheduler with nothing left to stop it on a subsequent demotion — mitigated by retaining a scheduler-side independent controller-liveness check as a backstop (Requirement 6), so the Scheduler never depends solely on `xce-node-manager` staying alive.
- `xce-node-manager` restarting could double-spawn a component that is already running — mitigated by a small persisted per-role record (role, pid, start time, binary path) it consults on startup before spawning (see `design.md` Data Models); a stale/mismatched record falls back to a fresh start rather than blindly trusting the file, and each component's own listen-port bind-failure is a natural backstop against an accidental true duplicate.
- Sizing the heartbeat interval/timeout thresholds is an implementation-time tuning decision (same posture the superseded spec already took for its own liveness thresholds), not a design decision.

## References
- `.kiro/specs/scheduler-primary-lifecycle/design.md` — superseded design; source of the `posix_spawn` mutex-inheritance-deadlock finding and the wedge-detection real-progress principle, both carried forward here.
- `.wolf/buglog.json` bug-1760 — motivating incident for the scheduler-lifecycle requirements this spec restates.