# XCE Controller

## Overview
C daemon managing cluster-level job and queue operations. Acts as the central coordinator between the scheduler, worker nodes, and the Go API server.

## Build
```bash
/xce-build controller    # debug + ASAN + clean build + runs tests
```
For full build system details, see `.kiro/steering/build-deployment.md`.

## Architecture
Core source files handle distinct domains:
- `job.c` — Job lifecycle management
- `queue.c` — Queue state and control
- `cluster.c` — Cluster topology
- `dataex.c` — Data exchange with other daemons

Links against `libxce` from `src/lib/`.

## Conventions
- C standard: gnu17
- Format with `clang-format`
- Component-specific headers in `include/`
- Shared types defined in `src/include/xce/` (libxce)

## Process Lifecycle (DESIGN INVARIANT — applies to ALL controller tasks)

**A controller process MUST NEVER self-terminate.** It exits ONLY on (a) a genuine crash (segfault/OOM) or (b) an admin/supervisor kill. A controller must NOT call `exit()`/`_exit()`/`abort()`/`tlog_exit()` (or any self-termination) as a **designed response to an operational condition** — e.g. a recovery-truncated/corrupt WAL, an inability to reach quorum, a peer being unreachable, a failed catch-up, or any cluster-state condition.

- **Rationale:** a self-exited controller requires **manual admin recovery** to come back and leaves quorum degraded in the meantime. A controller that **stays alive and keeps retrying self-heals automatically** when a peer returns — no admin in the loop.
- **"Fail loud" = refuse to participate, not exit.** When a controller detects it cannot safely act (e.g. it cannot reconstruct committed state, or would serve divergent state), it must: log a **loud, operator-visible ERROR** (with the actionable boundary, e.g. durable_floor vs cluster high-water), enter a **permanent non-promotable / fenced** state (never become authoritative, never serve divergent state), **stay alive**, and **keep retrying catch-up** so it recovers automatically when conditions allow. Clear the fenced state once a committed-consistent state is reached.
- **The only acceptable `assert()`/abort** is for a genuine *programming-invariant* violation (a bug that means the process is in an impossible state) — that is a "crash," not an operational response. Never use `assert()`/abort to handle expected operational/failure conditions.
- **When reviewing or writing controller code:** any new `exit`/`abort`/`tlog_exit`/`FATAL`-style terminal on an operational path is a design violation — replace it with the loud-fence-and-stay-alive pattern above. (Precedent: `ha-commit-durability` recovery-unrecoverable handling — `persist_recover_loud_fence_if_unrecoverable` + the `recovery_unrecoverable` sticky fence, which replaced an earlier incorrect `abort()` terminal.)

## HA Protocol

See `.kiro/steering/ha-async-io.md` for the full HA protocol reference: async I/O model (iomgr/vtaskmgr/delaymgr), connection ownership, Raft election, heartbeat model, two-tier replication consistency, dispatch gate, and TLA+ variable mapping. The TLA+ safety invariants and protocol verification checklist below are the authoritative controller-specific verification gates — keep them here.

### Raft TLA+ Formal Specification

The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft consensus algorithm. This is the authoritative reference for HA protocol correctness. **All HA protocol changes must be verified against this specification.**

### TLA+ Variable-to-Code Mapping

| TLA+ Variable | XCE Code | Location |
|--------------|----------|----------|
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | `ha.h:139` |
| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY,STANDBY} | `ha.h:139` |
| `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | `ha.h:161` |
| `log[i]` | WAL segments in `persist->wal_path` | `persist.c` |
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | `persist.h` |
| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) | `ha.h:141` |
| `nextIndex[i,j]` | `standby.join_applied_lsn` (replicator) | `repl.h` |
| `matchIndex[i,j]` | `standby.acked_lsn` (replicator) | `repl.h` |

### TLA+ Action-to-Code Mapping

| TLA+ Action | XCE Code | Location |
|------------|----------|----------|
| `Timeout(i)` | `raft_start_election` via delaymgr | `raft.c:57` |
| `RequestVote(i,j)` | `raft_on_vote_request` / reply path | `raft.c:106,169` |
| `BecomeLeader(i)` | `raft_cb_become_leader` | `ha.c:347` |
| `AdvanceCommitIndex(i)` | `repl_wait_majority_ack` → advance `committed_lsn` | `repl.c:338` |
| `AppendEntries(i,j)` — production send (LIVE) | `repl_broadcast` (per-entry fan-out on append, NACK-driven retry via `ha_raft_on_repl_nack`) + `ha_stream_wal_to_standby` (catch-up-range streaming) — the sole production `AppendEntries`-equivalent send/retry/backoff mechanism (raft-adapter-conformance Req 3.3) | `repl.c:295`, `config_repl.c:2351` |
| `AppendEntries(i,j)` — nextIndex-driven sender (DORMANT, documented) | `raft_send_append_entries` is fully protocol-conformant but has ZERO production callers (only `raft_sim.c` and an `XCE_TESTING` test helper); its registered `on_send_append_entries` callback `ha_cb_send_append_entries_stub` is reachable only through it. Kept as an intentionally-dormant reference implementation (retire/wire-in both rejected, see `research.md`), guarded against silent regression by `ha_send_append_entries_stub_call_count` staying 0 while the production path fires (`test_ha_core_safety.c` task 4.2) | `src/lib/raft/raft.c:595`, `ha.c:2590` (stub), `ha.c:2588` (guard counter) |
| `AppendEntries(i,j)` reply — WAL catch-up reject | Every WAL catch-up rejection sends an explicit `WalAckReply{success=0}` via `ha_send_wal_ack_reply_ex` back to the primary — never a silent drop (bug-1479; reply-construction boundary regression-locked by raft-adapter-conformance task 5.1) | `ha.c:3155` (fn), `ha.c:3190` (accept call site), `ha.c:3326` (reject call site) |
| `AppendEntries(i,j)` reply — configuration-push accept/reject | Every `handle_repl_config_push` outcome — all 4 reject branches plus the success/`-EAGAIN`-activation-deferred tail — sends exactly one explicit `ReplConfigPushReply{accepted, term}` via `send_config_push_reply`, so the primary can distinguish an active reject from a lost push (raft-adapter-conformance Req 2.2-2.5); the reply's `term` field feeds the primary's `raft_on_higher_term_observed` term-adoption alongside the heartbeat and WAL-NACK vectors (Req 7.4/7.5) | `config_repl.c:1267` (send fn), `config_repl.c:1308-1381` (handler), `cluster.c:750-780` (primary-side accept/reject branch + term-adoption) |
| `RequestVote`/`PreVote(i,j)` reply — endpoint-failure routing, all timings | A vote/PreVote endpoint failure is always counted through the production tally (`vote_reply_dispatch` / `pre_vote_reply_dispatch`), never the legacy `handle_request_vote_reply` path — true both at connect-time (bug-1423/1424, regression-locked) and post-connect, i.e. a failure after connect but before a reply arrives (raft-adapter-conformance Req 4.2/4.3, closing that one remaining failure-timing gap). `vote_ep_err_handler` (real-vote) and `pre_vote_ep_err_handler` (PreVote) are separate handlers sharing one `CTRL_PRIMARY`-cleanup helper | `ha.c:7667` (`vote_ep_err_handler`), `ha.c:7705` (`pre_vote_ep_err_handler`), `ha.c:7627` (shared cleanup helper), registered at `ha.c:692` / `ha.c:831` |
| `ClientRequest(i,v)` | Job submit → WAL append → `repl_wait_majority_ack` | `persist.c`, `repl.c` |
| `UpdateTerm(i,j,m)` | Term CAS in `handle_request_vote` / vote reply handler | `ha.c:1218-1230` |

> Full gap-analysis citations, options considered, and design rationale for the `raft-adapter-conformance` rows above (including the dormant-sender decision and the vote/PreVote handler-split decision): `.kiro/specs/raft-adapter-conformance/research.md` and `design.md`. This table stays the terse, current pointer; the evidentiary trail lives there.

### Safety Invariants Verified

**Election Safety** (≤1 leader per term): Enforced by `votedFor` — a server grants at most one vote per term (`raft_on_vote_request` line 145-151). A candidate needs majority votes; each server votes at most once per term.

**Leader Completeness** (committed entries survive): Freshness gate in `RequestVote` — candidate must have `lastLogTerm ≥ voter` AND `lastLogIndex ≥ voter`. Since committed entries are on a majority, the new leader must have all committed entries. The post-election truncation removes only a divergent tail **above the leader's own durable log**: the cutoff is `ha_promotion_truncate_floor(safe_committed_lsn, durable_tip) = max(safe_committed_lsn, persist_durable_floor)`, so `persist_truncate_wal_above` is a no-op on the leader's own durable entries. **A new leader MUST NEVER truncate its own log** — entries it holds durably were acked by it (as a standby) and so were committed by a prior leader; truncating to the bare majority element destroys them (bug-1060: granted `{self=47, peer=44}` → `safe=44` cut the leader's own committed 45..47 → divergent WALs + permanent prevote deadlock).

**State Machine Safety** (no divergent entries): Monotonic `committed_lsn` + truncation on promotion. Entries at or below `committed_lsn` are identical across all servers.

**Commit Rule** (current-term entry required): Each WAL entry is committed individually via `repl_wait_majority_ack()`. Since entries are single-writer (only the current primary writes), every committed entry is implicitly from the current leader's term. Equivalent to Raft's batched commit rule.

**Commit Ordering** (contiguous, in log order — RULE): `committed_lsn` advances only as a contiguous prefix in LSN/append order; entries are never committed or reordered out of log order. The single-writer append lock (`p->lock`) is the serialization point. Durability flushing (incl. group-commit batching) makes a contiguous **prefix** durable only — `durable_lsn` is a prefix watermark, never a per-entry set, so a later entry can never be durable while an earlier one is not. Log order must preserve **causal dependencies** (append a cause before its effect — e.g. job submit before job start); independent concurrent events take an arbitrary but consistent total order, which is linearizable via the single serialization point plus commit-gated reply. Out-of-order / parallel commit is NOT supported by this single-log architecture (it would require per-entity logs with dependency tracking — explicitly out of scope).

**Commit Durability** (acknowledged ⇒ durable on a majority — GUARANTEED): when the controller replies success to a client, the operation's entry is `fdatasync`'d on a fault-tolerant majority (the acking member + ⌊N/2⌋ others); a member counts toward the commit majority for an LSN only once that LSN is durable locally (self caps at `durable_lsn`, not the appended tip). **CLOSED — bug-661, spec `ha-commit-durability` (landed 2026-06-18).** Mechanisms: `majority_candidate_locked` caps the primary self-contribution at `persist_durable_lsn` (not `highest_appended_lsn`); `fsync_worker` is a demand-driven group-commit flusher that advances `durable_lsn` only after a successful `fdatasync` and wakes the commit eval (fail-loud rc-gate — `durable_lsn` never advances on `fdatasync` error); the standby `fdatasync`s before its WalAck and the ACK is gated on that sync's success; recovery distinguishes a clean vs corruption-truncated WAL (`recover_verdict_t`) and re-derives the durable floor (WAL tip); the vote/PreVote freshness floor and a candidate's advertised position are durable-backed (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, last_known_committed_lsn)`), and a pre-promotion gate fences a recovery-truncated/behind member (it catches up before leading; if genuinely unrecoverable it stays a loud, non-promotable, *alive* fence — never self-exits — and self-heals on peer return). Verified by unit/integration tests (2.x, 5.1), the deterministic ack-before-fdatasync gap E2E (RED→GREEN), and the rolling-failover soak (zero acked-job loss, cluster stays available) — NOT by `raft.tla`, which assumes a durable log (restores `log` intact from stable storage on restart, `raft.tla:165-174`) and so cannot model the durability invariant; that invariant rests on these tests + this checklist. **Total-cluster-outage (cold-restart-all) — CLOSED 2026-07-02 (bug-795, `ha-commit-durability` Phase 6):** the original overlayfs-sandbox sighting (a committed job lost after killing ALL members + restarting) was investigated by reproducing the exact `cold-restart-all` scenario 5 times against a live 3-node cluster on a genuine ext4 filesystem (task 6.1) — zero instances of job loss across 5 full runs (40 rolling-failover rounds + 5 cold-restart-all cycles). Confirmed an **overlayfs `fdatasync` artifact**, not a genuine durability gap; no production fix was needed (6.2/6.3 N/A). The guarantee above is now proven for **both** rolling single-node failover **and** simultaneous total-cluster outage. (A separate, unrelated anomaly — an extra, unexpected committed job appearing after one cold-restart-all cycle, the opposite of loss — was logged as bug-1396, open/low-priority, not yet root-caused.)

### Post-Election Safe Commit Point (Critical)

After winning election, the new primary determines `safe_committed_lsn` from voter state collected during the election. This eliminates a round-trip vs waiting for the first heartbeat reply:

```
safe_committed_lsn = max(lsn : count of voters with peerLastLogLsn >= lsn) + 1 >= majority)
```

The `RequestVoteReply` message already carries `peerConfigTag` and `peerLastLogLsn` from each voter. The winning candidate uses these (from `ha_elect_ctx_t.votes[]`) to compute `safe_committed_lsn` immediately in `raft_cb_become_leader`, then truncates at `ha_promotion_truncate_floor(safe_committed_lsn, persist_durable_floor)` — i.e. `max(safe_committed_lsn, own durable tip)`, so it never truncates the leader's own durable log (bug-1060 Leader Completeness; `safe_committed_lsn` is still used as the commit lower-bound and the promotion-gate HWM).

**Never** use `last_known_committed_lsn` from the previous primary's ctrlhello for truncation — it may be stale (the old primary may have committed entries after the last ctrlhello was sent).

### Protocol Verification Checklist

When modifying HA protocol code (election, replication, heartbeat), verify:
- [ ] `votedFor` is updated atomically with term increment before granting any vote (library: `raft_on_vote_request`, `src/lib/raft/raft.c:178-270` — term update before persist, `voted_for` set only after persist succeeds; adapter: `raft_cb_persist`, `ha.c:1784-1824`, writes term+voted_for atomically under `ha->lock`. `handle_request_vote` in `ha.c` is `#ifdef XCE_TESTING`-only; production dispatch is `cluster.c:477` → `raft_on_vote_request`. Corrected 2026-07-18 — the prior `ha.c:1218-1230` citation had drifted onto unrelated post-election code.)
- [ ] `currentTerm` is monotonic — never decreases
- [ ] `commitIndex` never decreases (monotonic)
- [ ] A leader only commits entries from its current term (or equivalent via per-entry majority ack)
- [ ] `safe_committed_lsn` is derived from voter replies, not from stale ctrlhello
- [ ] WAL truncation on promotion never falls below the leader's OWN durable tip — cutoff = `max(safe_committed_lsn, persist_durable_floor)` (`ha_promotion_truncate_floor`); a new leader never truncates its own committed log (bug-1060)
- [ ] Freshness gate in RequestVote: candidate must have `lastLogTerm >= voter` AND `lastLogIndex >= voter`
- [ ] AppendEntries carries TWO distinct terms: the HEADER term = leader's CURRENT term (UpdateTerm/stale-check) and each ENTRY's STORED term (logOk/persist). For catch-up of historical entries they DIFFER — the AE header term must be the leader's current term (`wal_catchup_entry_t.leader_current_term`), never the entry's stored term, or the library stale-rejects every cross-term entry (bug-1067)
- [ ] `committed_lsn` advances only as a contiguous LSN-ordered prefix — no out-of-order or per-entry commit
- [ ] Durability flushing makes a contiguous prefix durable; `durable_lsn` is a prefix watermark (never a per-entry set)
- [ ] Causal dependencies are appended in order (cause before effect, e.g. submit before start)
- [x] (post `ha-commit-durability`, landed 2026-06-18) a member counts toward the commit majority only at its durably-persisted LSN; acknowledged ⇒ durable on a majority
- [x] (post `raft-adapter-conformance`, landed 2026-07-29) every reject of a replication-class message (WAL catch-up entry, configuration push) sends an explicit reply distinguishable from a lost message — never a silent drop (`ha_send_wal_ack_reply_ex(success=0)` for WAL catch-up, bug-1479; `send_config_push_reply(accepted=0)` for config-push, Req 2.2-2.5)
- [x] (post `raft-adapter-conformance`, landed 2026-07-29) every vote/PreVote endpoint failure — connect-time or post-connect (a failure after connect but before a reply) — is counted through the production tally (`vote_reply_dispatch`/`pre_vote_reply_dispatch`), never the legacy `handle_request_vote_reply` path (Req 4.1-4.4)
- [x] (post `raft-adapter-conformance`, landed 2026-07-29) `raft_send_append_entries`/`ha_cb_send_append_entries_stub` remains dormant in production (zero callers) by deliberate documented decision, not oversight — `ha_send_append_entries_stub_call_count` stays 0 while `repl_broadcast`/`ha_stream_wal_to_standby` fire for the same scenario (Req 3.3)

## HA Design Constraints

Covered in `.kiro/steering/ha-async-io.md` (connection ownership, async I/O, heartbeat, two-tier replication, dispatch gate) and `.kiro/steering/build-deployment.md` (deploy commands, cluster startup/restart/verification). The TLA+ safety invariants above remain the controller-specific verification reference.

## Key Files

## Current Functional Status (2026-08-15)

### Completed (HA / reliability layer is mature)
- **Raft/HA family fully landed**: term-safety, commit-durability (ack ⇒ durable on majority), commit-barrier-rollout, commit-engine, wal-replication, apply-decoupling (canonical pre-hooks → commit → apply → notify), single-leader-role, convergence-recovery, config-bootstrap, libxce-failover, controller-status (18/18 tasks).
- **Job-array single WAL record** (job-array-single-wal-record).
- **Scheduler message-gap** G-4/G-5/G-6/G-8 largely implemented (`sched_update_existing_job`, rusage storage, distinct UPDATE_JOB_FIELDS op).
- **job_ctrl** (suspend/resume/hold/release/kill) and **queue_ctrl** (open/close/inactivate) exist in `job.c`/`queue.c`.

### Gaps / missing functionality
These are the main functional shortfalls (see `.kiro/steering/roadmap.md` for full backlog):

| Area | Gap | Location / note |
|---|---|---|
| **Job control residual** | `job_ctrl` direct-mutate path (admin suspend/resume) still mutates state then fires hook — not commit-then-apply (same family as bug-795/828/851/1060, narrowly scoped to admin suspend/resume) | roadmap Hardening: "investigate: job_ctrl direct-mutate path" |
| **Worker rejoin resync** | On rejoin/failover, worker does NOT re-report running/finished job states (`xce_join_req` carries only `hostattrs_t`, no job state) → facts sent during disconnect/failover window are lost | roadmap backlog: worker-rejoin-resync |
| **Component failover adoption** | Daemons still pass single endpoint (`conf->masters[0]`) to libxce; bounded-retry liveness not yet verified per component | roadmap backlog: component-failover-adoption |
| **Config-upgrade snapshot** | `do_upgrade_config_with_bytes` does config-push+restart+per-node replay — can leave divergent materialized state (bug-857); needs entity-bearing snapshots (HARD dep, still 16-byte LSN sentinel) | roadmap backlog: config-upgrade-snapshot, entity-bearing-snapshots |
| **Job-array bulk query/cancel** | Single client op to query/cancel whole array not implemented (per-element only) | roadmap backlog: bulk job-array query/cancel |
| **Preemption** | No mechanism to preempt lower-priority running jobs (high-priority waits for free resources) | roadmap backlog / scheduler-preemption spec |
| **Group building** | `cluster_build_groups()` is an empty TODO stub | `cluster.c:997` |
| **API-server fact delivery** | FIXMEs: job pid / exit code / fail reason not sent to apiserver | `apiserver.c:248,283,318` |
| **Peer auth** | `validate_host()` is a stub returning success for all peers; `OP_upgradeConfig`/`OP_replConfig` unauthenticated (trusted-LAN model) | roadmap Hardening: validate_host, peer-authentication-hardening (security-critical) |
| **Admin login** | xceadmin uses zero-config TLS, no login/token/session/audit | roadmap Hardening: peer-authentication-hardening (d) |

### Note
Reliability (HA/durability/consistency) is the mature layer; the gaps above are mostly **business/operational features** (bulk ops, preemption, auth, rejoin resync) plus a few correctness residuals (job_ctrl mutate-before-commit). See also `src/scheduler/CLAUDE.md` for scheduler-side status.

## See Also
- Shared library: `src/lib/CLAUDE.md`
- Root guidelines: `CLAUDE.md` (repo root)