# XCE Controller

## Overview
C daemon managing cluster-level job and queue operations. Acts as the central coordinator between the scheduler, worker nodes, and the Go API server.

## Build
```bash
/xce-build controller    # debug + ASAN + clean build + runs tests
```
For full build system details, see `.kiro/steering/build-deployment.md`.

## Architecture
Core source files handle distinct domains:
- `job.c` — Job lifecycle management
- `queue.c` — Queue state and control
- `cluster.c` — Cluster topology
- `dataex.c` — Data exchange with other daemons

Links against `libxce` from `src/lib/`.

## Conventions
- C standard: gnu17
- Format with `clang-format`
- Component-specific headers in `include/`
- Shared types defined in `src/include/xce/` (libxce)

## Process Lifecycle (DESIGN INVARIANT — applies to ALL controller tasks)

**A controller process MUST NEVER self-terminate.** It exits ONLY on (a) a genuine crash (segfault/OOM) or (b) an admin/supervisor kill. A controller must NOT call `exit()`/`_exit()`/`abort()`/`tlog_exit()` (or any self-termination) as a **designed response to an operational condition** — e.g. a recovery-truncated/corrupt WAL, an inability to reach quorum, a peer being unreachable, a failed catch-up, or any cluster-state condition.

- **Rationale:** a self-exited controller requires **manual admin recovery** to come back and leaves quorum degraded in the meantime. A controller that **stays alive and keeps retrying self-heals automatically** when a peer returns — no admin in the loop.
- **"Fail loud" = refuse to participate, not exit.** When a controller detects it cannot safely act (e.g. it cannot reconstruct committed state, or would serve divergent state), it must: log a **loud, operator-visible ERROR** (with the actionable boundary, e.g. durable_floor vs cluster high-water), enter a **permanent non-promotable / fenced** state (never become authoritative, never serve divergent state), **stay alive**, and **keep retrying catch-up** so it recovers automatically when conditions allow. Clear the fenced state once a committed-consistent state is reached.
- **The only acceptable `assert()`/abort** is for a genuine *programming-invariant* violation (a bug that means the process is in an impossible state) — that is a "crash," not an operational response. Never use `assert()`/abort to handle expected operational/failure conditions.
- **When reviewing or writing controller code:** any new `exit`/`abort`/`tlog_exit`/`FATAL`-style terminal on an operational path is a design violation — replace it with the loud-fence-and-stay-alive pattern above. (Precedent: `ha-commit-durability` recovery-unrecoverable handling — `persist_recover_loud_fence_if_unrecoverable` + the `recovery_unrecoverable` sticky fence, which replaced an earlier incorrect `abort()` terminal.)

## HA Protocol

See `.kiro/steering/ha-async-io.md` for the full HA protocol reference: async I/O model (iomgr/vtaskmgr/delaymgr), connection ownership, Raft election, heartbeat model, two-tier replication consistency, dispatch gate, and TLA+ variable mapping. The TLA+ safety invariants and protocol verification checklist below are the authoritative controller-specific verification gates — keep them here.

### Raft TLA+ Formal Specification

The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft consensus algorithm. This is the authoritative reference for HA protocol correctness. **All HA protocol changes must be verified against this specification.**

### TLA+ Variable-to-Code Mapping

| TLA+ Variable | XCE Code | Location |
|--------------|----------|----------|
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | `ha.h:139` |
| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY,STANDBY} | `ha.h:139` |
| `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | `ha.h:161` |
| `log[i]` | WAL segments in `persist->wal_path` | `persist.c` |
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | `persist.h` |
| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) | `ha.h:141` |
| `nextIndex[i,j]` | `standby.join_applied_lsn` (replicator) | `repl.h` |
| `matchIndex[i,j]` | `standby.acked_lsn` (replicator) | `repl.h` |

### TLA+ Action-to-Code Mapping

| TLA+ Action | XCE Code | Location |
|------------|----------|----------|
| `Timeout(i)` | `raft_start_election` via delaymgr | `raft.c:57` |
| `RequestVote(i,j)` | `raft_on_vote_request` / reply path | `raft.c:106,169` |
| `BecomeLeader(i)` | `raft_cb_become_leader` | `ha.c:347` |
| `AdvanceCommitIndex(i)` | `repl_wait_majority_ack` → advance `committed_lsn` | `repl.c:338` |
| `AppendEntries(i,j)` | `OPCode_nop` heartbeat + WAL streaming | `ha.c:2880`, `config_repl.c:1489` |
| `ClientRequest(i,v)` | Job submit → WAL append → `repl_wait_majority_ack` | `persist.c`, `repl.c` |
| `UpdateTerm(i,j,m)` | Term CAS in `handle_request_vote` / vote reply handler | `ha.c:1218-1230` |

### Safety Invariants Verified

**Election Safety** (≤1 leader per term): Enforced by `votedFor` — a server grants at most one vote per term (`raft_on_vote_request` line 145-151). A candidate needs majority votes; each server votes at most once per term.

**Leader Completeness** (committed entries survive): Freshness gate in `RequestVote` — candidate must have `lastLogTerm ≥ voter` AND `lastLogIndex ≥ voter`. Since committed entries are on a majority, the new leader must have all committed entries. The post-election truncation removes only a divergent tail **above the leader's own durable log**: the cutoff is `ha_promotion_truncate_floor(safe_committed_lsn, durable_tip) = max(safe_committed_lsn, persist_durable_floor)`, so `persist_truncate_wal_above` is a no-op on the leader's own durable entries. **A new leader MUST NEVER truncate its own log** — entries it holds durably were acked by it (as a standby) and so were committed by a prior leader; truncating to the bare majority element destroys them (bug-1060: granted `{self=47, peer=44}` → `safe=44` cut the leader's own committed 45..47 → divergent WALs + permanent prevote deadlock).

**State Machine Safety** (no divergent entries): Monotonic `committed_lsn` + truncation on promotion. Entries at or below `committed_lsn` are identical across all servers.

**Commit Rule** (current-term entry required): Each WAL entry is committed individually via `repl_wait_majority_ack()`. Since entries are single-writer (only the current primary writes), every committed entry is implicitly from the current leader's term. Equivalent to Raft's batched commit rule.

**Commit Ordering** (contiguous, in log order — RULE): `committed_lsn` advances only as a contiguous prefix in LSN/append order; entries are never committed or reordered out of log order. The single-writer append lock (`p->lock`) is the serialization point. Durability flushing (incl. group-commit batching) makes a contiguous **prefix** durable only — `durable_lsn` is a prefix watermark, never a per-entry set, so a later entry can never be durable while an earlier one is not. Log order must preserve **causal dependencies** (append a cause before its effect — e.g. job submit before job start); independent concurrent events take an arbitrary but consistent total order, which is linearizable via the single serialization point plus commit-gated reply. Out-of-order / parallel commit is NOT supported by this single-log architecture (it would require per-entity logs with dependency tracking — explicitly out of scope).

**Commit Durability** (acknowledged ⇒ durable on a majority — GUARANTEED): when the controller replies success to a client, the operation's entry is `fdatasync`'d on a fault-tolerant majority (the acking member + ⌊N/2⌋ others); a member counts toward the commit majority for an LSN only once that LSN is durable locally (self caps at `durable_lsn`, not the appended tip). **CLOSED — bug-661, spec `ha-commit-durability` (landed 2026-06-18).** Mechanisms: `majority_candidate_locked` caps the primary self-contribution at `persist_durable_lsn` (not `highest_appended_lsn`); `fsync_worker` is a demand-driven group-commit flusher that advances `durable_lsn` only after a successful `fdatasync` and wakes the commit eval (fail-loud rc-gate — `durable_lsn` never advances on `fdatasync` error); the standby `fdatasync`s before its WalAck and the ACK is gated on that sync's success; recovery distinguishes a clean vs corruption-truncated WAL (`recover_verdict_t`) and re-derives the durable floor (WAL tip); the vote/PreVote freshness floor and a candidate's advertised position are durable-backed (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, last_known_committed_lsn)`), and a pre-promotion gate fences a recovery-truncated/behind member (it catches up before leading; if genuinely unrecoverable it stays a loud, non-promotable, *alive* fence — never self-exits — and self-heals on peer return). Verified by unit/integration tests (2.x, 5.1), the deterministic ack-before-fdatasync gap E2E (RED→GREEN), and the rolling-failover soak (zero acked-job loss, cluster stays available) — NOT by `raft.tla`, which assumes a durable log (restores `log` intact from stable storage on restart, `raft.tla:165-174`) and so cannot model the durability invariant; that invariant rests on these tests + this checklist. **Total-cluster-outage (cold-restart-all) — CLOSED 2026-07-02 (bug-795, `ha-commit-durability` Phase 6):** the original overlayfs-sandbox sighting (a committed job lost after killing ALL members + restarting) was investigated by reproducing the exact `cold-restart-all` scenario 5 times against a live 3-node cluster on a genuine ext4 filesystem (task 6.1) — zero instances of job loss across 5 full runs (40 rolling-failover rounds + 5 cold-restart-all cycles). Confirmed an **overlayfs `fdatasync` artifact**, not a genuine durability gap; no production fix was needed (6.2/6.3 N/A). The guarantee above is now proven for **both** rolling single-node failover **and** simultaneous total-cluster outage. (A separate, unrelated anomaly — an extra, unexpected committed job appearing after one cold-restart-all cycle, the opposite of loss — was logged as bug-1396, open/low-priority, not yet root-caused.)

### Post-Election Safe Commit Point (Critical)

After winning election, the new primary determines `safe_committed_lsn` from voter state collected during the election. This eliminates a round-trip vs waiting for the first heartbeat reply:

```
safe_committed_lsn = max(lsn : count of voters with peerLastLogLsn >= lsn) + 1 >= majority)
```

The `RequestVoteReply` message already carries `peerConfigTag` and `peerLastLogLsn` from each voter. The winning candidate uses these (from `ha_elect_ctx_t.votes[]`) to compute `safe_committed_lsn` immediately in `raft_cb_become_leader`, then truncates at `ha_promotion_truncate_floor(safe_committed_lsn, persist_durable_floor)` — i.e. `max(safe_committed_lsn, own durable tip)`, so it never truncates the leader's own durable log (bug-1060 Leader Completeness; `safe_committed_lsn` is still used as the commit lower-bound and the promotion-gate HWM).

**Never** use `last_known_committed_lsn` from the previous primary's ctrlhello for truncation — it may be stale (the old primary may have committed entries after the last ctrlhello was sent).

### Protocol Verification Checklist

When modifying HA protocol code (election, replication, heartbeat), verify:
- [ ] `votedFor` is updated atomically with term increment before granting any vote
- [ ] `currentTerm` is monotonic — never decreases
- [ ] `commitIndex` never decreases (monotonic)
- [ ] A leader only commits entries from its current term (or equivalent via per-entry majority ack)
- [ ] `safe_committed_lsn` is derived from voter replies, not from stale ctrlhello
- [ ] WAL truncation on promotion never falls below the leader's OWN durable tip — cutoff = `max(safe_committed_lsn, persist_durable_floor)` (`ha_promotion_truncate_floor`); a new leader never truncates its own committed log (bug-1060)
- [ ] Freshness gate in RequestVote: candidate must have `lastLogTerm >= voter` AND `lastLogIndex >= voter`
- [ ] AppendEntries carries TWO distinct terms: the HEADER term = leader's CURRENT term (UpdateTerm/stale-check) and each ENTRY's STORED term (logOk/persist). For catch-up of historical entries they DIFFER — the AE header term must be the leader's current term (`wal_catchup_entry_t.leader_current_term`), never the entry's stored term, or the library stale-rejects every cross-term entry (bug-1067)
- [ ] `committed_lsn` advances only as a contiguous LSN-ordered prefix — no out-of-order or per-entry commit
- [ ] Durability flushing makes a contiguous prefix durable; `durable_lsn` is a prefix watermark (never a per-entry set)
- [ ] Causal dependencies are appended in order (cause before effect, e.g. submit before start)
- [x] (post `ha-commit-durability`, landed 2026-06-18) a member counts toward the commit majority only at its durably-persisted LSN; acknowledged ⇒ durable on a majority

## HA Design Constraints

Covered in `.kiro/steering/ha-async-io.md` (connection ownership, async I/O, heartbeat, two-tier replication, dispatch gate) and `.kiro/steering/build-deployment.md` (deploy commands, cluster startup/restart/verification). The TLA+ safety invariants above remain the controller-specific verification reference.

## Key Files

## See Also
- Shared library: `src/lib/CLAUDE.md`
- Root guidelines: `CLAUDE.md` (repo root)