# XCE Controller

## Overview
C daemon managing cluster-level job and queue operations. Acts as the central coordinator between the scheduler, worker nodes, and the Go API server.

## Build
```bash
/xce-build controller    # debug + ASAN + clean build + runs tests
```
For full build system details, see `.kiro/steering/build-deployment.md`.

## Architecture
Core source files handle distinct domains:
- `job.c` — Job lifecycle management
- `queue.c` — Queue state and control
- `cluster.c` — Cluster topology
- `dataex.c` — Data exchange with other daemons

Links against `libxce` from `src/lib/`.

## Conventions
- C standard: gnu17
- Format with `clang-format`
- Component-specific headers in `include/`
- Shared types defined in `src/include/xce/` (libxce)

## HA Protocol

See `.kiro/steering/ha-async-io.md` for the full HA protocol reference: async I/O model (iomgr/vtaskmgr/delaymgr), connection ownership, Raft election, heartbeat model, two-tier replication consistency, dispatch gate, and TLA+ variable mapping. The TLA+ safety invariants and protocol verification checklist below are the authoritative controller-specific verification gates — keep them here.

### Raft TLA+ Formal Specification

The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft consensus algorithm. This is the authoritative reference for HA protocol correctness. **All HA protocol changes must be verified against this specification.**

### TLA+ Variable-to-Code Mapping

| TLA+ Variable | XCE Code | Location |
|--------------|----------|----------|
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | `ha.h:139` |
| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY_COMMITTING,PRIMARY,STANDBY} | `ha.h:139` |
| `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | `ha.h:161` |
| `log[i]` | WAL segments in `persist->wal_path` | `persist.c` |
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | `persist.h` |
| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) | `ha.h:141` |
| `nextIndex[i,j]` | `standby.join_applied_lsn` (replicator) | `repl.h` |
| `matchIndex[i,j]` | `standby.acked_lsn` (replicator) | `repl.h` |

### TLA+ Action-to-Code Mapping

| TLA+ Action | XCE Code | Location |
|------------|----------|----------|
| `Timeout(i)` | `raft_start_election` via delaymgr | `raft.c:57` |
| `RequestVote(i,j)` | `raft_on_vote_request` / reply path | `raft.c:106,169` |
| `BecomeLeader(i)` | `raft_cb_become_leader` | `ha.c:347` |
| `AdvanceCommitIndex(i)` | `repl_wait_majority_ack` → advance `committed_lsn` | `repl.c:338` |
| `AppendEntries(i,j)` | `OPCode_nop` heartbeat + WAL streaming | `ha.c:2880`, `config_repl.c:1489` |
| `ClientRequest(i,v)` | Job submit → WAL append → `repl_wait_majority_ack` | `persist.c`, `repl.c` |
| `UpdateTerm(i,j,m)` | Term CAS in `handle_request_vote` / vote reply handler | `ha.c:1218-1230` |

### Safety Invariants Verified

**Election Safety** (≤1 leader per term): Enforced by `votedFor` — a server grants at most one vote per term (`raft_on_vote_request` line 145-151). A candidate needs majority votes; each server votes at most once per term.

**Leader Completeness** (committed entries survive): Freshness gate in `RequestVote` — candidate must have `lastLogTerm ≥ voter` AND `lastLogIndex ≥ voter`. Since committed entries are on a majority, the new leader must have all committed entries. `persist_truncate_wal_above(safe_committed_lsn)` removes any divergent entries above the safe commit point.

**State Machine Safety** (no divergent entries): Monotonic `committed_lsn` + truncation on promotion. Entries at or below `committed_lsn` are identical across all servers.

**Commit Rule** (current-term entry required): Each WAL entry is committed individually via `repl_wait_majority_ack()`. Since entries are single-writer (only the current primary writes), every committed entry is implicitly from the current leader's term. Equivalent to Raft's batched commit rule.

### Post-Election Safe Commit Point (Critical)

After winning election, the new primary determines `safe_committed_lsn` from voter state collected during the election. This eliminates a round-trip vs waiting for the first heartbeat reply:

```
safe_committed_lsn = max(lsn : count of voters with peerLastLogLsn >= lsn) + 1 >= majority)
```

The `RequestVoteReply` message already carries `peerConfigTag` and `peerLastLogLsn` from each voter. The winning candidate uses these (from `ha_elect_ctx_t.votes[]`) to compute `safe_committed_lsn` immediately in `raft_cb_become_leader`, then calls `persist_truncate_wal_above(safe_committed_lsn)`.

**Never** use `last_known_committed_lsn` from the previous primary's ctrlhello for truncation — it may be stale (the old primary may have committed entries after the last ctrlhello was sent).

### Protocol Verification Checklist

When modifying HA protocol code (election, replication, heartbeat), verify:
- [ ] `votedFor` is updated atomically with term increment before granting any vote
- [ ] `currentTerm` is monotonic — never decreases
- [ ] `commitIndex` never decreases (monotonic)
- [ ] A leader only commits entries from its current term (or equivalent via per-entry majority ack)
- [ ] `safe_committed_lsn` is derived from voter replies, not from stale ctrlhello
- [ ] WAL truncation on promotion only removes entries above the safe commit point
- [ ] Freshness gate in RequestVote: candidate must have `lastLogTerm >= voter` AND `lastLogIndex >= voter`

## HA Design Constraints

Covered in `.kiro/steering/ha-async-io.md` (connection ownership, async I/O, heartbeat, two-tier replication, dispatch gate) and `.kiro/steering/build-deployment.md` (deploy commands, cluster startup/restart/verification). The TLA+ safety invariants above remain the controller-specific verification reference.

## Key Files

## See Also
- Shared library: `src/lib/CLAUDE.md`
- Root guidelines: `CLAUDE.md` (repo root)