myxce/src/controller/CLAUDE.md

XCE Controller

Overview

C daemon managing cluster-level job and queue operations. Acts as the central coordinator between the scheduler, worker nodes, and the Go API server.

Build

/xce-build controller    # debug + ASAN + clean build + runs tests

For full build system details, see .kiro/steering/build-deployment.md.

Architecture

Core source files handle distinct domains:

• job.c — Job lifecycle management
• queue.c — Queue state and control
• cluster.c — Cluster topology
• dataex.c — Data exchange with other daemons

Links against libxce from src/lib/.

Conventions

• C standard: gnu17
• Format with clang-format
• Component-specific headers in include/
• Shared types defined in src/include/xce/ (libxce)

Process Lifecycle (DESIGN INVARIANT — applies to ALL controller tasks)

A controller process MUST NEVER self-terminate. It exits ONLY on (a) a genuine crash (segfault/OOM) or (b) an admin/supervisor kill. A controller must NOT call exit()/_exit()/abort()/tlog_exit() (or any self-termination) as a designed response to an operational condition — e.g. a recovery-truncated/corrupt WAL, an inability to reach quorum, a peer being unreachable, a failed catch-up, or any cluster-state condition.

• Rationale: a self-exited controller requires manual admin recovery to come back and leaves quorum degraded in the meantime. A controller that stays alive and keeps retrying self-heals automatically when a peer returns — no admin in the loop.
• "Fail loud" = refuse to participate, not exit. When a controller detects it cannot safely act (e.g. it cannot reconstruct committed state, or would serve divergent state), it must: log a loud, operator-visible ERROR (with the actionable boundary, e.g. durable_floor vs cluster high-water), enter a permanent non-promotable / fenced state (never become authoritative, never serve divergent state), stay alive, and keep retrying catch-up so it recovers automatically when conditions allow. Clear the fenced state once a committed-consistent state is reached.
• The only acceptable assert()/abort is for a genuine programming-invariant violation (a bug that means the process is in an impossible state) — that is a "crash," not an operational response. Never use assert()/abort to handle expected operational/failure conditions.
• When reviewing or writing controller code: any new exit/abort/tlog_exit/FATAL-style terminal on an operational path is a design violation — replace it with the loud-fence-and-stay-alive pattern above. (Precedent: ha-commit-durability recovery-unrecoverable handling — persist_recover_loud_fence_if_unrecoverable + the recovery_unrecoverable sticky fence, which replaced an earlier incorrect abort() terminal.)

HA Protocol

See .kiro/steering/ha-async-io.md for the full HA protocol reference: async I/O model (iomgr/vtaskmgr/delaymgr), connection ownership, Raft election, heartbeat model, two-tier replication consistency, dispatch gate, and TLA+ variable mapping. The TLA+ safety invariants and protocol verification checklist below are the authoritative controller-specific verification gates — keep them here.

Raft TLA+ Formal Specification

The file raft/raft.tla contains Diego Ongaro's TLA+ specification of the Raft consensus algorithm. This is the authoritative reference for HA protocol correctness. All HA protocol changes must be verified against this specification.

TLA+ Variable-to-Code Mapping

TLA+ Variable	XCE Code	Location
currentTerm[i]	ha->term (_Atomic uint64_t)	ha.h:139
state[i] ∈ {Follower,Candidate,Leader}	ha->role ∈ {INIT,ELECTING,PRIMARY,STANDBY}	ha.h:139
votedFor[i]	ha->voted_for_name[XCE_MAXNAME_LEN]	ha.h:161
log[i]	WAL segments in persist->wal_path	persist.c
commitIndex[i]	persist->committed_lsn (_Atomic uint64_t)	persist.h
lastApplied[i]	ha->applied_lsn (_Atomic uint64_t)	ha.h:141
nextIndex[i,j]	standby.join_applied_lsn (replicator)	repl.h
matchIndex[i,j]	standby.acked_lsn (replicator)	repl.h

TLA+ Action-to-Code Mapping

TLA+ Action	XCE Code	Location
Timeout(i)	raft_start_election via delaymgr	raft.c:57
RequestVote(i,j)	raft_on_vote_request / reply path	raft.c:106,169
BecomeLeader(i)	raft_cb_become_leader	ha.c:347
AdvanceCommitIndex(i)	repl_wait_majority_ack → advance committed_lsn	repl.c:338
AppendEntries(i,j)	OPCode_nop heartbeat + WAL streaming	ha.c:2880, config_repl.c:1489
ClientRequest(i,v)	Job submit → WAL append → repl_wait_majority_ack	persist.c, repl.c
UpdateTerm(i,j,m)	Term CAS in handle_request_vote / vote reply handler	ha.c:1218-1230

Safety Invariants Verified

Election Safety (≤1 leader per term): Enforced by votedFor — a server grants at most one vote per term (raft_on_vote_request line 145-151). A candidate needs majority votes; each server votes at most once per term.

Leader Completeness (committed entries survive): Freshness gate in RequestVote — candidate must have lastLogTerm ≥ voter AND lastLogIndex ≥ voter. Since committed entries are on a majority, the new leader must have all committed entries. The post-election truncation removes only a divergent tail above the leader's own durable log: the cutoff is ha_promotion_truncate_floor(safe_committed_lsn, durable_tip) = max(safe_committed_lsn, persist_durable_floor), so persist_truncate_wal_above is a no-op on the leader's own durable entries. A new leader MUST NEVER truncate its own log — entries it holds durably were acked by it (as a standby) and so were committed by a prior leader; truncating to the bare majority element destroys them (bug-1060: granted {self=47, peer=44} → safe=44 cut the leader's own committed 45..47 → divergent WALs + permanent prevote deadlock).

State Machine Safety (no divergent entries): Monotonic committed_lsn + truncation on promotion. Entries at or below committed_lsn are identical across all servers.

Commit Rule (current-term entry required): Each WAL entry is committed individually via repl_wait_majority_ack(). Since entries are single-writer (only the current primary writes), every committed entry is implicitly from the current leader's term. Equivalent to Raft's batched commit rule.

Commit Ordering (contiguous, in log order — RULE): committed_lsn advances only as a contiguous prefix in LSN/append order; entries are never committed or reordered out of log order. The single-writer append lock (p->lock) is the serialization point. Durability flushing (incl. group-commit batching) makes a contiguous prefix durable only — durable_lsn is a prefix watermark, never a per-entry set, so a later entry can never be durable while an earlier one is not. Log order must preserve causal dependencies (append a cause before its effect — e.g. job submit before job start); independent concurrent events take an arbitrary but consistent total order, which is linearizable via the single serialization point plus commit-gated reply. Out-of-order / parallel commit is NOT supported by this single-log architecture (it would require per-entity logs with dependency tracking — explicitly out of scope).

Commit Durability (acknowledged ⇒ durable on a majority — GUARANTEED): when the controller replies success to a client, the operation's entry is fdatasync'd on a fault-tolerant majority (the acking member + ⌊N/2⌋ others); a member counts toward the commit majority for an LSN only once that LSN is durable locally (self caps at durable_lsn, not the appended tip). CLOSED — bug-661, spec ha-commit-durability (landed 2026-06-18). Mechanisms: majority_candidate_locked caps the primary self-contribution at persist_durable_lsn (not highest_appended_lsn); fsync_worker is a demand-driven group-commit flusher that advances durable_lsn only after a successful fdatasync and wakes the commit eval (fail-loud rc-gate — durable_lsn never advances on fdatasync error); the standby fdatasyncs before its WalAck and the ACK is gated on that sync's success; recovery distinguishes a clean vs corruption-truncated WAL (recover_verdict_t) and re-derives the durable floor (WAL tip); the vote/PreVote freshness floor and a candidate's advertised position are durable-backed (candidate advertises bare persist_durable_floor, voter floor = max(durable_floor, last_known_committed_lsn)), and a pre-promotion gate fences a recovery-truncated/behind member (it catches up before leading; if genuinely unrecoverable it stays a loud, non-promotable, alive fence — never self-exits — and self-heals on peer return). Verified by unit/integration tests (2.x, 5.1), the deterministic ack-before-fdatasync gap E2E (RED→GREEN), and the rolling-failover soak (zero acked-job loss, cluster stays available) — NOT by raft.tla, which assumes a durable log (restores log intact from stable storage on restart, raft.tla:165-174) and so cannot model the durability invariant; that invariant rests on these tests + this checklist. Total-cluster-outage (cold-restart-all) — CLOSED 2026-07-02 (bug-795, ha-commit-durability Phase 6): the original overlayfs-sandbox sighting (a committed job lost after killing ALL members + restarting) was investigated by reproducing the exact cold-restart-all scenario 5 times against a live 3-node cluster on a genuine ext4 filesystem (task 6.1) — zero instances of job loss across 5 full runs (40 rolling-failover rounds + 5 cold-restart-all cycles). Confirmed an overlayfs fdatasync artifact, not a genuine durability gap; no production fix was needed (6.2/6.3 N/A). The guarantee above is now proven for both rolling single-node failover and simultaneous total-cluster outage. (A separate, unrelated anomaly — an extra, unexpected committed job appearing after one cold-restart-all cycle, the opposite of loss — was logged as bug-1396, open/low-priority, not yet root-caused.)

Post-Election Safe Commit Point (Critical)

After winning election, the new primary determines safe_committed_lsn from voter state collected during the election. This eliminates a round-trip vs waiting for the first heartbeat reply:

safe_committed_lsn = max(lsn : count of voters with peerLastLogLsn >= lsn) + 1 >= majority)

The RequestVoteReply message already carries peerConfigTag and peerLastLogLsn from each voter. The winning candidate uses these (from ha_elect_ctx_t.votes[]) to compute safe_committed_lsn immediately in raft_cb_become_leader, then truncates at ha_promotion_truncate_floor(safe_committed_lsn, persist_durable_floor) — i.e. max(safe_committed_lsn, own durable tip), so it never truncates the leader's own durable log (bug-1060 Leader Completeness; safe_committed_lsn is still used as the commit lower-bound and the promotion-gate HWM).

Never use last_known_committed_lsn from the previous primary's ctrlhello for truncation — it may be stale (the old primary may have committed entries after the last ctrlhello was sent).

Protocol Verification Checklist

When modifying HA protocol code (election, replication, heartbeat), verify:

• votedFor is updated atomically with term increment before granting any vote
• currentTerm is monotonic — never decreases
• commitIndex never decreases (monotonic)
• A leader only commits entries from its current term (or equivalent via per-entry majority ack)
• safe_committed_lsn is derived from voter replies, not from stale ctrlhello
• WAL truncation on promotion never falls below the leader's OWN durable tip — cutoff = max(safe_committed_lsn, persist_durable_floor) (ha_promotion_truncate_floor); a new leader never truncates its own committed log (bug-1060)
• Freshness gate in RequestVote: candidate must have lastLogTerm >= voter AND lastLogIndex >= voter
• AppendEntries carries TWO distinct terms: the HEADER term = leader's CURRENT term (UpdateTerm/stale-check) and each ENTRY's STORED term (logOk/persist). For catch-up of historical entries they DIFFER — the AE header term must be the leader's current term (wal_catchup_entry_t.leader_current_term), never the entry's stored term, or the library stale-rejects every cross-term entry (bug-1067)
• committed_lsn advances only as a contiguous LSN-ordered prefix — no out-of-order or per-entry commit
• Durability flushing makes a contiguous prefix durable; durable_lsn is a prefix watermark (never a per-entry set)
• Causal dependencies are appended in order (cause before effect, e.g. submit before start)
• (post ha-commit-durability, landed 2026-06-18) a member counts toward the commit majority only at its durably-persisted LSN; acknowledged ⇒ durable on a majority

HA Design Constraints

Covered in .kiro/steering/ha-async-io.md (connection ownership, async I/O, heartbeat, two-tier replication, dispatch gate) and .kiro/steering/build-deployment.md (deploy commands, cluster startup/restart/verification). The TLA+ safety invariants above remain the controller-specific verification reference.

Key Files

See Also

• Shared library: src/lib/CLAUDE.md
• Root guidelines: CLAUDE.md (repo root)