myxce/src/controller/CLAUDE.md

XCE Controller

Overview

C daemon managing cluster-level job and queue operations. Acts as the central coordinator between the scheduler, worker nodes, and the Go API server.

Build

/xce-build controller    # debug + ASAN + clean build + runs tests

For full build system details, see .kiro/steering/build-deployment.md.

Architecture

Core source files handle distinct domains:

• job.c — Job lifecycle management
• queue.c — Queue state and control
• cluster.c — Cluster topology
• dataex.c — Data exchange with other daemons

Links against libxce from src/lib/.

Conventions

• C standard: gnu17
• Format with clang-format
• Component-specific headers in include/
• Shared types defined in src/include/xce/ (libxce)

HA Protocol

See .kiro/steering/ha-async-io.md for the full HA protocol reference: async I/O model (iomgr/vtaskmgr/delaymgr), connection ownership, Raft election, heartbeat model, two-tier replication consistency, dispatch gate, and TLA+ variable mapping. The TLA+ safety invariants and protocol verification checklist below are the authoritative controller-specific verification gates — keep them here.

Raft TLA+ Formal Specification

The file raft/raft.tla contains Diego Ongaro's TLA+ specification of the Raft consensus algorithm. This is the authoritative reference for HA protocol correctness. All HA protocol changes must be verified against this specification.

TLA+ Variable-to-Code Mapping

TLA+ Variable	XCE Code	Location
currentTerm[i]	ha->term (_Atomic uint64_t)	ha.h:139
state[i] ∈ {Follower,Candidate,Leader}	ha->role ∈ {INIT,ELECTING,PRIMARY_COMMITTING,PRIMARY,STANDBY}	ha.h:139
votedFor[i]	ha->voted_for_name[XCE_MAXNAME_LEN]	ha.h:161
log[i]	WAL segments in persist->wal_path	persist.c
commitIndex[i]	persist->committed_lsn (_Atomic uint64_t)	persist.h
lastApplied[i]	ha->applied_lsn (_Atomic uint64_t)	ha.h:141
nextIndex[i,j]	standby.join_applied_lsn (replicator)	repl.h
matchIndex[i,j]	standby.acked_lsn (replicator)	repl.h

TLA+ Action-to-Code Mapping

TLA+ Action	XCE Code	Location
Timeout(i)	raft_start_election via delaymgr	raft.c:57
RequestVote(i,j)	raft_on_vote_request / reply path	raft.c:106,169
BecomeLeader(i)	raft_cb_become_leader	ha.c:347
AdvanceCommitIndex(i)	repl_wait_majority_ack → advance committed_lsn	repl.c:338
AppendEntries(i,j)	OPCode_nop heartbeat + WAL streaming	ha.c:2880, config_repl.c:1489
ClientRequest(i,v)	Job submit → WAL append → repl_wait_majority_ack	persist.c, repl.c
UpdateTerm(i,j,m)	Term CAS in handle_request_vote / vote reply handler	ha.c:1218-1230

Safety Invariants Verified

Election Safety (≤1 leader per term): Enforced by votedFor — a server grants at most one vote per term (raft_on_vote_request line 145-151). A candidate needs majority votes; each server votes at most once per term.

Leader Completeness (committed entries survive): Freshness gate in RequestVote — candidate must have lastLogTerm ≥ voter AND lastLogIndex ≥ voter. Since committed entries are on a majority, the new leader must have all committed entries. persist_truncate_wal_above(safe_committed_lsn) removes any divergent entries above the safe commit point.

State Machine Safety (no divergent entries): Monotonic committed_lsn + truncation on promotion. Entries at or below committed_lsn are identical across all servers.

Commit Rule (current-term entry required): Each WAL entry is committed individually via repl_wait_majority_ack(). Since entries are single-writer (only the current primary writes), every committed entry is implicitly from the current leader's term. Equivalent to Raft's batched commit rule.

Post-Election Safe Commit Point (Critical)

After winning election, the new primary determines safe_committed_lsn from voter state collected during the election. This eliminates a round-trip vs waiting for the first heartbeat reply:

safe_committed_lsn = max(lsn : count of voters with peerLastLogLsn >= lsn) + 1 >= majority)

The RequestVoteReply message already carries peerConfigTag and peerLastLogLsn from each voter. The winning candidate uses these (from ha_elect_ctx_t.votes[]) to compute safe_committed_lsn immediately in raft_cb_become_leader, then calls persist_truncate_wal_above(safe_committed_lsn).

Never use last_known_committed_lsn from the previous primary's ctrlhello for truncation — it may be stale (the old primary may have committed entries after the last ctrlhello was sent).

Protocol Verification Checklist

When modifying HA protocol code (election, replication, heartbeat), verify:

• votedFor is updated atomically with term increment before granting any vote
• currentTerm is monotonic — never decreases
• commitIndex never decreases (monotonic)
• A leader only commits entries from its current term (or equivalent via per-entry majority ack)
• safe_committed_lsn is derived from voter replies, not from stale ctrlhello
• WAL truncation on promotion only removes entries above the safe commit point
• Freshness gate in RequestVote: candidate must have lastLogTerm >= voter AND lastLogIndex >= voter

HA Design Constraints

Covered in .kiro/steering/ha-async-io.md (connection ownership, async I/O, heartbeat, two-tier replication, dispatch gate) and .kiro/steering/build-deployment.md (deploy commands, cluster startup/restart/verification). The TLA+ safety invariants above remain the controller-specific verification reference.

Key Files

See Also

• Shared library: src/lib/CLAUDE.md
• Root guidelines: CLAUDE.md (repo root)