myxce/src/controller/CLAUDE.md

XCE Controller

Overview

C daemon managing cluster-level job and queue operations. Acts as the central coordinator between the scheduler, worker nodes, and the Go API server.

Build

/xce-build controller    # debug + ASAN + clean build + runs tests

For full build system details, see .kiro/steering/build-deployment.md.

Architecture

Core source files handle distinct domains:

• job.c — Job lifecycle management
• queue.c — Queue state and control
• cluster.c — Cluster topology
• dataex.c — Data exchange with other daemons

Links against libxce from src/lib/.

Conventions

• C standard: gnu17
• Format with clang-format
• Component-specific headers in include/
• Shared types defined in src/include/xce/ (libxce)

Process Lifecycle (DESIGN INVARIANT — applies to ALL controller tasks)

A controller process MUST NEVER self-terminate. It exits ONLY on (a) a genuine crash (segfault/OOM) or (b) an admin/supervisor kill. A controller must NOT call exit()/_exit()/abort()/tlog_exit() (or any self-termination) as a designed response to an operational condition — e.g. a recovery-truncated/corrupt WAL, an inability to reach quorum, a peer being unreachable, a failed catch-up, or any cluster-state condition.

• Rationale: a self-exited controller requires manual admin recovery to come back and leaves quorum degraded in the meantime. A controller that stays alive and keeps retrying self-heals automatically when a peer returns — no admin in the loop.
• "Fail loud" = refuse to participate, not exit. When a controller detects it cannot safely act (e.g. it cannot reconstruct committed state, or would serve divergent state), it must: log a loud, operator-visible ERROR (with the actionable boundary, e.g. durable_floor vs cluster high-water), enter a permanent non-promotable / fenced state (never become authoritative, never serve divergent state), stay alive, and keep retrying catch-up so it recovers automatically when conditions allow. Clear the fenced state once a committed-consistent state is reached.
• The only acceptable assert()/abort is for a genuine programming-invariant violation (a bug that means the process is in an impossible state) — that is a "crash," not an operational response. Never use assert()/abort to handle expected operational/failure conditions.
• When reviewing or writing controller code: any new exit/abort/tlog_exit/FATAL-style terminal on an operational path is a design violation — replace it with the loud-fence-and-stay-alive pattern above. (Precedent: ha-commit-durability recovery-unrecoverable handling — persist_recover_loud_fence_if_unrecoverable + the recovery_unrecoverable sticky fence, which replaced an earlier incorrect abort() terminal.)

HA Protocol

See .kiro/steering/ha-async-io.md for the full HA protocol reference: async I/O model (iomgr/vtaskmgr/delaymgr), connection ownership, Raft election, heartbeat model, two-tier replication consistency, dispatch gate, and TLA+ variable mapping. The TLA+ safety invariants and protocol verification checklist below are the authoritative controller-specific verification gates — keep them here.

Raft TLA+ Formal Specification

The file raft/raft.tla contains Diego Ongaro's TLA+ specification of the Raft consensus algorithm. This is the authoritative reference for HA protocol correctness. All HA protocol changes must be verified against this specification.

TLA+ Variable-to-Code Mapping

TLA+ Variable	XCE Code	Location
currentTerm[i]	ha->term (_Atomic uint64_t)	ha.h:139
state[i] ∈ {Follower,Candidate,Leader}	ha->role ∈ {INIT,ELECTING,PRIMARY,STANDBY}	ha.h:139
votedFor[i]	ha->voted_for_name[XCE_MAXNAME_LEN]	ha.h:161
log[i]	WAL segments in persist->wal_path	persist.c
commitIndex[i]	persist->committed_lsn (_Atomic uint64_t)	persist.h
lastApplied[i]	ha->applied_lsn (_Atomic uint64_t)	ha.h:141
nextIndex[i,j]	standby.join_applied_lsn (replicator)	repl.h
matchIndex[i,j]	standby.acked_lsn (replicator)	repl.h

TLA+ Action-to-Code Mapping

TLA+ Action	XCE Code	Location
Timeout(i)	raft_start_election via delaymgr	raft.c:57
RequestVote(i,j)	raft_on_vote_request / reply path	raft.c:106,169
BecomeLeader(i)	raft_cb_become_leader	ha.c:347
AdvanceCommitIndex(i)	repl_wait_majority_ack → advance committed_lsn	repl.c:338
AppendEntries(i,j) — production send (LIVE)	repl_broadcast (per-entry fan-out on append, NACK-driven retry via ha_raft_on_repl_nack) + ha_stream_wal_to_standby (catch-up-range streaming) — the sole production AppendEntries-equivalent send/retry/backoff mechanism (raft-adapter-conformance Req 3.3)	repl.c:295, config_repl.c:2351
AppendEntries(i,j) — nextIndex-driven sender (DORMANT, documented)	raft_send_append_entries is fully protocol-conformant but has ZERO production callers (only raft_sim.c and an XCE_TESTING test helper); its registered on_send_append_entries callback ha_cb_send_append_entries_stub is reachable only through it. Kept as an intentionally-dormant reference implementation (retire/wire-in both rejected, see research.md), guarded against silent regression by ha_send_append_entries_stub_call_count staying 0 while the production path fires (test_ha_core_safety.c task 4.2)	src/lib/raft/raft.c:595, ha.c:2590 (stub), ha.c:2588 (guard counter)
AppendEntries(i,j) reply — WAL catch-up reject	Every WAL catch-up rejection sends an explicit WalAckReply{success=0} via ha_send_wal_ack_reply_ex back to the primary — never a silent drop (bug-1479; reply-construction boundary regression-locked by raft-adapter-conformance task 5.1)	ha.c:3155 (fn), ha.c:3190 (accept call site), ha.c:3326 (reject call site)
AppendEntries(i,j) reply — configuration-push accept/reject	Every handle_repl_config_push outcome — all 4 reject branches plus the success/-EAGAIN-activation-deferred tail — sends exactly one explicit ReplConfigPushReply{accepted, term} via send_config_push_reply, so the primary can distinguish an active reject from a lost push (raft-adapter-conformance Req 2.2-2.5); the reply's term field feeds the primary's raft_on_higher_term_observed term-adoption alongside the heartbeat and WAL-NACK vectors (Req 7.4/7.5)	config_repl.c:1267 (send fn), config_repl.c:1308-1381 (handler), cluster.c:750-780 (primary-side accept/reject branch + term-adoption)
RequestVote/PreVote(i,j) reply — endpoint-failure routing, all timings	A vote/PreVote endpoint failure is always counted through the production tally (vote_reply_dispatch / pre_vote_reply_dispatch), never the legacy handle_request_vote_reply path — true both at connect-time (bug-1423/1424, regression-locked) and post-connect, i.e. a failure after connect but before a reply arrives (raft-adapter-conformance Req 4.2/4.3, closing that one remaining failure-timing gap). vote_ep_err_handler (real-vote) and pre_vote_ep_err_handler (PreVote) are separate handlers sharing one CTRL_PRIMARY-cleanup helper	ha.c:7667 (vote_ep_err_handler), ha.c:7705 (pre_vote_ep_err_handler), ha.c:7627 (shared cleanup helper), registered at ha.c:692 / ha.c:831
ClientRequest(i,v)	Job submit → WAL append → repl_wait_majority_ack	persist.c, repl.c
UpdateTerm(i,j,m)	Term CAS in handle_request_vote / vote reply handler	ha.c:1218-1230

Full gap-analysis citations, options considered, and design rationale for the raft-adapter-conformance rows above (including the dormant-sender decision and the vote/PreVote handler-split decision): .kiro/specs/raft-adapter-conformance/research.md and design.md. This table stays the terse, current pointer; the evidentiary trail lives there.

Safety Invariants Verified

Election Safety (≤1 leader per term): Enforced by votedFor — a server grants at most one vote per term (raft_on_vote_request line 145-151). A candidate needs majority votes; each server votes at most once per term.

Leader Completeness (committed entries survive): Freshness gate in RequestVote — candidate must have lastLogTerm ≥ voter AND lastLogIndex ≥ voter. Since committed entries are on a majority, the new leader must have all committed entries. The post-election truncation removes only a divergent tail above the leader's own durable log: the cutoff is ha_promotion_truncate_floor(safe_committed_lsn, durable_tip) = max(safe_committed_lsn, persist_durable_floor), so persist_truncate_wal_above is a no-op on the leader's own durable entries. A new leader MUST NEVER truncate its own log — entries it holds durably were acked by it (as a standby) and so were committed by a prior leader; truncating to the bare majority element destroys them (bug-1060: granted {self=47, peer=44} → safe=44 cut the leader's own committed 45..47 → divergent WALs + permanent prevote deadlock).

State Machine Safety (no divergent entries): Monotonic committed_lsn + truncation on promotion. Entries at or below committed_lsn are identical across all servers.

Commit Rule (current-term entry required): Each WAL entry is committed individually via repl_wait_majority_ack(). Since entries are single-writer (only the current primary writes), every committed entry is implicitly from the current leader's term. Equivalent to Raft's batched commit rule.

Commit Ordering (contiguous, in log order — RULE): committed_lsn advances only as a contiguous prefix in LSN/append order; entries are never committed or reordered out of log order. The single-writer append lock (p->lock) is the serialization point. Durability flushing (incl. group-commit batching) makes a contiguous prefix durable only — durable_lsn is a prefix watermark, never a per-entry set, so a later entry can never be durable while an earlier one is not. Log order must preserve causal dependencies (append a cause before its effect — e.g. job submit before job start); independent concurrent events take an arbitrary but consistent total order, which is linearizable via the single serialization point plus commit-gated reply. Out-of-order / parallel commit is NOT supported by this single-log architecture (it would require per-entity logs with dependency tracking — explicitly out of scope).

Commit Durability (acknowledged ⇒ durable on a majority — GUARANTEED): when the controller replies success to a client, the operation's entry is fdatasync'd on a fault-tolerant majority (the acking member + ⌊N/2⌋ others); a member counts toward the commit majority for an LSN only once that LSN is durable locally (self caps at durable_lsn, not the appended tip). CLOSED — bug-661, spec ha-commit-durability (landed 2026-06-18). Mechanisms: majority_candidate_locked caps the primary self-contribution at persist_durable_lsn (not highest_appended_lsn); fsync_worker is a demand-driven group-commit flusher that advances durable_lsn only after a successful fdatasync and wakes the commit eval (fail-loud rc-gate — durable_lsn never advances on fdatasync error); the standby fdatasyncs before its WalAck and the ACK is gated on that sync's success; recovery distinguishes a clean vs corruption-truncated WAL (recover_verdict_t) and re-derives the durable floor (WAL tip); the vote/PreVote freshness floor and a candidate's advertised position are durable-backed (candidate advertises bare persist_durable_floor, voter floor = max(durable_floor, last_known_committed_lsn)), and a pre-promotion gate fences a recovery-truncated/behind member (it catches up before leading; if genuinely unrecoverable it stays a loud, non-promotable, alive fence — never self-exits — and self-heals on peer return). Verified by unit/integration tests (2.x, 5.1), the deterministic ack-before-fdatasync gap E2E (RED→GREEN), and the rolling-failover soak (zero acked-job loss, cluster stays available) — NOT by raft.tla, which assumes a durable log (restores log intact from stable storage on restart, raft.tla:165-174) and so cannot model the durability invariant; that invariant rests on these tests + this checklist. Total-cluster-outage (cold-restart-all) — CLOSED 2026-07-02 (bug-795, ha-commit-durability Phase 6): the original overlayfs-sandbox sighting (a committed job lost after killing ALL members + restarting) was investigated by reproducing the exact cold-restart-all scenario 5 times against a live 3-node cluster on a genuine ext4 filesystem (task 6.1) — zero instances of job loss across 5 full runs (40 rolling-failover rounds + 5 cold-restart-all cycles). Confirmed an overlayfs fdatasync artifact, not a genuine durability gap; no production fix was needed (6.2/6.3 N/A). The guarantee above is now proven for both rolling single-node failover and simultaneous total-cluster outage. (A separate, unrelated anomaly — an extra, unexpected committed job appearing after one cold-restart-all cycle, the opposite of loss — was logged as bug-1396, open/low-priority, not yet root-caused.)

Post-Election Safe Commit Point (Critical)

After winning election, the new primary determines safe_committed_lsn from voter state collected during the election. This eliminates a round-trip vs waiting for the first heartbeat reply:

safe_committed_lsn = max(lsn : count of voters with peerLastLogLsn >= lsn) + 1 >= majority)

The RequestVoteReply message already carries peerConfigTag and peerLastLogLsn from each voter. The winning candidate uses these (from ha_elect_ctx_t.votes[]) to compute safe_committed_lsn immediately in raft_cb_become_leader, then truncates at ha_promotion_truncate_floor(safe_committed_lsn, persist_durable_floor) — i.e. max(safe_committed_lsn, own durable tip), so it never truncates the leader's own durable log (bug-1060 Leader Completeness; safe_committed_lsn is still used as the commit lower-bound and the promotion-gate HWM).

Never use last_known_committed_lsn from the previous primary's ctrlhello for truncation — it may be stale (the old primary may have committed entries after the last ctrlhello was sent).

Protocol Verification Checklist

When modifying HA protocol code (election, replication, heartbeat), verify:

• votedFor is updated atomically with term increment before granting any vote (library: raft_on_vote_request, src/lib/raft/raft.c:178-270 — term update before persist, voted_for set only after persist succeeds; adapter: raft_cb_persist, ha.c:1784-1824, writes term+voted_for atomically under ha->lock. handle_request_vote in ha.c is #ifdef XCE_TESTING-only; production dispatch is cluster.c:477 → raft_on_vote_request. Corrected 2026-07-18 — the prior ha.c:1218-1230 citation had drifted onto unrelated post-election code.)
• currentTerm is monotonic — never decreases
• commitIndex never decreases (monotonic)
• A leader only commits entries from its current term (or equivalent via per-entry majority ack)
• safe_committed_lsn is derived from voter replies, not from stale ctrlhello
• WAL truncation on promotion never falls below the leader's OWN durable tip — cutoff = max(safe_committed_lsn, persist_durable_floor) (ha_promotion_truncate_floor); a new leader never truncates its own committed log (bug-1060)
• Freshness gate in RequestVote: candidate must have lastLogTerm >= voter AND lastLogIndex >= voter
• AppendEntries carries TWO distinct terms: the HEADER term = leader's CURRENT term (UpdateTerm/stale-check) and each ENTRY's STORED term (logOk/persist). For catch-up of historical entries they DIFFER — the AE header term must be the leader's current term (wal_catchup_entry_t.leader_current_term), never the entry's stored term, or the library stale-rejects every cross-term entry (bug-1067)
• committed_lsn advances only as a contiguous LSN-ordered prefix — no out-of-order or per-entry commit
• Durability flushing makes a contiguous prefix durable; durable_lsn is a prefix watermark (never a per-entry set)
• Causal dependencies are appended in order (cause before effect, e.g. submit before start)
• (post ha-commit-durability, landed 2026-06-18) a member counts toward the commit majority only at its durably-persisted LSN; acknowledged ⇒ durable on a majority
• (post raft-adapter-conformance, landed 2026-07-29) every reject of a replication-class message (WAL catch-up entry, configuration push) sends an explicit reply distinguishable from a lost message — never a silent drop (ha_send_wal_ack_reply_ex(success=0) for WAL catch-up, bug-1479; send_config_push_reply(accepted=0) for config-push, Req 2.2-2.5)
• (post raft-adapter-conformance, landed 2026-07-29) every vote/PreVote endpoint failure — connect-time or post-connect (a failure after connect but before a reply) — is counted through the production tally (vote_reply_dispatch/pre_vote_reply_dispatch), never the legacy handle_request_vote_reply path (Req 4.1-4.4)
• (post raft-adapter-conformance, landed 2026-07-29) raft_send_append_entries/ha_cb_send_append_entries_stub remains dormant in production (zero callers) by deliberate documented decision, not oversight — ha_send_append_entries_stub_call_count stays 0 while repl_broadcast/ha_stream_wal_to_standby fire for the same scenario (Req 3.3)

HA Design Constraints

Covered in .kiro/steering/ha-async-io.md (connection ownership, async I/O, heartbeat, two-tier replication, dispatch gate) and .kiro/steering/build-deployment.md (deploy commands, cluster startup/restart/verification). The TLA+ safety invariants above remain the controller-specific verification reference.

Key Files

Current Functional Status (2026-08-15)

Completed (HA / reliability layer is mature)

• Raft/HA family fully landed: term-safety, commit-durability (ack ⇒ durable on majority), commit-barrier-rollout, commit-engine, wal-replication, apply-decoupling (canonical pre-hooks → commit → apply → notify), single-leader-role, convergence-recovery, config-bootstrap, libxce-failover, controller-status (18/18 tasks).
• Job-array single WAL record (job-array-single-wal-record).
• Scheduler message-gap G-4/G-5/G-6/G-8 largely implemented (sched_update_existing_job, rusage storage, distinct UPDATE_JOB_FIELDS op).
• job_ctrl (suspend/resume/hold/release/kill) and queue_ctrl (open/close/inactivate) exist in job.c/queue.c.

Gaps / missing functionality

These are the main functional shortfalls (see .kiro/steering/roadmap.md for full backlog):

Area	Gap	Location / note
Job control residual	job_ctrl direct-mutate path (admin suspend/resume) still mutates state then fires hook — not commit-then-apply (same family as bug-795/828/851/1060, narrowly scoped to admin suspend/resume)	roadmap Hardening: "investigate: job_ctrl direct-mutate path"
Worker rejoin resync	On rejoin/failover, worker does NOT re-report running/finished job states (xce_join_req carries only hostattrs_t, no job state) → facts sent during disconnect/failover window are lost	roadmap backlog: worker-rejoin-resync
Component failover adoption	Daemons still pass single endpoint (conf->masters[0]) to libxce; bounded-retry liveness not yet verified per component	roadmap backlog: component-failover-adoption
Config-upgrade snapshot	do_upgrade_config_with_bytes does config-push+restart+per-node replay — can leave divergent materialized state (bug-857); needs entity-bearing snapshots (HARD dep, still 16-byte LSN sentinel)	roadmap backlog: config-upgrade-snapshot, entity-bearing-snapshots
Job-array bulk query/cancel	Single client op to query/cancel whole array not implemented (per-element only)	roadmap backlog: bulk job-array query/cancel
Preemption	No mechanism to preempt lower-priority running jobs (high-priority waits for free resources)	roadmap backlog / scheduler-preemption spec
Group building	cluster_build_groups() is an empty TODO stub	cluster.c:997
API-server fact delivery	FIXMEs: job pid / exit code / fail reason not sent to apiserver	apiserver.c:248,283,318
Peer auth	validate_host() is a stub returning success for all peers; OP_upgradeConfig/OP_replConfig unauthenticated (trusted-LAN model)	roadmap Hardening: validate_host, peer-authentication-hardening (security-critical)
Admin login	xceadmin uses zero-config TLS, no login/token/session/audit	roadmap Hardening: peer-authentication-hardening (d)

Note

Reliability (HA/durability/consistency) is the mature layer; the gaps above are mostly business/operational features (bulk ops, preemption, auth, rejoin resync) plus a few correctness residuals (job_ctrl mutate-before-commit). See also src/scheduler/CLAUDE.md for scheduler-side status.

See Also

• Shared library: src/lib/CLAUDE.md
• Root guidelines: CLAUDE.md (repo root)