update documents

src/controller/CLAUDE.md

@@ -13,9 +13,18 @@ For full build system details, see `.kiro/steering/build-deployment.md`.
 Core source files handle distinct domains:
 - `job.c` — Job lifecycle management
 - `queue.c` — Queue state and control
-- `cluster.c` — Cluster topology
+- `cluster.c` — Cluster topology, request dispatch
 - `dataex.c` — Data exchange with other daemons
 
+HA / replication / persistence subsystem (see HA Protocol section below for the election/replication/commit design; functions from these files are what the TLA+ mapping and safety invariants below actually cite):
+- `ha.c` — HA state machine, election glue, promotion/fencing logic
+- `ha_convergence.c` — post-election convergence and recovery
+- `persist.c` — WAL persistence, durability, recovery
+- `repl.c` — replication broadcast, majority-ack, WAL streaming
+- `config_repl.c` — configuration replication and catch-up
+
+`src/controller/` has ~27 `.c` files total (`ls src/controller/*.c`); the remainder (`commit.c`, `ctrlstatus.c`, `join.c`, `worker.c`, `resources.c`, `task.c`, and others) cover commit-index advancement, status reporting, node join, and worker/resource/task bookkeeping.
+
 Links against `libxce` from `src/lib/`.
 
 ## Conventions
@@ -45,12 +54,12 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft
 
 | TLA+ Variable | XCE Code | Location |
 |--------------|----------|----------|
-| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | `ha.h:139` |
-| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY,STANDBY} | `ha.h:139` |
-| `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | `ha.h:161` |
+| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | `ha.h:157` |
+| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY,STANDBY} | `ha.h:158` |
+| `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | `ha.h:179` |
 | `log[i]` | WAL segments in `persist->wal_path` | `persist.c` |
 | `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | `persist.h` |
-| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) | `ha.h:141` |
+| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) | `ha.h:160` |
 | `nextIndex[i,j]` | `standby.join_applied_lsn` (replicator) | `repl.h` |
 | `matchIndex[i,j]` | `standby.acked_lsn` (replicator) | `repl.h` |
 
@@ -58,23 +67,23 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft
 
 | TLA+ Action | XCE Code | Location |
 |------------|----------|----------|
-| `Timeout(i)` | `raft_start_election` via delaymgr | `raft.c:57` |
-| `RequestVote(i,j)` | `raft_on_vote_request` / reply path | `raft.c:106,169` |
-| `BecomeLeader(i)` | `raft_cb_become_leader` | `ha.c:347` |
-| `AdvanceCommitIndex(i)` | `repl_wait_majority_ack` → advance `committed_lsn` | `repl.c:338` |
+| `Timeout(i)` | `raft_start_election` via delaymgr | `raft.c:105` |
+| `RequestVote(i,j)` | `raft_on_vote_request` / reply path | `raft.c:178,433` |
+| `BecomeLeader(i)` | `raft_cb_become_leader` | `ha.c:1615` |
+| `AdvanceCommitIndex(i)` | `repl_wait_majority_ack` → advance `committed_lsn` | `repl.c:589` |
 | `AppendEntries(i,j)` — production send (LIVE) | `repl_broadcast` (per-entry fan-out on append, NACK-driven retry via `ha_raft_on_repl_nack`) + `ha_stream_wal_to_standby` (catch-up-range streaming) — the sole production `AppendEntries`-equivalent send/retry/backoff mechanism (raft-adapter-conformance Req 3.3) | `repl.c:295`, `config_repl.c:2351` |
 | `AppendEntries(i,j)` — nextIndex-driven sender (DORMANT, documented) | `raft_send_append_entries` is fully protocol-conformant but has ZERO production callers (only `raft_sim.c` and an `XCE_TESTING` test helper); its registered `on_send_append_entries` callback `ha_cb_send_append_entries_stub` is reachable only through it. Kept as an intentionally-dormant reference implementation (retire/wire-in both rejected, see `research.md`), guarded against silent regression by `ha_send_append_entries_stub_call_count` staying 0 while the production path fires (`test_ha_core_safety.c` task 4.2) | `src/lib/raft/raft.c:595`, `ha.c:2590` (stub), `ha.c:2588` (guard counter) |
 | `AppendEntries(i,j)` reply — WAL catch-up reject | Every WAL catch-up rejection sends an explicit `WalAckReply{success=0}` via `ha_send_wal_ack_reply_ex` back to the primary — never a silent drop (bug-1479; reply-construction boundary regression-locked by raft-adapter-conformance task 5.1) | `ha.c:3155` (fn), `ha.c:3190` (accept call site), `ha.c:3326` (reject call site) |
-| `AppendEntries(i,j)` reply — configuration-push accept/reject | Every `handle_repl_config_push` outcome — all 4 reject branches plus the success/`-EAGAIN`-activation-deferred tail — sends exactly one explicit `ReplConfigPushReply{accepted, term}` via `send_config_push_reply`, so the primary can distinguish an active reject from a lost push (raft-adapter-conformance Req 2.2-2.5); the reply's `term` field feeds the primary's `raft_on_higher_term_observed` term-adoption alongside the heartbeat and WAL-NACK vectors (Req 7.4/7.5) | `config_repl.c:1267` (send fn), `config_repl.c:1308-1381` (handler), `cluster.c:750-780` (primary-side accept/reject branch + term-adoption) |
+| `AppendEntries(i,j)` reply — configuration-push accept/reject | Every `handle_repl_config_push` outcome — all 4 reject branches plus the success/`-EAGAIN`-activation-deferred tail — sends exactly one explicit `ReplConfigPushReply{accepted, term}` via `send_config_push_reply`, so the primary can distinguish an active reject from a lost push (raft-adapter-conformance Req 2.2-2.5); the reply's `term` field feeds the primary's `raft_on_higher_term_observed` term-adoption alongside the heartbeat and WAL-NACK vectors (Req 7.4/7.5) | `config_repl.c:1267` (send fn), `config_repl.c:1308-1381` (handler), `cluster.c:1030-1055` (primary-side accept/reject branch + term-adoption) |
 | `RequestVote`/`PreVote(i,j)` reply — endpoint-failure routing, all timings | A vote/PreVote endpoint failure is always counted through the production tally (`vote_reply_dispatch` / `pre_vote_reply_dispatch`), never the legacy `handle_request_vote_reply` path — true both at connect-time (bug-1423/1424, regression-locked) and post-connect, i.e. a failure after connect but before a reply arrives (raft-adapter-conformance Req 4.2/4.3, closing that one remaining failure-timing gap). `vote_ep_err_handler` (real-vote) and `pre_vote_ep_err_handler` (PreVote) are separate handlers sharing one `CTRL_PRIMARY`-cleanup helper | `ha.c:7667` (`vote_ep_err_handler`), `ha.c:7705` (`pre_vote_ep_err_handler`), `ha.c:7627` (shared cleanup helper), registered at `ha.c:692` / `ha.c:831` |
 | `ClientRequest(i,v)` | Job submit → WAL append → `repl_wait_majority_ack` | `persist.c`, `repl.c` |
-| `UpdateTerm(i,j,m)` | Term CAS in `handle_request_vote` / vote reply handler | `ha.c:1218-1230` |
+| `UpdateTerm(i,j,m)` | Term CAS before granting/casting a vote — now lives in the Raft library, not the adapter | `raft.c:178-270` (`raft_on_vote_request`) via `raft_cb_persist`, `ha.c:2257` |
 
 > Full gap-analysis citations, options considered, and design rationale for the `raft-adapter-conformance` rows above (including the dormant-sender decision and the vote/PreVote handler-split decision): `.kiro/specs/raft-adapter-conformance/research.md` and `design.md`. This table stays the terse, current pointer; the evidentiary trail lives there.
 
 ### Safety Invariants Verified
 
-**Election Safety** (≤1 leader per term): Enforced by `votedFor` — a server grants at most one vote per term (`raft_on_vote_request` line 145-151). A candidate needs majority votes; each server votes at most once per term.
+**Election Safety** (≤1 leader per term): Enforced by `votedFor` — a server grants at most one vote per term (`raft_on_vote_request`'s duplicate-vote check, `raft.c:231-237`). A candidate needs majority votes; each server votes at most once per term.
 
 **Leader Completeness** (committed entries survive): Freshness gate in `RequestVote` — candidate must have `lastLogTerm ≥ voter` AND `lastLogIndex ≥ voter`. Since committed entries are on a majority, the new leader must have all committed entries. The post-election truncation removes only a divergent tail **above the leader's own durable log**: the cutoff is `ha_promotion_truncate_floor(safe_committed_lsn, durable_tip) = max(safe_committed_lsn, persist_durable_floor)`, so `persist_truncate_wal_above` is a no-op on the leader's own durable entries. **A new leader MUST NEVER truncate its own log** — entries it holds durably were acked by it (as a standby) and so were committed by a prior leader; truncating to the bare majority element destroys them (bug-1060: granted `{self=47, peer=44}` → `safe=44` cut the leader's own committed 45..47 → divergent WALs + permanent prevote deadlock).
 
@@ -101,7 +110,7 @@ The `RequestVoteReply` message already carries `peerConfigTag` and `peerLastLogL
 ### Protocol Verification Checklist
 
 When modifying HA protocol code (election, replication, heartbeat), verify:
-- [ ] `votedFor` is updated atomically with term increment before granting any vote (library: `raft_on_vote_request`, `src/lib/raft/raft.c:178-270` — term update before persist, `voted_for` set only after persist succeeds; adapter: `raft_cb_persist`, `ha.c:1784-1824`, writes term+voted_for atomically under `ha->lock`. `handle_request_vote` in `ha.c` is `#ifdef XCE_TESTING`-only; production dispatch is `cluster.c:477` → `raft_on_vote_request`. Corrected 2026-07-18 — the prior `ha.c:1218-1230` citation had drifted onto unrelated post-election code.)
+- [ ] `votedFor` is updated atomically with term increment before granting any vote (library: `raft_on_vote_request`, `src/lib/raft/raft.c:178-270` — term update before persist, `voted_for` set only after persist succeeds; adapter: `raft_cb_persist`, `ha.c:2257`, writes term+voted_for atomically under `ha->lock`. `handle_request_vote` in `ha.c` is `#ifdef XCE_TESTING`-only; production dispatch is `cluster.c:582` → `raft_on_vote_request`. Re-corrected 2026-08-08 — the `ha.c:1784-1824` and `cluster.c:477` citations from the prior 2026-07-18 correction had themselves drifted onto unrelated promotion/recovery-gate code.)
 - [ ] `currentTerm` is monotonic — never decreases
 - [ ] `commitIndex` never decreases (monotonic)
 - [ ] A leader only commits entries from its current term (or equivalent via per-entry majority ack)
@@ -121,8 +130,6 @@ When modifying HA protocol code (election, replication, heartbeat), verify:
 
 Covered in `.kiro/steering/ha-async-io.md` (connection ownership, async I/O, heartbeat, two-tier replication, dispatch gate) and `.kiro/steering/build-deployment.md` (deploy commands, cluster startup/restart/verification). The TLA+ safety invariants above remain the controller-specific verification reference.
 
-## Key Files
-
 ## See Also
 - Shared library: `src/lib/CLAUDE.md`
 - Root guidelines: `CLAUDE.md` (repo root)