Clean failover was truncating committed jobs because the post-election
safe-commit-from-votes computation was dead: vote_reply_dispatch decoded each
granting voter's advertised peerLastLogLsn but discarded it, so ha_compute_safe
_committed_lsn saw only the self-vote (the 'from 1 granted voters' fingerprint)
and truncated to self_lsn.
Part 1: vote_reply_dispatch now records the granting voter's (lsn,tag,grant) into
ctx->votes[] via a record-only ha_record_vote_reply seam (no double-promotion;
raft_on_vote_reply stays the sole production promotion driver) — so safe-commit
reflects the true voted majority.
Part 2: the candidate/voter advertise the real DURABLE (fsync'd) tip; persist_wal
_write_replicated advances durable_lsn after its fail-loud inline fdatasync.
Reviewer-verified SAFE (advertises only durable positions; safe-commit bounded by
the durable majority, preserving more committed entries, never marking a
non-majority-durable LSN safe). 177 unit tests; new test_ha_vote_gate + election
tests green.
PARTIAL: does NOT yet make the live clean-failover pass. A deeper slice of
bug-828/795 remains — a catch-up'd standby still under-advertises its durable
position in the live run (the documented architectural catch-up/durability gap),
so a job-less node can still win and truncate. Tracked for a dedicated effort.
Two coupled fixes for a live commit-stall (committed_lsn pinned at the own-term
NOOP while a 3-node cluster was durable to LSN 5 — jobs acked but never committed):
bug-991: the WAL catch-up applied_lsn was advanced to lsn+1 (next-to-apply) in
the three live/catch-up apply paths, but recovery/checkpoint/init and the leader's
catch-up streamer treat it as last-held. The over-advertise made the streamer skip
an LSN -> library logOk wedge -> no WalAck. Fixed to last-held (= lsn).
Option A (design reconsideration of REVIEW ISSUE 3): removed the
committed = min(durable-majority, raft_commit_index) ceiling. Under single-writer
the durable-majority watermark is already (term,lsn)-current-term-safe (term-carrying
replication + (term,lsn) reconcile/truncate-before-durable + own-term NOOP gate +
post-election truncate-to-safe), so the library commit_index was a redundant,
starvation-prone gate that pinned committed below the safe durable majority whenever
the library lagged. The durable-majority barrier + own-term writable gate are now
authoritative; raft_commit_index is inert-advisory.
Safety: reviewer-verified the durable-majority watermark is (term,lsn)-safe and
Figure-8 holds via the own-term gate (new RED->GREEN test); bug-851 cannot recur.
Live: commit trace 4->5 + walcompare committed-prefix identical across 3 nodes.
177 unit tests pass.
Closes bug-990: the live replicator repl_broadcast wire-encoded term=0
(entry.term was never set), so standbys persisted hdr.term=0 and took the
legacy direct-apply path instead of the library (term,lsn) intake. Now the
live broadcast and catch-up streamer carry the leader's real term + AE prev
tuple; ha_apply_wal_catchup_entry routes every term-carrying entry through
raft_on_append_entries_request -> on_persist_entry (leader term stamped),
retiring the term-0 direct-apply. Standbys now persist the leader's real term
and reconcile by (term,lsn); the live commit chain is end-to-end.
848/849/850 + persist-before-ACK + commit-barrier + bug-960 preserved.
Proven at integration level; the definitive live STRICT_TERM failover run is
env-blocked here (bug-924 overlayfs) and recommended on a real fs.
Wires config_committed_marker to a production consumer (config_activate_committed):
verifies pushed bytes' sha256 against the committed marker before activation, fences
(loud, non-serving, stay-alive) on mismatch, and activates the latest committed marker
on recovery. Enforced via ha_dispatch_gate + promotion gate.
Fixes bug-981: a 2nd+ config upgrade could permanently fence a standby on a
push-before-commit ordering. Adds an off-iomgr delaymgr retry scheduled from the
commit-advance hook (atomic read + non-blocking enqueue only on the ACK thread) that
re-activates and clears the stale fence once the new marker commits and bytes verify.
Includes fix for bug-960: endpoint-type-aware peer-index resolution
(promoted-vote relative vs heartbeat-standby absolute) with fail-loud
bounds check, preventing an out-of-bounds match_index[] write / wrong-peer
false-quorum on the replication ACK path.
Replaces the stub: builds a WalCatchupEntry AppendEntries to standbys[peer_idx].ep
carrying prev_log_index/prev_log_term; for n_entries==1 sources the entry at
prev_log_index+1 from the local WAL (term via persist_entry_term, type+payload via
ha_read_wal_entry_at); n_entries==0 emits a zero-entry heartbeat. Async send
(xce_wal_batch_req_raw), payload freed on all paths. 2 tests via a capture seam.
Review-applied: corrected the misleading repl.c comment — leader_commit is accepted
but NOT yet wire-encoded (deferred to task 3.2: WalCatchupEntry leaderCommit field +
standby intake). Reviewer APPROVED; per-group 0 failures.
on_get_entry_term → persist_entry_term (recorded term, 0=absent, legacy-0→1; caveat
documented per 1.3 review). on_truncate_from → persist_truncate_wal_above(p, index-1)
(delete LSN>=index), returns rc for library/leader retry; non-promotable fence intact;
replaces the committed-boundary heuristic on the library-driven path. New
truncate_term_index_locked keeps the in-memory per-LSN term index consistent on both
truncate paths (under lock, cutoff-aligned) → no stale-term read after truncation.
3 tests, reviewer APPROVED, per-group 0 failures.
Replaces the iter-1 stub: persist_set_term(p, term) stamps the leader/library term
into hdr.term, then role-routes the write — primary via persist_wal_async, standby
via persist_wal_write_replicated(p, index, ...). Persist failure returns non-zero
(no false ACK; durable_lsn unchanged); bug-850 contiguity guard + persist-before-ACK
preserved beneath. In-flight raft_entry_blob_t {type,payload_len} envelope added to
wal_types.h (never enters the 40-byte header). 3 tests, reviewer APPROVED.
Deferred (recorded in tasks.md): task 3.1 must retire the hand-rolled primary append
so current_lsn tracks the library index; task 3.2 must not call on_persist_entry on
heartbeats. Per-group 0 failures (pre-existing test_commit_barrier_reply leak aside).
1.1: WalCatchupEntry gains term@4/prevLogIndex@5/prevLogTerm@6 (append-only).
1.2: RequestVote/PreVote gain lastLogTerm@4/lastLogIndex@5; cluster.c vote callers
fixed to the extended library arity (RESTORES xce_controller + controller_unit_test
compile+link, broken by the raft-commit-safety signature extension); ha.c emits the
candidate's last-log tuple on send.
5 minimal stub log/commit callbacks wired into s_real_cbs so raft_init's new
non-NULL requirement is satisfied — tasks 2.x replace them with real bodies; the
logOk freshness gate is inert until the term seam (1.3) + on_get_entry_term (2.2).
Pre-existing test_repl_unit.c wal_catchup -EAGAIN failures + ASAN leaks are an
overlayfs /tmp fsync env artifact (bug-850), not from this task (untouched paths;
append-only schema cannot affect the @3 entries-list decode). Build/baseline
isolation is blocked by a c-capnproto flake-hash drift — see Implementation Notes.
primary_heartbeat_fn now sends xce_ha_heartbeat_req with the primary's OWN
atomic ha->term / persist.committed_lsn / committed_config_tag (NOT last_known_*),
snapshotted once per tick (Req 1.1, 3.4). XCE_TESTING seam adds a capturing
ha_heartbeat_req_hook (precedence new->legacy->real) so existing ha_nop_req_hook
liveness tests stay green; send-failure WARN + last_push_time unchanged. New test
poisons last_known_* to prove own-committed. Direct binary 119/0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Append a committed-consume branch to the OP_nop handler (existing liveness/reply
byte-unchanged): ha_consume_heartbeat_committed applies the non-primary gate
(3.4) + stale-term guard (accept hb->term >= ha->term, 3.2), then delegates to
the monotonic ha_observe_heartbeat_committed (1.2). Absent body / zero / failed
gate are no-ops (4.2). No vote-floor edit — safety is the term-guard. 4 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Strict-monotonic updater for a standby's learned committed position: advance
last_known_committed_lsn only when strictly greater (zero/lower no-op), set the
config tag together (tag-before-lsn release ordering for reader safety). No
ha_rejoin_reconcile, no truncation, no other mutation (Req 2.1-2.3, 3.1, 3.3).
4 revert-detecting unit tests; controller suite 0 failures.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On the standby replicated-apply path, ha_apply_single_entry wrote the entry via
persist_wal_write_replicated (which correctly set the self-computed CRC chain
link p->last_crc), then called apply_wal_entry with a SYNTHETIC header whose
crc=0. apply_wal_entry's `p->last_crc = hdr->crc` clobbered the chain link with
0, so every subsequent replicated write stamped prev_crc=0 on disk. The on-disk
CRC chain broke at the second replicated entry — invisible while live (standbys
serve in-memory state) but fatal on COLD RESTART: replay hit the break, returned
RECOVER_TRUNCATED, recovered only the config prefix, and lost all committed jobs
(joblist=0). The sticky truncated verdict then drove a promotion-fence
availability wedge (bug-777) when every survivor was truncated.
Fix — single ownership of the WAL header + CRC chain:
- apply_wal_entry is now LOGICAL entity-restore only: signature
(p, type, lsn, payload, payload_len); no header. It advances the current_lsn
high-water (monotonic) but NEVER touches p->last_crc.
- The physical layer is the sole owner of the CRC chain: persist_wal_write_-
replicated on the live path, replay_wal/persist_recover on recovery.
- ha_apply_single_entry drops the fabricated header and passes entry fields
directly. The persist_recover replay caller is updated to the new signature.
Verified: new RED->GREEN test test_replicated_apply_chain_survives_cold_restart
(drives the real ha_apply_wal_catchup_entry path, cold-replays -> RECOVER_CLEAN,
all N recovered); controller suite 0 failures; 3-node rolling-SIGKILL soak on a
real ext2/3 fs now passes 5/5 (was wedge+loss at round 3); full cold restart
recovers all 6 committed jobs (was 0) with CLEAN verdicts and a normal election.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per design invariant (a controller exits only on crash or admin kill), replace the
self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable
fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual
admin recovery). Records the never-self-terminate invariant as project-wide guidance in
src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.
Desync FAILOVER re-elections by node priority (ha_election_stagger_ms /
ha_failover_stagger_fn) mirroring the startup stagger, with seed jitter and
async prevote start; allow prevote in voter roles through the dispatch gate.
Adds HA role and vote-gate tests (stagger desync by priority, prevote on dead
primary, stickiness-lapse grants).
Pre-existing uncommitted work, committed to establish a clean baseline before
ha-commit-durability implementation.
The cluster.c denial-reason elimination ordered catch_up before fresh_heartbeat,
disagreeing with raft_cb_vote_eligible (which checks fresh-heartbeat-on-STANDBY
before catch-up). In the rare STANDBY catch-up + fresh-heartbeat overlap the log
could report catch_up while the gate denied on fresh_heartbeat. Expose the pure
read predicate ha_heartbeat_is_fresh and mirror the gate's exact order so the
logged reason can no longer disagree with the actual denial. Decision unchanged;
log-string fidelity only. prevote/vote-gate/failover-prevote suites green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
peer_ctrl_msg_handler gains a CTRL_STANDBY + pre_vote_in_progress branch that
decodes an incoming preVoteReply (no term adoption) and feeds it to
pre_vote_reply_dispatch with the absolute peer index, paralleling the real-vote
reply path. Gives pre_vote_reply_dispatch its production caller (drops the used
attribute). STANDBY-no-round still drops; ELECTING/PRIMARY routing unchanged.
PreVote now drives rounds to a decision end-to-end over the wire.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ha_on_primary_ctrl_fd_rdhup + liveness_poll_fn now begin a PreVote round via
ha_begin_failover -> ha_start_prevote_async instead of electing directly; WAL
truncate + CTRL_ELECTING deferred to PreVote WON (ha_enter_real_election).
Connection teardown + config-less voter guard preserved at the trigger.
ha_on_heartbeat_received cancels an in-progress round and re-attaches as standby
with no term change. Single-node immediate-WON synchronously enters the real
election. PreVote now live; full controller suite 563 pass / 0 fail; bug-339/340
preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two distinct HA livelocks, both verified on a live 3-node ASAN deploy
(kill+restart of primary and standby, plus SIGSTOP wedge injection).
bug-339 — election freshness-floor asymmetry (RDHUP failover path):
After the primary dies the two survivors could never elect a leader — both
only ever became follower while terms climbed in lockstep. The RDHUP candidate
advertised c->persist.committed_lsn, which a standby never advances (only the
primary commit engine writes it), while every voter computed its freshness floor
from max(applied_lsn, last_known_committed_lsn). So each survivor advertised a
stale-low LSN against a high voter floor and the freshness gate denied on both
sides. Fix: compute the candidate LSN via ha_vote_self_freshness so the
advertised value equals the voter floor (ha_on_primary_ctrl_fd_rdhup); also route
the boot-path election (myapp_run) through the same helper for symmetry.
bug-340 — wedged standby endpoint suppressed reconnect (rejoin path):
A killed/restarted peer's outbound endpoint on the primary could wedge (epoll
re-arm ENOENT) yet stay in standby_endpoints, so primary_heartbeat_fn's reconnect
scan treated the peer as still-connected and never re-established a ctrlhello
channel — the restarted node never heard from the primary and re-elected every
heartbeat_timeout (term climbing unbounded). xce_nop_req returns success on
enqueue even when the send can't progress, so the heartbeat layer never noticed.
Fix: primary_heartbeat_fn now evicts a standby that was responsive once
(ts_last_in > 0) but has been silent past heartbeat_timeout — symmetric with the
standby's own liveness poll — and the reconnect scan re-establishes the channel.
Uses existing endpoint_t.ts_last_in (no libnetio change).
Tests (RED->GREEN):
- test_ha_rdhup_integration: candidate advertises freshness floor, not stale committed_lsn
- test_ha_liveness: evicts_silent_standby + keeps_live/keeps_unconfirmed (no false eviction)
Live symptom: every write (jobsub) on a healthy 3-node cluster returned
E_XCE_NOT_READY (10013) — the primary had no write quorum and committed_lsn was
stuck at 0, even though both standbys were connected, heartbeating, streamed, and
the primary RECEIVED their WAL ACKs.
Root cause: raft_cb_become_leader registers a surviving vote endpoint with
repl_add_standby only when ectx->votes[i].peer != NULL, but raft_cb_send_vote_request
set votes[i].ep and never votes[i].peer. So at promotion the guard failed, the
standbys were added to ha->standby_endpoints (streaming) but NOT to the replicator
array c->persist.repl.standbys[] (commit/quorum). Consequently repl_on_ack could
not find the ack's ep (slot_idx=-1) -> commit_on_ack never ran -> committed_lsn
never advanced; majority_candidate/repl_active_standby_count saw 0 standbys ->
no write quorum -> the dispatch gate rejected every write with -XCE_ERR_NOT_READY.
(Streaming/persistence worked, which masked it: reslist/workerlist succeed, only
writes fail.)
Fix: raft_cb_send_vote_request now also sets ectx->votes[votes_idx].peer = peer
(the &candidates[peer_idx] pointer already computed for the connect) at the same
success site as .ep. peer points into the persistent myconf candidates[];
repl_add_standby copies host/port, so no lifetime dependency. Promotion now
registers each surviving vote ep -> standbys enter the replicator -> WAL ACKs
advance committed_lsn -> own-term gate opens -> write quorum established.
TDD: drive the real raft_cb_become_leader over a real persist/repl/commit fixture
and assert repl_active_standby_count post-promotion == N when votes[].peer is set,
== 0 when NULL (the bug). Full controller suite green under ASAN; election/commit/
repl/wire untouched apart from the one-line .peer assignment + a test seam.
A standby applied replicated WAL entries via apply_wal_entry (the replay/
in-memory path) which restores entities + advances applied_lsn but NEVER writes
the entry to the standby's WAL segment. The standby then ACKed (feeding the
primary's commit majority) without persisting — a Raft durability violation:
entries counted toward a commit could be lost on standby restart, and standby
WAL files stayed empty (contradicting steering: log[i] = WAL segments for every
node). This is why ctrl-2/ctrl-3 received + acked the streamed own-term NOOP yet
kept empty seg-*.wal.
Fix: new persist_wal_write_replicated(p, lsn, type, payload, len) writes the
entry to the local WAL at the PRIMARY's lsn (no lsn-reassignment), with a
self-consistent intra-segment CRC chain (the streamed entry carries no term/crc;
each node chains independently and recovery validates intra-segment), fdatasyncs
for durability, advances current_lsn = max(current_lsn, lsn+1), rotates segments,
and does NOT re-broadcast. Wired into ha_apply_single_entry as
persist(durable) -> apply(in-memory) -> ACK; on persist failure: no apply, no ACK
(a standby never ACKs an entry it didn't persist). Covers single + batch
catch-up; OPCode_replEntry has no separate apply path (allow-list only), so all
live WAL apply now persists. Dedup guard (lsn+1<=current_lsn) makes re-streams
idempotent while still ACKing.
TDD: standby write -> drop persist_t -> reopen + persist_recover replays it
(single, 3-entry crc chain, post-dedup); checkpoint/oversized rejected. Full
controller suite green under ASAN; persist_wal_async / recovery / truncation /
streamer / wire untouched.
After reserve-lsn-0 (oldest retained WAL entry = lsn 1 on a fresh cluster), a
freshly-joined empty standby (applied_lsn=0) hit the snapshot-fallback branch
'standby_lsn < oldest_wal_lsn' (0 < 1) and was sent a non-existent snapshot
instead of streaming — so the own-term NOOP (lsn 1) never replicated, committed_lsn
stayed 0, the own-term gate (writable_lsn=1) stayed closed, and standbys kept
empty WALs. Live evidence:
ha_dispatch_repl_action_for_standby: standby[0] lsn=0 < oldest_wal_lsn=1 -> requesting snapshot transfer
The snapshot fallback should fire only when the standby's NEXT-needed entry
(standby_lsn+1) has been GC'd below the oldest retained entry. Fix the boundary
to 'standby_lsn + 1 < oldest_wal_lsn' in ha_evaluate_repl_action (and the parallel
dead repl_needs_snapshot predicate). An empty standby (next-needed=1, oldest=1)
now STREAMs lsn 1 -> acks -> committed_lsn 0->1 -> own-term gate opens -> standbys
receive the WAL. Snapshot still fires for genuinely GC'd-behind standbys
(standby_lsn+1 < oldest). No underflow (oldest=0 -> 1<0 false). Pre-reserve-lsn-0
this off-by-one was masked by oldest_wal_lsn=0 (0<0 false).
TDD: test_empty_standby_fresh_cluster_streams_not_snapshot (RED->GREEN);
snapshot + heartbeat pins strengthened to assert SEND_SNAPSHOT only for genuinely
GC'd standbys and STREAM_WAL at the empty/boundary case. Full controller suite
green under ASAN; commit engine / streamer / apply path / wire untouched.
Root cause (found via ha-convergence-recovery task 8 live 3-node E2E): a
controller with no local config (config_version_tag==0) could become PRIMARY
and push an empty config, breaking replication.
Two upstream election-safety gaps, both outside ha-convergence-recovery:
1. RDHUP re-election bypassed the no-config 'voter only' guard. The startup
path (ha_start_election_async) refuses to start an election when
my_config_tag==0 && n_peers>1, but the RDHUP failover path
(ha_on_primary_ctrl_fd_rdhup) called raft_on_leader_lost unconditionally.
Fix: mirror the guard before elect_ctx alloc / hook / raft_on_leader_lost;
a config-less node stays CTRL_STANDBY (voter only) and grants the
configured peer instead of starting (and winning) its own election.
2. Voter freshness gate compared the candidate only against the voter's LOCAL
config, ignoring the cluster's known committed state. Fix: ha_vote_self_freshness
floors the comparison at max(local, learned-committed config_tag / committed_lsn),
so a voter denies any candidate whose config/cluster state is outdated vs
what the cluster has committed. Strictly additive (never denies the freshest
node — Leader Completeness preserves liveness).
TDD: test_rdhup_noconfig_node_does_not_elect + configured-node contrast;
test_ha_vote_gate learned-floor deny + liveness + backward-compat. Full
controller suite green under ASAN. commit.c and ha-convergence-recovery
files untouched.
configAck branch: per-push majority (acked_config_lsn >= push_tag,
1+count > n_peers/2 over r->ce->n_peers, n_peers<=1 immediate,
ce==NULL -EINVAL); legacy >0 counting and n_active==0 shortcut dead.
push_config_to_standbys returns the barrier result; upgrade failure ->
send_upgrade_error (rc forwarded), no success ACK, no restart,
committed_config_tag success-only.
Remediation r1 (two CRITICAL review findings):
- F1: wire production config-ACK ingestion — replconfigpushreplymsg
decode branch in repl_ack_decode_and_dispatch -> repl_on_config_ack
on the promoted-vote ep; + defensive cluster.c OPCode_reply case.
Closes the unlanded half of ha-config-bootstrap 8.1; without this the
barrier would brick every multi-member config upgrade. No schema change.
- F2: keep the barrier off the iomgr thread — export
push_config_transport_only (non-blocking) for the 5 ha.c + join.c
streaming sites; barrier path now vtask-only (do_upgrade_config).
Wire-path test drives real capnp ACK bytes through the production decode
branch. 30/30 rollout, 6/6 repl_majority, 46/46 config_repl; census 0
FAIL; pre-existing fixture leak only (less than baseline).
notify/update/setpreason/clrpreason admitted in CTRL_PRIMARY without
write quorum (Req 3.2 / umbrella 2.8 / bug-396): all four terminate in
receipt-style RETAIN fact paths — reviewer-traced no-block proof (no
admitted opcode reaches a barrier/WAIT path; request opcodes stay
rejected). 21/21 group; census 494/494 zero FAIL; RED→GREEN
independently reproduced via stash-rebuild.
repl_active_standby_count: registry truth (ep != NULL && !in_catch_up)
under r->lock. ha_state_t gains repl back-pointer (wired at init by
task 5.3); ha_has_write_quorum uses registry truth when wired, exact
legacy fallback when NULL; formula unchanged. Test 29 (wired gate,
catch-up-only standby → NOT_READY, admits after clear) captured RED
pre-fix. 29/29 group tests; suite at the 3-failure baseline.
Handle incoming WalAckReply in repl_ack_decode_and_dispatch: extract
lsn+success, call repl_on_ack on success to update standby acked_lsn
and signal majority-ack waiter. NACK is WARN-logged with no state
change. Two unit tests verify positive path (acked_lsn updated, waiter
unblocked) and negative path (NACK leaves acked_lsn=0, wait times out).
Standby sends WalAckReply via xce_reply after each successfully
applied WAL entry. ha_apply_wal_catchup_entry signature extended
with endpoint_t* and cluster_t* parameters. walackreplymsg union
arm added to RequestReply capnp schema and xce_reply_t. Error
entries skip ACK. Dispatch gate test confirms OPCode_walCatchupEntry
allowed in CTRL_STANDBY/ELECTING/INIT.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
