Rongsong Shen
34ab78d72d
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node rotation) can leave a standby log-conflicted after a rejoin. Fixing this needed three compounding layers, each unit-green but still failing the identical live scenario until the next layer landed: 1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft library's nextIndex backoff and immediately repl_start_catchup's the standby at the corrected position, instead of the 16-strike silent eviction with no catch-up driver after it. Correct from the start, but live soak still failed identically: nothing ever produced the NACK this code was waiting for. 2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of silently dropping the reject. ha_send_wal_ack_reply had hardcoded success=1 unconditionally -- the only WalAckReply sender in the controller -- making (1)'s NACK-routing correct but unreachable dead code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe the NACK without a live endpoint. Live-confirmed the NACK now reaches the primary -- but the scenario still failed for a third reason. 3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's catch-up range was hard-bounded at committed_lsn, which can never advance in this exact scenario because advancing it requires an ACK from the very standby that's stuck catching up (2-of-3 alive, this standby is the only path to a new majority) -- a self-referential deadlock. Widens the STREAM_WAL ceiling (and the convergence check gating it) to the leader's own log tail, matching raft.tla's real AppendEntries semantics (leaderCommit is advisory, never a send-cap), while reusing bug-1478's exact guard so this doesn't reopen that trap (never widen onto a trailing local-only checkpoint's own unstreamable LSN). Verification: controller_unit_test 1053/1053 PASS, clean twice, including new regression tests for all three layers (test_config_repl.c, test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its pre-fix behavior before the real fix was restored. Three successive live run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the final run passed end-to-end -- all 3 force_elect_node rounds succeeded (ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never previously reached), zero materialized-state divergence, zero job loss. Unblocks config-upgrade-snapshot task 8.6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
5.6 KiB
Requirements Document
Project Description (Input)
Systematic conformance gap-validation of the XCE controller's Raft replication ADAPTER
(src/controller/ha.c, repl.c, config_repl.c — the real wire-level send/receive/reply
code) against raft.tla's action model, action by action (AppendEntries request/response,
RequestVote, ClientRequest, AdvanceCommitIndex, term rules). This is distinct from and
complements the existing raft-sim-conformance spec, whose boundary is explicitly scoped to
src/lib/raft/raft_sim.c only (an in-process, in-memory harness validating the LIBRARY's
protocol semantics) and explicitly does not touch the controller adapter code.
Motivating finding (already confirmed live — seed this spec with it as a known first
defect rather than rediscovering it): during a fresh live-soak re-verification of
bug-1479's fix (config-upgrade-snapshot task 8.6's --config-upgrade scenario,
run-ha-soak.sh, deploy preserved at ~/tmp/xce_ha_bug1479_verify), the fix's new
primary-side NACK-routing code (ha_raft_on_repl_nack + repl_start_catchup in
repl_ack_decode_and_dispatch's WAL-ACK-reply-not-success branch, ha.c ~6680-6720) was
confirmed CORRECT but UNREACHABLE in production: ha.c's ha_apply_wal_catchup_entry has
three reject paths (~3230-3232 for the term-carrying single/batch entry via
ha_apply_term_entry_via_library, ~3320-3321 for a batch sub-entry apply failure, ~3373-3374
for the legacy single-entry path) that all explicitly send NO reply at all on a logOk reject
/ forward-gap (not even a NACK) — and ha_send_wal_ack_reply (ha.c:3084-3106), the ONLY
function anywhere in the controller that ever constructs a WalAckReply, hardcodes
success=1 unconditionally. There is no production code path that ever sends a failing
WalAckReply. In raft.tla, AppendEntries always yields a response (success true or false)
that drives nextIndex backoff; this codebase's silent-drop-on-reject deviates from that and
reproduces bug-1479's original permanent-quorum-loss symptom via a different mechanism
(confirmed live: ctrl-2 standby stuck rejecting ae_reject_logmismatch at growing
prevIdx forever, ctrl-1 primary logs zero WAL-NACK messages the entire time,
committed_lsn frozen, job submissions permanently stuck at 0/3 acked).
Scope for this spec:
- Fix the concrete missing-NACK-reply defect at the three reject sites so a real
WalAckReply{success=0}reaches the primary and the bug-1479 NACK-routing code can actually engage. - Beyond that one confirmed defect, perform the broader systematic pass — walk
raft.tla's action list (AppendEntries send+response, RequestVote/PreVote send+response, ClientRequest/log append, AdvanceCommitIndex, term/UpdateTerm handling, Restart/recovery) against the corresponding controller adapter code paths inha.c/repl.c/config_repl.cand flag every other place a response, retry, or backoff the spec model assumes is silently missing or short-circuited — the same class of gap this first finding represents.
Depends on / builds on: raft-log-replication, raft-commit-safety, ha-term-safety,
raft-sim-conformance (consumes their library entry points and prior validate_impl
known_limitations, several of which already flagged "documentation drift" around this
exact send/receive boundary — cross-reference those notes rather than re-deriving them).
Related bugs: bug-1479 (this session's re-verification, buglog.json, occurrence 2 —
still not fully fixed), bug-1466/1471/1472/1474/1475/1477/1478 (same HA-replication defect
family).
Status Update (2026-07-29)
Scope item 1 (the concrete missing-NACK-reply defect) is FIXED, as part of closing out
config-upgrade-snapshot task 8.6 / bug-1479. ha_apply_term_entry_via_library's logOk-reject
path (the one live-reachable reject site of the three named above — the other two are
documented dormant/unreachable with real senders, see ha_apply_wal_catchup_entry's own
comments) now sends a real WalAckReply{success=0} via a new ha_send_wal_ack_reply_ex
helper, confirmed live: the primary now genuinely receives and routes the NACK
(ha_raft_on_repl_nack, previously-correct-but-unreachable code, now actually fires).
Closing that gap alone was still not sufficient — it exposed a SECOND, independent adapter
gap (also now fixed, also a raft.tla-deviation of exactly the kind this spec's scope item 2
is meant to find systematically): ha_stream_wal_to_standby's catch-up range was hard-bounded
at committed_lsn rather than the leader's real log tail, which is a raft.tla deviation
(leaderCommit is advisory to the follower in the real spec, never a send-cap) and caused a
self-referential deadlock whenever the streamed-to standby's own ack was required to advance
committed_lsn in the first place. Fixed via a new ha_compute_repl_stream_ceiling_lsn pure
function. Full live re-verification (run-ha-soak.sh --config-upgrade, 3/3 force_elect_node
rounds) now PASSES end-to-end. See .wolf/buglog.json bug-1479 and
config-upgrade-snapshot/tasks.md task 8.6's Implementation Notes for the complete fix
narrative and evidence.
Scope item 2 (the broader systematic raft.tla action-by-action pass) remains open and
valid — the fact that fixing item 1 alone immediately surfaced a second, independent
sibling gap (the committed_lsn-vs-log-tail conflation) is itself evidence this class of
adapter/spec-model divergence is not a one-off, and a systematic pass is still worth doing
to find any remaining siblings before they're each independently rediscovered via a live
soak the way these two were.