Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:
1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
library's nextIndex backoff and immediately repl_start_catchup's the
standby at the corrected position, instead of the 16-strike silent
eviction with no catch-up driver after it. Correct from the start, but
live soak still failed identically: nothing ever produced the NACK this
code was waiting for.
2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
silently dropping the reject. ha_send_wal_ack_reply had hardcoded
success=1 unconditionally -- the only WalAckReply sender in the
controller -- making (1)'s NACK-routing correct but unreachable dead
code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
the NACK without a live endpoint. Live-confirmed the NACK now reaches
the primary -- but the scenario still failed for a third reason.
3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
catch-up range was hard-bounded at committed_lsn, which can never advance
in this exact scenario because advancing it requires an ACK from the very
standby that's stuck catching up (2-of-3 alive, this standby is the only
path to a new majority) -- a self-referential deadlock. Widens the
STREAM_WAL ceiling (and the convergence check gating it) to the leader's
own log tail, matching raft.tla's real AppendEntries semantics
(leaderCommit is advisory, never a send-cap), while reusing bug-1478's
exact guard so this doesn't reopen that trap (never widen onto a trailing
local-only checkpoint's own unstreamable LSN).
Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
repl_add_standby()'s "find first empty slot" scan treated any ep==NULL
slot as available, with no awareness that a disconnected slot can still
hold an unexpired bug-1147 WAL-retention guard (last_disconnected_lsn,
set by repl_remove_standby, meant to be cleared only by
repl_expire_disconnect_slots once snap_interval_s genuinely elapses).
The scan could pick a guard-holding slot over a genuinely virgin one
purely by lower index, then unconditionally memset it -- destroying the
guard before trim_wal_segments/repl_trim_lsn could ever honor it.
Confirmed via 2/2 reproducible live 3-node soak runs (config-upgrade-
snapshot task 8.9's new --config-upgrade-wal-trim mode): a killed
standby's still-needed WAL segment got trimmed out from under it.
Now a two-pass scan: prefer a genuinely virgin slot; only fall back to
reusing a guard-holding slot (oldest last_disconnect_time first, to
discard the least remaining protection) when no virgin slot exists --
logged via WARN() so the discarded guard is operator-visible rather
than silent. Defense-in-depth at the slot-allocation layer, independent
of whatever triggers a given repl_add_standby call (a separate,
not-fully-root-caused ha.c reconnect-sweep defect is the specific
trigger seen in the soak, out of this fix's scope).
Independently reviewed across two rounds: round 1 flagged the
exhausted-fallback path silently discarding a guard with zero logging;
remediated with a WARN() matching this file's existing repl_broadcast
convention, empirically confirmed to fire only on the fallback path
with correct content. Suite: 1028 -> 1030 tests, 0 failures (net of
this fix; final combined total with bug-1472 landing concurrently:
1033 tests, 0 failures).
Closes bug-990: the live replicator repl_broadcast wire-encoded term=0
(entry.term was never set), so standbys persisted hdr.term=0 and took the
legacy direct-apply path instead of the library (term,lsn) intake. Now the
live broadcast and catch-up streamer carry the leader's real term + AE prev
tuple; ha_apply_wal_catchup_entry routes every term-carrying entry through
raft_on_append_entries_request -> on_persist_entry (leader term stamped),
retiring the term-0 direct-apply. Standbys now persist the leader's real term
and reconcile by (term,lsn); the live commit chain is end-to-end.
848/849/850 + persist-before-ACK + commit-barrier + bug-960 preserved.
Proven at integration level; the definitive live STRICT_TERM failover run is
env-blocked here (bug-924 overlayfs) and recommended on a real fs.
Replaces the stub: builds a WalCatchupEntry AppendEntries to standbys[peer_idx].ep
carrying prev_log_index/prev_log_term; for n_entries==1 sources the entry at
prev_log_index+1 from the local WAL (term via persist_entry_term, type+payload via
ha_read_wal_entry_at); n_entries==0 emits a zero-entry heartbeat. Async send
(xce_wal_batch_req_raw), payload freed on all paths. 2 tests via a capture seam.
Review-applied: corrected the misleading repl.c comment — leader_commit is accepted
but NOT yet wire-encoded (deferred to task 3.2: WalCatchupEntry leaderCommit field +
standby intake). Reviewer APPROVED; per-group 0 failures.
After reserve-lsn-0 (oldest retained WAL entry = lsn 1 on a fresh cluster), a
freshly-joined empty standby (applied_lsn=0) hit the snapshot-fallback branch
'standby_lsn < oldest_wal_lsn' (0 < 1) and was sent a non-existent snapshot
instead of streaming — so the own-term NOOP (lsn 1) never replicated, committed_lsn
stayed 0, the own-term gate (writable_lsn=1) stayed closed, and standbys kept
empty WALs. Live evidence:
ha_dispatch_repl_action_for_standby: standby[0] lsn=0 < oldest_wal_lsn=1 -> requesting snapshot transfer
The snapshot fallback should fire only when the standby's NEXT-needed entry
(standby_lsn+1) has been GC'd below the oldest retained entry. Fix the boundary
to 'standby_lsn + 1 < oldest_wal_lsn' in ha_evaluate_repl_action (and the parallel
dead repl_needs_snapshot predicate). An empty standby (next-needed=1, oldest=1)
now STREAMs lsn 1 -> acks -> committed_lsn 0->1 -> own-term gate opens -> standbys
receive the WAL. Snapshot still fires for genuinely GC'd-behind standbys
(standby_lsn+1 < oldest). No underflow (oldest=0 -> 1<0 false). Pre-reserve-lsn-0
this off-by-one was masked by oldest_wal_lsn=0 (0<0 false).
TDD: test_empty_standby_fresh_cluster_streams_not_snapshot (RED->GREEN);
snapshot + heartbeat pins strengthened to assert SEND_SNAPSHOT only for genuinely
GC'd standbys and STREAM_WAL at the empty/boundary case. Full controller suite
green under ASAN; commit engine / streamer / apply path / wire untouched.
configAck branch: per-push majority (acked_config_lsn >= push_tag,
1+count > n_peers/2 over r->ce->n_peers, n_peers<=1 immediate,
ce==NULL -EINVAL); legacy >0 counting and n_active==0 shortcut dead.
push_config_to_standbys returns the barrier result; upgrade failure ->
send_upgrade_error (rc forwarded), no success ACK, no restart,
committed_config_tag success-only.
Remediation r1 (two CRITICAL review findings):
- F1: wire production config-ACK ingestion — replconfigpushreplymsg
decode branch in repl_ack_decode_and_dispatch -> repl_on_config_ack
on the promoted-vote ep; + defensive cluster.c OPCode_reply case.
Closes the unlanded half of ha-config-bootstrap 8.1; without this the
barrier would brick every multi-member config upgrade. No schema change.
- F2: keep the barrier off the iomgr thread — export
push_config_transport_only (non-blocking) for the 5 ha.c + join.c
streaming sites; barrier path now vtask-only (do_upgrade_config).
Wire-path test drives real capnp ACK bytes through the production decode
branch. 30/30 rollout, 6/6 repl_majority, 46/46 config_repl; census 0
FAIL; pre-existing fixture leak only (less than baseline).
replicator_t gains ce pointer (set-once-before-traffic contract).
repl_on_ack feeds commit_on_ack after releasing r->lock (iomgr-safe);
four membership mutators feed commit_on_membership_change post-unlock,
gated on actual mutation. walAck branch delegates to commit_submit(WAIT)
when ce wired AND entry_lsn != 0 (LSN-0 interim guard until 5.3);
legacy counting kept for ce==NULL, marked DEPRECATED. commit.c linked
into production target (inert until 5.3 wiring). Tests 30-31 + all
repl_majority walAck cases green; 33/34 remain RED for 5.3; baseline
unchanged.
repl_active_standby_count: registry truth (ep != NULL && !in_catch_up)
under r->lock. ha_state_t gains repl back-pointer (wired at init by
task 5.3); ha_has_write_quorum uses registry truth when wired, exact
legacy fallback when NULL; formula unchanged. Test 29 (wired gate,
catch-up-only standby → NOT_READY, admits after clear) captured RED
pre-fix. 29/29 group tests; suite at the 3-failure baseline.
Tests 21-28: repl_active_standby_count contract (5 RED vs inert -1
stub) + 3 gate-level pins (n_peers<=1 always-true, zero-connected
reject, stateless re-accept) that 4.2 must preserve. Declaration in
repl.h; ha.c/ha.h untouched (4.2 boundary). 28 tests / 5 RED; suite
at the 3-failure baseline with unchanged signatures.
Implement the catch-up state machine as testable pure-logic functions:
- repl_start_catchup(): marks standby in_catch_up=1, stores join_applied_lsn
- repl_complete_catchup(): clears in_catch_up=0, broadcasts ack_cond so any
blocked repl_wait_ack() re-checks the ACK condition immediately
- repl_needs_snapshot(): pure predicate — returns 1 when standby LSN predates
oldest surviving WAL segment (snapshot transfer required), 0 otherwise
Also fixes pre-existing linker error: repl.c was missing from controller_src
(xce_controller production binary), causing undefined references to repl_*
symbols introduced in P2-4.
7 new TDD tests in test_repl_catchup.c — all pass. Existing tests green.
Implements sync replication wait: primary blocks until at least one non-catching-up
standby ACKs the target LSN, or times out. Uses pthread_cond_timedwait with
CLOCK_REALTIME deadline. Inner loop rechecks all standbys after each wakeup to
guard against spurious wakeups.
ACK deadlock prevention: repl_on_ack() is called exclusively from the io_worker
read callback (not vtaskmgr), so it only does atomic_store + pthread_cond_signal —
safe to call while vcpu threads block in repl_wait_ack().
4 new TDD tests (all green, 9 total in test_repl_unit):
- test_wait_ack_no_standbys_returns_enodev
- test_wait_ack_already_acked
- test_wait_ack_timeout (verifies >=80ms elapsed)
- test_wait_ack_woken_by_background_ack (pthread thread signals after 30ms)
Add replicator_t structure with per-standby connection slots (standby_conn_t),
lifecycle (repl_init/repl_fini), slot management (repl_add_standby,
repl_remove_standby, repl_expire_disconnect_slots), ACK tracking
(repl_on_ack), WAL trim guard (repl_trim_lsn), and stubs for
repl_broadcast/repl_wait_ack (wired in P2-3/P2-4).
Uses mylock_t (pthread_mutex_t alias) with mylock_lock/mylock_unlock.
snap_interval_s==0 in repl_expire_disconnect_slots forces immediate expiry
to support deterministic unit testing.
5 new Unity tests all pass; 115 pre-existing controller tests unchanged.
