lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)

Browse source 
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:

1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
   library's nextIndex backoff and immediately repl_start_catchup's the
   standby at the corrected position, instead of the 16-strike silent
   eviction with no catch-up driver after it. Correct from the start, but
   live soak still failed identically: nothing ever produced the NACK this
   code was waiting for.

2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
   path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
   silently dropping the reject. ha_send_wal_ack_reply had hardcoded
   success=1 unconditionally -- the only WalAckReply sender in the
   controller -- making (1)'s NACK-routing correct but unreachable dead
   code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
   the NACK without a live endpoint. Live-confirmed the NACK now reaches
   the primary -- but the scenario still failed for a third reason.

3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
   catch-up range was hard-bounded at committed_lsn, which can never advance
   in this exact scenario because advancing it requires an ACK from the very
   standby that's stuck catching up (2-of-3 alive, this standby is the only
   path to a new majority) -- a self-referential deadlock. Widens the
   STREAM_WAL ceiling (and the convergence check gating it) to the leader's
   own log tail, matching raft.tla's real AppendEntries semantics
   (leaderCommit is advisory, never a send-cap), while reusing bug-1478's
   exact guard so this doesn't reopen that trap (never widen onto a trailing
   local-only checkpoint's own unstreamable LSN).

Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-07-29 04:11:32 +00:00
parent aecd8766f4
commit 34ab78d72d
14 changed files with 1256 additions and 377 deletions
Download patch file Download diff file Expand all files Collapse all files

174
src/controller/ha.c
View file

@ -3081,19 +3081,36 @@ static int ha_apply_single_entry(persist_t *p, wal_catchup_entry_t *e)
return 0;
}
/* ha_send_wal_ack_reply: send WalAckReply {term, lsn, success=1} back to
* the primary over ep. Skipped when ep or c is NULL (test/offline path).
* Task 3.1 (ha-wal-replication): ACK each successfully applied WAL entry. */
static void ha_send_wal_ack_reply(ha_state_t *ha, struct endpoint_t *ep,
cluster_t *c, uint64_t lsn)
#ifdef XCE_TESTING
void (*ha_wal_ack_send_hook)(uint64_t term, uint64_t lsn, int success) = NULL;
#endif
/* ha_send_wal_ack_reply_ex: send WalAckReply {term, lsn, success} back to
* the primary over ep. Skipped when ep or c is NULL (test/offline path),
* unless ha_wal_ack_send_hook is set (XCE_TESTING), in which case the hook is
* called with the built {term, lsn, success} INSTEAD of sending over ep --
* lets a unit test observe a NACK (success=0) without a live endpoint
* (bug-1479). Task 3.1 (ha-wal-replication): ACK/NACK each applied/rejected
* WAL entry. */
static void ha_send_wal_ack_reply_ex(ha_state_t *ha, struct endpoint_t *ep,
cluster_t *c, uint64_t lsn, int success)
{
uint64_t term = (uint64_t)atomic_load_explicit(&ha->term, memory_order_acquire);
#ifdef XCE_TESTING
if (ha_wal_ack_send_hook) {
ha_wal_ack_send_hook(term, lsn, success);
return;
}
#endif
if (!ep || !c)
return;
wal_ack_reply_t ack;
ack.term = (uint64_t)atomic_load_explicit(&ha->term, memory_order_acquire);
ack.term = term;
ack.lsn = lsn;
ack.success = 1;
ack.success = success ? 1 : 0;
xce_reply_t reply;
memset(&reply, 0, sizeof(reply));
@ -3105,6 +3122,14 @@ static void ha_send_wal_ack_reply(ha_state_t *ha, struct endpoint_t *ep,
xce_reply(ep, c->myhost, &reply);
}
/* ha_send_wal_ack_reply: send a successful WalAckReply {term, lsn,
* success=1} -- the common case, every existing call site. */
static void ha_send_wal_ack_reply(ha_state_t *ha, struct endpoint_t *ep,
cluster_t *c, uint64_t lsn)
{
ha_send_wal_ack_reply_ex(ha, ep, c, lsn, /*success*/1);
}
/* ha_apply_term_entry_via_library: route ONE term-carrying replicated entry
* (a live broadcast or a catch-up sub-entry) through the raft library
* (raft_on_append_entries_request via ha_consume_append_entries) so the LIBRARY
@ -3227,8 +3252,18 @@ static int ha_apply_term_entry_via_library(ha_state_t *ha, persist_t *p,
ha_send_wal_ack_reply(ha, ep, c, e->lsn);
return 1;
}
/* logOk reject (e.g. forward gap before the prefix arrives): no ACK; the
* ACK-driven catch-up continuation re-streams from current_lsn in order. */
/* logOk reject (e.g. forward gap before the prefix arrives, or a genuine
* (term,lsn) conflict): bug-1479 -- send a NACK (success=0) so the
* primary's ha_raft_on_repl_nack can back off nextIndex and immediately
* mark this standby for catch-up, matching raft.tla's AppendEntries
* action (the receiver ALWAYS replies, success = logOk). Before this fix
* NOTHING was sent on reject: the primary's NACK-handling branch
* (repl_ack_decode_and_dispatch) was correct but unreachable, since no
* production path ever produced the success=0 reply it was waiting for
* -- the standby would loop rejecting the identical re-broadcast entry
* forever, discovered live via the config-upgrade soak's force-elect
* rotation. */
ha_send_wal_ack_reply_ex(ha, ep, c, e->lsn, /*success*/0);
return 0;
}
@ -3843,6 +3878,20 @@ uint64_t ha_compute_repl_target_lsn(uint64_t committed_lsn,
return committed_lsn;
}
/* bug-1479 (gap 3): see full doc comment in ha.h. Distinct from
* ha_compute_repl_target_lsn above (the CONVERGENCE target) -- this is the
* STREAM ceiling: how far a leader may actually push WAL to a standby, which
* must extend past committed_lsn or a standby whose own ack is required to
* advance committed_lsn can never receive the entries it needs to ack. */
uint64_t ha_compute_repl_stream_ceiling_lsn(uint64_t committed_lsn,
uint64_t last_written,
uint64_t checkpoint_lsn)
{
if (last_written == checkpoint_lsn && last_written > committed_lsn)
last_written = committed_lsn;
return (last_written > committed_lsn) ? last_written : committed_lsn;
}
/* ───────────────────────────────────────────────────────────────────────
* raft-drive-async Task 3.2 — SnapshotSend slow-lane offload (bug-1198).
*
@ -4233,8 +4282,38 @@ int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
uint64_t standby_tag = ha->standby_reported_tag[peer_idx];
uint64_t standby_lsn = ha->standby_reported_lsn[peer_idx];
/* bug-1479: a standby marked in_catch_up (via ha_raft_on_repl_nack's
* immediate trigger on log-conflict rejection, or repl_broadcast's
* MAX_PENDING_ACKS eviction) can never correct its OWN self-reported
* standby_reported_lsn — it's stuck rejecting every AppendEntries, so it
* never sends a stateReport reflecting genuine progress. Use the
* corrected join_applied_lsn instead: this heartbeat-driven cycle (runs
* every heartbeat_interval_ms regardless of ACK/reject outcome) becomes
* the actual catch-up driver for this case — the ACK-driven continuation
* (ha_walack_on_ack, config_repl.c) can only fire on a REAL ack, which a
* log-conflicted standby can never produce until streamed from the
* corrected position at least once. */
if (c) {
uint64_t join_lsn;
if (repl_get_catchup_join_lsn(&c->persist.repl, ep, &join_lsn))
standby_lsn = join_lsn;
}
/* bug-1479 (gap 3): stream (and evaluate convergence against) the
* leader's own log tail, not just committed_lsn -- see
* ha_compute_repl_stream_ceiling_lsn's doc comment (ha.h) for why capping
* at committed_lsn is a permanent deadlock whenever this exact standby's
* own ack is required to advance committed_lsn in the first place. */
uint64_t stream_ceiling_lsn = committed_lsn;
if (c) {
uint64_t last_written = (c->persist.current_lsn > 0)
? c->persist.current_lsn - 1 : 0;
stream_ceiling_lsn = ha_compute_repl_stream_ceiling_lsn(
committed_lsn, last_written, c->persist.checkpoint_lsn);
}
ha_repl_action_t action = ha_evaluate_repl_action(standby_tag, standby_lsn,
committed_tag, committed_lsn,
committed_tag, stream_ceiling_lsn,
oldest_wal_lsn);
switch (action) {
@ -4283,16 +4362,19 @@ int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
case HA_REPL_STREAM_WAL:
/* Task 3.4: batch WAL streaming.
* Enqueues all batch OPCode_walCatchupEntry messages for the range
* [standby_lsn+1, committed_lsn] via ha_stream_wal_to_standby().
* [standby_lsn+1, stream_ceiling_lsn] via ha_stream_wal_to_standby()
* (bug-1479 gap 3: the ceiling extends past committed_lsn to the
* leader's own log tail — see ha_compute_repl_stream_ceiling_lsn).
* All xce_request() calls are non-blocking (enqueue-only). */
INFO("HA repl: standby[%d] lsn=%" PRIu64
" < committed_lsn=%" PRIu64 " → stream WAL batches",
peer_idx, standby_lsn, committed_lsn);
" < stream_ceiling_lsn=%" PRIu64 " (committed_lsn=%" PRIu64
") → stream WAL batches",
peer_idx, standby_lsn, stream_ceiling_lsn, committed_lsn);
#ifdef XCE_TESTING
if (ha_repl_stream_wal_hook != NULL) {
const char *wal_path = (c && c->persist.wal_path[0])
? c->persist.wal_path : "";
ha_repl_stream_wal_hook(ep, wal_path, standby_lsn, committed_lsn);
ha_repl_stream_wal_hook(ep, wal_path, standby_lsn, stream_ceiling_lsn);
break;
}
#endif
@ -4311,7 +4393,7 @@ int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
uint64_t leader_term =
atomic_load_explicit(&c->ha.term, memory_order_acquire);
ha_stream_wal_to_standby(ep, c->persist.wal_path, &c->persist,
leader_term, standby_lsn, committed_lsn);
leader_term, standby_lsn, stream_ceiling_lsn);
}
break;
@ -4820,6 +4902,38 @@ int ha_raft_on_repl_ack(cluster_t *c, int abs_peer_idx, uint64_t acked_lsn,
return 0;
}
int ha_raft_on_repl_nack(cluster_t *c, int abs_peer_idx, uint64_t reply_term,
uint64_t *corrected_lsn_out)
{
if (!c)
return -EINVAL;
/* Same defense-in-depth bounds/self check as ha_raft_on_repl_ack. */
int self_idx = c->ha.my_priority;
int n_peers = c->ha.raft.n_peers;
if (abs_peer_idx < 0 || abs_peer_idx >= n_peers || abs_peer_idx == self_idx) {
ERROR("ha_raft_on_repl_nack: rejecting out-of-range peer index %d "
"(n_peers=%d self_idx=%d) — skipping library NACK accounting",
abs_peer_idx, n_peers, self_idx);
return -EINVAL;
}
/* A rejected AppendEntries reports a log conflict at the sent
* prevLogIndex/prevLogTerm. match_index is unused on the failure branch
* (raft.c: "failure: next_index[peer] = Max(next_index[peer]-1, 1). Does
* NOT advance commit.") -- pass 0. The library applies the backoff only
* when reply_term == s->term (stale-term drop), same as the ack path. */
(void)raft_on_append_entries_reply(&c->ha.raft, abs_peer_idx, /*success*/0,
/*match_index unused*/0, reply_term);
if (corrected_lsn_out) {
uint64_t next = c->ha.raft.next_index[abs_peer_idx];
*corrected_lsn_out = (next > 0) ? next - 1 : 0;
}
return 0;
}
/*
* ha_arm_own_term_gate: append the own-term WAL_NOOP and arm the commit
* engine's writable_lsn gate (ha-single-leader-role task 2.2, Req 1.4/2.1/2.2).
@ -6638,8 +6752,36 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx,
pthread_mutex_lock(&ha->raft_lock);
(void)raft_on_higher_term_observed(&ha->raft, reply_term);
pthread_mutex_unlock(&ha->raft_lock);
/* bug-1479: a same-or-lower-term NACK means an ordinary Raft
* log conflict (ae_reject_logmismatch), NOT a stale-term
* primary — the case above doesn't cover it. Before this fix
* the reject was silently dropped: the library's own,
* already-correct nextIndex backoff (raft_on_append_entries_
* reply) was never invoked, so the primary kept re-
* broadcasting NEW entries assuming agreement at an index
* the standby had already rejected, forever — the standby
* would eventually get silently evicted via
* MAX_PENDING_ACKS, but nothing would ever stream it back
* into sync (repl_start_catchup's own driver, the ACK-driven
* continuation, needs a real ACK to fire, and a log-
* conflicted standby can never send one while stuck).
* Immediately (not after 16 silent failures) back off via
* the library and mark catch-up at the corrected position —
* ha_dispatch_repl_action_for_standby's heartbeat-driven
* decision (below) is what actually streams to it. */
if (c) {
int ack_abs_idx = ha_repl_ack_abs_peer_idx(
&c->ha, /*ep_is_vote*/ vctx->ctx != NULL, vctx->peer_idx);
uint64_t corrected_lsn = 0;
if (ha_raft_on_repl_nack(c, ack_abs_idx, reply_term,
&corrected_lsn) == 0) {
repl_start_catchup(&c->persist.repl, ep, corrected_lsn);
}
}
WARN("repl_ack_decode_and_dispatch: WAL NACK from standby"
" lsn=%lu success=0 — ignoring", (unsigned long)acked_lsn);
" lsn=%lu success=0 — marking catch-up",
(unsigned long)acked_lsn);
}
} else if (rep_s.data_which == RequestReply_data_replconfigpushreplymsg) {
/* Config-push ACK from standby (ha-commit-barrier-rollout task

92
src/controller/ha.h
View file

@ -596,6 +596,32 @@ int ha_primary_log_append(cluster_t *c, uint32_t type,
int ha_raft_on_repl_ack(cluster_t *c, int abs_peer_idx, uint64_t acked_lsn,
uint64_t reply_term);
/*
* ha_raft_on_repl_nack (bug-1479): drive the library on a REJECTED
* AppendEntries (WalAckReply.success == 0, i.e. the standby's log conflicts
* at the sent prevLogIndex/prevLogTerm). Routes the rejection through
* raft_on_append_entries_reply(success=0), which backs off
* next_index[abs_peer_idx] by one (raft.tla's AppendEntries failure branch,
* floored at 1) -- mirrors ha_raft_on_repl_ack's bounds-checked peer-index
* resolution exactly, but does NOT call raft_advance_commit (a rejection
* can never advance commit).
*
* Before this fix, a NACK was only ever used for stale-term discovery
* (raft_on_higher_term_observed) and otherwise silently dropped -- the
* library's own, already-correct nextIndex backoff mechanism was never
* invoked, so a standby stuck in a genuine log conflict (an ordinary
* post-election occurrence) never had its resend position corrected: the
* primary kept broadcasting new entries assuming agreement at an index the
* standby had already rejected, forever.
*
* Returns the corrected resume LSN (next_index[abs_peer_idx] - 1, i.e. the
* position the standby is now believed caught up through) via
* *corrected_lsn_out on success; returns -EINVAL (leaving *corrected_lsn_out
* untouched) on a no-cluster context or a rejected peer index, matching
* ha_raft_on_repl_ack's defense-in-depth bounds check. */
int ha_raft_on_repl_nack(cluster_t *c, int abs_peer_idx, uint64_t reply_term,
uint64_t *corrected_lsn_out);
/*
* ha_repl_ack_abs_peer_idx (ha-term-safety task 3.3 remediation): resolve the
* ABSOLUTE library peer index from an ACK endpoint's stored peer_idx, branching
@ -917,15 +943,36 @@ int ha_schedule_phase2_retry(cluster_t *c);
* On success (rc == 0): applied_lsn advances to entry->lsn+1 and a
* WalAckReply {term, lsn, success=1} is sent back to the primary via
* xce_reply(ep, c->myhost, &reply) (Task 3.1, Req 3.1).
* On error (rc < 0): logs ERROR, returns error; no ACK sent (Task 3.2).
* On a genuine apply/persist ERROR (rc < 0): logs ERROR, returns error; no ACK
* sent at all (Task 3.2) -- there is nothing coherent to report back yet.
* On a term-carrying entry's AppendEntries REJECT (logOk fails -- forward gap
* or a genuine (term,lsn) conflict; rc == 0, entry not accepted): bug-1479 --
* a WalAckReply {term, lsn, success=0} (a NACK) IS now sent, so the primary's
* ha_raft_on_repl_nack can back off nextIndex and immediately mark this
* standby for catch-up. Before this fix the reject was dropped with no reply
* at all, so the primary's NACK-handling could never engage no matter how
* correct it was -- the standby would loop rejecting the identical
* re-broadcast entry forever.
* applied_lsn is in-memory only — not persisted to disk.
* ep and c may be NULL in unit-test builds (XCE_TESTING); ACK is skipped.
* ep and c may be NULL in unit-test builds (XCE_TESTING); ACK/NACK is skipped
* (or captured via ha_wal_ack_send_hook when set).
* Requirements: 3.1, 3.2, 3.3, 3.4, 3.5
*/
int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
wal_catchup_entry_t *entry,
struct endpoint_t *ep, cluster_t *c);
#ifdef XCE_TESTING
/*
* ha_wal_ack_send_hook: test-only seam for the outbound WalAckReply built by
* ha_send_wal_ack_reply_ex (bug-1479). When non-NULL, called with
* (term, lsn, success) INSTEAD OF building/sending the real reply over ep --
* lets a unit test assert that an AppendEntries reject now sends a NACK
* (success=0), without needing a live endpoint. NULL in production.
*/
extern void (*ha_wal_ack_send_hook)(uint64_t term, uint64_t lsn, int success);
#endif
/* ha_nop_reply_state_report — populate a heartbeat reply with stateReport.
* Fills *sr from ha->applied_lsn (atomic read) and myconf->config_version_tag,
* then sets reply_out->data_which = stateReportMsg and
@ -1195,6 +1242,36 @@ void ha_process_standby_state(ha_state_t *ha, int peer_idx,
uint64_t ha_compute_repl_target_lsn(uint64_t committed_lsn,
uint64_t last_written);
/*
* ha_compute_repl_stream_ceiling_lsn (bug-1479, gap 3): the upper bound of
* what a leader may actually STREAM to a standby via ha_stream_wal_to_standby
* — distinct from ha_compute_repl_target_lsn's CONVERGENCE target above.
*
* Bounding the stream range at committed_lsn is a permanent deadlock whenever
* the very standby being streamed to must ACK before committed_lsn can
* advance at all (e.g. a 3-node cluster with one member down and this
* standby log-conflicted after a chaotic election): it can never receive the
* entries it needs to ack to unblock the majority. raft.tla's real
* AppendEntries replicates the leader's FULL log — leaderCommit is advisory
* to the follower, never a cap on what gets sent. Streaming past commit is
* always safe: a later leadership change's own (term,lsn) conflict-truncate
* (ha_consume_append_entries) correctly unwinds anything that turns out not
* to have been committed.
*
* bug-1478 guard: never widen the ceiling to exactly the trailing local-only
* checkpoint's own consumed LSN slot — no standby's applied_lsn can ever
* legitimately reach that value (WAL_CHECKPOINT is never streamed or applied
* by any standby, by design), so doing so would reintroduce bug-1478's
* permanent-non-convergence trap. Falling back to committed_lsn for just
* that transient instant is safe — the very next real entry appended after
* the checkpoint widens the ceiling normally again.
*
* Pure, side-effect-free — safe to unit test directly.
*/
uint64_t ha_compute_repl_stream_ceiling_lsn(uint64_t committed_lsn,
uint64_t last_written,
uint64_t checkpoint_lsn);
/* ── Task 3.5c: Standby snapshot receive handler ──────────────────────────── */
/*
@ -1864,8 +1941,15 @@ uint32_t ha_jittered_election_delay(ha_state_t *ha, uint32_t base);
* - HA_REPL_PUSH_CONFIG: In production builds, calls push_config_to_standbys(c, …).
* In XCE_TESTING builds, calls ha_repl_push_config_for_standby_hook(c, peer_idx).
* - HA_REPL_STREAM_WAL: calls ha_stream_wal_to_standby(ep, wal_path, standby_lsn,
* committed_lsn) to batch-stream WAL entries via xce_request() (non-blocking).
* In XCE_TESTING builds, calls ha_repl_stream_wal_hook if non-NULL.
* stream_ceiling_lsn) to batch-stream WAL entries via xce_request()
* (non-blocking). stream_ceiling_lsn (bug-1479 gap 3) is
* ha_compute_repl_stream_ceiling_lsn(committed_lsn, last_written,
* checkpoint_lsn) when c is non-NULL — the leader's own log tail, NOT
* just committed_lsn, so a standby whose own ack is required to advance
* committed_lsn can still receive the entries it needs to ack. Falls
* back to committed_lsn when c is NULL (test fixtures with no real
* persist_t). In XCE_TESTING builds, calls ha_repl_stream_wal_hook if
* non-NULL.
* - HA_REPL_SEND_SNAPSHOT: calls ha_send_snapshot_to_standby(ep, snap_path, lsn).
* In XCE_TESTING builds, calls ha_repl_send_snapshot_hook(peer_idx) if non-NULL.
* - HA_REPL_CONVERGED: no-op.

28
src/controller/repl.c
View file

@ -775,6 +775,12 @@ void repl_complete_catchup(replicator_t *r, repl_ep_t ep)
if (r->standbys[i].ep == ep) {
atomic_store_explicit(&r->standbys[i].in_catch_up, 0,
memory_order_release);
/* bug-1479: reset the eviction counter on re-entry to live
* broadcast -- otherwise a stale pending_ack_count carried over
* from before catch-up could immediately re-trip
* repl_broadcast's MAX_PENDING_ACKS eviction on the very next
* send, undoing the catch-up this function just completed. */
r->standbys[i].pending_ack_count = 0;
changed = 1;
break;
}
@ -794,6 +800,28 @@ void repl_complete_catchup(replicator_t *r, repl_ep_t ep)
pthread_mutex_unlock(&r->ack_lock);
}
int repl_get_catchup_join_lsn(replicator_t *r, repl_ep_t ep,
uint64_t *join_applied_lsn_out)
{
if (r == NULL || join_applied_lsn_out == NULL)
return 0;
int found = 0;
mylock_lock(&r->lock);
for (int i = 0; i < MAX_STANDBYS; i++) {
if (r->standbys[i].ep == ep) {
if (atomic_load_explicit(&r->standbys[i].in_catch_up,
memory_order_acquire)) {
*join_applied_lsn_out = r->standbys[i].join_applied_lsn;
found = 1;
}
break;
}
}
mylock_unlock(&r->lock);
return found;
}
int repl_needs_snapshot(replicator_t *r, uint64_t standby_lsn,
uint64_t oldest_wal_lsn)
{

88
src/controller/testsuite/test_config_repl.c
View file

@ -3922,6 +3922,93 @@ static void test_wal_catchup_entry_error_skipped_no_crash(void)
printf("test_wal_catchup_entry_error_skipped_no_crash: PASS\n");
}
/* bug-1479: a term-carrying entry that fails logOk (forward gap / genuine
* (term,lsn) conflict) must now send a NACK {success=0} back to the primary,
* not silence the reject entirely -- otherwise the primary's NACK-handling
* (ha_raft_on_repl_nack, ha.c) can never engage, no matter how correct it is,
* because nothing ever produces the reply it is waiting for. This is exactly
* the live gap the config-upgrade soak's force-elect rotation exposed: a
* standby that fell behind (a genuine forward gap against its empty log)
* looped rejecting the identical re-broadcast entry forever, with no NACK
* ever reaching the primary to trigger backoff + catch-up.
*
* ha_wal_ack_send_hook (ha.h, XCE_TESTING) captures the {term, lsn, success}
* that ha_send_wal_ack_reply_ex would have sent, without a live endpoint. */
static uint64_t g_b1479_hook_term;
static uint64_t g_b1479_hook_lsn;
static int g_b1479_hook_success;
static int g_b1479_hook_called;
static void b1479_wal_ack_capture(uint64_t term, uint64_t lsn, int success)
{
g_b1479_hook_called = 1;
g_b1479_hook_term = term;
g_b1479_hook_lsn = lsn;
g_b1479_hook_success = success;
}
static void test_bug1479_ae_reject_sends_nack_not_silence(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_cr_b1479_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
cluster_t *c = t42_make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
/* Bring the node up as a raft FOLLOWER at term 3 with an EMPTY log
* (log_len=0) -- mirrors t_bug1475_make_follower further below in this
* file (that fixture is defined later; duplicated inline here to avoid
* an ordering dependency). */
atomic_store(&c->ha.role, CTRL_STANDBY);
atomic_store(&c->ha.term, 3);
ha_init_raft(&c->ha, c, /*n_peers*/3);
c->ha.raft.term = 3;
c->ha.raft.role = RAFT_FOLLOWER;
TEST_ASSERT_EQUAL_UINT64(0, c->ha.raft.log_len);
g_b1479_hook_called = 0;
ha_wal_ack_send_hook = b1479_wal_ack_capture;
/* A term-carrying entry at lsn=5 whose prev_log_index=4 is a forward gap
* against this node's empty log (0 < 4) -- logOk MUST reject it. */
wal_catchup_entry_t inner;
memset(&inner, 0, sizeof(inner));
inner.lsn = 5;
inner.term = 3;
inner.prev_log_index = 4;
inner.prev_log_term = 3;
inner.entry_type = (uint8_t)WAL_JOB_UPDATE;
wal_catchup_entry_t *sub_arr[1] = { &inner };
wal_catchup_entry_t outer;
memset(&outer, 0, sizeof(outer));
outer.n_entries = 1;
outer.entries = sub_arr;
int rc = ha_apply_wal_catchup_entry(&c->ha, &c->persist, &outer,
/*ep*/NULL, c);
ha_wal_ack_send_hook = NULL; /* uninstall before any assertion can fail */
TEST_ASSERT_EQUAL_INT(0, rc); /* reject, not a hard error */
TEST_ASSERT_EQUAL_UINT64(0, c->ha.raft.log_len); /* nothing was accepted */
TEST_ASSERT_TRUE_MESSAGE(g_b1479_hook_called,
"bug-1479: an AppendEntries reject must send a WalAckReply "
"(a NACK), not silence it entirely");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_b1479_hook_success,
"the reply for a rejected entry must carry success=0");
TEST_ASSERT_EQUAL_UINT64(3, g_b1479_hook_term);
TEST_ASSERT_EQUAL_UINT64(5, g_b1479_hook_lsn);
t42_teardown(c);
char cmd[512];
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp);
system(cmd);
printf("test_bug1479_ae_reject_sends_nack_not_silence: PASS\n");
}
/* Test 2.1-C: NULL entry is handled gracefully (null guard) */
static void test_wal_catchup_entry_null_entry_no_crash(void)
{
@ -6764,6 +6851,7 @@ void test_config_repl(void)
/* Task 2.1: OPCode_walCatchupEntry handler */
RUN_TEST(test_wal_catchup_entry_advances_applied_lsn);
RUN_TEST(test_wal_catchup_entry_error_skipped_no_crash);
RUN_TEST(test_bug1479_ae_reject_sends_nack_not_silence);
RUN_TEST(test_wal_catchup_entry_null_entry_no_crash);
/* Task 2.2: batch WAL catch-up entries */
RUN_TEST(test_wal_catchup_batch_advances_applied_lsn);

60
src/controller/testsuite/test_ha_role.c
View file

@ -718,6 +718,61 @@ static void test_compute_repl_target_lsn_committed_already_ahead(void)
printf("test_compute_repl_target_lsn_committed_already_ahead: PASS\n");
}
/* ── bug-1479 (gap 3): ha_compute_repl_stream_ceiling_lsn ─────────────────
*
* Distinct from ha_compute_repl_target_lsn above (the CONVERGENCE target,
* bootstrap-only fallback). This is the STREAM ceiling: a live 3-node HA
* soak (config-upgrade-snapshot task 8.6, force-elect round) caught
* committed_lsn-bounded streaming in a permanent deadlock -- with one member
* down and the other standby log-conflicted after a chaotic election, that
* standby's own ack is the ONLY path to a new majority, but it can never
* receive any entry past committed_lsn to ack in the first place, since
* ha_stream_wal_to_standby refuses to send anything beyond it. The ceiling
* must extend to the leader's real log tail so catch-up can make progress
* past a stalled commit point, while still excluding a trailing local-only
* checkpoint LSN (the exact bug-1478 trap) since no standby can ever
* legitimately ack that exact slot either. */
static void test_compute_repl_stream_ceiling_lsn_widens_past_committed(void)
{
/* The core bug-1479 gap-3 fix: committed_lsn is stalled at 14, but the
* leader has real, streamable entries up to 33 -- the ceiling must
* widen to 33, not stay capped at 14. */
TEST_ASSERT_EQUAL_UINT64(33,
ha_compute_repl_stream_ceiling_lsn(14, 33, /*checkpoint_lsn*/0));
printf("test_compute_repl_stream_ceiling_lsn_widens_past_committed: PASS\n");
}
static void test_compute_repl_stream_ceiling_lsn_excludes_trailing_checkpoint(void)
{
/* bug-1478's exact trap, reintroduced here if unguarded: last_written
* (15) IS the trailing local-only checkpoint's own consumed LSN slot
* (checkpoint_lsn=15) -- no standby can ever legitimately ack that exact
* value, so the ceiling must fall back to committed_lsn (14), not widen
* to the unreachable checkpoint slot. */
TEST_ASSERT_EQUAL_UINT64(14,
ha_compute_repl_stream_ceiling_lsn(14, 15, /*checkpoint_lsn*/15));
printf("test_compute_repl_stream_ceiling_lsn_excludes_trailing_checkpoint: PASS\n");
}
static void test_compute_repl_stream_ceiling_lsn_committed_already_ahead(void)
{
/* Normal steady state: committed_lsn is already the higher value (a
* standby that hasn't caught up to a recently-advanced commit). The
* ceiling must simply be committed_lsn, never go backward. */
TEST_ASSERT_EQUAL_UINT64(100,
ha_compute_repl_stream_ceiling_lsn(100, 50, /*checkpoint_lsn*/0));
printf("test_compute_repl_stream_ceiling_lsn_committed_already_ahead: PASS\n");
}
static void test_compute_repl_stream_ceiling_lsn_zero_zero(void)
{
/* Genuinely fresh cluster, nothing appended yet. */
TEST_ASSERT_EQUAL_UINT64(0,
ha_compute_repl_stream_ceiling_lsn(0, 0, /*checkpoint_lsn*/0));
printf("test_compute_repl_stream_ceiling_lsn_zero_zero: PASS\n");
}
/* The transitional-leader timeout default test was removed: the config knob
* and its default macro are deleted by ha-single-leader-role (the wall-clock
* force-promotion watchdog it configured is gone — Req 1.5). */
@ -955,6 +1010,11 @@ void test_ha_role(void)
RUN_TEST(test_compute_repl_target_lsn_zero_zero);
RUN_TEST(test_compute_repl_target_lsn_ignores_trailing_checkpoint_lsn);
RUN_TEST(test_compute_repl_target_lsn_committed_already_ahead);
/* bug-1479 (gap 3): ha_compute_repl_stream_ceiling_lsn */
RUN_TEST(test_compute_repl_stream_ceiling_lsn_widens_past_committed);
RUN_TEST(test_compute_repl_stream_ceiling_lsn_excludes_trailing_checkpoint);
RUN_TEST(test_compute_repl_stream_ceiling_lsn_committed_already_ahead);
RUN_TEST(test_compute_repl_stream_ceiling_lsn_zero_zero);
/* Task 3.3: ha_dispatch_repl_action_for_standby PUSH_CONFIG wiring */
RUN_TEST(test_dispatch_repl_action_push_config_when_tag_behind);
RUN_TEST(test_dispatch_repl_action_no_push_when_converged);

180
src/controller/testsuite/test_term_safety_adapter.c
View file

@ -1983,6 +1983,181 @@ static void test_repl_ack_out_of_range_peer_rejected(void)
printf("test_repl_ack_out_of_range_peer_rejected: PASS\n");
}
/* ================================================================== */
/* bug-1479 — drive the library on a REJECTED AppendEntries (WAL NACK) */
/* ================================================================== */
/* Before this fix, a WAL NACK (WalAckReply.success == 0) for an ordinary
* Raft log conflict (ae_reject_logmismatch) was silently dropped after
* stale-term discovery -- the library's own, already-correct nextIndex
* backoff (raft_on_append_entries_reply's failure branch, raft.tla 401) was
* never invoked, so a standby stuck in log conflict never had its resend
* position corrected: the primary kept broadcasting new entries assuming
* agreement at an index the standby had already rejected, forever, until
* MAX_PENDING_ACKS silently evicted it into catch-up -- at which point
* NOTHING streamed it back into sync either (the ACK-driven continuation
* needs a real ACK to fire, which a log-conflicted standby can never send).
*
* ha_raft_on_repl_nack mirrors ha_raft_on_repl_ack's bounds-checked
* peer-index resolution, but routes success=0 into the library (backing
* off next_index[peer] by one, per raft.c's documented failure branch) and
* returns the corrected resume LSN instead of trying to advance commit. */
/* (a) A rejection backs off next_index[peer] by exactly one and returns the
* corrected LSN (next_index - 1) -- the position the standby is now
* believed caught up through. */
static void test_repl_nack_backs_off_next_index_and_returns_corrected_lsn(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_backoff_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/5);
/* make_leader pokes role=RAFT_LEADER directly without going through the
* library's real BecomeLeader transition (raft_init_leader_progress,
* raft.c:97-103), so next_index[] stays 0 for every peer regardless of
* log_len. Seed it here to what a genuine BecomeLeader would have set
* (log_len + 1), so this test exercises a realistic starting position. */
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
TEST_ASSERT_EQUAL_UINT64(6, c->ha.raft.next_index[1]);
uint64_t corrected_lsn = 999; /* sentinel -- must be overwritten */
int rc = ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T,
&corrected_lsn);
TEST_ASSERT_EQUAL_INT(0, rc);
/* next_index backed off by exactly one (raft.c: Max(next_index-1, 1)). */
TEST_ASSERT_EQUAL_UINT64(5, c->ha.raft.next_index[1]);
/* Corrected resume point is next_index - 1. */
TEST_ASSERT_EQUAL_UINT64(4, corrected_lsn);
/* A rejection never advances commit. */
TEST_ASSERT_EQUAL_UINT64(0, c->ha.raft.commit_index);
teardown_cluster(c);
printf("test_repl_nack_backs_off_next_index_and_returns_corrected_lsn: PASS\n");
}
/* (b) Repeated rejections keep backing off, floored at 1 -- never underflows
* (raft.c's Max(next_index-1, 1), matching raft.tla exactly). */
static void test_repl_nack_repeated_rejections_floor_at_one(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_floor_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/2);
/* Seed next_index[1] to what a real BecomeLeader would set (log_len+1 =
* 3) -- make_leader itself leaves it at 0 (see the backoff test above).
* Reject enough times to walk it down to 1 and confirm it never goes
* below. */
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
uint64_t corrected_lsn = 0;
for (int i = 0; i < 5; i++) {
TEST_ASSERT_EQUAL_INT(0,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T,
&corrected_lsn));
}
TEST_ASSERT_EQUAL_UINT64(1, c->ha.raft.next_index[1]);
TEST_ASSERT_EQUAL_UINT64(0, corrected_lsn);
teardown_cluster(c);
printf("test_repl_nack_repeated_rejections_floor_at_one: PASS\n");
}
/* (c) Defense-in-depth: same out-of-range/self rejection as the ACK seam --
* no library accounting on a bogus peer index. */
static void test_repl_nack_out_of_range_peer_rejected(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_oob_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/2);
/* Seed next_index[1] as a real BecomeLeader would (log_len+1 = 3); see
* the backoff test above for why make_leader itself leaves it at 0. */
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
uint64_t corrected_lsn = 0;
TEST_ASSERT_EQUAL_INT(-EINVAL,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/3, /*reply_term*/T, &corrected_lsn));
TEST_ASSERT_EQUAL_INT(-EINVAL,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/0, /*reply_term*/T, &corrected_lsn));
/* next_index untouched by either rejected call. */
TEST_ASSERT_EQUAL_UINT64(3, c->ha.raft.next_index[1]);
teardown_cluster(c);
printf("test_repl_nack_out_of_range_peer_rejected: PASS\n");
}
/* (d) THE CORE REGRESSION: composing ha_raft_on_repl_nack + repl_start_catchup
* -- exactly what repl_ack_decode_and_dispatch's NACK branch now does --
* marks the standby in_catch_up at the CORRECTED position on the very FIRST
* rejection. Before this fix, nothing marked it at all until repl_broadcast's
* own MAX_PENDING_ACKS=16 eviction fired (silently, using the standby's last
* ACKED lsn rather than a library-backed-off position) -- and even then,
* nothing would ever stream to it (the ACK-driven continuation requires a
* real ACK a log-conflicted standby can never send). This test proves the
* immediate half of the fix: no 16-strike wait, and the resume point comes
* from the library's own nextIndex backoff, not a guess. */
static void test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_catchup_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/5);
TEST_ASSERT_EQUAL_INT(0,
repl_add_standby(&c->persist.repl, &g_ae_dummy_ep, "sb0", 7000));
uint64_t join_lsn_before;
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
repl_get_catchup_join_lsn(&c->persist.repl, &g_ae_dummy_ep, &join_lsn_before),
"a freshly-added standby must not already be in catch-up");
/* THE FIRST rejection -- not the 16th. */
uint64_t corrected_lsn = 999;
TEST_ASSERT_EQUAL_INT(0,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T, &corrected_lsn));
repl_start_catchup(&c->persist.repl, &g_ae_dummy_ep, corrected_lsn);
uint64_t join_lsn_after;
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
repl_get_catchup_join_lsn(&c->persist.repl, &g_ae_dummy_ep, &join_lsn_after),
"a single rejection must immediately mark the standby in catch-up "
"(bug-1479 -- no 16-strike wait)");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(corrected_lsn, join_lsn_after,
"the catch-up resume point must be the library's backed-off "
"next_index - 1, not a guess");
teardown_cluster(c);
printf("test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes: PASS\n");
}
/* ================================================================== */
/* Task 5.1 — every-callback + marker-path end-to-end integration */
/* ================================================================== */
@ -2209,6 +2384,11 @@ void test_term_safety_adapter(void)
RUN_TEST(test_repl_ack_abs_peer_idx_endpoint_type_roundtrip);
RUN_TEST(test_repl_ack_heartbeat_standby_absolute_slot);
RUN_TEST(test_repl_ack_out_of_range_peer_rejected);
/* bug-1479: NACK seam drives the library's nextIndex backoff. */
RUN_TEST(test_repl_nack_backs_off_next_index_and_returns_corrected_lsn);
RUN_TEST(test_repl_nack_repeated_rejections_floor_at_one);
RUN_TEST(test_repl_nack_out_of_range_peer_rejected);
RUN_TEST(test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes);
/* Task 5.1: every-callback + marker-path end-to-end integration. */
RUN_TEST(test_every_callback_and_marker_path_end_to_end);
}

10
src/include/repl.h
View file

@ -137,6 +137,16 @@ void repl_start_catchup(replicator_t *r, repl_ep_t ep, uint64_t join_applied_lsn
/* Mark standby catch-up complete (called on OP_replCatchUpDone receipt) */
void repl_complete_catchup(replicator_t *r, repl_ep_t ep);
/* bug-1479: look up whether the given standby is currently in catch-up and,
* if so, its corrected resume point (join_applied_lsn). Returns 1 and sets
* *join_applied_lsn_out when in_catch_up is set; returns 0 (leaves
* *join_applied_lsn_out untouched) otherwise. Lets a caller (the heartbeat-
* driven replication decision) use the corrected position instead of a
* standby's own self-reported LSN, which is unreliable for a standby stuck
* rejecting every AppendEntries due to a log conflict. */
int repl_get_catchup_join_lsn(replicator_t *r, repl_ep_t ep,
uint64_t *join_applied_lsn_out);
/* Returns 1 if standby needs a full snapshot transfer (its LSN predates oldest WAL).
* oldest_wal_lsn == 0 means no WAL exists — no snapshot needed for LSN 0 standbys. */
int repl_needs_snapshot(replicator_t *r, uint64_t standby_lsn, uint64_t oldest_wal_lsn);