forked from lei.chen/myxce
Rongsong Shen
34ab78d72d
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node rotation) can leave a standby log-conflicted after a rejoin. Fixing this needed three compounding layers, each unit-green but still failing the identical live scenario until the next layer landed: 1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft library's nextIndex backoff and immediately repl_start_catchup's the standby at the corrected position, instead of the 16-strike silent eviction with no catch-up driver after it. Correct from the start, but live soak still failed identically: nothing ever produced the NACK this code was waiting for. 2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of silently dropping the reject. ha_send_wal_ack_reply had hardcoded success=1 unconditionally -- the only WalAckReply sender in the controller -- making (1)'s NACK-routing correct but unreachable dead code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe the NACK without a live endpoint. Live-confirmed the NACK now reaches the primary -- but the scenario still failed for a third reason. 3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's catch-up range was hard-bounded at committed_lsn, which can never advance in this exact scenario because advancing it requires an ACK from the very standby that's stuck catching up (2-of-3 alive, this standby is the only path to a new majority) -- a self-referential deadlock. Widens the STREAM_WAL ceiling (and the convergence check gating it) to the leader's own log tail, matching raft.tla's real AppendEntries semantics (leaderCommit is advisory, never a send-cap), while reusing bug-1478's exact guard so this doesn't reopen that trap (never widen onto a trailing local-only checkpoint's own unstreamable LSN). Verification: controller_unit_test 1053/1053 PASS, clean twice, including new regression tests for all three layers (test_config_repl.c, test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its pre-fix behavior before the real fix was restored. Three successive live run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the final run passed end-to-end -- all 3 force_elect_node rounds succeeded (ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never previously reached), zero materialized-state divergence, zero job loss. Unblocks config-upgrade-snapshot task 8.6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| xce | ||
| attrs.h | ||
| bitset.h | ||
| byteorder.h | ||
| client.h | ||
| clusterconfig.h | ||
| config.h | ||
| configobj.h | ||
| ctrl_metrics.h | ||
| dataq.h | ||
| delay.h | ||
| group.h | ||
| helper.h | ||
| hook.h | ||
| htab.h | ||
| idgen.h | ||
| jobj.h | ||
| kv.h | ||
| kvmgr.h | ||
| misc.h | ||
| myconf.h | ||
| mylock.h | ||
| netio.h | ||
| objs.h | ||
| ordq.h | ||
| persist.h | ||
| preason.h | ||
| qobj.h | ||
| repl.h | ||
| request.h | ||
| resobj.h | ||
| resreq.h | ||
| typemap.h | ||
| utils.h | ||
| vtaskmgr.h | ||
| wal_types.h | ||
| wobj.h | ||
| xcealloc.h | ||
| xceapi.h | ||
| xcectx.h | ||
| xcelimit.h | ||
| xcelog.h | ||
| xlist.h | ||
| xstring.h | ||
| xvec.h | ||
| zarray.h | ||