lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

test(ha-term-safety): harness post-failover client probes target a live node (bug-854 workaround)

Browse source 
The post-failover wait-primary/expect-job probes used the full masters list led
by the just-killed node, triggering libxce bug-854 (positional discovery can't
fail over from a dead-first master). Now they use a live-first list excluding the
killed primary.

NOTE: this exposes a deeper, separate post-failover AVAILABILITY issue (not data
loss): the new primary stays stuck in a catch-up loop with a lagging standby
(standby[0] lsn=5 < committed_lsn=6) and does not serve clients, so wait-primary
still times out. That is a catch-up/serving-readiness liveness concern orthogonal
to bug-828's committed-job DATA SAFETY, which is fixed and verified (walcompare
committed-prefix identical, jobs preserved on disk).
This commit is contained in:
Rongsong Shen 2026-06-23 04:12:32 +00:00
parent 51d8a4c981
commit 88823c59a7
1 changed files with 14 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

16
src/controller/testsuite/clean_failover_preservation.sh
View file

@ -195,6 +195,18 @@ done
[ -n "$PRIMARY" ] || fail "could not identify the elected primary from logs"
log "primary is $PRIMARY"
# Post-failover client probes must NOT lead the masters list with the node we are
# about to kill: libxce positional primary-discovery (bug-854) cannot fail over to
# the next master when the FIRST one is dead, so a client pointed at a dead-first
# list hangs in wait-primary regardless of server-side health. Build a masters
# list that excludes the (to-be-killed) primary so the post-failover client checks
# reach a live node. (This is a CLIENT-side test concern, orthogonal to the
# server-side committed-prefix safety that walcompare verifies.)
name_to_addr() { case "$1" in ctrl-1) echo "$P1";; ctrl-2) echo "$P2";; ctrl-3) echo "$P3";; esac; }
DEAD_ADDR="$(name_to_addr "$PRIMARY")"
MASTERS_LIVE="$(printf '%s\n' "$P1" "$P2" "$P3" | grep -vxF "$DEAD_ADDR" | paste -sd,)"
log "post-failover client masters (excludes killed $PRIMARY): $MASTERS_LIVE"
# ── 2b. Wait for the jobs to COMMIT on a majority BEFORE killing ──────────────
# A clean-failover test must kill the primary AFTER the submitted jobs are
# committed (durable on a majority). `jobsub` returns the JobId on acceptance,
@ -263,7 +275,7 @@ log "new primary is $NEW_PRIMARY"
# hard failure — the authoritative liveness signal is the log-confirmed
# promotion + the job-presence check at step 6b.
SERVE_OUT="$(printf 'set masters %s\nset myname failover-test\nclient\nwait-primary %s\nquit\n' \
"$MASTERS" "$WAIT_PRIMARY_MS" | xtest)"
"$MASTERS_LIVE" "$WAIT_PRIMARY_MS" | xtest)"
grep -q 'serving' <<<"$SERVE_OUT" \
|| log "WARN: post-failover wait-primary probe did not observe 'serving' within ${WAIT_PRIMARY_MS}ms (timing-sensitive; the authoritative checks are the log-confirmed promotion + expect-job at step 6b)"
@ -328,7 +340,7 @@ log "walcompare: committed prefix identical"
# ── 6b. every acked-committed job still present (R5.1 / R6.4) ─────────────────
log "verifying all ${#JOBIDS[@]} submitted jobs survive the failover"
EXPECT_SCRIPT="set masters $MASTERS
EXPECT_SCRIPT="set masters $MASTERS_LIVE
set myname failover-test
client
wait-primary $WAIT_PRIMARY_MS"