lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

test(ha-term-safety): live-harness fixes + record bug-991 (commit stall via catch-up + 2.4 ceiling)

Browse source 
Ran clean_failover_preservation.sh on a real fs ($HOME/tmp/xce-ha-deploy, ext4).
PAYLOAD run CONFIRMED bug-990 fixed live: standbys persist the leader's real
term (2) for the committed prefix; walcompare OK.

Harness fixes: robust failover detection (any post-kill CTRL_PRIMARY promotion,
45s settle) + a wait-for-commit-before-kill gate (jobsub returns on accept not
commit, so the prior harness raced replication vs kill).

That commit-wait gate exposed bug-991: on a STABLE cluster, jobs replicate to
all members (durable LSN 5) but committed_lsn pins at 1. Catch-up-delivered
entries don't advance the library commit_index, so task-2.4's ceiling
min(durable-majority, raft_commit_index) starves commit below the durable
majority. A real regression from the commit-source switch (2.4+3.3), invisible
to the steady-state-only unit tests.
This commit is contained in:
Rongsong Shen 2026-06-22 21:10:48 +00:00
parent 5fa394f598
commit df4bc4f86a
2 changed files with 39 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

40
src/controller/testsuite/clean_failover_preservation.sh
View file

@ -92,8 +92,12 @@ PEERS="$P1,$P2,$P3"
N_JOBS="${XCE_FAILOVER_NJOBS:-2}"
WAIT_PRIMARY_MS="${XCE_FAILOVER_WAIT_MS:-30000}"
# Liveness timeout in the controller is ~10s; allow margin for failover.
FAILOVER_SETTLE_S="${XCE_FAILOVER_SETTLE_S:-20}"
# Liveness timeout in the controller is ~10s; allow generous margin for failover.
# A standby that booted with no local config participates voter-only on the first
# RDHUP (ha-config-bootstrap Req 2.8) and only self-promotes on a later failover
# trigger once it holds config — so a clean-failover promotion can legitimately
# take ~25-35s (RDHUP detect + PreVote stagger + retry). 45s gives margin.
FAILOVER_SETTLE_S="${XCE_FAILOVER_SETTLE_S:-45}"
CATCHUP_SETTLE_S="${XCE_FAILOVER_CATCHUP_S:-8}"
STRICT_TERM="${XCE_FAILOVER_STRICT_TERM:-0}"
@ -191,6 +195,31 @@ done
[ -n "$PRIMARY" ] || fail "could not identify the elected primary from logs"
log "primary is $PRIMARY"
# ── 2b. Wait for the jobs to COMMIT on a majority BEFORE killing ──────────────
# A clean-failover test must kill the primary AFTER the submitted jobs are
# committed (durable on a majority). `jobsub` returns the JobId on acceptance,
# NOT after commit, so killing immediately would discard the still-UNCOMMITTED
# job entries on the new primary's promotion-truncate (correct Raft, NOT a job
# loss) — making the "no committed-job lost" assertion a race (it passed/failed
# nondeterministically on whether replication won the race with the kill). The
# primary logs each standby's learned committed_lsn via ha_process_standby_state;
# require it to reach the committed boundary (own-term noop LSN 1 + the committed
# JOB_SUBMIT entries; bug-845 doubles each submit => 1 + 2*N_JOBS).
COMMITTED_BOUNDARY="${XCE_FAILOVER_COMMITTED:-$((1 + 2 * N_JOBS))}"
COMMIT_WAIT_S="${XCE_FAILOVER_COMMIT_WAIT_S:-40}"
log "waiting up to ${COMMIT_WAIT_S}s for jobs to commit on a majority (committed_lsn >= $COMMITTED_BOUNDARY)"
COMMIT_HIGH=0
for _ in $(seq 1 "$COMMIT_WAIT_S"); do
COMMIT_HIGH="$(grep -oE 'committed_lsn=[0-9]+' "$TOPDIR/logs/${PRIMARY}.log" 2>/dev/null \
| awk -F= '{print $2}' | sort -n | tail -1)"
COMMIT_HIGH="${COMMIT_HIGH:-0}"
[ "$COMMIT_HIGH" -ge "$COMMITTED_BOUNDARY" ] && break
sleep 1
done
[ "$COMMIT_HIGH" -ge "$COMMITTED_BOUNDARY" ] \
|| fail "jobs did not commit on a majority within ${COMMIT_WAIT_S}s (max committed_lsn=$COMMIT_HIGH < $COMMITTED_BOUNDARY); killing now would test in-flight replication loss, not committed-job preservation (durability concern if the jobs truly never commit on a stable cluster)"
log "jobs committed on a majority (committed_lsn=$COMMIT_HIGH >= $COMMITTED_BOUNDARY)"
# ── 3. KILL the primary ──────────────────────────────────────────────────────
PRIMARY_PID="$(node_pid "$PRIMARY")"
[ -n "$PRIMARY_PID" ] || fail "could not resolve pid of primary $PRIMARY"
@ -204,7 +233,12 @@ for _ in $(seq 1 "$FAILOVER_SETTLE_S"); do
sleep 1
for n in ctrl-1 ctrl-2 ctrl-3; do
[ "$n" = "$PRIMARY" ] && continue
if grep -q 'raft_cb_become_leader.*CTRL_PRIMARY.*first_boot=0' "$TOPDIR/logs/${n}.log" 2>/dev/null; then
# A standby never logs become_leader before the kill, so ANY
# CTRL_PRIMARY promotion line on a non-original-primary node is the
# failover promotion (the first_boot flag varies with whether the
# standby recovered its WAL vs fell back to election-path recovery —
# it is NOT a reliable promotion discriminator).
if grep -q 'raft_cb_become_leader.*CTRL_PRIMARY' "$TOPDIR/logs/${n}.log" 2>/dev/null; then
NEW_PRIMARY="$n"
break
fi