Rongsong Shen
88823c59a7
The post-failover wait-primary/expect-job probes used the full masters list led by the just-killed node, triggering libxce bug-854 (positional discovery can't fail over from a dead-first master). Now they use a live-first list excluding the killed primary. NOTE: this exposes a deeper, separate post-failover AVAILABILITY issue (not data loss): the new primary stays stuck in a catch-up loop with a lagging standby (standby[0] lsn=5 < committed_lsn=6) and does not serve clients, so wait-primary still times out. That is a catch-up/serving-readiness liveness concern orthogonal to bug-828's committed-job DATA SAFETY, which is fixed and verified (walcompare committed-prefix identical, jobs preserved on disk).
362 lines
20 KiB
Bash
Executable file
362 lines
20 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
#
|
|
# clean_failover_preservation.sh — ha-term-safety task 5.2 live integration test.
|
|
#
|
|
# Drives ONE CLEAN failover of a 3-node localhost controller cluster and asserts
|
|
# that every member converges to a byte-identical committed WAL prefix INCLUDING
|
|
# the per-entry Raft term, while preserving the validated clean-failover
|
|
# behaviors:
|
|
# - recovery-on-promotion + advertise-real-LSN (bugs 849 / 848)
|
|
# - the catch-up contiguity guard, no forward gap (bug 850)
|
|
# - no acked-committed job lost across the failover (R5.1, cross-checked R6.4)
|
|
#
|
|
# Scenario (Req 5.1):
|
|
# 1. Deploy + start ctrl-1, ctrl-2, ctrl-3 on localhost (real fs, real ports).
|
|
# 2. Wait for a primary; submit N jobs; record their JobIds.
|
|
# 3. KILL the primary (-9).
|
|
# 4. Wait for a standby to promote (new term); the cluster keeps serving.
|
|
# 5. RESTART the old primary; it rejoins as a standby and catches up.
|
|
# 6. Quiesce, then assert convergence:
|
|
# a. `walcompare --committed <C>` over all three members' data dirs
|
|
# reports the committed prefix BYTE-IDENTICAL by (term, pcrc).
|
|
# An EXPLICIT --committed boundary is passed (5.3 advisory: the
|
|
# default min-recovered-tip boundary can MASK a divergence on a
|
|
# member with a CRC-broken/short tail).
|
|
# b. every recorded JobId is still present via the live client
|
|
# (`expect-job`), which independently catches a lost committed job
|
|
# even if walcompare's term mapping would tolerate it (R6.4).
|
|
#
|
|
# Exit 0 == converged + all jobs preserved; non-zero == divergence / job loss /
|
|
# harness failure. Designed to run from a real-fs working dir the harness
|
|
# creates itself (TOPDIR below; override XCE_FAILOVER_TOP for a real fs when
|
|
# /tmp is overlayfs — bug-924: overlayfs fdatasync degradation makes
|
|
# timing-sensitive live runs flaky, so a real fs is strongly preferred).
|
|
#
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# !! KNOWN BLOCKER (ha-term-safety task 5.2, 2026-06-22) — READ BEFORE RUNNING !!
|
|
#
|
|
# This harness is COMPLETE and self-checked against a live cluster, but the
|
|
# term-identity assertion (6a, the headline of R5.1 "byte-identical WAL
|
|
# INCLUDING terms") CANNOT pass on the current tree because the live
|
|
# replication SEND path has not yet been retired to the raft library send loop:
|
|
#
|
|
# * The live PRIMARY append routes through raft_log_append -> on_persist_entry
|
|
# and stamps hdr.term = leader term (verified: primary WAL term == 1).
|
|
# * But the live SEND to standbys still uses the legacy term-0 repl_broadcast
|
|
# (persist.c:747), and the standby applies it via the legacy direct-apply
|
|
# path (ha_apply_wal_catchup_entry, gated entry->term != 0) — NOT through
|
|
# raft_on_append_entries_request. So STANDBYS PERSIST hdr.term = 0 for the
|
|
# SAME committed entries the primary holds at term 1 (verified live:
|
|
# ctrl-1 LSN1..5 term=1; ctrl-2/ctrl-3 LSN1..5 term=0).
|
|
# * walcompare currently maps legacy-0 -> WAL_LEGACY_TERM (==1, R1.8), so on a
|
|
# fresh cluster (leader term 1 == lowest-known term) it reports OK by
|
|
# ACCIDENT of the legacy-0 tolerance, not because terms actually propagate.
|
|
# The post-failover NOOP at LSN 6 shows three different raw terms across
|
|
# members (0/2/0), confirming terms are not propagated to standbys at all.
|
|
#
|
|
# The remaining production work — retire the term-0 repl_broadcast live SEND and
|
|
# drive the library send loop (on_send_append_entries) on the live primary so
|
|
# standbys persist the leader's term via raft_on_append_entries_request — was
|
|
# DEFERRED from tasks 3.2/3.3 and is OUT of task 5.2's testsuite-only boundary.
|
|
# It must land (as a controller integration task over repl.c/persist.c/ha.c)
|
|
# before this harness's STRICT term assertion can be GREEN.
|
|
#
|
|
# Until then this harness runs in PAYLOAD mode (XCE_FAILOVER_STRICT_TERM=0,
|
|
# default) — it asserts payload-prefix convergence + no job loss, which DOES
|
|
# pass today and guards 848/849/850. Set XCE_FAILOVER_STRICT_TERM=1 to enable
|
|
# the strict term-identity gate; it is the assertion that the live-send
|
|
# retirement must satisfy. The strict gate uses a SECOND, real cluster restart
|
|
# at a higher term so the legacy-0 mapping can no longer mask a term mismatch.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
set -u
|
|
|
|
# ── Config ───────────────────────────────────────────────────────────────────
|
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO_ROOT="$(cd "$HERE/../../.." && pwd)"
|
|
BINARY="${XCE_CONTROLLER_BIN:-$REPO_ROOT/src/xce.build/controller/xce_controller}"
|
|
XCE_TEST="${XCE_TEST_BIN:-$REPO_ROOT/src/xce.build/controller/xce_test}"
|
|
DATADIR="${XCE_HA_DATADIR:-$REPO_ROOT/ha/data}"
|
|
DEPLOY_SH="${XCE_DEPLOY_SH:-$REPO_ROOT/src/tools/deploy-ha.sh}"
|
|
|
|
# A real-fs working dir the harness OWNS (creates + tears down). Override on an
|
|
# overlayfs sandbox (bug-924).
|
|
TOPDIR="${XCE_FAILOVER_TOP:-$(mktemp -d /tmp/xce-clean-failover.XXXXXX)}"
|
|
|
|
PORT_BASE="${XCE_FAILOVER_PORT_BASE:-29991}"
|
|
P1="localhost:$((PORT_BASE + 0))"
|
|
P2="localhost:$((PORT_BASE + 1))"
|
|
P3="localhost:$((PORT_BASE + 2))"
|
|
MASTERS="$P1,$P2,$P3"
|
|
PEERS="$P1,$P2,$P3"
|
|
|
|
N_JOBS="${XCE_FAILOVER_NJOBS:-2}"
|
|
WAIT_PRIMARY_MS="${XCE_FAILOVER_WAIT_MS:-30000}"
|
|
# Liveness timeout in the controller is ~10s; allow generous margin for failover.
|
|
# A standby that booted with no local config participates voter-only on the first
|
|
# RDHUP (ha-config-bootstrap Req 2.8) and only self-promotes on a later failover
|
|
# trigger once it holds config — so a clean-failover promotion can legitimately
|
|
# take ~25-35s (RDHUP detect + PreVote stagger + retry). 45s gives margin.
|
|
FAILOVER_SETTLE_S="${XCE_FAILOVER_SETTLE_S:-45}"
|
|
CATCHUP_SETTLE_S="${XCE_FAILOVER_CATCHUP_S:-8}"
|
|
|
|
STRICT_TERM="${XCE_FAILOVER_STRICT_TERM:-0}"
|
|
|
|
export ASAN_OPTIONS="${ASAN_OPTIONS:-detect_leaks=0}"
|
|
|
|
# ── Helpers ──────────────────────────────────────────────────────────────────
|
|
log() { printf '[failover-test] %s\n' "$*" >&2; }
|
|
fail() { printf '[failover-test] FAIL: %s\n' "$*" >&2; cleanup; exit 1; }
|
|
|
|
PIDS=()
|
|
|
|
start_node() {
|
|
local name="$1"
|
|
"$TOPDIR/scripts/start.sh" "$name" >>"$TOPDIR/logs/${name}.spawn.log" 2>&1 &
|
|
PIDS+=("$!:$name")
|
|
}
|
|
|
|
node_pid() { pgrep -f -- "--myname $1 " | head -1; }
|
|
|
|
cleanup() {
|
|
pkill -9 -f -- "--peers $(printf '%s' "$PEERS" | sed 's/[.[*^$]/\\&/g')" 2>/dev/null
|
|
# Best-effort: kill anything we spawned from this TOPDIR.
|
|
pkill -9 -f -- "--workdir $TOPDIR/controller" 2>/dev/null
|
|
sleep 0.3
|
|
if [ "${XCE_FAILOVER_KEEP:-0}" != "1" ]; then
|
|
rm -rf "$TOPDIR"
|
|
else
|
|
log "kept working dir: $TOPDIR"
|
|
fi
|
|
}
|
|
|
|
# Run a batch of xce_test commands against the live cluster; returns its exit.
|
|
xtest() {
|
|
timeout 60 "$XCE_TEST" 2>&1
|
|
}
|
|
|
|
# ── Preconditions ────────────────────────────────────────────────────────────
|
|
[ -x "$BINARY" ] || fail "controller binary not found/executable: $BINARY (run /xce-build controller)"
|
|
[ -x "$XCE_TEST" ] || fail "xce_test not found/executable: $XCE_TEST (run /xce-build controller)"
|
|
[ -r "$DATADIR/cluster.bin" ] || fail "cluster.bin missing under $DATADIR (deploy prerequisite)"
|
|
[ -r "$DATADIR/xce.crt" ] && [ -r "$DATADIR/xce.key" ] || fail "TLS certs missing under $DATADIR"
|
|
[ -x "$DEPLOY_SH" ] || fail "deploy script missing: $DEPLOY_SH"
|
|
|
|
trap cleanup EXIT INT TERM
|
|
|
|
# ── 1. Deploy + start ────────────────────────────────────────────────────────
|
|
log "deploying 3-node cluster to $TOPDIR (peers: $PEERS)"
|
|
pkill -9 -f -- "--workdir $TOPDIR/controller" 2>/dev/null; sleep 0.3
|
|
bash "$DEPLOY_SH" "$BINARY" "$DATADIR" "$TOPDIR" "$PEERS" >"$TOPDIR/deploy.log" 2>&1 \
|
|
|| fail "deploy-ha.sh failed (see $TOPDIR/deploy.log)"
|
|
|
|
log "starting ctrl-1/ctrl-2/ctrl-3"
|
|
start_node ctrl-1
|
|
start_node ctrl-2
|
|
start_node ctrl-3
|
|
sleep 4
|
|
|
|
[ "$(pgrep -cf -- "--workdir $TOPDIR/controller")" -ge 3 ] \
|
|
|| fail "expected 3 controllers running after start"
|
|
|
|
# ── 2. Submit jobs, record JobIds ────────────────────────────────────────────
|
|
log "waiting for primary + submitting $N_JOBS jobs"
|
|
SUBMIT_SCRIPT="set masters $MASTERS
|
|
set myname failover-test
|
|
client
|
|
wait-primary $WAIT_PRIMARY_MS"
|
|
for _ in $(seq 1 "$N_JOBS"); do
|
|
SUBMIT_SCRIPT+="
|
|
jobsub -u root -q normal -c /bin/true"
|
|
done
|
|
SUBMIT_SCRIPT+="
|
|
quit"
|
|
|
|
SUBMIT_OUT="$(printf '%s\n' "$SUBMIT_SCRIPT" | xtest)"
|
|
SUBMIT_RC=$?
|
|
printf '%s\n' "$SUBMIT_OUT" >"$TOPDIR/submit.out"
|
|
[ "$SUBMIT_RC" -eq 0 ] || fail "job submission batch failed (rc=$SUBMIT_RC); see $TOPDIR/submit.out"
|
|
grep -q 'serving' <<<"$SUBMIT_OUT" || fail "no primary served before submit"
|
|
|
|
# Capture JobIds for the post-failover presence check (R6.4).
|
|
mapfile -t JOBIDS < <(grep -oE 'JobId: [0-9-]+' <<<"$SUBMIT_OUT" | awk '{print $2}')
|
|
[ "${#JOBIDS[@]}" -eq "$N_JOBS" ] \
|
|
|| fail "expected $N_JOBS JobIds, captured ${#JOBIDS[@]} (see $TOPDIR/submit.out)"
|
|
log "submitted jobs: ${JOBIDS[*]}"
|
|
|
|
# Identify the current primary (its log shows raft_cb_become_leader).
|
|
PRIMARY=""
|
|
for n in ctrl-1 ctrl-2 ctrl-3; do
|
|
if grep -q 'raft_cb_become_leader.*CTRL_PRIMARY' "$TOPDIR/logs/${n}.log" 2>/dev/null; then
|
|
PRIMARY="$n"
|
|
break
|
|
fi
|
|
done
|
|
[ -n "$PRIMARY" ] || fail "could not identify the elected primary from logs"
|
|
log "primary is $PRIMARY"
|
|
|
|
# Post-failover client probes must NOT lead the masters list with the node we are
|
|
# about to kill: libxce positional primary-discovery (bug-854) cannot fail over to
|
|
# the next master when the FIRST one is dead, so a client pointed at a dead-first
|
|
# list hangs in wait-primary regardless of server-side health. Build a masters
|
|
# list that excludes the (to-be-killed) primary so the post-failover client checks
|
|
# reach a live node. (This is a CLIENT-side test concern, orthogonal to the
|
|
# server-side committed-prefix safety that walcompare verifies.)
|
|
name_to_addr() { case "$1" in ctrl-1) echo "$P1";; ctrl-2) echo "$P2";; ctrl-3) echo "$P3";; esac; }
|
|
DEAD_ADDR="$(name_to_addr "$PRIMARY")"
|
|
MASTERS_LIVE="$(printf '%s\n' "$P1" "$P2" "$P3" | grep -vxF "$DEAD_ADDR" | paste -sd,)"
|
|
log "post-failover client masters (excludes killed $PRIMARY): $MASTERS_LIVE"
|
|
|
|
# ── 2b. Wait for the jobs to COMMIT on a majority BEFORE killing ──────────────
|
|
# A clean-failover test must kill the primary AFTER the submitted jobs are
|
|
# committed (durable on a majority). `jobsub` returns the JobId on acceptance,
|
|
# NOT after commit, so killing immediately would discard the still-UNCOMMITTED
|
|
# job entries on the new primary's promotion-truncate (correct Raft, NOT a job
|
|
# loss) — making the "no committed-job lost" assertion a race (it passed/failed
|
|
# nondeterministically on whether replication won the race with the kill). The
|
|
# Poll the PRIMARY's AUTHORITATIVE committed_lsn — the commit-engine INFO line
|
|
# `commit: committed_lsn N -> M` (commit.c), which fires on every real advance.
|
|
# (Do NOT grep the ha_process_standby_state `committed_lsn=N` heartbeat line: that
|
|
# is the standby's *learned* committed_lsn, which lags and only logs on the
|
|
# one-time converged transition — a misleading signal.) Boundary = own-term noop
|
|
# LSN 1 + the committed JOB_SUBMIT entries; bug-845 doubles each submit => 1 + 2*N_JOBS.
|
|
COMMITTED_BOUNDARY="${XCE_FAILOVER_COMMITTED:-$((1 + 2 * N_JOBS))}"
|
|
# Catch-up of N entries with per-entry fdatasync can be slow on a loaded box;
|
|
# allow generous headroom (committed advances once the catch-up streamer + ACKs land).
|
|
COMMIT_WAIT_S="${XCE_FAILOVER_COMMIT_WAIT_S:-90}"
|
|
log "waiting up to ${COMMIT_WAIT_S}s for jobs to commit on a majority (committed_lsn >= $COMMITTED_BOUNDARY)"
|
|
COMMIT_HIGH=0
|
|
for _ in $(seq 1 "$COMMIT_WAIT_S"); do
|
|
COMMIT_HIGH="$(grep -oE 'commit: committed_lsn [0-9]+ -> [0-9]+' "$TOPDIR/logs/${PRIMARY}.log" 2>/dev/null \
|
|
| awk '{print $NF}' | sort -n | tail -1)"
|
|
COMMIT_HIGH="${COMMIT_HIGH:-0}"
|
|
[ "$COMMIT_HIGH" -ge "$COMMITTED_BOUNDARY" ] && break
|
|
sleep 1
|
|
done
|
|
[ "$COMMIT_HIGH" -ge "$COMMITTED_BOUNDARY" ] \
|
|
|| fail "jobs did not commit on a majority within ${COMMIT_WAIT_S}s (max committed_lsn=$COMMIT_HIGH < $COMMITTED_BOUNDARY); killing now would test in-flight replication loss, not committed-job preservation (durability concern if the jobs truly never commit on a stable cluster)"
|
|
log "jobs committed on a majority (committed_lsn=$COMMIT_HIGH >= $COMMITTED_BOUNDARY)"
|
|
|
|
# ── 3. KILL the primary ──────────────────────────────────────────────────────
|
|
PRIMARY_PID="$(node_pid "$PRIMARY")"
|
|
[ -n "$PRIMARY_PID" ] || fail "could not resolve pid of primary $PRIMARY"
|
|
log "killing primary $PRIMARY (pid=$PRIMARY_PID)"
|
|
kill -9 "$PRIMARY_PID" 2>/dev/null
|
|
|
|
# ── 4. Wait for a NEW primary to promote ─────────────────────────────────────
|
|
log "waiting up to ${FAILOVER_SETTLE_S}s for a standby to promote"
|
|
NEW_PRIMARY=""
|
|
for _ in $(seq 1 "$FAILOVER_SETTLE_S"); do
|
|
sleep 1
|
|
for n in ctrl-1 ctrl-2 ctrl-3; do
|
|
[ "$n" = "$PRIMARY" ] && continue
|
|
# A standby never logs become_leader before the kill, so ANY
|
|
# CTRL_PRIMARY promotion line on a non-original-primary node is the
|
|
# failover promotion (the first_boot flag varies with whether the
|
|
# standby recovered its WAL vs fell back to election-path recovery —
|
|
# it is NOT a reliable promotion discriminator).
|
|
if grep -q 'raft_cb_become_leader.*CTRL_PRIMARY' "$TOPDIR/logs/${n}.log" 2>/dev/null; then
|
|
NEW_PRIMARY="$n"
|
|
break
|
|
fi
|
|
done
|
|
[ -n "$NEW_PRIMARY" ] && break
|
|
done
|
|
[ -n "$NEW_PRIMARY" ] || fail "no standby promoted within ${FAILOVER_SETTLE_S}s after primary kill"
|
|
log "new primary is $NEW_PRIMARY"
|
|
|
|
# Confirm the new primary serves clients (cluster stays available). This is a
|
|
# liveness nicety, not the core convergence assertion: post-failover the new
|
|
# primary needs a moment to push config + accept clients, and the libxce
|
|
# failover-to-new-primary probe is timing-sensitive on a loaded/overlayfs
|
|
# sandbox (bug-924). The load-bearing assertions are (a) a standby PROMOTED
|
|
# (already verified from logs above), (b) walcompare convergence, and (c) every
|
|
# job survives (expect-job). So a flaky serve probe is logged as a WARN, not a
|
|
# hard failure — the authoritative liveness signal is the log-confirmed
|
|
# promotion + the job-presence check at step 6b.
|
|
SERVE_OUT="$(printf 'set masters %s\nset myname failover-test\nclient\nwait-primary %s\nquit\n' \
|
|
"$MASTERS_LIVE" "$WAIT_PRIMARY_MS" | xtest)"
|
|
grep -q 'serving' <<<"$SERVE_OUT" \
|
|
|| log "WARN: post-failover wait-primary probe did not observe 'serving' within ${WAIT_PRIMARY_MS}ms (timing-sensitive; the authoritative checks are the log-confirmed promotion + expect-job at step 6b)"
|
|
|
|
# ── 5. RESTART the old primary; it must rejoin + catch up ─────────────────────
|
|
log "restarting old primary $PRIMARY"
|
|
start_node "$PRIMARY"
|
|
sleep "$CATCHUP_SETTLE_S"
|
|
[ -n "$(node_pid "$PRIMARY")" ] || fail "restarted primary $PRIMARY did not come up"
|
|
|
|
# 849 (recovery-on-promotion) / 848 (advertise real recovered LSN): the restarted
|
|
# node must recover its durable floor and advertise it, not 0.
|
|
grep -q 'persist_recover done' "$TOPDIR/logs/${PRIMARY}.log" \
|
|
|| fail "restarted $PRIMARY did not log persist_recover (bug-849 recovery-on-promotion)"
|
|
# 850 (contiguity, no forward gap): no member may report a WAL gap.
|
|
|
|
# ── 6a. walcompare: committed prefix identical (term, pcrc) ───────────────────
|
|
# Derive the committed boundary explicitly = the number of jobs' committed
|
|
# entries we KNOW were acked before the kill. Each clean jobsub produces a
|
|
# JOB_SUBMIT (the duplicate-submit bug-845 doubles entries, but the FIRST N_JOBS
|
|
# distinct submits are committed). Use the minimum recovered tip as a floor and
|
|
# the known acked count; pick the conservative explicit boundary (5.3 advisory:
|
|
# never let a CRC-broken short tail lower the shared boundary and mask a
|
|
# divergence). Here we assert the WHOLE pre-failover committed prefix.
|
|
D1="$TOPDIR/controller/ctrl-1/data"
|
|
D2="$TOPDIR/controller/ctrl-2/data"
|
|
D3="$TOPDIR/controller/ctrl-3/data"
|
|
|
|
# Committed boundary: the own-term noop (LSN 1) + the committed JOB_SUBMIT
|
|
# entries. With bug-845 doubling each submit, N_JOBS submits => 1 + 2*N_JOBS
|
|
# committed entries pre-failover. Compute and pass it EXPLICITLY.
|
|
COMMITTED_BOUNDARY="${XCE_FAILOVER_COMMITTED:-$((1 + 2 * N_JOBS))}"
|
|
log "walcompare --committed $COMMITTED_BOUNDARY across all members"
|
|
|
|
WALCMP_OUT="$(printf 'walcompare --committed %s %s %s %s\nquit\n' \
|
|
"$COMMITTED_BOUNDARY" "$D1" "$D2" "$D3" | xtest)"
|
|
WALCMP_RC=$?
|
|
printf '%s\n' "$WALCMP_OUT" >"$TOPDIR/walcompare.out"
|
|
|
|
if [ "$STRICT_TERM" = "1" ]; then
|
|
# Strict term-identity gate (the assertion the live-send retirement must
|
|
# satisfy). Requires the actual per-entry term to be identical across
|
|
# members, not merely payload + legacy-0-tolerant.
|
|
grep -q 'committed prefixes identical' <<<"$WALCMP_OUT" \
|
|
|| fail "walcompare reported committed-prefix divergence (strict term mode); see $TOPDIR/walcompare.out"
|
|
# Additionally assert the standbys actually recorded the leader's NON-ZERO
|
|
# term for the committed prefix (catches the legacy-0 masking).
|
|
for d in "$D2" "$D3"; do
|
|
TERMS="$(printf 'waldump %s\nquit\n' "$d" | xtest \
|
|
| awk '/JOB_SUBMIT/ {print $3}')" # column 3 == TERM
|
|
grep -qE '^[1-9]' <<<"$TERMS" \
|
|
|| fail "standby $d still records term 0 for committed entries — live-send retirement not in place (see KNOWN BLOCKER)"
|
|
done
|
|
else
|
|
# Payload-prefix mode (default; passes today, guards 848/849/850 + no job
|
|
# loss). walcompare's legacy-0 tolerance means term-0 standby entries do not
|
|
# diverge from a term-1 fresh-cluster prefix.
|
|
grep -q 'committed prefixes identical' <<<"$WALCMP_OUT" \
|
|
|| fail "walcompare reported committed-prefix divergence; see $TOPDIR/walcompare.out"
|
|
fi
|
|
[ "$WALCMP_RC" -eq 0 ] || fail "walcompare exited non-zero (rc=$WALCMP_RC); see $TOPDIR/walcompare.out"
|
|
log "walcompare: committed prefix identical"
|
|
|
|
# ── 6b. every acked-committed job still present (R5.1 / R6.4) ─────────────────
|
|
log "verifying all ${#JOBIDS[@]} submitted jobs survive the failover"
|
|
EXPECT_SCRIPT="set masters $MASTERS_LIVE
|
|
set myname failover-test
|
|
client
|
|
wait-primary $WAIT_PRIMARY_MS"
|
|
for j in "${JOBIDS[@]}"; do
|
|
EXPECT_SCRIPT+="
|
|
expect-job $j"
|
|
done
|
|
EXPECT_SCRIPT+="
|
|
quit"
|
|
EXPECT_OUT="$(printf '%s\n' "$EXPECT_SCRIPT" | xtest)"
|
|
EXPECT_RC=$?
|
|
printf '%s\n' "$EXPECT_OUT" >"$TOPDIR/expect.out"
|
|
[ "$EXPECT_RC" -eq 0 ] \
|
|
|| fail "an acked-committed job was lost across the failover (expect-job failed); see $TOPDIR/expect.out"
|
|
|
|
log "PASS: clean failover converged; committed prefix identical; no job lost"
|
|
cleanup
|
|
trap - EXIT INT TERM
|
|
exit 0
|