forked from lei.chen/myxce
feat(xce-test-ha): support deploying HA cluster to a specific directory
Add an optional [dir] argument to the /xce-test-ha skill so the 3-node soak cluster can be deployed to a caller-chosen path. - run-ha-soak.sh: deploy dir resolves positional $1 > $XCE_HA_TOP > default; relative paths absolutized in pure bash (this host's busybox realpath lacks -m and would mangle nonexistent/relative paths). - Fail-loud rm -rf guard: writes a .xce-ha-deploy sentinel at deploy time and refuses to wipe a non-empty dir lacking it (protects a typo'd home/source path); empty/nonexistent dirs stay safe to (re)deploy into. - SKILL.md: document /xce-test-ha [rounds] [dir] and the safety semantics. Verified: guard refuses an unmarked non-empty dir (dir survives, exit 1); full 5-round soak into a custom dir PASS (18 jobs, zero job loss), cluster landed in the named dir while the default dir was untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
This commit is contained in:
parent
0eafed3999
commit
6067ba6c94
2 changed files with 578 additions and 0 deletions
189
.claude/skills/xce-test-ha/SKILL.md
Normal file
189
.claude/skills/xce-test-ha/SKILL.md
Normal file
|
|
@ -0,0 +1,189 @@
|
|||
---
|
||||
name: xce-test-ha
|
||||
description: "End-to-end HA failover soak test for an XCE controller cluster: deploy 3 nodes, submit jobs, then repeatedly kill the primary and verify zero committed-job loss across 5-10 rounds. Usage: /xce-test-ha [rounds] [dir]"
|
||||
---
|
||||
|
||||
# XCE HA Failover Soak Test
|
||||
|
||||
Deploys a 3-node XCE controller cluster, submits jobs, then drives repeated
|
||||
primary kills + restarts, verifying after every failover that **no committed job
|
||||
is ever lost**. PASS only if all rounds complete with zero loss.
|
||||
|
||||
This is the live regression for the HA failover work. It deterministically
|
||||
exercises the full single-failover lifecycle (kill → elect → discover → verify →
|
||||
restart → reconverge) and has surfaced/validated these bugs:
|
||||
- **bug-828** committed-job data safety,
|
||||
- **bug-854** client primary discovery (incl. dead-first-master initial connect),
|
||||
- **bug-1028** post-failover availability (streamer term-source),
|
||||
- **bug-1042** standby-rejoin never converges (raft log_len not synced after recovery),
|
||||
- **bug-1043** catch-up apply gap (replicated entries not applied to the job table).
|
||||
|
||||
A green run is `PASS: N rounds, M jobs submitted, zero job loss` (exit 0).
|
||||
|
||||
## Arguments
|
||||
|
||||
- Optional `[rounds]` — number of kill/restart rounds (clamped to 5–10, default 6).
|
||||
- Optional `[dir]` — deploy directory for the 3-node cluster
|
||||
(default `/home/claude/tmp/xce-ha-deploy`). Maps to `XCE_HA_TOP` /
|
||||
`run-ha-soak.sh`'s positional dir arg. Relative paths are resolved to
|
||||
absolute. **Safety:** the driver `rm -rf`s this dir on each run, so it refuses
|
||||
to wipe a non-empty directory that wasn't created by a prior `/xce-test-ha`
|
||||
run (it looks for a `.xce-ha-deploy` marker) — point it at a new/empty/scratch
|
||||
path, not a real source or home directory.
|
||||
|
||||
```
|
||||
/xce-test-ha # 6 rounds, default deploy dir
|
||||
/xce-test-ha 10 # 10 rounds, default deploy dir
|
||||
/xce-test-ha 10 /tmp/my-ha-run # 10 rounds, deploy to /tmp/my-ha-run
|
||||
```
|
||||
|
||||
## Step 1 — Ensure binaries are built
|
||||
|
||||
The test needs `xce_controller` and `xce_test`:
|
||||
|
||||
```bash
|
||||
BIN=src/xce.build/controller/xce_controller
|
||||
XT=src/xce.build/controller/xce_test
|
||||
ls -lh "$BIN" "$XT"
|
||||
```
|
||||
|
||||
If either is missing, build them first with the `/xce-build` skill:
|
||||
|
||||
```
|
||||
/xce-build controller
|
||||
```
|
||||
|
||||
(Do NOT proceed to the soak until both binaries exist.)
|
||||
|
||||
## Step 2 — Run the soak driver
|
||||
|
||||
The full workflow (deploy → start → elect → submit → kill-loop → verify) is
|
||||
encoded in `run-ha-soak.sh` in this skill directory. It MUST run under the nix
|
||||
dev shell so the controller/xce_test runtime libs (openssl, etc.) are on the
|
||||
library path. Run it in the **background** (it takes several minutes) and tail
|
||||
the log:
|
||||
|
||||
```bash
|
||||
SKILL_DIR=.claude/skills/xce-test-ha
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" 2>&1 | tee /tmp/xce-test-ha.log
|
||||
```
|
||||
|
||||
Pass the round count and other knobs via env. The deploy directory may be given
|
||||
either as the script's first positional argument or via `XCE_HA_TOP` (positional
|
||||
wins):
|
||||
|
||||
```bash
|
||||
# 10 rounds, default deploy dir
|
||||
XCE_HA_ROUNDS=10 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh"
|
||||
|
||||
# 10 rounds, deploy to a specific directory (positional arg)
|
||||
XCE_HA_ROUNDS=10 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" /tmp/my-ha-run
|
||||
|
||||
# equivalent via env
|
||||
XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh"
|
||||
```
|
||||
|
||||
| Env var | Meaning | Default |
|
||||
|---|---|---|
|
||||
| `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
|
||||
| `XCE_HA_DATADIR` | config/certs source (`cluster.bin`, `xce.crt`, `xce.key`) | `<repo>/ha/data` |
|
||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–10) | `6` |
|
||||
| `XCE_HA_JOBS` | jobs per submit batch | `3` |
|
||||
| `XCE_HA_WAITMS` | `wait-primary` budget per probe | `30000` |
|
||||
|
||||
## What the driver does (maps to the requested steps)
|
||||
|
||||
1. **Preconditions** — verifies `xce_controller`, `xce_test`, `cluster.bin`, and certs exist.
|
||||
2. **Deploy** — `src/tools/deploy-ha.sh` into `XCE_HA_TOP` with nodes
|
||||
`ctrl-1=localhost:9991, ctrl-2=localhost:9992, ctrl-3=localhost:9993`,
|
||||
using config/certs from `XCE_HA_DATADIR`.
|
||||
3. **Start** — `scripts/start.sh ctrl-N -vvv` for all three, 0.5s staggered.
|
||||
4. **Elect** — waits (≤45s) for `raft_cb_become_leader … CTRL_PRIMARY` in any
|
||||
`logs/ctrl-N.log`, then confirms the client can reach the primary via
|
||||
`ctrlstatus`.
|
||||
5. **Initial workload** — drives `xce_test`:
|
||||
`set masters … / client / wait-primary / jobsub ×N / joblist (expect-job) / walcompare`.
|
||||
Every acked `JobId` is tracked.
|
||||
6–11. **Kill loop** (5–10 rounds): discover the live primary → `kill -9` it →
|
||||
restart the node downed in the *previous* round (so the cluster returns to 3
|
||||
members) → wait for a new primary → `verify_no_loss` (expect-job every tracked
|
||||
JobId) → submit another batch → `verify_no_loss` again → `walcompare`.
|
||||
12. **Verdict** — exits `0` (PASS) only if every round verified zero job loss;
|
||||
any missing job, failed election, or unreachable primary exits non-zero (FAIL).
|
||||
|
||||
## Step 3 — Report the result
|
||||
|
||||
- Exit `0` and a final `PASS: N rounds, M jobs submitted, zero job loss` line ⇒ **test PASSED**.
|
||||
- Any `FAIL: …` line / non-zero exit ⇒ **test FAILED**. Surface the failing
|
||||
round and reason; the per-node logs under `$XCE_HA_TOP/logs/` and the deploy
|
||||
log `$XCE_HA_TOP/deploy.log` hold the detail.
|
||||
|
||||
## Goal-driven loop integration (`/goal`)
|
||||
|
||||
This skill is designed to be the **success oracle** for a goal-driven loop whose
|
||||
goal is "the HA soak test passes". The driver is built for that contract:
|
||||
|
||||
- **Single machine-checkable verdict.** Exactly one terminal line — either
|
||||
`[xce-test-ha] PASS: …` (exit 0) or `[xce-test-ha] FAIL: …` (exit ≠ 0). A loop
|
||||
decides done/continue by grepping that line / checking the exit code; no
|
||||
human-style parsing of intermediate output is needed.
|
||||
- **Fully idempotent & self-contained.** Every run `rm -rf`s `$XCE_HA_TOP`,
|
||||
redeploys from `ha/data`, kills any stale `xce_controller`, and cleans up on
|
||||
exit (trap). A loop can invoke it any number of times with no manual reset.
|
||||
- **Deterministic-enough.** Cold-start election ~15s, each round ~20-30s; a
|
||||
5-round run is ~3-5 min, 10-round ~6-10 min. Budget the loop's per-iteration
|
||||
timeout to `rounds × 60s + 60s`.
|
||||
|
||||
**Loop shape** (the goal is reached when the driver exits 0):
|
||||
|
||||
```
|
||||
goal: `/xce-test-ha` exits 0 with a "PASS:" line
|
||||
each iteration:
|
||||
1. run the driver (background, capture log + exit code)
|
||||
2. if exit == 0 and log has "PASS:" → GOAL REACHED, stop
|
||||
3. else: read the FAIL line + $XCE_HA_TOP/logs to find the root cause
|
||||
4. fix the underlying bug in src/ (controller or libxce)
|
||||
5. /xce-build controller (rebuild the binaries the next run uses)
|
||||
6. repeat
|
||||
```
|
||||
|
||||
The driver itself does NOT fix code — it is the *oracle*. The loop's job each
|
||||
round is: run oracle → if FAIL, diagnose from the logs, patch `src/`, rebuild,
|
||||
re-run. The goal is reached the first time the oracle returns PASS.
|
||||
|
||||
## Job-loss model (why expect-job is the authority)
|
||||
|
||||
`walcompare --committed` checks the committed WAL prefix is byte-identical
|
||||
(term+crc) across members; it is logged as a note. The **authoritative**
|
||||
loss check is `expect-job <JobId>` against the live primary for every job ever
|
||||
acked — a committed job that an elected primary cannot list is a real loss
|
||||
(bug-828 class), regardless of walcompare's term-mapping tolerance.
|
||||
|
||||
The driver runs `xce_test` with `ASAN_OPTIONS=detect_leaks=0`: the debug/ASAN
|
||||
build reports harmless heap leaks at exit (non-zero exit code) that are NOT job
|
||||
loss. Verdicts are therefore judged on command **output** (`expect-job: ok` count
|
||||
and `not found`), never the leak-polluted process exit code.
|
||||
|
||||
## Manual driving (debug)
|
||||
|
||||
To poke the cluster by hand after the driver deploys it (or instead of it):
|
||||
|
||||
```bash
|
||||
# discover primary + submit one job
|
||||
printf 'set masters localhost:9991,localhost:9992,localhost:9993\nset myname m\nclient\nwait-primary 30000\nctrlstatus\njobsub -u root -q normal -c /bin/true\njoblist\nquit\n' \
|
||||
| nix develop ./src --command "$XT"
|
||||
|
||||
# dump a node's WAL
|
||||
printf 'waldump %s/controller/ctrl-1/data\nquit\n' "$XCE_HA_TOP" \
|
||||
| nix develop ./src --command "$XT"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---|---|---|
|
||||
| `controller binary missing` | not built | `/xce-build controller` |
|
||||
| `no primary elected within 45s` | cold-start election stalled / port clash | check `logs/ctrl-*.log`; ensure ports 9991-9993 free |
|
||||
| `primary elected in logs but client could not reach it` | client discovery (bug-854 regression) | check `client.c` rotation fix is in the binary |
|
||||
| `a committed job was LOST after failover` | data-safety regression (bug-828 class) | inspect WAL via `waldump`; this is a real FAIL |
|
||||
| driver hangs | a controller wedged; `xtest` 60s timeout should break it | check for stuck `xce_test`/`xce_controller` procs |
|
||||
389
.claude/skills/xce-test-ha/run-ha-soak.sh
Executable file
389
.claude/skills/xce-test-ha/run-ha-soak.sh
Executable file
|
|
@ -0,0 +1,389 @@
|
|||
#!/usr/bin/env bash
|
||||
#
|
||||
# run-ha-soak.sh — driver for /xce-test-ha: deploy a 3-node XCE HA cluster, then
|
||||
# repeatedly kill the primary, verify no committed job is ever lost, restart the
|
||||
# downed node, and loop.
|
||||
#
|
||||
# Exit 0 == PASS (no job loss across all rounds); non-zero == FAIL.
|
||||
#
|
||||
# Must be invoked under `nix develop ./src --command bash ...` so the controller
|
||||
# and xce_test runtime libs (openssl etc.) are on the library path.
|
||||
#
|
||||
# Usage:
|
||||
# run-ha-soak.sh [DEPLOY_DIR]
|
||||
# DEPLOY_DIR optional deploy directory (positional). Precedence:
|
||||
# positional arg > $XCE_HA_TOP env > default. Relative paths
|
||||
# are resolved to absolute.
|
||||
#
|
||||
# Env overrides:
|
||||
# XCE_HA_TOP deploy dir (default /home/claude/tmp/xce-ha-deploy)
|
||||
# XCE_HA_DATADIR config/certs source (default <repo>/ha/data)
|
||||
# XCE_HA_ROUNDS kill/restart rounds (default 6, clamp 5..10)
|
||||
# XCE_HA_JOBS jobs per submit batch (default 3)
|
||||
# XCE_HA_WAITMS wait-primary budget (default 30000)
|
||||
set -u
|
||||
|
||||
# ── Paths ────────────────────────────────────────────────────────────────────
|
||||
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
|
||||
# Deploy dir: explicit positional arg wins, then $XCE_HA_TOP, then the default.
|
||||
TOP="${1:-${XCE_HA_TOP:-/home/claude/tmp/xce-ha-deploy}}"
|
||||
# Make a relative dir absolute (against cwd) so it is unambiguous to deploy-ha.sh
|
||||
# and the per-node scripts. Pure-bash — no realpath dependency (busybox realpath
|
||||
# here lacks -m and would mangle nonexistent/relative paths).
|
||||
case "$TOP" in
|
||||
/*) : ;; # already absolute
|
||||
*) TOP="$PWD/$TOP" ;; # resolve relative path against current directory
|
||||
esac
|
||||
# Sentinel written into a deploy we own; the guard before `rm -rf` only wipes a
|
||||
# non-empty dir that carries this marker (see "Step 2: deploy" below).
|
||||
MARKER_NAME=".xce-ha-deploy"
|
||||
DATADIR="${XCE_HA_DATADIR:-$REPO/ha/data}"
|
||||
BIN="$REPO/src/xce.build/controller/xce_controller"
|
||||
XT="$REPO/src/xce.build/controller/xce_test"
|
||||
DEPLOY="$REPO/src/tools/deploy-ha.sh"
|
||||
|
||||
ROUNDS="${XCE_HA_ROUNDS:-6}"
|
||||
[ "$ROUNDS" -lt 5 ] && ROUNDS=5
|
||||
[ "$ROUNDS" -gt 10 ] && ROUNDS=10
|
||||
NJOBS="${XCE_HA_JOBS:-3}"
|
||||
WAITMS="${XCE_HA_WAITMS:-30000}"
|
||||
|
||||
NODES=(ctrl-1 ctrl-2 ctrl-3)
|
||||
declare -A PORT=( [ctrl-1]=9991 [ctrl-2]=9992 [ctrl-3]=9993 )
|
||||
PEERS="ctrl-1=localhost:9991,ctrl-2=localhost:9992,ctrl-3=localhost:9993"
|
||||
MASTERS="localhost:9991,localhost:9992,localhost:9993"
|
||||
|
||||
ALL_JOBIDS=() # every JobId we have ever successfully submitted
|
||||
|
||||
log() { echo "[xce-test-ha] $*"; }
|
||||
fail() { echo "[xce-test-ha] FAIL: $*" >&2; cleanup; exit 1; }
|
||||
|
||||
cleanup() { pkill -9 -x xce_controller 2>/dev/null; sleep 0.3; }
|
||||
|
||||
# ── Preconditions ────────────────────────────────────────────────────────────
|
||||
[ -x "$BIN" ] || { echo "[xce-test-ha] FAIL: controller binary missing: $BIN (run /xce-build controller)" >&2; exit 2; }
|
||||
[ -x "$XT" ] || { echo "[xce-test-ha] FAIL: xce_test missing: $XT (run /xce-build controller)" >&2; exit 2; }
|
||||
[ -r "$DATADIR/cluster.bin" ] || { echo "[xce-test-ha] FAIL: cluster.bin missing under $DATADIR" >&2; exit 2; }
|
||||
[ -r "$DATADIR/xce.crt" ] && [ -r "$DATADIR/xce.key" ] || { echo "[xce-test-ha] FAIL: certs missing under $DATADIR" >&2; exit 2; }
|
||||
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
# ── Helpers ──────────────────────────────────────────────────────────────────
|
||||
|
||||
# Run a batch of xce_test commands (stdin script) with a hard timeout.
|
||||
# Disable ASAN leak detection: the debug/ASAN xce_test build reports harmless
|
||||
# heap leaks at exit (non-zero exit code) that would otherwise be misread as a
|
||||
# command failure. Job-loss is judged by the command OUTPUT (expect-job "not
|
||||
# found"), not the leak-polluted process exit code.
|
||||
xtest() { ASAN_OPTIONS=detect_leaks=0 timeout 60 "$XT" 2>&1; }
|
||||
|
||||
start_node() { bash "$TOP/scripts/start.sh" "$1" -vvv >/dev/null 2>&1 & }
|
||||
|
||||
# The INFO-level promotion marker each elected primary logs (ha.c: raft_cb_
|
||||
# become_leader → "HA FSM (raft): → CTRL_PRIMARY (term=N, ...)"). Present even
|
||||
# without -vvv. We track election by this line + its term, which is the most
|
||||
# reliable "a new primary was elected" signal (independent of client-side
|
||||
# discovery). NOTE: the arrow is a UTF-8 "→"; grep with a wildcard to avoid
|
||||
# encoding pitfalls.
|
||||
PRIMARY_LOG_RE='HA FSM .raft.: . CTRL_PRIMARY .term='
|
||||
|
||||
# Highest term any node has announced CTRL_PRIMARY for, across all logs.
|
||||
latest_primary_term() {
|
||||
grep -ohE "$PRIMARY_LOG_RE[0-9]+" "$TOP"/logs/*.log 2>/dev/null \
|
||||
| grep -oE 'term=[0-9]+' | grep -oE '[0-9]+' | sort -n | tail -1
|
||||
}
|
||||
|
||||
# The node name that most recently logged CTRL_PRIMARY (the elected primary).
|
||||
# Echoes "ctrl-N" or "" — purely from the logs, no client round-trip.
|
||||
primary_from_log() {
|
||||
local n best_term="" best_node="" t
|
||||
for n in "${NODES[@]}"; do
|
||||
t="$(grep -oE "$PRIMARY_LOG_RE[0-9]+" "$TOP/logs/$n.log" 2>/dev/null \
|
||||
| grep -oE 'term=[0-9]+' | grep -oE '[0-9]+' | sort -n | tail -1)"
|
||||
[ -z "$t" ] && continue
|
||||
if [ -z "$best_term" ] || [ "$t" -ge "$best_term" ]; then
|
||||
best_term="$t"; best_node="$n"
|
||||
fi
|
||||
done
|
||||
echo "$best_node"
|
||||
}
|
||||
|
||||
# Block until a node announces CTRL_PRIMARY at a term STRICTLY GREATER than
|
||||
# `$1` (the term before the kill), i.e. a genuinely NEW election has happened.
|
||||
# Echoes the new primary's node name on success; returns non-zero on timeout.
|
||||
wait_for_new_primary() {
|
||||
local prev_term="${1:-0}" i t node
|
||||
for i in $(seq 1 45); do
|
||||
t="$(latest_primary_term)"
|
||||
if [ -n "$t" ] && [ "$t" -gt "$prev_term" ]; then
|
||||
node="$(primary_from_log)"
|
||||
[ -n "$node" ] && { echo "$node"; return 0; }
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Wait until SOME node logs CTRL_PRIMARY (cold-start election), up to ~45s.
|
||||
wait_for_primary_log() {
|
||||
local i
|
||||
for i in $(seq 1 45); do
|
||||
[ -n "$(latest_primary_term)" ] && return 0
|
||||
sleep 1
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Identify the current primary by asking the cluster (authoritative ctrlstatus,
|
||||
# which internally rotates to the serving primary). Echoes "ctrl-N" or "".
|
||||
# Retries the whole wait-primary+ctrlstatus probe for up to ~60s so it spans a
|
||||
# full failover election (liveness timeout ~10s + PreVote stagger + election).
|
||||
# This is the SERVING check (the client can actually reach the primary), distinct
|
||||
# from the log-based election check above.
|
||||
discover_primary() {
|
||||
local out name i
|
||||
for i in $(seq 1 6); do
|
||||
out="$(printf 'set masters %s\nset myname probe\nclient\nwait-primary %s\nctrlstatus\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" | xtest)"
|
||||
name="$(echo "$out" | sed -n 's/^Primary: \(ctrl-[0-9]\)$/\1/p' | head -1)"
|
||||
if [ -n "$name" ]; then
|
||||
echo "$name"
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Wait until the cluster is FULLY converged: all 3 controllers are alive AND the
|
||||
# current primary is no longer actively streaming catch-up to any standby. This
|
||||
# is a HARD precondition before killing the next primary — killing while a just-
|
||||
# restarted node is still catching up tests an UNSUPPORTED double-fault-within-
|
||||
# one-reconvergence window (a momentary 2-of-3 with a stale member), not the
|
||||
# single-failover guarantee.
|
||||
#
|
||||
# Convergence signal: a lagging standby makes the primary log "stream WAL batches"
|
||||
# every heartbeat (~3s); once every standby is caught up the primary STOPS
|
||||
# emitting that line (action becomes HA_REPL_CONVERGED, a no-op). So "no new
|
||||
# 'stream WAL batches' line on the primary for STABLE_S seconds" == all standbys
|
||||
# converged. This is robust to the per-term re-emit quirk of the "standby[i]
|
||||
# converged" transition log (which only fires for a standby that was previously
|
||||
# NOT converged under the current primary term). Bounded ~90s.
|
||||
wait_for_full_convergence() {
|
||||
local i alive primary line_now line_prev stable=0
|
||||
local STABLE_TICKS=3 # 3 ticks × 2s = 6s with no new stream line ⇒ converged
|
||||
line_prev=""
|
||||
for i in $(seq 1 45); do
|
||||
alive="$(pgrep -c -f -- "xce_controller --myname ctrl-")"
|
||||
if [ "$alive" -lt 3 ]; then
|
||||
stable=0; sleep 2; continue
|
||||
fi
|
||||
primary="$(primary_from_log)"
|
||||
[ -z "$primary" ] && { sleep 2; continue; }
|
||||
# Most recent "stream WAL batches" line in the primary's log (empty if
|
||||
# the primary has never streamed / has stopped). Compare across ticks:
|
||||
# if it has not changed for STABLE_TICKS, no catch-up is in flight.
|
||||
line_now="$(grep -n "stream WAL batches" "$TOP/logs/$primary.log" 2>/dev/null | tail -1)"
|
||||
if [ "$line_now" = "$line_prev" ]; then
|
||||
stable=$((stable + 1))
|
||||
[ "$stable" -ge "$STABLE_TICKS" ] && return 0
|
||||
else
|
||||
stable=0
|
||||
line_prev="$line_now"
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Submit jobs through the live primary until N are cleanly acked, appending each
|
||||
# acked JobId to ALL_JOBIDS. A FRESHLY-promoted primary is briefly not write-able
|
||||
# (it appends its own-term NOOP but cannot commit until a standby reconnects and
|
||||
# write-quorum is restored — log: "HA quorum lost/restored"), so a submit issued
|
||||
# in that ~1-3s window fails its 500ms commit WAIT. That is NOT a job loss (the
|
||||
# job was never acked, so it is correctly not tracked) — it is a transient that
|
||||
# clears once quorum returns. Retry the SHORTFALL up to a few times.
|
||||
submit_jobs() {
|
||||
local want="$1" attempt acked=0 got script out jid
|
||||
for attempt in 1 2 3 4 5; do
|
||||
local need=$((want - acked))
|
||||
[ "$need" -le 0 ] && break
|
||||
script="set masters $MASTERS
|
||||
set myname soak
|
||||
client
|
||||
wait-primary $WAITMS"
|
||||
local i
|
||||
for i in $(seq 1 "$need"); do
|
||||
script+="
|
||||
jobsub -u root -q normal -c /bin/true"
|
||||
done
|
||||
script+="
|
||||
quit"
|
||||
out="$(printf '%s\n' "$script" | xtest)"
|
||||
got=0
|
||||
while IFS= read -r jid; do
|
||||
ALL_JOBIDS+=("$jid")
|
||||
acked=$((acked + 1)); got=$((got + 1))
|
||||
done < <(echo "$out" | sed -n 's/^jobsubmit ok: JobId: \(.*\)$/\1/p')
|
||||
[ "$acked" -ge "$want" ] && break
|
||||
# Shortfall: the primary likely just promoted and is regaining write-
|
||||
# quorum. Wait a moment and retry the remainder.
|
||||
log "submit attempt $attempt: $got/$need acked (total $acked/$want); waiting for write-quorum"
|
||||
sleep 3
|
||||
done
|
||||
log "submitted $acked/$want jobs"
|
||||
[ "$acked" -eq "$want" ]
|
||||
}
|
||||
|
||||
# Verify EVERY JobId in ALL_JOBIDS is present via expect-job on the live primary.
|
||||
# Judged by OUTPUT, not exit code (ASAN leaks pollute the exit code): require one
|
||||
# "expect-job: ok" per tracked job and zero "not found". Returns non-zero on any
|
||||
# missing job or short count.
|
||||
verify_no_loss() {
|
||||
local script out want oks
|
||||
want="${#ALL_JOBIDS[@]}"
|
||||
script="set masters $MASTERS
|
||||
set myname verify
|
||||
client
|
||||
wait-primary $WAITMS"
|
||||
local jid
|
||||
for jid in "${ALL_JOBIDS[@]}"; do
|
||||
script+="
|
||||
expect-job $jid"
|
||||
done
|
||||
script+="
|
||||
quit"
|
||||
out="$(printf '%s\n' "$script" | xtest)"
|
||||
if echo "$out" | grep -q "not found"; then
|
||||
echo "$out" | grep "not found" >&2
|
||||
return 1
|
||||
fi
|
||||
oks="$(echo "$out" | grep -c "expect-job: ok")"
|
||||
if [ "$oks" -ne "$want" ]; then
|
||||
echo "verify_no_loss: only $oks/$want jobs confirmed present" >&2
|
||||
echo "$out" | tail -8 >&2
|
||||
return 1
|
||||
fi
|
||||
log "verified all $want jobs present"
|
||||
return 0
|
||||
}
|
||||
|
||||
# Cross-check committed WAL prefixes across all live members are byte-identical
|
||||
# (term+crc). Non-fatal warn-only on the known term-identity caveat; data loss
|
||||
# (a member MISSING a committed entry) is the real failure surfaced by expect-job.
|
||||
walcheck() {
|
||||
local boundary="$1" d1 d2 d3 out
|
||||
d1="$TOP/controller/ctrl-1/data"
|
||||
d2="$TOP/controller/ctrl-2/data"
|
||||
d3="$TOP/controller/ctrl-3/data"
|
||||
out="$(printf 'walcompare --committed %s %s %s %s\nquit\n' \
|
||||
"$boundary" "$d1" "$d2" "$d3" | xtest)"
|
||||
if echo "$out" | grep -qiE "identical|OK"; then
|
||||
log "walcompare: committed prefix consistent (boundary=$boundary)"
|
||||
else
|
||||
log "walcompare note (boundary=$boundary): $(echo "$out" | tail -2 | tr '\n' ' ')"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Step 2: deploy ───────────────────────────────────────────────────────────
|
||||
log "deploying 3-node cluster to $TOP"
|
||||
cleanup
|
||||
# Safety guard: the deploy is destructive (rm -rf). Since the deploy dir can now
|
||||
# be any path the caller names, refuse to wipe a non-empty directory unless it
|
||||
# carries the sentinel from a prior xce-test-ha run — protects an arbitrary
|
||||
# user-named dir (e.g. a typo'd home/project path) from being nuked. An empty or
|
||||
# non-existent dir is always safe to (re)deploy into.
|
||||
if [ -d "$TOP" ] && [ -n "$(ls -A "$TOP" 2>/dev/null)" ] && [ ! -e "$TOP/$MARKER_NAME" ]; then
|
||||
fail "refusing to wipe non-empty $TOP — not a prior xce-test-ha deploy (no $MARKER_NAME marker). Choose an empty/new dir or remove it yourself."
|
||||
fi
|
||||
rm -rf "$TOP"
|
||||
mkdir -p "$TOP" || fail "could not create deploy dir $TOP"
|
||||
touch "$TOP/$MARKER_NAME" # claim ownership so a later run may safely re-wipe it
|
||||
bash "$DEPLOY" "$BIN" "$DATADIR" "$TOP" "localhost:9991,localhost:9992,localhost:9993" \
|
||||
>"$TOP/deploy.log" 2>&1 || fail "deploy-ha.sh failed (see $TOP/deploy.log)"
|
||||
|
||||
# ── Step 3: start all nodes ──────────────────────────────────────────────────
|
||||
log "starting ctrl-1 / ctrl-2 / ctrl-3 (-vvv)"
|
||||
for n in "${NODES[@]}"; do
|
||||
start_node "$n"
|
||||
sleep 0.5
|
||||
done
|
||||
|
||||
# ── Step 4: confirm a primary was elected ────────────────────────────────────
|
||||
log "waiting for primary election (cold start ~15s)"
|
||||
wait_for_primary_log || fail "no primary elected within 45s (check $TOP/logs)"
|
||||
PRIMARY="$(discover_primary)"
|
||||
[ -n "$PRIMARY" ] || fail "primary elected in logs but client could not reach it"
|
||||
log "primary is $PRIMARY"
|
||||
|
||||
# ── Step 5: initial submit + verify + waldump ────────────────────────────────
|
||||
submit_jobs "$NJOBS" || fail "initial job submission did not get $NJOBS clean acks"
|
||||
verify_no_loss || fail "initial joblist verification failed"
|
||||
walcheck "$((1 + 2 * ${#ALL_JOBIDS[@]}))"
|
||||
|
||||
# ── Steps 7–10 looped: kill primary, verify, submit, restart, reconverge ─────
|
||||
#
|
||||
# Each round kills the CURRENT primary, then restarts the node downed in the
|
||||
# previous round, then BLOCKS until the cluster is fully 3-of-3 converged before
|
||||
# the next round's kill. The convergence gate is essential: only SINGLE-failover
|
||||
# is a supported guarantee — killing a second primary while a just-restarted node
|
||||
# is still catching up is a double-fault-within-one-reconvergence window (a
|
||||
# momentary 2-of-3 with a stale member) that no Raft cluster guarantees against.
|
||||
DOWN="" # the node killed in this round, to be restarted+reconverged next round
|
||||
for round in $(seq 1 "$ROUNDS"); do
|
||||
log "=== round $round/$ROUNDS ==="
|
||||
|
||||
# Step 10 (from the previous round): restart the previously-downed node and
|
||||
# WAIT for the cluster to return to a fully-converged 3-of-3 before we kill
|
||||
# again. (Round 1 starts from the freshly-deployed converged cluster.)
|
||||
if [ -n "$DOWN" ]; then
|
||||
log "round $round: restarting previously-downed $DOWN, waiting for full convergence"
|
||||
start_node "$DOWN"
|
||||
wait_for_full_convergence \
|
||||
|| fail "round $round: cluster did not reconverge to 3-of-3 after restarting $DOWN"
|
||||
log "round $round: cluster reconverged (3-of-3)"
|
||||
fi
|
||||
|
||||
# Re-discover the live primary (it rotates as we kill nodes).
|
||||
PRIMARY="$(discover_primary)"
|
||||
[ -n "$PRIMARY" ] || fail "round $round: no serving primary before kill"
|
||||
log "round $round: current primary $PRIMARY"
|
||||
|
||||
# Record the election term BEFORE the kill so we can detect a STRICTLY-newer
|
||||
# CTRL_PRIMARY announcement (a genuine new election) afterwards.
|
||||
PREV_TERM="$(latest_primary_term)"
|
||||
PREV_TERM="${PREV_TERM:-0}"
|
||||
|
||||
# Step 6: kill the current primary. Match on the BINARY + --myname so the
|
||||
# pattern cannot also hit this driver's own shell snapshot (which carries the
|
||||
# "--myname ctrl-N" substring in its argv and would otherwise be matched by a
|
||||
# bare `pgrep -f -- "--myname ctrl-N"`, returning extra PIDs).
|
||||
PID="$(pgrep -f -- "xce_controller --myname $PRIMARY ")"
|
||||
[ -n "$PID" ] || fail "round $round: could not find pid for $PRIMARY"
|
||||
log "round $round: killing primary $PRIMARY (term=$PREV_TERM, pid=$(echo $PID | tr '\n' ' '))"
|
||||
kill -9 $PID 2>/dev/null
|
||||
DOWN="$PRIMARY"
|
||||
sleep 2
|
||||
|
||||
# Step 7: a NEW primary must be elected (CTRL_PRIMARY at a higher term in the
|
||||
# logs) AND be reachable by the client; then joblist must show no loss.
|
||||
log "round $round: waiting for new primary (term > $PREV_TERM)"
|
||||
NEWP="$(wait_for_new_primary "$PREV_TERM")"
|
||||
[ -n "$NEWP" ] || fail "round $round: no new primary elected after kill (no CTRL_PRIMARY term>$PREV_TERM)"
|
||||
log "round $round: new primary elected: $NEWP (term=$(latest_primary_term))"
|
||||
PRIMARY="$(discover_primary)"
|
||||
[ -n "$PRIMARY" ] || fail "round $round: new primary $NEWP elected but client cannot reach it (bug-854 class)"
|
||||
verify_no_loss || fail "round $round: a committed job was LOST after failover"
|
||||
|
||||
# Step 8: submit another batch on the new primary.
|
||||
submit_jobs "$NJOBS" || fail "round $round: post-failover submit did not get $NJOBS acks"
|
||||
|
||||
# Step 9: confirm still no loss (old + new jobs).
|
||||
verify_no_loss || fail "round $round: job loss after post-failover submit"
|
||||
walcheck "$((1 + 2 * ${#ALL_JOBIDS[@]}))"
|
||||
done
|
||||
|
||||
log "PASS: $ROUNDS rounds, ${#ALL_JOBIDS[@]} jobs submitted, zero job loss"
|
||||
cleanup
|
||||
trap - EXIT INT TERM
|
||||
exit 0
|
||||
Loading…
Add table
Add a link
Reference in a new issue