From 6067ba6c94adf56d9cfa6f0ccff23bec8733d902 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 23 Jun 2026 22:13:40 +0000 Subject: [PATCH] feat(xce-test-ha): support deploying HA cluster to a specific directory Add an optional [dir] argument to the /xce-test-ha skill so the 3-node soak cluster can be deployed to a caller-chosen path. - run-ha-soak.sh: deploy dir resolves positional $1 > $XCE_HA_TOP > default; relative paths absolutized in pure bash (this host's busybox realpath lacks -m and would mangle nonexistent/relative paths). - Fail-loud rm -rf guard: writes a .xce-ha-deploy sentinel at deploy time and refuses to wipe a non-empty dir lacking it (protects a typo'd home/source path); empty/nonexistent dirs stay safe to (re)deploy into. - SKILL.md: document /xce-test-ha [rounds] [dir] and the safety semantics. Verified: guard refuses an unmarked non-empty dir (dir survives, exit 1); full 5-round soak into a custom dir PASS (18 jobs, zero job loss), cluster landed in the named dir while the default dir was untouched. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V --- .claude/skills/xce-test-ha/SKILL.md | 189 +++++++++++ .claude/skills/xce-test-ha/run-ha-soak.sh | 389 ++++++++++++++++++++++ 2 files changed, 578 insertions(+) create mode 100644 .claude/skills/xce-test-ha/SKILL.md create mode 100755 .claude/skills/xce-test-ha/run-ha-soak.sh diff --git a/.claude/skills/xce-test-ha/SKILL.md b/.claude/skills/xce-test-ha/SKILL.md new file mode 100644 index 0000000..319b610 --- /dev/null +++ b/.claude/skills/xce-test-ha/SKILL.md @@ -0,0 +1,189 @@ +--- +name: xce-test-ha +description: "End-to-end HA failover soak test for an XCE controller cluster: deploy 3 nodes, submit jobs, then repeatedly kill the primary and verify zero committed-job loss across 5-10 rounds. Usage: /xce-test-ha [rounds] [dir]" +--- + +# XCE HA Failover Soak Test + +Deploys a 3-node XCE controller cluster, submits jobs, then drives repeated +primary kills + restarts, verifying after every failover that **no committed job +is ever lost**. PASS only if all rounds complete with zero loss. + +This is the live regression for the HA failover work. It deterministically +exercises the full single-failover lifecycle (kill → elect → discover → verify → +restart → reconverge) and has surfaced/validated these bugs: +- **bug-828** committed-job data safety, +- **bug-854** client primary discovery (incl. dead-first-master initial connect), +- **bug-1028** post-failover availability (streamer term-source), +- **bug-1042** standby-rejoin never converges (raft log_len not synced after recovery), +- **bug-1043** catch-up apply gap (replicated entries not applied to the job table). + +A green run is `PASS: N rounds, M jobs submitted, zero job loss` (exit 0). + +## Arguments + +- Optional `[rounds]` — number of kill/restart rounds (clamped to 5–10, default 6). +- Optional `[dir]` — deploy directory for the 3-node cluster + (default `/home/claude/tmp/xce-ha-deploy`). Maps to `XCE_HA_TOP` / + `run-ha-soak.sh`'s positional dir arg. Relative paths are resolved to + absolute. **Safety:** the driver `rm -rf`s this dir on each run, so it refuses + to wipe a non-empty directory that wasn't created by a prior `/xce-test-ha` + run (it looks for a `.xce-ha-deploy` marker) — point it at a new/empty/scratch + path, not a real source or home directory. + +``` +/xce-test-ha # 6 rounds, default deploy dir +/xce-test-ha 10 # 10 rounds, default deploy dir +/xce-test-ha 10 /tmp/my-ha-run # 10 rounds, deploy to /tmp/my-ha-run +``` + +## Step 1 — Ensure binaries are built + +The test needs `xce_controller` and `xce_test`: + +```bash +BIN=src/xce.build/controller/xce_controller +XT=src/xce.build/controller/xce_test +ls -lh "$BIN" "$XT" +``` + +If either is missing, build them first with the `/xce-build` skill: + +``` +/xce-build controller +``` + +(Do NOT proceed to the soak until both binaries exist.) + +## Step 2 — Run the soak driver + +The full workflow (deploy → start → elect → submit → kill-loop → verify) is +encoded in `run-ha-soak.sh` in this skill directory. It MUST run under the nix +dev shell so the controller/xce_test runtime libs (openssl, etc.) are on the +library path. Run it in the **background** (it takes several minutes) and tail +the log: + +```bash +SKILL_DIR=.claude/skills/xce-test-ha +nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" 2>&1 | tee /tmp/xce-test-ha.log +``` + +Pass the round count and other knobs via env. The deploy directory may be given +either as the script's first positional argument or via `XCE_HA_TOP` (positional +wins): + +```bash +# 10 rounds, default deploy dir +XCE_HA_ROUNDS=10 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" + +# 10 rounds, deploy to a specific directory (positional arg) +XCE_HA_ROUNDS=10 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" /tmp/my-ha-run + +# equivalent via env +XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" +``` + +| Env var | Meaning | Default | +|---|---|---| +| `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` | +| `XCE_HA_DATADIR` | config/certs source (`cluster.bin`, `xce.crt`, `xce.key`) | `/ha/data` | +| `XCE_HA_ROUNDS` | kill/restart rounds (5–10) | `6` | +| `XCE_HA_JOBS` | jobs per submit batch | `3` | +| `XCE_HA_WAITMS` | `wait-primary` budget per probe | `30000` | + +## What the driver does (maps to the requested steps) + +1. **Preconditions** — verifies `xce_controller`, `xce_test`, `cluster.bin`, and certs exist. +2. **Deploy** — `src/tools/deploy-ha.sh` into `XCE_HA_TOP` with nodes + `ctrl-1=localhost:9991, ctrl-2=localhost:9992, ctrl-3=localhost:9993`, + using config/certs from `XCE_HA_DATADIR`. +3. **Start** — `scripts/start.sh ctrl-N -vvv` for all three, 0.5s staggered. +4. **Elect** — waits (≤45s) for `raft_cb_become_leader … CTRL_PRIMARY` in any + `logs/ctrl-N.log`, then confirms the client can reach the primary via + `ctrlstatus`. +5. **Initial workload** — drives `xce_test`: + `set masters … / client / wait-primary / jobsub ×N / joblist (expect-job) / walcompare`. + Every acked `JobId` is tracked. +6–11. **Kill loop** (5–10 rounds): discover the live primary → `kill -9` it → + restart the node downed in the *previous* round (so the cluster returns to 3 + members) → wait for a new primary → `verify_no_loss` (expect-job every tracked + JobId) → submit another batch → `verify_no_loss` again → `walcompare`. +12. **Verdict** — exits `0` (PASS) only if every round verified zero job loss; + any missing job, failed election, or unreachable primary exits non-zero (FAIL). + +## Step 3 — Report the result + +- Exit `0` and a final `PASS: N rounds, M jobs submitted, zero job loss` line ⇒ **test PASSED**. +- Any `FAIL: …` line / non-zero exit ⇒ **test FAILED**. Surface the failing + round and reason; the per-node logs under `$XCE_HA_TOP/logs/` and the deploy + log `$XCE_HA_TOP/deploy.log` hold the detail. + +## Goal-driven loop integration (`/goal`) + +This skill is designed to be the **success oracle** for a goal-driven loop whose +goal is "the HA soak test passes". The driver is built for that contract: + +- **Single machine-checkable verdict.** Exactly one terminal line — either + `[xce-test-ha] PASS: …` (exit 0) or `[xce-test-ha] FAIL: …` (exit ≠ 0). A loop + decides done/continue by grepping that line / checking the exit code; no + human-style parsing of intermediate output is needed. +- **Fully idempotent & self-contained.** Every run `rm -rf`s `$XCE_HA_TOP`, + redeploys from `ha/data`, kills any stale `xce_controller`, and cleans up on + exit (trap). A loop can invoke it any number of times with no manual reset. +- **Deterministic-enough.** Cold-start election ~15s, each round ~20-30s; a + 5-round run is ~3-5 min, 10-round ~6-10 min. Budget the loop's per-iteration + timeout to `rounds × 60s + 60s`. + +**Loop shape** (the goal is reached when the driver exits 0): + +``` +goal: `/xce-test-ha` exits 0 with a "PASS:" line +each iteration: + 1. run the driver (background, capture log + exit code) + 2. if exit == 0 and log has "PASS:" → GOAL REACHED, stop + 3. else: read the FAIL line + $XCE_HA_TOP/logs to find the root cause + 4. fix the underlying bug in src/ (controller or libxce) + 5. /xce-build controller (rebuild the binaries the next run uses) + 6. repeat +``` + +The driver itself does NOT fix code — it is the *oracle*. The loop's job each +round is: run oracle → if FAIL, diagnose from the logs, patch `src/`, rebuild, +re-run. The goal is reached the first time the oracle returns PASS. + +## Job-loss model (why expect-job is the authority) + +`walcompare --committed` checks the committed WAL prefix is byte-identical +(term+crc) across members; it is logged as a note. The **authoritative** +loss check is `expect-job ` against the live primary for every job ever +acked — a committed job that an elected primary cannot list is a real loss +(bug-828 class), regardless of walcompare's term-mapping tolerance. + +The driver runs `xce_test` with `ASAN_OPTIONS=detect_leaks=0`: the debug/ASAN +build reports harmless heap leaks at exit (non-zero exit code) that are NOT job +loss. Verdicts are therefore judged on command **output** (`expect-job: ok` count +and `not found`), never the leak-polluted process exit code. + +## Manual driving (debug) + +To poke the cluster by hand after the driver deploys it (or instead of it): + +```bash +# discover primary + submit one job +printf 'set masters localhost:9991,localhost:9992,localhost:9993\nset myname m\nclient\nwait-primary 30000\nctrlstatus\njobsub -u root -q normal -c /bin/true\njoblist\nquit\n' \ + | nix develop ./src --command "$XT" + +# dump a node's WAL +printf 'waldump %s/controller/ctrl-1/data\nquit\n' "$XCE_HA_TOP" \ + | nix develop ./src --command "$XT" +``` + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `controller binary missing` | not built | `/xce-build controller` | +| `no primary elected within 45s` | cold-start election stalled / port clash | check `logs/ctrl-*.log`; ensure ports 9991-9993 free | +| `primary elected in logs but client could not reach it` | client discovery (bug-854 regression) | check `client.c` rotation fix is in the binary | +| `a committed job was LOST after failover` | data-safety regression (bug-828 class) | inspect WAL via `waldump`; this is a real FAIL | +| driver hangs | a controller wedged; `xtest` 60s timeout should break it | check for stuck `xce_test`/`xce_controller` procs | diff --git a/.claude/skills/xce-test-ha/run-ha-soak.sh b/.claude/skills/xce-test-ha/run-ha-soak.sh new file mode 100755 index 0000000..d379774 --- /dev/null +++ b/.claude/skills/xce-test-ha/run-ha-soak.sh @@ -0,0 +1,389 @@ +#!/usr/bin/env bash +# +# run-ha-soak.sh — driver for /xce-test-ha: deploy a 3-node XCE HA cluster, then +# repeatedly kill the primary, verify no committed job is ever lost, restart the +# downed node, and loop. +# +# Exit 0 == PASS (no job loss across all rounds); non-zero == FAIL. +# +# Must be invoked under `nix develop ./src --command bash ...` so the controller +# and xce_test runtime libs (openssl etc.) are on the library path. +# +# Usage: +# run-ha-soak.sh [DEPLOY_DIR] +# DEPLOY_DIR optional deploy directory (positional). Precedence: +# positional arg > $XCE_HA_TOP env > default. Relative paths +# are resolved to absolute. +# +# Env overrides: +# XCE_HA_TOP deploy dir (default /home/claude/tmp/xce-ha-deploy) +# XCE_HA_DATADIR config/certs source (default /ha/data) +# XCE_HA_ROUNDS kill/restart rounds (default 6, clamp 5..10) +# XCE_HA_JOBS jobs per submit batch (default 3) +# XCE_HA_WAITMS wait-primary budget (default 30000) +set -u + +# ── Paths ──────────────────────────────────────────────────────────────────── +REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +# Deploy dir: explicit positional arg wins, then $XCE_HA_TOP, then the default. +TOP="${1:-${XCE_HA_TOP:-/home/claude/tmp/xce-ha-deploy}}" +# Make a relative dir absolute (against cwd) so it is unambiguous to deploy-ha.sh +# and the per-node scripts. Pure-bash — no realpath dependency (busybox realpath +# here lacks -m and would mangle nonexistent/relative paths). +case "$TOP" in + /*) : ;; # already absolute + *) TOP="$PWD/$TOP" ;; # resolve relative path against current directory +esac +# Sentinel written into a deploy we own; the guard before `rm -rf` only wipes a +# non-empty dir that carries this marker (see "Step 2: deploy" below). +MARKER_NAME=".xce-ha-deploy" +DATADIR="${XCE_HA_DATADIR:-$REPO/ha/data}" +BIN="$REPO/src/xce.build/controller/xce_controller" +XT="$REPO/src/xce.build/controller/xce_test" +DEPLOY="$REPO/src/tools/deploy-ha.sh" + +ROUNDS="${XCE_HA_ROUNDS:-6}" +[ "$ROUNDS" -lt 5 ] && ROUNDS=5 +[ "$ROUNDS" -gt 10 ] && ROUNDS=10 +NJOBS="${XCE_HA_JOBS:-3}" +WAITMS="${XCE_HA_WAITMS:-30000}" + +NODES=(ctrl-1 ctrl-2 ctrl-3) +declare -A PORT=( [ctrl-1]=9991 [ctrl-2]=9992 [ctrl-3]=9993 ) +PEERS="ctrl-1=localhost:9991,ctrl-2=localhost:9992,ctrl-3=localhost:9993" +MASTERS="localhost:9991,localhost:9992,localhost:9993" + +ALL_JOBIDS=() # every JobId we have ever successfully submitted + +log() { echo "[xce-test-ha] $*"; } +fail() { echo "[xce-test-ha] FAIL: $*" >&2; cleanup; exit 1; } + +cleanup() { pkill -9 -x xce_controller 2>/dev/null; sleep 0.3; } + +# ── Preconditions ──────────────────────────────────────────────────────────── +[ -x "$BIN" ] || { echo "[xce-test-ha] FAIL: controller binary missing: $BIN (run /xce-build controller)" >&2; exit 2; } +[ -x "$XT" ] || { echo "[xce-test-ha] FAIL: xce_test missing: $XT (run /xce-build controller)" >&2; exit 2; } +[ -r "$DATADIR/cluster.bin" ] || { echo "[xce-test-ha] FAIL: cluster.bin missing under $DATADIR" >&2; exit 2; } +[ -r "$DATADIR/xce.crt" ] && [ -r "$DATADIR/xce.key" ] || { echo "[xce-test-ha] FAIL: certs missing under $DATADIR" >&2; exit 2; } + +trap cleanup EXIT INT TERM + +# ── Helpers ────────────────────────────────────────────────────────────────── + +# Run a batch of xce_test commands (stdin script) with a hard timeout. +# Disable ASAN leak detection: the debug/ASAN xce_test build reports harmless +# heap leaks at exit (non-zero exit code) that would otherwise be misread as a +# command failure. Job-loss is judged by the command OUTPUT (expect-job "not +# found"), not the leak-polluted process exit code. +xtest() { ASAN_OPTIONS=detect_leaks=0 timeout 60 "$XT" 2>&1; } + +start_node() { bash "$TOP/scripts/start.sh" "$1" -vvv >/dev/null 2>&1 & } + +# The INFO-level promotion marker each elected primary logs (ha.c: raft_cb_ +# become_leader → "HA FSM (raft): → CTRL_PRIMARY (term=N, ...)"). Present even +# without -vvv. We track election by this line + its term, which is the most +# reliable "a new primary was elected" signal (independent of client-side +# discovery). NOTE: the arrow is a UTF-8 "→"; grep with a wildcard to avoid +# encoding pitfalls. +PRIMARY_LOG_RE='HA FSM .raft.: . CTRL_PRIMARY .term=' + +# Highest term any node has announced CTRL_PRIMARY for, across all logs. +latest_primary_term() { + grep -ohE "$PRIMARY_LOG_RE[0-9]+" "$TOP"/logs/*.log 2>/dev/null \ + | grep -oE 'term=[0-9]+' | grep -oE '[0-9]+' | sort -n | tail -1 +} + +# The node name that most recently logged CTRL_PRIMARY (the elected primary). +# Echoes "ctrl-N" or "" — purely from the logs, no client round-trip. +primary_from_log() { + local n best_term="" best_node="" t + for n in "${NODES[@]}"; do + t="$(grep -oE "$PRIMARY_LOG_RE[0-9]+" "$TOP/logs/$n.log" 2>/dev/null \ + | grep -oE 'term=[0-9]+' | grep -oE '[0-9]+' | sort -n | tail -1)" + [ -z "$t" ] && continue + if [ -z "$best_term" ] || [ "$t" -ge "$best_term" ]; then + best_term="$t"; best_node="$n" + fi + done + echo "$best_node" +} + +# Block until a node announces CTRL_PRIMARY at a term STRICTLY GREATER than +# `$1` (the term before the kill), i.e. a genuinely NEW election has happened. +# Echoes the new primary's node name on success; returns non-zero on timeout. +wait_for_new_primary() { + local prev_term="${1:-0}" i t node + for i in $(seq 1 45); do + t="$(latest_primary_term)" + if [ -n "$t" ] && [ "$t" -gt "$prev_term" ]; then + node="$(primary_from_log)" + [ -n "$node" ] && { echo "$node"; return 0; } + fi + sleep 1 + done + return 1 +} + +# Wait until SOME node logs CTRL_PRIMARY (cold-start election), up to ~45s. +wait_for_primary_log() { + local i + for i in $(seq 1 45); do + [ -n "$(latest_primary_term)" ] && return 0 + sleep 1 + done + return 1 +} + +# Identify the current primary by asking the cluster (authoritative ctrlstatus, +# which internally rotates to the serving primary). Echoes "ctrl-N" or "". +# Retries the whole wait-primary+ctrlstatus probe for up to ~60s so it spans a +# full failover election (liveness timeout ~10s + PreVote stagger + election). +# This is the SERVING check (the client can actually reach the primary), distinct +# from the log-based election check above. +discover_primary() { + local out name i + for i in $(seq 1 6); do + out="$(printf 'set masters %s\nset myname probe\nclient\nwait-primary %s\nctrlstatus\nquit\n' \ + "$MASTERS" "$WAITMS" | xtest)" + name="$(echo "$out" | sed -n 's/^Primary: \(ctrl-[0-9]\)$/\1/p' | head -1)" + if [ -n "$name" ]; then + echo "$name" + return 0 + fi + sleep 2 + done + return 1 +} + +# Wait until the cluster is FULLY converged: all 3 controllers are alive AND the +# current primary is no longer actively streaming catch-up to any standby. This +# is a HARD precondition before killing the next primary — killing while a just- +# restarted node is still catching up tests an UNSUPPORTED double-fault-within- +# one-reconvergence window (a momentary 2-of-3 with a stale member), not the +# single-failover guarantee. +# +# Convergence signal: a lagging standby makes the primary log "stream WAL batches" +# every heartbeat (~3s); once every standby is caught up the primary STOPS +# emitting that line (action becomes HA_REPL_CONVERGED, a no-op). So "no new +# 'stream WAL batches' line on the primary for STABLE_S seconds" == all standbys +# converged. This is robust to the per-term re-emit quirk of the "standby[i] +# converged" transition log (which only fires for a standby that was previously +# NOT converged under the current primary term). Bounded ~90s. +wait_for_full_convergence() { + local i alive primary line_now line_prev stable=0 + local STABLE_TICKS=3 # 3 ticks × 2s = 6s with no new stream line ⇒ converged + line_prev="" + for i in $(seq 1 45); do + alive="$(pgrep -c -f -- "xce_controller --myname ctrl-")" + if [ "$alive" -lt 3 ]; then + stable=0; sleep 2; continue + fi + primary="$(primary_from_log)" + [ -z "$primary" ] && { sleep 2; continue; } + # Most recent "stream WAL batches" line in the primary's log (empty if + # the primary has never streamed / has stopped). Compare across ticks: + # if it has not changed for STABLE_TICKS, no catch-up is in flight. + line_now="$(grep -n "stream WAL batches" "$TOP/logs/$primary.log" 2>/dev/null | tail -1)" + if [ "$line_now" = "$line_prev" ]; then + stable=$((stable + 1)) + [ "$stable" -ge "$STABLE_TICKS" ] && return 0 + else + stable=0 + line_prev="$line_now" + fi + sleep 2 + done + return 1 +} + +# Submit jobs through the live primary until N are cleanly acked, appending each +# acked JobId to ALL_JOBIDS. A FRESHLY-promoted primary is briefly not write-able +# (it appends its own-term NOOP but cannot commit until a standby reconnects and +# write-quorum is restored — log: "HA quorum lost/restored"), so a submit issued +# in that ~1-3s window fails its 500ms commit WAIT. That is NOT a job loss (the +# job was never acked, so it is correctly not tracked) — it is a transient that +# clears once quorum returns. Retry the SHORTFALL up to a few times. +submit_jobs() { + local want="$1" attempt acked=0 got script out jid + for attempt in 1 2 3 4 5; do + local need=$((want - acked)) + [ "$need" -le 0 ] && break + script="set masters $MASTERS +set myname soak +client +wait-primary $WAITMS" + local i + for i in $(seq 1 "$need"); do + script+=" +jobsub -u root -q normal -c /bin/true" + done + script+=" +quit" + out="$(printf '%s\n' "$script" | xtest)" + got=0 + while IFS= read -r jid; do + ALL_JOBIDS+=("$jid") + acked=$((acked + 1)); got=$((got + 1)) + done < <(echo "$out" | sed -n 's/^jobsubmit ok: JobId: \(.*\)$/\1/p') + [ "$acked" -ge "$want" ] && break + # Shortfall: the primary likely just promoted and is regaining write- + # quorum. Wait a moment and retry the remainder. + log "submit attempt $attempt: $got/$need acked (total $acked/$want); waiting for write-quorum" + sleep 3 + done + log "submitted $acked/$want jobs" + [ "$acked" -eq "$want" ] +} + +# Verify EVERY JobId in ALL_JOBIDS is present via expect-job on the live primary. +# Judged by OUTPUT, not exit code (ASAN leaks pollute the exit code): require one +# "expect-job: ok" per tracked job and zero "not found". Returns non-zero on any +# missing job or short count. +verify_no_loss() { + local script out want oks + want="${#ALL_JOBIDS[@]}" + script="set masters $MASTERS +set myname verify +client +wait-primary $WAITMS" + local jid + for jid in "${ALL_JOBIDS[@]}"; do + script+=" +expect-job $jid" + done + script+=" +quit" + out="$(printf '%s\n' "$script" | xtest)" + if echo "$out" | grep -q "not found"; then + echo "$out" | grep "not found" >&2 + return 1 + fi + oks="$(echo "$out" | grep -c "expect-job: ok")" + if [ "$oks" -ne "$want" ]; then + echo "verify_no_loss: only $oks/$want jobs confirmed present" >&2 + echo "$out" | tail -8 >&2 + return 1 + fi + log "verified all $want jobs present" + return 0 +} + +# Cross-check committed WAL prefixes across all live members are byte-identical +# (term+crc). Non-fatal warn-only on the known term-identity caveat; data loss +# (a member MISSING a committed entry) is the real failure surfaced by expect-job. +walcheck() { + local boundary="$1" d1 d2 d3 out + d1="$TOP/controller/ctrl-1/data" + d2="$TOP/controller/ctrl-2/data" + d3="$TOP/controller/ctrl-3/data" + out="$(printf 'walcompare --committed %s %s %s %s\nquit\n' \ + "$boundary" "$d1" "$d2" "$d3" | xtest)" + if echo "$out" | grep -qiE "identical|OK"; then + log "walcompare: committed prefix consistent (boundary=$boundary)" + else + log "walcompare note (boundary=$boundary): $(echo "$out" | tail -2 | tr '\n' ' ')" + fi +} + +# ── Step 2: deploy ─────────────────────────────────────────────────────────── +log "deploying 3-node cluster to $TOP" +cleanup +# Safety guard: the deploy is destructive (rm -rf). Since the deploy dir can now +# be any path the caller names, refuse to wipe a non-empty directory unless it +# carries the sentinel from a prior xce-test-ha run — protects an arbitrary +# user-named dir (e.g. a typo'd home/project path) from being nuked. An empty or +# non-existent dir is always safe to (re)deploy into. +if [ -d "$TOP" ] && [ -n "$(ls -A "$TOP" 2>/dev/null)" ] && [ ! -e "$TOP/$MARKER_NAME" ]; then + fail "refusing to wipe non-empty $TOP — not a prior xce-test-ha deploy (no $MARKER_NAME marker). Choose an empty/new dir or remove it yourself." +fi +rm -rf "$TOP" +mkdir -p "$TOP" || fail "could not create deploy dir $TOP" +touch "$TOP/$MARKER_NAME" # claim ownership so a later run may safely re-wipe it +bash "$DEPLOY" "$BIN" "$DATADIR" "$TOP" "localhost:9991,localhost:9992,localhost:9993" \ + >"$TOP/deploy.log" 2>&1 || fail "deploy-ha.sh failed (see $TOP/deploy.log)" + +# ── Step 3: start all nodes ────────────────────────────────────────────────── +log "starting ctrl-1 / ctrl-2 / ctrl-3 (-vvv)" +for n in "${NODES[@]}"; do + start_node "$n" + sleep 0.5 +done + +# ── Step 4: confirm a primary was elected ──────────────────────────────────── +log "waiting for primary election (cold start ~15s)" +wait_for_primary_log || fail "no primary elected within 45s (check $TOP/logs)" +PRIMARY="$(discover_primary)" +[ -n "$PRIMARY" ] || fail "primary elected in logs but client could not reach it" +log "primary is $PRIMARY" + +# ── Step 5: initial submit + verify + waldump ──────────────────────────────── +submit_jobs "$NJOBS" || fail "initial job submission did not get $NJOBS clean acks" +verify_no_loss || fail "initial joblist verification failed" +walcheck "$((1 + 2 * ${#ALL_JOBIDS[@]}))" + +# ── Steps 7–10 looped: kill primary, verify, submit, restart, reconverge ───── +# +# Each round kills the CURRENT primary, then restarts the node downed in the +# previous round, then BLOCKS until the cluster is fully 3-of-3 converged before +# the next round's kill. The convergence gate is essential: only SINGLE-failover +# is a supported guarantee — killing a second primary while a just-restarted node +# is still catching up is a double-fault-within-one-reconvergence window (a +# momentary 2-of-3 with a stale member) that no Raft cluster guarantees against. +DOWN="" # the node killed in this round, to be restarted+reconverged next round +for round in $(seq 1 "$ROUNDS"); do + log "=== round $round/$ROUNDS ===" + + # Step 10 (from the previous round): restart the previously-downed node and + # WAIT for the cluster to return to a fully-converged 3-of-3 before we kill + # again. (Round 1 starts from the freshly-deployed converged cluster.) + if [ -n "$DOWN" ]; then + log "round $round: restarting previously-downed $DOWN, waiting for full convergence" + start_node "$DOWN" + wait_for_full_convergence \ + || fail "round $round: cluster did not reconverge to 3-of-3 after restarting $DOWN" + log "round $round: cluster reconverged (3-of-3)" + fi + + # Re-discover the live primary (it rotates as we kill nodes). + PRIMARY="$(discover_primary)" + [ -n "$PRIMARY" ] || fail "round $round: no serving primary before kill" + log "round $round: current primary $PRIMARY" + + # Record the election term BEFORE the kill so we can detect a STRICTLY-newer + # CTRL_PRIMARY announcement (a genuine new election) afterwards. + PREV_TERM="$(latest_primary_term)" + PREV_TERM="${PREV_TERM:-0}" + + # Step 6: kill the current primary. Match on the BINARY + --myname so the + # pattern cannot also hit this driver's own shell snapshot (which carries the + # "--myname ctrl-N" substring in its argv and would otherwise be matched by a + # bare `pgrep -f -- "--myname ctrl-N"`, returning extra PIDs). + PID="$(pgrep -f -- "xce_controller --myname $PRIMARY ")" + [ -n "$PID" ] || fail "round $round: could not find pid for $PRIMARY" + log "round $round: killing primary $PRIMARY (term=$PREV_TERM, pid=$(echo $PID | tr '\n' ' '))" + kill -9 $PID 2>/dev/null + DOWN="$PRIMARY" + sleep 2 + + # Step 7: a NEW primary must be elected (CTRL_PRIMARY at a higher term in the + # logs) AND be reachable by the client; then joblist must show no loss. + log "round $round: waiting for new primary (term > $PREV_TERM)" + NEWP="$(wait_for_new_primary "$PREV_TERM")" + [ -n "$NEWP" ] || fail "round $round: no new primary elected after kill (no CTRL_PRIMARY term>$PREV_TERM)" + log "round $round: new primary elected: $NEWP (term=$(latest_primary_term))" + PRIMARY="$(discover_primary)" + [ -n "$PRIMARY" ] || fail "round $round: new primary $NEWP elected but client cannot reach it (bug-854 class)" + verify_no_loss || fail "round $round: a committed job was LOST after failover" + + # Step 8: submit another batch on the new primary. + submit_jobs "$NJOBS" || fail "round $round: post-failover submit did not get $NJOBS acks" + + # Step 9: confirm still no loss (old + new jobs). + verify_no_loss || fail "round $round: job loss after post-failover submit" + walcheck "$((1 + 2 * ${#ALL_JOBIDS[@]}))" +done + +log "PASS: $ROUNDS rounds, ${#ALL_JOBIDS[@]} jobs submitted, zero job loss" +cleanup +trap - EXIT INT TERM +exit 0