forked from lei.chen/myxce
test(raft-ha-core-safety): chaotic soak mode + isolated verdict (task 1.2)
Opt-in overlapping-kill mode (kills inside election windows). Verdict counts committed walcompare divergence + committed-job loss; bug-854 (client discovery) and bug-924 (overlayfs fdatasync) counted separately and excluded from PASS/FAIL. 17/17 verdict self-test GREEN. Live smoke corroborated chaotic regime + live bug-854 isolation.
This commit is contained in:
parent
b028d25dbd
commit
02c0fcb6aa
5 changed files with 537 additions and 2 deletions
|
|
@ -31,6 +31,74 @@ A green run is `PASS: N rounds, M jobs submitted, zero job loss` (exit 0).
|
|||
run (it looks for a `.xce-ha-deploy` marker) — point it at a new/empty/scratch
|
||||
path, not a real source or home directory.
|
||||
|
||||
## Modes: sequential (default) vs chaotic
|
||||
|
||||
The driver has two kill schedules, selected by a flag (or `XCE_HA_MODE`):
|
||||
|
||||
- **`--sequential` (default)** — supported single-failover lifecycle: kill the
|
||||
primary → wait for a new primary → restart the downed node → **wait for full
|
||||
3-of-3 reconvergence** before the next kill. Any committed-job loss or failed
|
||||
election is a hard `FAIL` (exit ≠ 0). Backward-compatible; the leadership-
|
||||
coverage maneuver (all three nodes must lead) runs only in this mode.
|
||||
- **`--chaotic`** — **overlapping** kills that land *inside* election / catch-up
|
||||
windows: the previously-downed node is restarted but the driver does **not**
|
||||
wait for it to reconverge before killing the next primary. This deliberately
|
||||
exercises the unsupported double-fault-within-one-reconvergence window that
|
||||
surfaces committed-prefix divergence (bug-851 family). Quorum is preserved —
|
||||
only the single current primary is killed per round and the prior node is
|
||||
restarted, so at most one node is down at a kill instant.
|
||||
|
||||
```bash
|
||||
SKILL_DIR=.claude/skills/xce-test-ha
|
||||
# chaotic soak, 6 rounds, scratch dir
|
||||
XCE_HA_ROUNDS=6 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --chaotic /tmp/ha-chaos
|
||||
# equivalent via env
|
||||
XCE_HA_MODE=chaotic XCE_HA_ROUNDS=6 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" /tmp/ha-chaos
|
||||
```
|
||||
|
||||
For the **acceptance gate (task 7.2)** run chaotic mode for ≥6 rounds against
|
||||
**real-fs (ext4) release binaries** (point `XCE_HA_BIN` / `XCE_HA_XT` at the
|
||||
release `xce_controller` / `xce_test`).
|
||||
|
||||
### Chaotic verdict (bug-854 / bug-924 isolated)
|
||||
|
||||
Chaotic mode does not hard-fail on data outcomes; it **counts** them and emits a
|
||||
single machine-parseable verdict line (`soak-verdict.sh`):
|
||||
|
||||
```
|
||||
[xce-test-ha] VERDICT mode=chaotic rounds=6 jobs=18 committed_divergence=0 committed_job_loss=0 excluded_bug854_discovery=2 excluded_bug924_fdatasync=0 result=PASS
|
||||
```
|
||||
|
||||
- **`committed_divergence`** — committed-prefix `walcompare` divergences, counted
|
||||
from task 1.1's `WALCMP_DIVERGENCE … class=committed` machine line
|
||||
(`walcompare --report`). Uncommitted-tail divergences are non-fatal and not
|
||||
counted.
|
||||
- **`committed_job_loss`** — a tracked, acked job reported `not found` on a
|
||||
**reachable** primary (a real server-side loss, bug-828 class).
|
||||
- **`excluded_bug854_discovery`** — the client could not LOCATE the primary
|
||||
(libxce primary-discovery fault). A discovery miss, **not** a server losing
|
||||
committed state — reported but **excluded** from the verdict.
|
||||
- **`excluded_bug924_fdatasync`** — overlayfs `fdatasync FAILED (errno=5/EIO)`
|
||||
ERROR lines in the controller logs (environment defect). Reported but
|
||||
**excluded**; the real-fs acceptance run does not hit it.
|
||||
|
||||
`result=PASS` iff `committed_divergence == 0` **and** `committed_job_loss == 0`,
|
||||
regardless of the excluded bug-854/bug-924 counts. The driver also prints the
|
||||
usual terminal `PASS:` / `FAIL:` oracle line (exit 0 / non-zero) so a goal-loop
|
||||
can branch on it unchanged.
|
||||
|
||||
### Verdict self-test (no cluster)
|
||||
|
||||
The verdict parser has a deterministic, pure-bash self-test that feeds fixture
|
||||
output (a committed divergence, a bug-854 discovery failure, and bug-924
|
||||
`fdatasync` lines) and asserts the divergence/job-loss counts are unaffected by
|
||||
bug-854/bug-924. It needs no binaries and runs in under a second:
|
||||
|
||||
```bash
|
||||
bash .claude/skills/xce-test-ha/run-ha-soak.sh --selftest # or:
|
||||
bash .claude/skills/xce-test-ha/soak-verdict-selftest.sh
|
||||
```
|
||||
|
||||
```
|
||||
/xce-test-ha # 6 rounds, default deploy dir
|
||||
/xce-test-ha 10 # 10 rounds, default deploy dir
|
||||
|
|
@ -87,6 +155,9 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
|---|---|---|
|
||||
| `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
|
||||
| `XCE_HA_DATADIR` | config/certs source (`cluster.bin`, `xce.crt`, `xce.key`) | `<repo>/ha/data` |
|
||||
| `XCE_HA_MODE` | `sequential` or `chaotic` (`--chaotic` flag wins) | `sequential` |
|
||||
| `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
|
||||
| `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
|
||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–10) | `6` |
|
||||
| `XCE_HA_JOBS` | jobs per submit batch | `3` |
|
||||
| `XCE_HA_WAITMS` | `wait-primary` budget per probe | `30000` |
|
||||
|
|
|
|||
|
|
@ -10,19 +10,50 @@
|
|||
# and xce_test runtime libs (openssl etc.) are on the library path.
|
||||
#
|
||||
# Usage:
|
||||
# run-ha-soak.sh [DEPLOY_DIR]
|
||||
# run-ha-soak.sh [--chaotic|--sequential] [--selftest] [DEPLOY_DIR]
|
||||
# DEPLOY_DIR optional deploy directory (positional). Precedence:
|
||||
# positional arg > $XCE_HA_TOP env > default. Relative paths
|
||||
# are resolved to absolute.
|
||||
# --chaotic overlapping-kill mode: kills land INSIDE election/catch-up
|
||||
# windows (it does NOT wait for full 3-of-3 reconvergence before
|
||||
# the next kill). Outcomes are COUNTED into a verdict, not hard-
|
||||
# failed; bug-854 (client discovery) and bug-924 (overlayfs
|
||||
# fdatasync) are isolated from the divergence/job-loss result.
|
||||
# --sequential default single-failover mode (kill → wait-new-primary →
|
||||
# restart → reconverge). Backward-compatible.
|
||||
# --selftest run the verdict-parser self-test (soak-verdict-selftest.sh)
|
||||
# and exit — no cluster, no binaries needed.
|
||||
#
|
||||
# Env overrides:
|
||||
# XCE_HA_TOP deploy dir (default /home/claude/tmp/xce-ha-deploy)
|
||||
# XCE_HA_DATADIR config/certs source (default <repo>/ha/data)
|
||||
# XCE_HA_MODE sequential | chaotic (default sequential; --chaotic wins)
|
||||
# XCE_HA_ROUNDS kill/restart rounds (default 6, clamp 5..10)
|
||||
# XCE_HA_JOBS jobs per submit batch (default 3)
|
||||
# XCE_HA_WAITMS wait-primary budget (default 30000)
|
||||
set -u
|
||||
|
||||
# ── Mode / flag parsing (flags may precede the positional DEPLOY_DIR) ─────────
|
||||
# `--selftest` short-circuits to the verdict self-test before any cluster setup.
|
||||
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
MODE="${XCE_HA_MODE:-sequential}"
|
||||
ARGS=()
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--chaotic) MODE="chaotic" ;;
|
||||
--sequential) MODE="sequential" ;;
|
||||
--selftest) exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;;
|
||||
*) ARGS+=("$1") ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
set -- ${ARGS[@]+"${ARGS[@]}"}
|
||||
|
||||
# Verdict helpers (pure parsers; see soak-verdict.sh). Sourced so the chaotic
|
||||
# loop can count committed divergence / job-loss with bug-854/bug-924 isolated.
|
||||
# shellcheck source=soak-verdict.sh
|
||||
. "$SELF_DIR/soak-verdict.sh"
|
||||
|
||||
# ── Paths ────────────────────────────────────────────────────────────────────
|
||||
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
|
||||
# Deploy dir: explicit positional arg wins, then $XCE_HA_TOP, then the default.
|
||||
|
|
@ -56,6 +87,14 @@ MASTERS="localhost:9991,localhost:9992,localhost:9993"
|
|||
ALL_JOBIDS=() # every JobId we have ever successfully submitted
|
||||
declare -A SEEN_PRIMARY # node name -> 1 once it has served as primary
|
||||
|
||||
# ── Chaotic-mode verdict counters (see soak-verdict.sh) ──────────────────────
|
||||
# committed_divergence and committed_job_loss are the REAL server-side verdict;
|
||||
# bug854/bug924 are EXCLUDED (counted separately, never affect PASS/FAIL).
|
||||
V_DIVERGENCE=0 # committed `walcompare` divergences observed across the run
|
||||
V_JOBLOSS=0 # tracked committed jobs "not found" on a REACHABLE primary
|
||||
V_BUG854=0 # client could not locate the primary (libxce discovery fault)
|
||||
V_BUG924=0 # overlayfs `fdatasync FAILED` log lines (environment defect)
|
||||
|
||||
log() { echo "[xce-test-ha] $*"; }
|
||||
fail() { echo "[xce-test-ha] FAIL: $*" >&2; cleanup; exit 1; }
|
||||
|
||||
|
|
@ -300,6 +339,153 @@ walcheck() {
|
|||
fi
|
||||
}
|
||||
|
||||
# ── Chaotic-mode primitives (count outcomes into the verdict, never hard-fail) ─
|
||||
|
||||
# Run `walcompare --report` across all three members and ADD any committed-class
|
||||
# divergence to V_DIVERGENCE (task 1.1's WALCMP_DIVERGENCE class=committed line).
|
||||
# Non-fatal: a divergence is COUNTED, not a hard exit — the verdict decides PASS.
|
||||
walcheck_count() {
|
||||
local boundary="$1" d1 d2 d3 out n
|
||||
d1="$TOP/controller/ctrl-1/data"
|
||||
d2="$TOP/controller/ctrl-2/data"
|
||||
d3="$TOP/controller/ctrl-3/data"
|
||||
out="$(printf 'walcompare --report --committed %s %s %s %s\nquit\n' \
|
||||
"$boundary" "$d1" "$d2" "$d3" | xtest)"
|
||||
n="$(printf '%s' "$out" | verdict_count_committed_divergence)"
|
||||
if [ "${n:-0}" -gt 0 ]; then
|
||||
V_DIVERGENCE=$((V_DIVERGENCE + n))
|
||||
log "walcompare: COMMITTED DIVERGENCE x$n (boundary=$boundary) — COUNTED"
|
||||
printf '%s\n' "$out" | grep 'WALCMP_DIVERGENCE' >&2
|
||||
else
|
||||
log "walcompare: committed prefix consistent (boundary=$boundary)"
|
||||
fi
|
||||
}
|
||||
|
||||
# Chaotic-mode verify: discover the serving primary (best effort), then classify
|
||||
# the expect-job result via the verdict parser. Increments V_JOBLOSS on a REAL
|
||||
# loss (reachable primary, tracked job "not found") and V_BUG854 when the client
|
||||
# cannot locate the primary (discovery fault — EXCLUDED). Never hard-fails.
|
||||
verify_classify() {
|
||||
local primary out cls want="${#ALL_JOBIDS[@]}"
|
||||
[ "$want" -eq 0 ] && return 0
|
||||
primary="$(discover_primary)"
|
||||
if [ -z "$primary" ]; then
|
||||
V_BUG854=$((V_BUG854 + 1))
|
||||
log "verify: client could not locate a primary (bug-854 class — EXCLUDED)"
|
||||
return 0
|
||||
fi
|
||||
local script jid
|
||||
script="set masters $MASTERS
|
||||
set myname verify
|
||||
client
|
||||
wait-primary $WAITMS"
|
||||
for jid in "${ALL_JOBIDS[@]}"; do
|
||||
script+="
|
||||
expect-job $jid"
|
||||
done
|
||||
script+="
|
||||
quit"
|
||||
out="$(printf '%s\n' "$script" | xtest)"
|
||||
cls="$(printf '%s' "$out" | verdict_classify_verify 1)"
|
||||
case "$cls" in
|
||||
LOSS)
|
||||
V_JOBLOSS=$((V_JOBLOSS + 1))
|
||||
log "verify: COMMITTED JOB LOSS on reachable primary $primary — COUNTED"
|
||||
printf '%s\n' "$out" | grep 'not found' >&2
|
||||
;;
|
||||
BUG854)
|
||||
V_BUG854=$((V_BUG854 + 1))
|
||||
log "verify: primary vanished mid-verify (bug-854 class — EXCLUDED)"
|
||||
;;
|
||||
OK)
|
||||
log "verify: all $want tracked jobs present on $primary"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Tally bug-924 overlayfs `fdatasync FAILED` ERROR lines across all node logs
|
||||
# into V_BUG924 (EXCLUDED from the divergence/job-loss verdict).
|
||||
scan_bug924() {
|
||||
V_BUG924="$(cat "$TOP"/logs/*.log 2>/dev/null | verdict_count_fdatasync_bug924)"
|
||||
[ "${V_BUG924:-0}" -gt 0 ] && \
|
||||
log "bug-924: $V_BUG924 overlayfs fdatasync errors in logs — EXCLUDED from verdict"
|
||||
return 0
|
||||
}
|
||||
|
||||
# Chaotic / overlapping-kill loop. Unlike the sequential loop it does NOT wait
|
||||
# for full 3-of-3 reconvergence before the next kill: it restarts the previously-
|
||||
# downed node and proceeds to kill the NEW primary as soon as one is ELECTED
|
||||
# (log-based), so kills land inside election / catch-up windows (overlapping
|
||||
# failovers — the unsupported double-fault window this spec must prove safe).
|
||||
# Quorum is preserved: only the single current primary is killed per round and
|
||||
# the prior node is restarted, so at most one node is down at a kill instant.
|
||||
# Every data outcome is COUNTED into the verdict, never hard-failed.
|
||||
run_chaotic_rounds() {
|
||||
local round PRIMARY PREV_TERM NEWP pid DOWN=""
|
||||
for round in $(seq 1 "$ROUNDS"); do
|
||||
log "=== chaotic round $round/$ROUNDS ==="
|
||||
|
||||
# Overlap: restart the previously-downed node but DO NOT wait for it to
|
||||
# converge — the next kill may land while it is still catching up.
|
||||
if [ -n "$DOWN" ]; then
|
||||
log "round $round: restarting $DOWN (no convergence wait — overlapping)"
|
||||
start_node "$DOWN"
|
||||
fi
|
||||
|
||||
# Identify the primary from the LOGS (server truth), independent of the
|
||||
# client: a discovery miss here is bug-854, not a server fault.
|
||||
PRIMARY="$(primary_from_log)"
|
||||
if [ -z "$PRIMARY" ]; then
|
||||
wait_for_primary_log || log "round $round: no primary elected yet (election lag/stall)"
|
||||
PRIMARY="$(primary_from_log)"
|
||||
fi
|
||||
if [ -z "$PRIMARY" ]; then
|
||||
log "round $round: no primary to kill this round — skipping"
|
||||
continue
|
||||
fi
|
||||
record_primary "$PRIMARY"
|
||||
PREV_TERM="$(latest_primary_term)"; PREV_TERM="${PREV_TERM:-0}"
|
||||
|
||||
# Best-effort submit so there is fresh committed state to (potentially)
|
||||
# lose. A freshly-promoted primary may briefly lack write-quorum; a
|
||||
# shortfall is NOT a loss (the job was never acked, so never tracked).
|
||||
submit_jobs "$NJOBS" || log "round $round: submit shortfall (transient — not a loss)"
|
||||
|
||||
# Kill the current primary, landing inside the next election window.
|
||||
pid="$(pgrep -f -- "xce_controller --myname $PRIMARY ")"
|
||||
if [ -n "$pid" ]; then
|
||||
log "round $round: killing primary $PRIMARY (term=$PREV_TERM, pid=$(echo $pid | tr '\n' ' '))"
|
||||
kill -9 $pid 2>/dev/null
|
||||
DOWN="$PRIMARY"
|
||||
else
|
||||
log "round $round: primary $PRIMARY already gone"
|
||||
fi
|
||||
sleep 1
|
||||
|
||||
# Wait ONLY for a new primary to be ELECTED (not full reconvergence).
|
||||
NEWP="$(wait_for_new_primary "$PREV_TERM")"
|
||||
if [ -n "$NEWP" ]; then
|
||||
log "round $round: new primary elected: $NEWP (term=$(latest_primary_term))"
|
||||
record_primary "$NEWP"
|
||||
else
|
||||
log "round $round: NO new primary within window (possible election stall)"
|
||||
fi
|
||||
|
||||
# Count loss/divergence for this round (verdict, not a hard fail).
|
||||
verify_classify
|
||||
walcheck_count "$((1 + 2 * ${#ALL_JOBIDS[@]}))"
|
||||
done
|
||||
|
||||
# Restore 3-of-3 and let it settle so the FINAL sweep sees every member's
|
||||
# committed prefix.
|
||||
if [ -n "$DOWN" ]; then
|
||||
log "chaotic: restarting last-downed $DOWN and settling before final verdict"
|
||||
start_node "$DOWN"
|
||||
wait_for_full_convergence \
|
||||
|| log "chaotic: did not fully reconverge before verdict (any residue is counted via walcompare)"
|
||||
fi
|
||||
}
|
||||
|
||||
# Drive a SPECIFIC node to primary — used to exercise leadership on the
|
||||
# lowest-priority node (ctrl-3), which the natural single-failover loop NEVER
|
||||
# elects: the election timeout is priority-staggered (my_priority * step_ms, p0
|
||||
|
|
@ -397,6 +583,34 @@ submit_jobs "$NJOBS" || fail "initial job submission did not get $NJOBS clean ac
|
|||
verify_no_loss || fail "initial joblist verification failed"
|
||||
walcheck "$((1 + 2 * ${#ALL_JOBIDS[@]}))"
|
||||
|
||||
# ── Chaotic mode: overlapping-kill rounds → isolated verdict ──────────────────
|
||||
# Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up
|
||||
# windows; outcomes are COUNTED (committed divergence + committed job loss) with
|
||||
# bug-854 (client discovery) and bug-924 (overlayfs fdatasync) ISOLATED. Emits a
|
||||
# single machine-parseable VERDICT line plus the PASS/FAIL oracle line, then
|
||||
# exits — it does NOT run the sequential loop or leadership-coverage maneuver.
|
||||
if [ "$MODE" = "chaotic" ]; then
|
||||
log "MODE=chaotic — overlapping-kill soak ($ROUNDS rounds)"
|
||||
run_chaotic_rounds
|
||||
# Final divergence + loss sweep on the settled cluster, then bug-924 tally.
|
||||
walcheck_count "$((1 + 2 * ${#ALL_JOBIDS[@]}))"
|
||||
verify_classify
|
||||
scan_bug924
|
||||
verdict_emit "chaotic" "$ROUNDS" "${#ALL_JOBIDS[@]}" \
|
||||
"$V_DIVERGENCE" "$V_JOBLOSS" "$V_BUG854" "$V_BUG924"
|
||||
VERDICT_RC=$?
|
||||
if [ "$VERDICT_RC" -eq 0 ]; then
|
||||
log "PASS (chaotic): $ROUNDS rounds, ${#ALL_JOBIDS[@]} jobs, zero committed divergence, zero committed job loss (bug854=$V_BUG854 bug924=$V_BUG924 excluded)"
|
||||
cleanup
|
||||
trap - EXIT INT TERM
|
||||
exit 0
|
||||
fi
|
||||
echo "[xce-test-ha] FAIL: chaotic verdict — committed_divergence=$V_DIVERGENCE committed_job_loss=$V_JOBLOSS (bug854=$V_BUG854 bug924=$V_BUG924 excluded)" >&2
|
||||
cleanup
|
||||
trap - EXIT INT TERM
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ── Steps 7–10 looped: kill primary, verify, submit, restart, reconverge ─────
|
||||
#
|
||||
# Each round kills the CURRENT primary, then restarts the node downed in the
|
||||
|
|
|
|||
159
.claude/skills/xce-test-ha/soak-verdict-selftest.sh
Normal file
159
.claude/skills/xce-test-ha/soak-verdict-selftest.sh
Normal file
|
|
@ -0,0 +1,159 @@
|
|||
#!/usr/bin/env bash
|
||||
#
|
||||
# soak-verdict-selftest.sh — deterministic unit test for the verdict parser in
|
||||
# soak-verdict.sh. Feeds FIXTURE output (a committed `walcompare` divergence, an
|
||||
# uncommitted-tail divergence, a bug-854 client-discovery failure, and bug-924
|
||||
# `fdatasync FAILED` log lines) and asserts the verdict:
|
||||
# - counts the committed divergence and committed job-loss (real failures), and
|
||||
# - EXCLUDES bug-854 (client cannot locate the primary) and bug-924 (overlayfs
|
||||
# fdatasync) from the divergence / job-loss result.
|
||||
#
|
||||
# No cluster, no binaries — pure bash, runs anywhere in well under a second. This
|
||||
# is the RED→GREEN proof for the verdict logic (task 1.2): the full live chaotic
|
||||
# soak is the separate acceptance gate (task 7.2).
|
||||
#
|
||||
# Usage: bash soak-verdict-selftest.sh (also reachable as run-ha-soak.sh --selftest)
|
||||
set -u
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=soak-verdict.sh
|
||||
. "$HERE/soak-verdict.sh" || {
|
||||
echo "SELFTEST FAIL: cannot source soak-verdict.sh (verdict library missing)" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
fails=0
|
||||
check() { # desc expected actual
|
||||
if [ "$2" = "$3" ]; then
|
||||
echo "ok - $1"
|
||||
else
|
||||
echo "FAIL - $1: expected [$2] got [$3]"
|
||||
fails=$((fails + 1))
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Fixtures (captured/synthetic soak output) ────────────────────────────────
|
||||
|
||||
# A real committed-prefix divergence: member 1 holds a different (term, pcrc) at
|
||||
# committed LSN 9 (the bug-851 family signature). Task 1.1's machine line marks
|
||||
# it class=committed → a FATAL, COUNTED divergence.
|
||||
WALCMP_COMMITTED_DIV="$(cat <<'EOF'
|
||||
== walcompare ==
|
||||
WALCMP member=0 lsn=9 term=3 pcrc=deadbeef status=present committed=1
|
||||
WALCMP member=1 lsn=9 term=4 pcrc=cafebabe status=present committed=1
|
||||
WALCMP member=2 lsn=9 term=3 pcrc=00000000 status=MISSING committed=1
|
||||
WALCMP_DIVERGENCE lsn=9 class=committed compared=3 present=2
|
||||
*** DIVERGENCE at committed LSN 9 (compared 3, present 2) ***
|
||||
walcompare: FAIL — committed-prefix divergence at LSN 9
|
||||
EOF
|
||||
)"
|
||||
|
||||
# An uncommitted-TAIL divergence (above the committed boundary). NON-fatal — it
|
||||
# must NOT be counted toward the committed-divergence verdict.
|
||||
WALCMP_UNCOMMITTED_DIV="$(cat <<'EOF'
|
||||
WALCMP member=0 lsn=11 term=5 pcrc=11111111 status=present committed=0
|
||||
WALCMP member=1 lsn=11 term=6 pcrc=22222222 status=present committed=0
|
||||
WALCMP_DIVERGENCE lsn=11 class=uncommitted compared=2 present=2
|
||||
walcompare: OK
|
||||
EOF
|
||||
)"
|
||||
|
||||
# A clean committed prefix (no divergence).
|
||||
WALCMP_CLEAN="$(cat <<'EOF'
|
||||
WALCMP member=0 lsn=3 term=2 pcrc=000000ab status=present committed=1
|
||||
WALCMP member=1 lsn=3 term=2 pcrc=000000ab status=present committed=1
|
||||
walcompare: OK
|
||||
EOF
|
||||
)"
|
||||
|
||||
# bug-924: overlayfs fdatasync EIO. Two ERROR lines from persist.c.
|
||||
FDATASYNC_LOG="$(cat <<'EOF'
|
||||
[INFO] persist: appended lsn=7
|
||||
[ERROR] persist: flusher fdatasync FAILED (errno=5 Input/output error) — durable_lsn held
|
||||
[ERROR] persist: persist_flush fdatasync FAILED (errno=5 Input/output error) — durable_lsn
|
||||
EOF
|
||||
)"
|
||||
|
||||
# A clean controller log (no fdatasync errors).
|
||||
CLEAN_LOG="$(cat <<'EOF'
|
||||
[INFO] HA FSM (raft): → CTRL_PRIMARY (term=4, ...)
|
||||
[INFO] persist: appended lsn=9 durable
|
||||
EOF
|
||||
)"
|
||||
|
||||
# expect-job output: one tracked job is genuinely missing on a reachable primary.
|
||||
EXPECT_LOSS="$(cat <<'EOF'
|
||||
expect-job: ok (job-1)
|
||||
expect-job: job job-2 not found
|
||||
EOF
|
||||
)"
|
||||
|
||||
# expect-job output: every tracked job present.
|
||||
EXPECT_OK="$(cat <<'EOF'
|
||||
expect-job: ok (job-1)
|
||||
expect-job: ok (job-2)
|
||||
EOF
|
||||
)"
|
||||
|
||||
# The client connected but the primary vanished mid-command → no expect-job
|
||||
# results at all (a discovery race — bug-854 class, EXCLUDED, not a loss).
|
||||
EXPECT_VANISHED="$(cat <<'EOF'
|
||||
wait-primary: 30000 ms elapsed, no primary
|
||||
EOF
|
||||
)"
|
||||
|
||||
# ── Assertions ───────────────────────────────────────────────────────────────
|
||||
|
||||
check "committed divergence is counted" \
|
||||
1 "$(printf '%s' "$WALCMP_COMMITTED_DIV" | verdict_count_committed_divergence)"
|
||||
check "uncommitted-tail divergence is EXCLUDED" \
|
||||
0 "$(printf '%s' "$WALCMP_UNCOMMITTED_DIV" | verdict_count_committed_divergence)"
|
||||
check "clean walcompare => 0 committed divergence" \
|
||||
0 "$(printf '%s' "$WALCMP_CLEAN" | verdict_count_committed_divergence)"
|
||||
|
||||
check "bug-924 fdatasync errors are counted" \
|
||||
2 "$(printf '%s' "$FDATASYNC_LOG" | verdict_count_fdatasync_bug924)"
|
||||
check "clean log => 0 bug-924 errors" \
|
||||
0 "$(printf '%s' "$CLEAN_LOG" | verdict_count_fdatasync_bug924)"
|
||||
|
||||
check "reachable + not-found => LOSS (real)" \
|
||||
LOSS "$(printf '%s' "$EXPECT_LOSS" | verdict_classify_verify 1)"
|
||||
check "unreachable => BUG854 (excluded, not loss)" \
|
||||
BUG854 "$(printf '%s' "$EXPECT_LOSS" | verdict_classify_verify 0)"
|
||||
check "reachable + all present => OK" \
|
||||
OK "$(printf '%s' "$EXPECT_OK" | verdict_classify_verify 1)"
|
||||
check "reachable but no expect-job results => BUG854 (mid-command discovery race)" \
|
||||
BUG854 "$(printf '%s' "$EXPECT_VANISHED" | verdict_classify_verify 1)"
|
||||
|
||||
# Verdict-level: a committed divergence is FAIL even amid bug-854/bug-924 noise.
|
||||
v="$(verdict_emit chaotic 6 18 1 0 3 2)"; rc=$?
|
||||
check "committed divergence => result=FAIL token" \
|
||||
"FAIL" "$(printf '%s' "$v" | grep -oE 'result=[A-Z]+' | cut -d= -f2)"
|
||||
check "committed divergence => non-zero rc" 1 "$rc"
|
||||
|
||||
# THE KEY ASSERTION: with zero real divergence and zero real loss, the verdict is
|
||||
# PASS even though bug-854 (3) and bug-924 (2) both fired — i.e. the verdict's
|
||||
# divergence/job-loss result is UNAFFECTED by bug-854/bug-924 (R8.4).
|
||||
v="$(verdict_emit chaotic 6 18 0 0 3 2)"; rc=$?
|
||||
check "div=0 loss=0 with bug854+bug924 noise => result=PASS token" \
|
||||
"PASS" "$(printf '%s' "$v" | grep -oE 'result=[A-Z]+' | cut -d= -f2)"
|
||||
check "div=0 loss=0 with bug854+bug924 noise => zero rc" 0 "$rc"
|
||||
check "verdict line carries the excluded bug854 count" \
|
||||
"excluded_bug854_discovery=3" "$(printf '%s' "$v" | grep -oE 'excluded_bug854_discovery=[0-9]+')"
|
||||
check "verdict line carries the excluded bug924 count" \
|
||||
"excluded_bug924_fdatasync=2" "$(printf '%s' "$v" | grep -oE 'excluded_bug924_fdatasync=[0-9]+')"
|
||||
|
||||
# A real committed job-loss is FAIL regardless of how few excluded faults fired.
|
||||
v="$(verdict_emit chaotic 6 18 0 2 0 0)"; rc=$?
|
||||
check "committed job loss => non-zero rc" 1 "$rc"
|
||||
check "committed job loss => result=FAIL token" \
|
||||
"FAIL" "$(printf '%s' "$v" | grep -oE 'result=[A-Z]+' | cut -d= -f2)"
|
||||
|
||||
# ── Verdict ──────────────────────────────────────────────────────────────────
|
||||
echo
|
||||
if [ "$fails" -eq 0 ]; then
|
||||
echo "SELFTEST PASS"
|
||||
exit 0
|
||||
fi
|
||||
echo "SELFTEST FAIL ($fails failed)"
|
||||
exit 1
|
||||
91
.claude/skills/xce-test-ha/soak-verdict.sh
Normal file
91
.claude/skills/xce-test-ha/soak-verdict.sh
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
#!/usr/bin/env bash
|
||||
#
|
||||
# soak-verdict.sh — pure (side-effect-free) verdict helpers for run-ha-soak.sh.
|
||||
#
|
||||
# These functions PARSE captured output and emit counts / a classification; they
|
||||
# never touch the cluster, so they are unit-testable against fixture text
|
||||
# (soak-verdict-selftest.sh). The chaotic soak sources this file and uses the
|
||||
# helpers to build a machine-parseable verdict whose committed-divergence and
|
||||
# committed-job-loss counts are UNAFFECTED by two known, out-of-scope defects:
|
||||
#
|
||||
# - bug-854 (libxce client primary-discovery): the client cannot LOCATE the
|
||||
# serving primary. This is a discovery fault, NOT a server losing committed
|
||||
# state — so a "could not reach the primary" outcome is counted separately
|
||||
# and EXCLUDED from the job-loss result.
|
||||
# - bug-924 (overlayfs `fdatasync FAILED (errno=5/EIO)`): an environment defect
|
||||
# of the sandbox filesystem, not the controller. Its log lines are counted
|
||||
# separately and EXCLUDED from the divergence/job-loss result (the acceptance
|
||||
# soak, task 7.2, runs on a real fs where bug-924 does not occur).
|
||||
#
|
||||
# Source this file; do not execute it directly. The terminal verdict is emitted
|
||||
# by verdict_emit().
|
||||
|
||||
# Count committed-class `walcompare` divergences in the text on stdin.
|
||||
# Consumes task 1.1's machine line:
|
||||
# WALCMP_DIVERGENCE lsn=<L> class=committed compared=<c> present=<p>
|
||||
# Only class=committed lines are counted; an uncommitted-tail divergence is
|
||||
# non-fatal (above the committed boundary) and is deliberately NOT counted.
|
||||
verdict_count_committed_divergence() {
|
||||
local n
|
||||
n="$(grep -c -E '^WALCMP_DIVERGENCE .*class=committed' || true)"
|
||||
echo "${n:-0}"
|
||||
}
|
||||
|
||||
# Count bug-924 overlayfs `fdatasync FAILED` ERROR lines in the log text on
|
||||
# stdin (persist.c emits "persist: ... fdatasync FAILED (errno=...)"). Reported
|
||||
# as an EXCLUDED count — it never feeds the divergence/job-loss verdict.
|
||||
verdict_count_fdatasync_bug924() {
|
||||
local n
|
||||
n="$(grep -c -E 'fdatasync FAILED' || true)"
|
||||
echo "${n:-0}"
|
||||
}
|
||||
|
||||
# Classify one verify (expect-job) attempt. Reads the expect-job command OUTPUT
|
||||
# on stdin; $1 = primary_reachable (1 iff client-side discovery already located a
|
||||
# serving primary). Echoes exactly one token:
|
||||
# BUG854 — the client could not reach a primary, OR the primary vanished
|
||||
# mid-command so NO expect-job result was produced (discovery fault,
|
||||
# EXCLUDED — never counted as committed-job loss)
|
||||
# LOSS — the primary was reachable but a tracked committed job is "not found"
|
||||
# (a REAL server-side committed-job loss, bug-828 class)
|
||||
# OK — the primary was reachable and every tracked job is present
|
||||
# This is the seam that separates "the server lost a committed job" (real) from
|
||||
# "the client could not locate the primary" (bug-854, excluded).
|
||||
verdict_classify_verify() {
|
||||
local reachable="$1" out
|
||||
out="$(cat)"
|
||||
if [ "$reachable" != "1" ]; then
|
||||
echo BUG854
|
||||
return 0
|
||||
fi
|
||||
# Reachable per discovery, but if the verify command yielded NO expect-job
|
||||
# results at all, the client lost the primary mid-command (a discovery race)
|
||||
# — classify as bug-854, not a loss.
|
||||
if ! printf '%s' "$out" | grep -q "expect-job:"; then
|
||||
echo BUG854
|
||||
return 0
|
||||
fi
|
||||
if printf '%s' "$out" | grep -q "not found"; then
|
||||
echo LOSS
|
||||
return 0
|
||||
fi
|
||||
echo OK
|
||||
}
|
||||
|
||||
# Emit the single machine-parseable verdict line. PASS iff committed_divergence
|
||||
# == 0 AND committed_job_loss == 0; the excluded bug-854/bug-924 counts are
|
||||
# REPORTED for visibility but do NOT affect the pass/fail decision. Returns 0 on
|
||||
# PASS, 1 on FAIL.
|
||||
# Args: mode rounds jobs committed_divergence committed_job_loss bug854 bug924
|
||||
verdict_emit() {
|
||||
local mode="$1" rounds="$2" jobs="$3" div="$4" loss="$5" b854="$6" b924="$7"
|
||||
local result="PASS"
|
||||
if [ "$div" -ne 0 ] || [ "$loss" -ne 0 ]; then
|
||||
result="FAIL"
|
||||
fi
|
||||
echo "[xce-test-ha] VERDICT mode=$mode rounds=$rounds jobs=$jobs" \
|
||||
"committed_divergence=$div committed_job_loss=$loss" \
|
||||
"excluded_bug854_discovery=$b854 excluded_bug924_fdatasync=$b924" \
|
||||
"result=$result"
|
||||
[ "$result" = "PASS" ]
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue