test(raft-ha-core-safety): chaotic soak mode + isolated verdict (task 1.2)
Opt-in overlapping-kill mode (kills inside election windows). Verdict counts committed walcompare divergence + committed-job loss; bug-854 (client discovery) and bug-924 (overlayfs fdatasync) counted separately and excluded from PASS/FAIL. 17/17 verdict self-test GREEN. Live smoke corroborated chaotic regime + live bug-854 isolation.
This commit is contained in:
parent
b028d25dbd
commit
02c0fcb6aa
5 changed files with 537 additions and 2 deletions
|
|
@ -31,6 +31,74 @@ A green run is `PASS: N rounds, M jobs submitted, zero job loss` (exit 0).
|
||||||
run (it looks for a `.xce-ha-deploy` marker) — point it at a new/empty/scratch
|
run (it looks for a `.xce-ha-deploy` marker) — point it at a new/empty/scratch
|
||||||
path, not a real source or home directory.
|
path, not a real source or home directory.
|
||||||
|
|
||||||
|
## Modes: sequential (default) vs chaotic
|
||||||
|
|
||||||
|
The driver has two kill schedules, selected by a flag (or `XCE_HA_MODE`):
|
||||||
|
|
||||||
|
- **`--sequential` (default)** — supported single-failover lifecycle: kill the
|
||||||
|
primary → wait for a new primary → restart the downed node → **wait for full
|
||||||
|
3-of-3 reconvergence** before the next kill. Any committed-job loss or failed
|
||||||
|
election is a hard `FAIL` (exit ≠ 0). Backward-compatible; the leadership-
|
||||||
|
coverage maneuver (all three nodes must lead) runs only in this mode.
|
||||||
|
- **`--chaotic`** — **overlapping** kills that land *inside* election / catch-up
|
||||||
|
windows: the previously-downed node is restarted but the driver does **not**
|
||||||
|
wait for it to reconverge before killing the next primary. This deliberately
|
||||||
|
exercises the unsupported double-fault-within-one-reconvergence window that
|
||||||
|
surfaces committed-prefix divergence (bug-851 family). Quorum is preserved —
|
||||||
|
only the single current primary is killed per round and the prior node is
|
||||||
|
restarted, so at most one node is down at a kill instant.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
SKILL_DIR=.claude/skills/xce-test-ha
|
||||||
|
# chaotic soak, 6 rounds, scratch dir
|
||||||
|
XCE_HA_ROUNDS=6 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --chaotic /tmp/ha-chaos
|
||||||
|
# equivalent via env
|
||||||
|
XCE_HA_MODE=chaotic XCE_HA_ROUNDS=6 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" /tmp/ha-chaos
|
||||||
|
```
|
||||||
|
|
||||||
|
For the **acceptance gate (task 7.2)** run chaotic mode for ≥6 rounds against
|
||||||
|
**real-fs (ext4) release binaries** (point `XCE_HA_BIN` / `XCE_HA_XT` at the
|
||||||
|
release `xce_controller` / `xce_test`).
|
||||||
|
|
||||||
|
### Chaotic verdict (bug-854 / bug-924 isolated)
|
||||||
|
|
||||||
|
Chaotic mode does not hard-fail on data outcomes; it **counts** them and emits a
|
||||||
|
single machine-parseable verdict line (`soak-verdict.sh`):
|
||||||
|
|
||||||
|
```
|
||||||
|
[xce-test-ha] VERDICT mode=chaotic rounds=6 jobs=18 committed_divergence=0 committed_job_loss=0 excluded_bug854_discovery=2 excluded_bug924_fdatasync=0 result=PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
- **`committed_divergence`** — committed-prefix `walcompare` divergences, counted
|
||||||
|
from task 1.1's `WALCMP_DIVERGENCE … class=committed` machine line
|
||||||
|
(`walcompare --report`). Uncommitted-tail divergences are non-fatal and not
|
||||||
|
counted.
|
||||||
|
- **`committed_job_loss`** — a tracked, acked job reported `not found` on a
|
||||||
|
**reachable** primary (a real server-side loss, bug-828 class).
|
||||||
|
- **`excluded_bug854_discovery`** — the client could not LOCATE the primary
|
||||||
|
(libxce primary-discovery fault). A discovery miss, **not** a server losing
|
||||||
|
committed state — reported but **excluded** from the verdict.
|
||||||
|
- **`excluded_bug924_fdatasync`** — overlayfs `fdatasync FAILED (errno=5/EIO)`
|
||||||
|
ERROR lines in the controller logs (environment defect). Reported but
|
||||||
|
**excluded**; the real-fs acceptance run does not hit it.
|
||||||
|
|
||||||
|
`result=PASS` iff `committed_divergence == 0` **and** `committed_job_loss == 0`,
|
||||||
|
regardless of the excluded bug-854/bug-924 counts. The driver also prints the
|
||||||
|
usual terminal `PASS:` / `FAIL:` oracle line (exit 0 / non-zero) so a goal-loop
|
||||||
|
can branch on it unchanged.
|
||||||
|
|
||||||
|
### Verdict self-test (no cluster)
|
||||||
|
|
||||||
|
The verdict parser has a deterministic, pure-bash self-test that feeds fixture
|
||||||
|
output (a committed divergence, a bug-854 discovery failure, and bug-924
|
||||||
|
`fdatasync` lines) and asserts the divergence/job-loss counts are unaffected by
|
||||||
|
bug-854/bug-924. It needs no binaries and runs in under a second:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash .claude/skills/xce-test-ha/run-ha-soak.sh --selftest # or:
|
||||||
|
bash .claude/skills/xce-test-ha/soak-verdict-selftest.sh
|
||||||
|
```
|
||||||
|
|
||||||
```
|
```
|
||||||
/xce-test-ha # 6 rounds, default deploy dir
|
/xce-test-ha # 6 rounds, default deploy dir
|
||||||
/xce-test-ha 10 # 10 rounds, default deploy dir
|
/xce-test-ha 10 # 10 rounds, default deploy dir
|
||||||
|
|
@ -87,6 +155,9 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
|
| `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
|
||||||
| `XCE_HA_DATADIR` | config/certs source (`cluster.bin`, `xce.crt`, `xce.key`) | `<repo>/ha/data` |
|
| `XCE_HA_DATADIR` | config/certs source (`cluster.bin`, `xce.crt`, `xce.key`) | `<repo>/ha/data` |
|
||||||
|
| `XCE_HA_MODE` | `sequential` or `chaotic` (`--chaotic` flag wins) | `sequential` |
|
||||||
|
| `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
|
||||||
|
| `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
|
||||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–10) | `6` |
|
| `XCE_HA_ROUNDS` | kill/restart rounds (5–10) | `6` |
|
||||||
| `XCE_HA_JOBS` | jobs per submit batch | `3` |
|
| `XCE_HA_JOBS` | jobs per submit batch | `3` |
|
||||||
| `XCE_HA_WAITMS` | `wait-primary` budget per probe | `30000` |
|
| `XCE_HA_WAITMS` | `wait-primary` budget per probe | `30000` |
|
||||||
|
|
|
||||||
|
|
@ -10,19 +10,50 @@
|
||||||
# and xce_test runtime libs (openssl etc.) are on the library path.
|
# and xce_test runtime libs (openssl etc.) are on the library path.
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# run-ha-soak.sh [DEPLOY_DIR]
|
# run-ha-soak.sh [--chaotic|--sequential] [--selftest] [DEPLOY_DIR]
|
||||||
# DEPLOY_DIR optional deploy directory (positional). Precedence:
|
# DEPLOY_DIR optional deploy directory (positional). Precedence:
|
||||||
# positional arg > $XCE_HA_TOP env > default. Relative paths
|
# positional arg > $XCE_HA_TOP env > default. Relative paths
|
||||||
# are resolved to absolute.
|
# are resolved to absolute.
|
||||||
|
# --chaotic overlapping-kill mode: kills land INSIDE election/catch-up
|
||||||
|
# windows (it does NOT wait for full 3-of-3 reconvergence before
|
||||||
|
# the next kill). Outcomes are COUNTED into a verdict, not hard-
|
||||||
|
# failed; bug-854 (client discovery) and bug-924 (overlayfs
|
||||||
|
# fdatasync) are isolated from the divergence/job-loss result.
|
||||||
|
# --sequential default single-failover mode (kill → wait-new-primary →
|
||||||
|
# restart → reconverge). Backward-compatible.
|
||||||
|
# --selftest run the verdict-parser self-test (soak-verdict-selftest.sh)
|
||||||
|
# and exit — no cluster, no binaries needed.
|
||||||
#
|
#
|
||||||
# Env overrides:
|
# Env overrides:
|
||||||
# XCE_HA_TOP deploy dir (default /home/claude/tmp/xce-ha-deploy)
|
# XCE_HA_TOP deploy dir (default /home/claude/tmp/xce-ha-deploy)
|
||||||
# XCE_HA_DATADIR config/certs source (default <repo>/ha/data)
|
# XCE_HA_DATADIR config/certs source (default <repo>/ha/data)
|
||||||
|
# XCE_HA_MODE sequential | chaotic (default sequential; --chaotic wins)
|
||||||
# XCE_HA_ROUNDS kill/restart rounds (default 6, clamp 5..10)
|
# XCE_HA_ROUNDS kill/restart rounds (default 6, clamp 5..10)
|
||||||
# XCE_HA_JOBS jobs per submit batch (default 3)
|
# XCE_HA_JOBS jobs per submit batch (default 3)
|
||||||
# XCE_HA_WAITMS wait-primary budget (default 30000)
|
# XCE_HA_WAITMS wait-primary budget (default 30000)
|
||||||
set -u
|
set -u
|
||||||
|
|
||||||
|
# ── Mode / flag parsing (flags may precede the positional DEPLOY_DIR) ─────────
|
||||||
|
# `--selftest` short-circuits to the verdict self-test before any cluster setup.
|
||||||
|
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
MODE="${XCE_HA_MODE:-sequential}"
|
||||||
|
ARGS=()
|
||||||
|
while [ $# -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
--chaotic) MODE="chaotic" ;;
|
||||||
|
--sequential) MODE="sequential" ;;
|
||||||
|
--selftest) exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;;
|
||||||
|
*) ARGS+=("$1") ;;
|
||||||
|
esac
|
||||||
|
shift
|
||||||
|
done
|
||||||
|
set -- ${ARGS[@]+"${ARGS[@]}"}
|
||||||
|
|
||||||
|
# Verdict helpers (pure parsers; see soak-verdict.sh). Sourced so the chaotic
|
||||||
|
# loop can count committed divergence / job-loss with bug-854/bug-924 isolated.
|
||||||
|
# shellcheck source=soak-verdict.sh
|
||||||
|
. "$SELF_DIR/soak-verdict.sh"
|
||||||
|
|
||||||
# ── Paths ────────────────────────────────────────────────────────────────────
|
# ── Paths ────────────────────────────────────────────────────────────────────
|
||||||
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
|
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
|
||||||
# Deploy dir: explicit positional arg wins, then $XCE_HA_TOP, then the default.
|
# Deploy dir: explicit positional arg wins, then $XCE_HA_TOP, then the default.
|
||||||
|
|
@ -56,6 +87,14 @@ MASTERS="localhost:9991,localhost:9992,localhost:9993"
|
||||||
ALL_JOBIDS=() # every JobId we have ever successfully submitted
|
ALL_JOBIDS=() # every JobId we have ever successfully submitted
|
||||||
declare -A SEEN_PRIMARY # node name -> 1 once it has served as primary
|
declare -A SEEN_PRIMARY # node name -> 1 once it has served as primary
|
||||||
|
|
||||||
|
# ── Chaotic-mode verdict counters (see soak-verdict.sh) ──────────────────────
|
||||||
|
# committed_divergence and committed_job_loss are the REAL server-side verdict;
|
||||||
|
# bug854/bug924 are EXCLUDED (counted separately, never affect PASS/FAIL).
|
||||||
|
V_DIVERGENCE=0 # committed `walcompare` divergences observed across the run
|
||||||
|
V_JOBLOSS=0 # tracked committed jobs "not found" on a REACHABLE primary
|
||||||
|
V_BUG854=0 # client could not locate the primary (libxce discovery fault)
|
||||||
|
V_BUG924=0 # overlayfs `fdatasync FAILED` log lines (environment defect)
|
||||||
|
|
||||||
log() { echo "[xce-test-ha] $*"; }
|
log() { echo "[xce-test-ha] $*"; }
|
||||||
fail() { echo "[xce-test-ha] FAIL: $*" >&2; cleanup; exit 1; }
|
fail() { echo "[xce-test-ha] FAIL: $*" >&2; cleanup; exit 1; }
|
||||||
|
|
||||||
|
|
@ -300,6 +339,153 @@ walcheck() {
|
||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# ── Chaotic-mode primitives (count outcomes into the verdict, never hard-fail) ─
|
||||||
|
|
||||||
|
# Run `walcompare --report` across all three members and ADD any committed-class
|
||||||
|
# divergence to V_DIVERGENCE (task 1.1's WALCMP_DIVERGENCE class=committed line).
|
||||||
|
# Non-fatal: a divergence is COUNTED, not a hard exit — the verdict decides PASS.
|
||||||
|
walcheck_count() {
|
||||||
|
local boundary="$1" d1 d2 d3 out n
|
||||||
|
d1="$TOP/controller/ctrl-1/data"
|
||||||
|
d2="$TOP/controller/ctrl-2/data"
|
||||||
|
d3="$TOP/controller/ctrl-3/data"
|
||||||
|
out="$(printf 'walcompare --report --committed %s %s %s %s\nquit\n' \
|
||||||
|
"$boundary" "$d1" "$d2" "$d3" | xtest)"
|
||||||
|
n="$(printf '%s' "$out" | verdict_count_committed_divergence)"
|
||||||
|
if [ "${n:-0}" -gt 0 ]; then
|
||||||
|
V_DIVERGENCE=$((V_DIVERGENCE + n))
|
||||||
|
log "walcompare: COMMITTED DIVERGENCE x$n (boundary=$boundary) — COUNTED"
|
||||||
|
printf '%s\n' "$out" | grep 'WALCMP_DIVERGENCE' >&2
|
||||||
|
else
|
||||||
|
log "walcompare: committed prefix consistent (boundary=$boundary)"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Chaotic-mode verify: discover the serving primary (best effort), then classify
|
||||||
|
# the expect-job result via the verdict parser. Increments V_JOBLOSS on a REAL
|
||||||
|
# loss (reachable primary, tracked job "not found") and V_BUG854 when the client
|
||||||
|
# cannot locate the primary (discovery fault — EXCLUDED). Never hard-fails.
|
||||||
|
verify_classify() {
|
||||||
|
local primary out cls want="${#ALL_JOBIDS[@]}"
|
||||||
|
[ "$want" -eq 0 ] && return 0
|
||||||
|
primary="$(discover_primary)"
|
||||||
|
if [ -z "$primary" ]; then
|
||||||
|
V_BUG854=$((V_BUG854 + 1))
|
||||||
|
log "verify: client could not locate a primary (bug-854 class — EXCLUDED)"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
local script jid
|
||||||
|
script="set masters $MASTERS
|
||||||
|
set myname verify
|
||||||
|
client
|
||||||
|
wait-primary $WAITMS"
|
||||||
|
for jid in "${ALL_JOBIDS[@]}"; do
|
||||||
|
script+="
|
||||||
|
expect-job $jid"
|
||||||
|
done
|
||||||
|
script+="
|
||||||
|
quit"
|
||||||
|
out="$(printf '%s\n' "$script" | xtest)"
|
||||||
|
cls="$(printf '%s' "$out" | verdict_classify_verify 1)"
|
||||||
|
case "$cls" in
|
||||||
|
LOSS)
|
||||||
|
V_JOBLOSS=$((V_JOBLOSS + 1))
|
||||||
|
log "verify: COMMITTED JOB LOSS on reachable primary $primary — COUNTED"
|
||||||
|
printf '%s\n' "$out" | grep 'not found' >&2
|
||||||
|
;;
|
||||||
|
BUG854)
|
||||||
|
V_BUG854=$((V_BUG854 + 1))
|
||||||
|
log "verify: primary vanished mid-verify (bug-854 class — EXCLUDED)"
|
||||||
|
;;
|
||||||
|
OK)
|
||||||
|
log "verify: all $want tracked jobs present on $primary"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# Tally bug-924 overlayfs `fdatasync FAILED` ERROR lines across all node logs
|
||||||
|
# into V_BUG924 (EXCLUDED from the divergence/job-loss verdict).
|
||||||
|
scan_bug924() {
|
||||||
|
V_BUG924="$(cat "$TOP"/logs/*.log 2>/dev/null | verdict_count_fdatasync_bug924)"
|
||||||
|
[ "${V_BUG924:-0}" -gt 0 ] && \
|
||||||
|
log "bug-924: $V_BUG924 overlayfs fdatasync errors in logs — EXCLUDED from verdict"
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# Chaotic / overlapping-kill loop. Unlike the sequential loop it does NOT wait
|
||||||
|
# for full 3-of-3 reconvergence before the next kill: it restarts the previously-
|
||||||
|
# downed node and proceeds to kill the NEW primary as soon as one is ELECTED
|
||||||
|
# (log-based), so kills land inside election / catch-up windows (overlapping
|
||||||
|
# failovers — the unsupported double-fault window this spec must prove safe).
|
||||||
|
# Quorum is preserved: only the single current primary is killed per round and
|
||||||
|
# the prior node is restarted, so at most one node is down at a kill instant.
|
||||||
|
# Every data outcome is COUNTED into the verdict, never hard-failed.
|
||||||
|
run_chaotic_rounds() {
|
||||||
|
local round PRIMARY PREV_TERM NEWP pid DOWN=""
|
||||||
|
for round in $(seq 1 "$ROUNDS"); do
|
||||||
|
log "=== chaotic round $round/$ROUNDS ==="
|
||||||
|
|
||||||
|
# Overlap: restart the previously-downed node but DO NOT wait for it to
|
||||||
|
# converge — the next kill may land while it is still catching up.
|
||||||
|
if [ -n "$DOWN" ]; then
|
||||||
|
log "round $round: restarting $DOWN (no convergence wait — overlapping)"
|
||||||
|
start_node "$DOWN"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Identify the primary from the LOGS (server truth), independent of the
|
||||||
|
# client: a discovery miss here is bug-854, not a server fault.
|
||||||
|
PRIMARY="$(primary_from_log)"
|
||||||
|
if [ -z "$PRIMARY" ]; then
|
||||||
|
wait_for_primary_log || log "round $round: no primary elected yet (election lag/stall)"
|
||||||
|
PRIMARY="$(primary_from_log)"
|
||||||
|
fi
|
||||||
|
if [ -z "$PRIMARY" ]; then
|
||||||
|
log "round $round: no primary to kill this round — skipping"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
record_primary "$PRIMARY"
|
||||||
|
PREV_TERM="$(latest_primary_term)"; PREV_TERM="${PREV_TERM:-0}"
|
||||||
|
|
||||||
|
# Best-effort submit so there is fresh committed state to (potentially)
|
||||||
|
# lose. A freshly-promoted primary may briefly lack write-quorum; a
|
||||||
|
# shortfall is NOT a loss (the job was never acked, so never tracked).
|
||||||
|
submit_jobs "$NJOBS" || log "round $round: submit shortfall (transient — not a loss)"
|
||||||
|
|
||||||
|
# Kill the current primary, landing inside the next election window.
|
||||||
|
pid="$(pgrep -f -- "xce_controller --myname $PRIMARY ")"
|
||||||
|
if [ -n "$pid" ]; then
|
||||||
|
log "round $round: killing primary $PRIMARY (term=$PREV_TERM, pid=$(echo $pid | tr '\n' ' '))"
|
||||||
|
kill -9 $pid 2>/dev/null
|
||||||
|
DOWN="$PRIMARY"
|
||||||
|
else
|
||||||
|
log "round $round: primary $PRIMARY already gone"
|
||||||
|
fi
|
||||||
|
sleep 1
|
||||||
|
|
||||||
|
# Wait ONLY for a new primary to be ELECTED (not full reconvergence).
|
||||||
|
NEWP="$(wait_for_new_primary "$PREV_TERM")"
|
||||||
|
if [ -n "$NEWP" ]; then
|
||||||
|
log "round $round: new primary elected: $NEWP (term=$(latest_primary_term))"
|
||||||
|
record_primary "$NEWP"
|
||||||
|
else
|
||||||
|
log "round $round: NO new primary within window (possible election stall)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Count loss/divergence for this round (verdict, not a hard fail).
|
||||||
|
verify_classify
|
||||||
|
walcheck_count "$((1 + 2 * ${#ALL_JOBIDS[@]}))"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Restore 3-of-3 and let it settle so the FINAL sweep sees every member's
|
||||||
|
# committed prefix.
|
||||||
|
if [ -n "$DOWN" ]; then
|
||||||
|
log "chaotic: restarting last-downed $DOWN and settling before final verdict"
|
||||||
|
start_node "$DOWN"
|
||||||
|
wait_for_full_convergence \
|
||||||
|
|| log "chaotic: did not fully reconverge before verdict (any residue is counted via walcompare)"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
# Drive a SPECIFIC node to primary — used to exercise leadership on the
|
# Drive a SPECIFIC node to primary — used to exercise leadership on the
|
||||||
# lowest-priority node (ctrl-3), which the natural single-failover loop NEVER
|
# lowest-priority node (ctrl-3), which the natural single-failover loop NEVER
|
||||||
# elects: the election timeout is priority-staggered (my_priority * step_ms, p0
|
# elects: the election timeout is priority-staggered (my_priority * step_ms, p0
|
||||||
|
|
@ -397,6 +583,34 @@ submit_jobs "$NJOBS" || fail "initial job submission did not get $NJOBS clean ac
|
||||||
verify_no_loss || fail "initial joblist verification failed"
|
verify_no_loss || fail "initial joblist verification failed"
|
||||||
walcheck "$((1 + 2 * ${#ALL_JOBIDS[@]}))"
|
walcheck "$((1 + 2 * ${#ALL_JOBIDS[@]}))"
|
||||||
|
|
||||||
|
# ── Chaotic mode: overlapping-kill rounds → isolated verdict ──────────────────
|
||||||
|
# Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up
|
||||||
|
# windows; outcomes are COUNTED (committed divergence + committed job loss) with
|
||||||
|
# bug-854 (client discovery) and bug-924 (overlayfs fdatasync) ISOLATED. Emits a
|
||||||
|
# single machine-parseable VERDICT line plus the PASS/FAIL oracle line, then
|
||||||
|
# exits — it does NOT run the sequential loop or leadership-coverage maneuver.
|
||||||
|
if [ "$MODE" = "chaotic" ]; then
|
||||||
|
log "MODE=chaotic — overlapping-kill soak ($ROUNDS rounds)"
|
||||||
|
run_chaotic_rounds
|
||||||
|
# Final divergence + loss sweep on the settled cluster, then bug-924 tally.
|
||||||
|
walcheck_count "$((1 + 2 * ${#ALL_JOBIDS[@]}))"
|
||||||
|
verify_classify
|
||||||
|
scan_bug924
|
||||||
|
verdict_emit "chaotic" "$ROUNDS" "${#ALL_JOBIDS[@]}" \
|
||||||
|
"$V_DIVERGENCE" "$V_JOBLOSS" "$V_BUG854" "$V_BUG924"
|
||||||
|
VERDICT_RC=$?
|
||||||
|
if [ "$VERDICT_RC" -eq 0 ]; then
|
||||||
|
log "PASS (chaotic): $ROUNDS rounds, ${#ALL_JOBIDS[@]} jobs, zero committed divergence, zero committed job loss (bug854=$V_BUG854 bug924=$V_BUG924 excluded)"
|
||||||
|
cleanup
|
||||||
|
trap - EXIT INT TERM
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
echo "[xce-test-ha] FAIL: chaotic verdict — committed_divergence=$V_DIVERGENCE committed_job_loss=$V_JOBLOSS (bug854=$V_BUG854 bug924=$V_BUG924 excluded)" >&2
|
||||||
|
cleanup
|
||||||
|
trap - EXIT INT TERM
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
# ── Steps 7–10 looped: kill primary, verify, submit, restart, reconverge ─────
|
# ── Steps 7–10 looped: kill primary, verify, submit, restart, reconverge ─────
|
||||||
#
|
#
|
||||||
# Each round kills the CURRENT primary, then restarts the node downed in the
|
# Each round kills the CURRENT primary, then restarts the node downed in the
|
||||||
|
|
|
||||||
159
.claude/skills/xce-test-ha/soak-verdict-selftest.sh
Normal file
159
.claude/skills/xce-test-ha/soak-verdict-selftest.sh
Normal file
|
|
@ -0,0 +1,159 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
#
|
||||||
|
# soak-verdict-selftest.sh — deterministic unit test for the verdict parser in
|
||||||
|
# soak-verdict.sh. Feeds FIXTURE output (a committed `walcompare` divergence, an
|
||||||
|
# uncommitted-tail divergence, a bug-854 client-discovery failure, and bug-924
|
||||||
|
# `fdatasync FAILED` log lines) and asserts the verdict:
|
||||||
|
# - counts the committed divergence and committed job-loss (real failures), and
|
||||||
|
# - EXCLUDES bug-854 (client cannot locate the primary) and bug-924 (overlayfs
|
||||||
|
# fdatasync) from the divergence / job-loss result.
|
||||||
|
#
|
||||||
|
# No cluster, no binaries — pure bash, runs anywhere in well under a second. This
|
||||||
|
# is the RED→GREEN proof for the verdict logic (task 1.2): the full live chaotic
|
||||||
|
# soak is the separate acceptance gate (task 7.2).
|
||||||
|
#
|
||||||
|
# Usage: bash soak-verdict-selftest.sh (also reachable as run-ha-soak.sh --selftest)
|
||||||
|
set -u
|
||||||
|
|
||||||
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
# shellcheck source=soak-verdict.sh
|
||||||
|
. "$HERE/soak-verdict.sh" || {
|
||||||
|
echo "SELFTEST FAIL: cannot source soak-verdict.sh (verdict library missing)" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
fails=0
|
||||||
|
check() { # desc expected actual
|
||||||
|
if [ "$2" = "$3" ]; then
|
||||||
|
echo "ok - $1"
|
||||||
|
else
|
||||||
|
echo "FAIL - $1: expected [$2] got [$3]"
|
||||||
|
fails=$((fails + 1))
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ── Fixtures (captured/synthetic soak output) ────────────────────────────────
|
||||||
|
|
||||||
|
# A real committed-prefix divergence: member 1 holds a different (term, pcrc) at
|
||||||
|
# committed LSN 9 (the bug-851 family signature). Task 1.1's machine line marks
|
||||||
|
# it class=committed → a FATAL, COUNTED divergence.
|
||||||
|
WALCMP_COMMITTED_DIV="$(cat <<'EOF'
|
||||||
|
== walcompare ==
|
||||||
|
WALCMP member=0 lsn=9 term=3 pcrc=deadbeef status=present committed=1
|
||||||
|
WALCMP member=1 lsn=9 term=4 pcrc=cafebabe status=present committed=1
|
||||||
|
WALCMP member=2 lsn=9 term=3 pcrc=00000000 status=MISSING committed=1
|
||||||
|
WALCMP_DIVERGENCE lsn=9 class=committed compared=3 present=2
|
||||||
|
*** DIVERGENCE at committed LSN 9 (compared 3, present 2) ***
|
||||||
|
walcompare: FAIL — committed-prefix divergence at LSN 9
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
|
||||||
|
# An uncommitted-TAIL divergence (above the committed boundary). NON-fatal — it
|
||||||
|
# must NOT be counted toward the committed-divergence verdict.
|
||||||
|
WALCMP_UNCOMMITTED_DIV="$(cat <<'EOF'
|
||||||
|
WALCMP member=0 lsn=11 term=5 pcrc=11111111 status=present committed=0
|
||||||
|
WALCMP member=1 lsn=11 term=6 pcrc=22222222 status=present committed=0
|
||||||
|
WALCMP_DIVERGENCE lsn=11 class=uncommitted compared=2 present=2
|
||||||
|
walcompare: OK
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
|
||||||
|
# A clean committed prefix (no divergence).
|
||||||
|
WALCMP_CLEAN="$(cat <<'EOF'
|
||||||
|
WALCMP member=0 lsn=3 term=2 pcrc=000000ab status=present committed=1
|
||||||
|
WALCMP member=1 lsn=3 term=2 pcrc=000000ab status=present committed=1
|
||||||
|
walcompare: OK
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
|
||||||
|
# bug-924: overlayfs fdatasync EIO. Two ERROR lines from persist.c.
|
||||||
|
FDATASYNC_LOG="$(cat <<'EOF'
|
||||||
|
[INFO] persist: appended lsn=7
|
||||||
|
[ERROR] persist: flusher fdatasync FAILED (errno=5 Input/output error) — durable_lsn held
|
||||||
|
[ERROR] persist: persist_flush fdatasync FAILED (errno=5 Input/output error) — durable_lsn
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
|
||||||
|
# A clean controller log (no fdatasync errors).
|
||||||
|
CLEAN_LOG="$(cat <<'EOF'
|
||||||
|
[INFO] HA FSM (raft): → CTRL_PRIMARY (term=4, ...)
|
||||||
|
[INFO] persist: appended lsn=9 durable
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
|
||||||
|
# expect-job output: one tracked job is genuinely missing on a reachable primary.
|
||||||
|
EXPECT_LOSS="$(cat <<'EOF'
|
||||||
|
expect-job: ok (job-1)
|
||||||
|
expect-job: job job-2 not found
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
|
||||||
|
# expect-job output: every tracked job present.
|
||||||
|
EXPECT_OK="$(cat <<'EOF'
|
||||||
|
expect-job: ok (job-1)
|
||||||
|
expect-job: ok (job-2)
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
|
||||||
|
# The client connected but the primary vanished mid-command → no expect-job
|
||||||
|
# results at all (a discovery race — bug-854 class, EXCLUDED, not a loss).
|
||||||
|
EXPECT_VANISHED="$(cat <<'EOF'
|
||||||
|
wait-primary: 30000 ms elapsed, no primary
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
|
||||||
|
# ── Assertions ───────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
check "committed divergence is counted" \
|
||||||
|
1 "$(printf '%s' "$WALCMP_COMMITTED_DIV" | verdict_count_committed_divergence)"
|
||||||
|
check "uncommitted-tail divergence is EXCLUDED" \
|
||||||
|
0 "$(printf '%s' "$WALCMP_UNCOMMITTED_DIV" | verdict_count_committed_divergence)"
|
||||||
|
check "clean walcompare => 0 committed divergence" \
|
||||||
|
0 "$(printf '%s' "$WALCMP_CLEAN" | verdict_count_committed_divergence)"
|
||||||
|
|
||||||
|
check "bug-924 fdatasync errors are counted" \
|
||||||
|
2 "$(printf '%s' "$FDATASYNC_LOG" | verdict_count_fdatasync_bug924)"
|
||||||
|
check "clean log => 0 bug-924 errors" \
|
||||||
|
0 "$(printf '%s' "$CLEAN_LOG" | verdict_count_fdatasync_bug924)"
|
||||||
|
|
||||||
|
check "reachable + not-found => LOSS (real)" \
|
||||||
|
LOSS "$(printf '%s' "$EXPECT_LOSS" | verdict_classify_verify 1)"
|
||||||
|
check "unreachable => BUG854 (excluded, not loss)" \
|
||||||
|
BUG854 "$(printf '%s' "$EXPECT_LOSS" | verdict_classify_verify 0)"
|
||||||
|
check "reachable + all present => OK" \
|
||||||
|
OK "$(printf '%s' "$EXPECT_OK" | verdict_classify_verify 1)"
|
||||||
|
check "reachable but no expect-job results => BUG854 (mid-command discovery race)" \
|
||||||
|
BUG854 "$(printf '%s' "$EXPECT_VANISHED" | verdict_classify_verify 1)"
|
||||||
|
|
||||||
|
# Verdict-level: a committed divergence is FAIL even amid bug-854/bug-924 noise.
|
||||||
|
v="$(verdict_emit chaotic 6 18 1 0 3 2)"; rc=$?
|
||||||
|
check "committed divergence => result=FAIL token" \
|
||||||
|
"FAIL" "$(printf '%s' "$v" | grep -oE 'result=[A-Z]+' | cut -d= -f2)"
|
||||||
|
check "committed divergence => non-zero rc" 1 "$rc"
|
||||||
|
|
||||||
|
# THE KEY ASSERTION: with zero real divergence and zero real loss, the verdict is
|
||||||
|
# PASS even though bug-854 (3) and bug-924 (2) both fired — i.e. the verdict's
|
||||||
|
# divergence/job-loss result is UNAFFECTED by bug-854/bug-924 (R8.4).
|
||||||
|
v="$(verdict_emit chaotic 6 18 0 0 3 2)"; rc=$?
|
||||||
|
check "div=0 loss=0 with bug854+bug924 noise => result=PASS token" \
|
||||||
|
"PASS" "$(printf '%s' "$v" | grep -oE 'result=[A-Z]+' | cut -d= -f2)"
|
||||||
|
check "div=0 loss=0 with bug854+bug924 noise => zero rc" 0 "$rc"
|
||||||
|
check "verdict line carries the excluded bug854 count" \
|
||||||
|
"excluded_bug854_discovery=3" "$(printf '%s' "$v" | grep -oE 'excluded_bug854_discovery=[0-9]+')"
|
||||||
|
check "verdict line carries the excluded bug924 count" \
|
||||||
|
"excluded_bug924_fdatasync=2" "$(printf '%s' "$v" | grep -oE 'excluded_bug924_fdatasync=[0-9]+')"
|
||||||
|
|
||||||
|
# A real committed job-loss is FAIL regardless of how few excluded faults fired.
|
||||||
|
v="$(verdict_emit chaotic 6 18 0 2 0 0)"; rc=$?
|
||||||
|
check "committed job loss => non-zero rc" 1 "$rc"
|
||||||
|
check "committed job loss => result=FAIL token" \
|
||||||
|
"FAIL" "$(printf '%s' "$v" | grep -oE 'result=[A-Z]+' | cut -d= -f2)"
|
||||||
|
|
||||||
|
# ── Verdict ──────────────────────────────────────────────────────────────────
|
||||||
|
echo
|
||||||
|
if [ "$fails" -eq 0 ]; then
|
||||||
|
echo "SELFTEST PASS"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
echo "SELFTEST FAIL ($fails failed)"
|
||||||
|
exit 1
|
||||||
91
.claude/skills/xce-test-ha/soak-verdict.sh
Normal file
91
.claude/skills/xce-test-ha/soak-verdict.sh
Normal file
|
|
@ -0,0 +1,91 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
#
|
||||||
|
# soak-verdict.sh — pure (side-effect-free) verdict helpers for run-ha-soak.sh.
|
||||||
|
#
|
||||||
|
# These functions PARSE captured output and emit counts / a classification; they
|
||||||
|
# never touch the cluster, so they are unit-testable against fixture text
|
||||||
|
# (soak-verdict-selftest.sh). The chaotic soak sources this file and uses the
|
||||||
|
# helpers to build a machine-parseable verdict whose committed-divergence and
|
||||||
|
# committed-job-loss counts are UNAFFECTED by two known, out-of-scope defects:
|
||||||
|
#
|
||||||
|
# - bug-854 (libxce client primary-discovery): the client cannot LOCATE the
|
||||||
|
# serving primary. This is a discovery fault, NOT a server losing committed
|
||||||
|
# state — so a "could not reach the primary" outcome is counted separately
|
||||||
|
# and EXCLUDED from the job-loss result.
|
||||||
|
# - bug-924 (overlayfs `fdatasync FAILED (errno=5/EIO)`): an environment defect
|
||||||
|
# of the sandbox filesystem, not the controller. Its log lines are counted
|
||||||
|
# separately and EXCLUDED from the divergence/job-loss result (the acceptance
|
||||||
|
# soak, task 7.2, runs on a real fs where bug-924 does not occur).
|
||||||
|
#
|
||||||
|
# Source this file; do not execute it directly. The terminal verdict is emitted
|
||||||
|
# by verdict_emit().
|
||||||
|
|
||||||
|
# Count committed-class `walcompare` divergences in the text on stdin.
|
||||||
|
# Consumes task 1.1's machine line:
|
||||||
|
# WALCMP_DIVERGENCE lsn=<L> class=committed compared=<c> present=<p>
|
||||||
|
# Only class=committed lines are counted; an uncommitted-tail divergence is
|
||||||
|
# non-fatal (above the committed boundary) and is deliberately NOT counted.
|
||||||
|
verdict_count_committed_divergence() {
|
||||||
|
local n
|
||||||
|
n="$(grep -c -E '^WALCMP_DIVERGENCE .*class=committed' || true)"
|
||||||
|
echo "${n:-0}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Count bug-924 overlayfs `fdatasync FAILED` ERROR lines in the log text on
|
||||||
|
# stdin (persist.c emits "persist: ... fdatasync FAILED (errno=...)"). Reported
|
||||||
|
# as an EXCLUDED count — it never feeds the divergence/job-loss verdict.
|
||||||
|
verdict_count_fdatasync_bug924() {
|
||||||
|
local n
|
||||||
|
n="$(grep -c -E 'fdatasync FAILED' || true)"
|
||||||
|
echo "${n:-0}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Classify one verify (expect-job) attempt. Reads the expect-job command OUTPUT
|
||||||
|
# on stdin; $1 = primary_reachable (1 iff client-side discovery already located a
|
||||||
|
# serving primary). Echoes exactly one token:
|
||||||
|
# BUG854 — the client could not reach a primary, OR the primary vanished
|
||||||
|
# mid-command so NO expect-job result was produced (discovery fault,
|
||||||
|
# EXCLUDED — never counted as committed-job loss)
|
||||||
|
# LOSS — the primary was reachable but a tracked committed job is "not found"
|
||||||
|
# (a REAL server-side committed-job loss, bug-828 class)
|
||||||
|
# OK — the primary was reachable and every tracked job is present
|
||||||
|
# This is the seam that separates "the server lost a committed job" (real) from
|
||||||
|
# "the client could not locate the primary" (bug-854, excluded).
|
||||||
|
verdict_classify_verify() {
|
||||||
|
local reachable="$1" out
|
||||||
|
out="$(cat)"
|
||||||
|
if [ "$reachable" != "1" ]; then
|
||||||
|
echo BUG854
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
# Reachable per discovery, but if the verify command yielded NO expect-job
|
||||||
|
# results at all, the client lost the primary mid-command (a discovery race)
|
||||||
|
# — classify as bug-854, not a loss.
|
||||||
|
if ! printf '%s' "$out" | grep -q "expect-job:"; then
|
||||||
|
echo BUG854
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
if printf '%s' "$out" | grep -q "not found"; then
|
||||||
|
echo LOSS
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
echo OK
|
||||||
|
}
|
||||||
|
|
||||||
|
# Emit the single machine-parseable verdict line. PASS iff committed_divergence
|
||||||
|
# == 0 AND committed_job_loss == 0; the excluded bug-854/bug-924 counts are
|
||||||
|
# REPORTED for visibility but do NOT affect the pass/fail decision. Returns 0 on
|
||||||
|
# PASS, 1 on FAIL.
|
||||||
|
# Args: mode rounds jobs committed_divergence committed_job_loss bug854 bug924
|
||||||
|
verdict_emit() {
|
||||||
|
local mode="$1" rounds="$2" jobs="$3" div="$4" loss="$5" b854="$6" b924="$7"
|
||||||
|
local result="PASS"
|
||||||
|
if [ "$div" -ne 0 ] || [ "$loss" -ne 0 ]; then
|
||||||
|
result="FAIL"
|
||||||
|
fi
|
||||||
|
echo "[xce-test-ha] VERDICT mode=$mode rounds=$rounds jobs=$jobs" \
|
||||||
|
"committed_divergence=$div committed_job_loss=$loss" \
|
||||||
|
"excluded_bug854_discovery=$b854 excluded_bug924_fdatasync=$b924" \
|
||||||
|
"result=$result"
|
||||||
|
[ "$result" = "PASS" ]
|
||||||
|
}
|
||||||
|
|
@ -13,7 +13,7 @@
|
||||||
- _Requirements: 7.1, 7.5_
|
- _Requirements: 7.1, 7.5_
|
||||||
- _Boundary: Deterministic Chaotic Harness_
|
- _Boundary: Deterministic Chaotic Harness_
|
||||||
|
|
||||||
- [ ] 1.2 Add a chaotic soak mode with an isolated verdict
|
- [x] 1.2 Add a chaotic soak mode with an isolated verdict
|
||||||
- Add an overlapping-kill mode to the soak driver (kills landing inside election windows, not only sequential kill→wait→rejoin); produce a verdict that counts `walcompare` divergences and committed-job loss while excluding client-discovery (bug-854) and overlayfs `fdatasync` (bug-924) failures from the divergence/job-loss result
|
- Add an overlapping-kill mode to the soak driver (kills landing inside election windows, not only sequential kill→wait→rejoin); produce a verdict that counts `walcompare` divergences and committed-job loss while excluding client-discovery (bug-854) and overlayfs `fdatasync` (bug-924) failures from the divergence/job-loss result
|
||||||
- Observable completion: the soak runs chaotic mode against real-fs release binaries and emits a verdict whose divergence/job-loss counts are unaffected by bug-854/bug-924
|
- Observable completion: the soak runs chaotic mode against real-fs release binaries and emits a verdict whose divergence/job-loss counts are unaffected by bug-854/bug-924
|
||||||
- _Requirements: 7.5, 8.4_
|
- _Requirements: 7.5, 8.4_
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue