From 02c0fcb6aa007cfa3f9b764f1052ebec66b4a969 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 27 Jun 2026 05:15:53 +0000 Subject: [PATCH] test(raft-ha-core-safety): chaotic soak mode + isolated verdict (task 1.2) Opt-in overlapping-kill mode (kills inside election windows). Verdict counts committed walcompare divergence + committed-job loss; bug-854 (client discovery) and bug-924 (overlayfs fdatasync) counted separately and excluded from PASS/FAIL. 17/17 verdict self-test GREEN. Live smoke corroborated chaotic regime + live bug-854 isolation. --- .claude/skills/xce-test-ha/SKILL.md | 71 ++++++ .claude/skills/xce-test-ha/run-ha-soak.sh | 216 +++++++++++++++++- .../xce-test-ha/soak-verdict-selftest.sh | 159 +++++++++++++ .claude/skills/xce-test-ha/soak-verdict.sh | 91 ++++++++ .kiro/specs/raft-ha-core-safety/tasks.md | 2 +- 5 files changed, 537 insertions(+), 2 deletions(-) create mode 100644 .claude/skills/xce-test-ha/soak-verdict-selftest.sh create mode 100644 .claude/skills/xce-test-ha/soak-verdict.sh diff --git a/.claude/skills/xce-test-ha/SKILL.md b/.claude/skills/xce-test-ha/SKILL.md index b6abc094..bb8d2e34 100644 --- a/.claude/skills/xce-test-ha/SKILL.md +++ b/.claude/skills/xce-test-ha/SKILL.md @@ -31,6 +31,74 @@ A green run is `PASS: N rounds, M jobs submitted, zero job loss` (exit 0). run (it looks for a `.xce-ha-deploy` marker) — point it at a new/empty/scratch path, not a real source or home directory. +## Modes: sequential (default) vs chaotic + +The driver has two kill schedules, selected by a flag (or `XCE_HA_MODE`): + +- **`--sequential` (default)** — supported single-failover lifecycle: kill the + primary → wait for a new primary → restart the downed node → **wait for full + 3-of-3 reconvergence** before the next kill. Any committed-job loss or failed + election is a hard `FAIL` (exit ≠ 0). Backward-compatible; the leadership- + coverage maneuver (all three nodes must lead) runs only in this mode. +- **`--chaotic`** — **overlapping** kills that land *inside* election / catch-up + windows: the previously-downed node is restarted but the driver does **not** + wait for it to reconverge before killing the next primary. This deliberately + exercises the unsupported double-fault-within-one-reconvergence window that + surfaces committed-prefix divergence (bug-851 family). Quorum is preserved — + only the single current primary is killed per round and the prior node is + restarted, so at most one node is down at a kill instant. + +```bash +SKILL_DIR=.claude/skills/xce-test-ha +# chaotic soak, 6 rounds, scratch dir +XCE_HA_ROUNDS=6 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --chaotic /tmp/ha-chaos +# equivalent via env +XCE_HA_MODE=chaotic XCE_HA_ROUNDS=6 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" /tmp/ha-chaos +``` + +For the **acceptance gate (task 7.2)** run chaotic mode for ≥6 rounds against +**real-fs (ext4) release binaries** (point `XCE_HA_BIN` / `XCE_HA_XT` at the +release `xce_controller` / `xce_test`). + +### Chaotic verdict (bug-854 / bug-924 isolated) + +Chaotic mode does not hard-fail on data outcomes; it **counts** them and emits a +single machine-parseable verdict line (`soak-verdict.sh`): + +``` +[xce-test-ha] VERDICT mode=chaotic rounds=6 jobs=18 committed_divergence=0 committed_job_loss=0 excluded_bug854_discovery=2 excluded_bug924_fdatasync=0 result=PASS +``` + +- **`committed_divergence`** — committed-prefix `walcompare` divergences, counted + from task 1.1's `WALCMP_DIVERGENCE … class=committed` machine line + (`walcompare --report`). Uncommitted-tail divergences are non-fatal and not + counted. +- **`committed_job_loss`** — a tracked, acked job reported `not found` on a + **reachable** primary (a real server-side loss, bug-828 class). +- **`excluded_bug854_discovery`** — the client could not LOCATE the primary + (libxce primary-discovery fault). A discovery miss, **not** a server losing + committed state — reported but **excluded** from the verdict. +- **`excluded_bug924_fdatasync`** — overlayfs `fdatasync FAILED (errno=5/EIO)` + ERROR lines in the controller logs (environment defect). Reported but + **excluded**; the real-fs acceptance run does not hit it. + +`result=PASS` iff `committed_divergence == 0` **and** `committed_job_loss == 0`, +regardless of the excluded bug-854/bug-924 counts. The driver also prints the +usual terminal `PASS:` / `FAIL:` oracle line (exit 0 / non-zero) so a goal-loop +can branch on it unchanged. + +### Verdict self-test (no cluster) + +The verdict parser has a deterministic, pure-bash self-test that feeds fixture +output (a committed divergence, a bug-854 discovery failure, and bug-924 +`fdatasync` lines) and asserts the divergence/job-loss counts are unaffected by +bug-854/bug-924. It needs no binaries and runs in under a second: + +```bash +bash .claude/skills/xce-test-ha/run-ha-soak.sh --selftest # or: +bash .claude/skills/xce-test-ha/soak-verdict-selftest.sh +``` + ``` /xce-test-ha # 6 rounds, default deploy dir /xce-test-ha 10 # 10 rounds, default deploy dir @@ -87,6 +155,9 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK |---|---|---| | `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` | | `XCE_HA_DATADIR` | config/certs source (`cluster.bin`, `xce.crt`, `xce.key`) | `/ha/data` | +| `XCE_HA_MODE` | `sequential` or `chaotic` (`--chaotic` flag wins) | `sequential` | +| `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `/src/xce.build/controller/xce_controller` | +| `XCE_HA_XT` | `xce_test` binary | `/src/xce.build/controller/xce_test` | | `XCE_HA_ROUNDS` | kill/restart rounds (5–10) | `6` | | `XCE_HA_JOBS` | jobs per submit batch | `3` | | `XCE_HA_WAITMS` | `wait-primary` budget per probe | `30000` | diff --git a/.claude/skills/xce-test-ha/run-ha-soak.sh b/.claude/skills/xce-test-ha/run-ha-soak.sh index b1db5a4a..aef3fb1a 100755 --- a/.claude/skills/xce-test-ha/run-ha-soak.sh +++ b/.claude/skills/xce-test-ha/run-ha-soak.sh @@ -10,19 +10,50 @@ # and xce_test runtime libs (openssl etc.) are on the library path. # # Usage: -# run-ha-soak.sh [DEPLOY_DIR] +# run-ha-soak.sh [--chaotic|--sequential] [--selftest] [DEPLOY_DIR] # DEPLOY_DIR optional deploy directory (positional). Precedence: # positional arg > $XCE_HA_TOP env > default. Relative paths # are resolved to absolute. +# --chaotic overlapping-kill mode: kills land INSIDE election/catch-up +# windows (it does NOT wait for full 3-of-3 reconvergence before +# the next kill). Outcomes are COUNTED into a verdict, not hard- +# failed; bug-854 (client discovery) and bug-924 (overlayfs +# fdatasync) are isolated from the divergence/job-loss result. +# --sequential default single-failover mode (kill → wait-new-primary → +# restart → reconverge). Backward-compatible. +# --selftest run the verdict-parser self-test (soak-verdict-selftest.sh) +# and exit — no cluster, no binaries needed. # # Env overrides: # XCE_HA_TOP deploy dir (default /home/claude/tmp/xce-ha-deploy) # XCE_HA_DATADIR config/certs source (default /ha/data) +# XCE_HA_MODE sequential | chaotic (default sequential; --chaotic wins) # XCE_HA_ROUNDS kill/restart rounds (default 6, clamp 5..10) # XCE_HA_JOBS jobs per submit batch (default 3) # XCE_HA_WAITMS wait-primary budget (default 30000) set -u +# ── Mode / flag parsing (flags may precede the positional DEPLOY_DIR) ───────── +# `--selftest` short-circuits to the verdict self-test before any cluster setup. +SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +MODE="${XCE_HA_MODE:-sequential}" +ARGS=() +while [ $# -gt 0 ]; do + case "$1" in + --chaotic) MODE="chaotic" ;; + --sequential) MODE="sequential" ;; + --selftest) exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;; + *) ARGS+=("$1") ;; + esac + shift +done +set -- ${ARGS[@]+"${ARGS[@]}"} + +# Verdict helpers (pure parsers; see soak-verdict.sh). Sourced so the chaotic +# loop can count committed divergence / job-loss with bug-854/bug-924 isolated. +# shellcheck source=soak-verdict.sh +. "$SELF_DIR/soak-verdict.sh" + # ── Paths ──────────────────────────────────────────────────────────────────── REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" # Deploy dir: explicit positional arg wins, then $XCE_HA_TOP, then the default. @@ -56,6 +87,14 @@ MASTERS="localhost:9991,localhost:9992,localhost:9993" ALL_JOBIDS=() # every JobId we have ever successfully submitted declare -A SEEN_PRIMARY # node name -> 1 once it has served as primary +# ── Chaotic-mode verdict counters (see soak-verdict.sh) ────────────────────── +# committed_divergence and committed_job_loss are the REAL server-side verdict; +# bug854/bug924 are EXCLUDED (counted separately, never affect PASS/FAIL). +V_DIVERGENCE=0 # committed `walcompare` divergences observed across the run +V_JOBLOSS=0 # tracked committed jobs "not found" on a REACHABLE primary +V_BUG854=0 # client could not locate the primary (libxce discovery fault) +V_BUG924=0 # overlayfs `fdatasync FAILED` log lines (environment defect) + log() { echo "[xce-test-ha] $*"; } fail() { echo "[xce-test-ha] FAIL: $*" >&2; cleanup; exit 1; } @@ -300,6 +339,153 @@ walcheck() { fi } +# ── Chaotic-mode primitives (count outcomes into the verdict, never hard-fail) ─ + +# Run `walcompare --report` across all three members and ADD any committed-class +# divergence to V_DIVERGENCE (task 1.1's WALCMP_DIVERGENCE class=committed line). +# Non-fatal: a divergence is COUNTED, not a hard exit — the verdict decides PASS. +walcheck_count() { + local boundary="$1" d1 d2 d3 out n + d1="$TOP/controller/ctrl-1/data" + d2="$TOP/controller/ctrl-2/data" + d3="$TOP/controller/ctrl-3/data" + out="$(printf 'walcompare --report --committed %s %s %s %s\nquit\n' \ + "$boundary" "$d1" "$d2" "$d3" | xtest)" + n="$(printf '%s' "$out" | verdict_count_committed_divergence)" + if [ "${n:-0}" -gt 0 ]; then + V_DIVERGENCE=$((V_DIVERGENCE + n)) + log "walcompare: COMMITTED DIVERGENCE x$n (boundary=$boundary) — COUNTED" + printf '%s\n' "$out" | grep 'WALCMP_DIVERGENCE' >&2 + else + log "walcompare: committed prefix consistent (boundary=$boundary)" + fi +} + +# Chaotic-mode verify: discover the serving primary (best effort), then classify +# the expect-job result via the verdict parser. Increments V_JOBLOSS on a REAL +# loss (reachable primary, tracked job "not found") and V_BUG854 when the client +# cannot locate the primary (discovery fault — EXCLUDED). Never hard-fails. +verify_classify() { + local primary out cls want="${#ALL_JOBIDS[@]}" + [ "$want" -eq 0 ] && return 0 + primary="$(discover_primary)" + if [ -z "$primary" ]; then + V_BUG854=$((V_BUG854 + 1)) + log "verify: client could not locate a primary (bug-854 class — EXCLUDED)" + return 0 + fi + local script jid + script="set masters $MASTERS +set myname verify +client +wait-primary $WAITMS" + for jid in "${ALL_JOBIDS[@]}"; do + script+=" +expect-job $jid" + done + script+=" +quit" + out="$(printf '%s\n' "$script" | xtest)" + cls="$(printf '%s' "$out" | verdict_classify_verify 1)" + case "$cls" in + LOSS) + V_JOBLOSS=$((V_JOBLOSS + 1)) + log "verify: COMMITTED JOB LOSS on reachable primary $primary — COUNTED" + printf '%s\n' "$out" | grep 'not found' >&2 + ;; + BUG854) + V_BUG854=$((V_BUG854 + 1)) + log "verify: primary vanished mid-verify (bug-854 class — EXCLUDED)" + ;; + OK) + log "verify: all $want tracked jobs present on $primary" + ;; + esac +} + +# Tally bug-924 overlayfs `fdatasync FAILED` ERROR lines across all node logs +# into V_BUG924 (EXCLUDED from the divergence/job-loss verdict). +scan_bug924() { + V_BUG924="$(cat "$TOP"/logs/*.log 2>/dev/null | verdict_count_fdatasync_bug924)" + [ "${V_BUG924:-0}" -gt 0 ] && \ + log "bug-924: $V_BUG924 overlayfs fdatasync errors in logs — EXCLUDED from verdict" + return 0 +} + +# Chaotic / overlapping-kill loop. Unlike the sequential loop it does NOT wait +# for full 3-of-3 reconvergence before the next kill: it restarts the previously- +# downed node and proceeds to kill the NEW primary as soon as one is ELECTED +# (log-based), so kills land inside election / catch-up windows (overlapping +# failovers — the unsupported double-fault window this spec must prove safe). +# Quorum is preserved: only the single current primary is killed per round and +# the prior node is restarted, so at most one node is down at a kill instant. +# Every data outcome is COUNTED into the verdict, never hard-failed. +run_chaotic_rounds() { + local round PRIMARY PREV_TERM NEWP pid DOWN="" + for round in $(seq 1 "$ROUNDS"); do + log "=== chaotic round $round/$ROUNDS ===" + + # Overlap: restart the previously-downed node but DO NOT wait for it to + # converge — the next kill may land while it is still catching up. + if [ -n "$DOWN" ]; then + log "round $round: restarting $DOWN (no convergence wait — overlapping)" + start_node "$DOWN" + fi + + # Identify the primary from the LOGS (server truth), independent of the + # client: a discovery miss here is bug-854, not a server fault. + PRIMARY="$(primary_from_log)" + if [ -z "$PRIMARY" ]; then + wait_for_primary_log || log "round $round: no primary elected yet (election lag/stall)" + PRIMARY="$(primary_from_log)" + fi + if [ -z "$PRIMARY" ]; then + log "round $round: no primary to kill this round — skipping" + continue + fi + record_primary "$PRIMARY" + PREV_TERM="$(latest_primary_term)"; PREV_TERM="${PREV_TERM:-0}" + + # Best-effort submit so there is fresh committed state to (potentially) + # lose. A freshly-promoted primary may briefly lack write-quorum; a + # shortfall is NOT a loss (the job was never acked, so never tracked). + submit_jobs "$NJOBS" || log "round $round: submit shortfall (transient — not a loss)" + + # Kill the current primary, landing inside the next election window. + pid="$(pgrep -f -- "xce_controller --myname $PRIMARY ")" + if [ -n "$pid" ]; then + log "round $round: killing primary $PRIMARY (term=$PREV_TERM, pid=$(echo $pid | tr '\n' ' '))" + kill -9 $pid 2>/dev/null + DOWN="$PRIMARY" + else + log "round $round: primary $PRIMARY already gone" + fi + sleep 1 + + # Wait ONLY for a new primary to be ELECTED (not full reconvergence). + NEWP="$(wait_for_new_primary "$PREV_TERM")" + if [ -n "$NEWP" ]; then + log "round $round: new primary elected: $NEWP (term=$(latest_primary_term))" + record_primary "$NEWP" + else + log "round $round: NO new primary within window (possible election stall)" + fi + + # Count loss/divergence for this round (verdict, not a hard fail). + verify_classify + walcheck_count "$((1 + 2 * ${#ALL_JOBIDS[@]}))" + done + + # Restore 3-of-3 and let it settle so the FINAL sweep sees every member's + # committed prefix. + if [ -n "$DOWN" ]; then + log "chaotic: restarting last-downed $DOWN and settling before final verdict" + start_node "$DOWN" + wait_for_full_convergence \ + || log "chaotic: did not fully reconverge before verdict (any residue is counted via walcompare)" + fi +} + # Drive a SPECIFIC node to primary — used to exercise leadership on the # lowest-priority node (ctrl-3), which the natural single-failover loop NEVER # elects: the election timeout is priority-staggered (my_priority * step_ms, p0 @@ -397,6 +583,34 @@ submit_jobs "$NJOBS" || fail "initial job submission did not get $NJOBS clean ac verify_no_loss || fail "initial joblist verification failed" walcheck "$((1 + 2 * ${#ALL_JOBIDS[@]}))" +# ── Chaotic mode: overlapping-kill rounds → isolated verdict ────────────────── +# Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up +# windows; outcomes are COUNTED (committed divergence + committed job loss) with +# bug-854 (client discovery) and bug-924 (overlayfs fdatasync) ISOLATED. Emits a +# single machine-parseable VERDICT line plus the PASS/FAIL oracle line, then +# exits — it does NOT run the sequential loop or leadership-coverage maneuver. +if [ "$MODE" = "chaotic" ]; then + log "MODE=chaotic — overlapping-kill soak ($ROUNDS rounds)" + run_chaotic_rounds + # Final divergence + loss sweep on the settled cluster, then bug-924 tally. + walcheck_count "$((1 + 2 * ${#ALL_JOBIDS[@]}))" + verify_classify + scan_bug924 + verdict_emit "chaotic" "$ROUNDS" "${#ALL_JOBIDS[@]}" \ + "$V_DIVERGENCE" "$V_JOBLOSS" "$V_BUG854" "$V_BUG924" + VERDICT_RC=$? + if [ "$VERDICT_RC" -eq 0 ]; then + log "PASS (chaotic): $ROUNDS rounds, ${#ALL_JOBIDS[@]} jobs, zero committed divergence, zero committed job loss (bug854=$V_BUG854 bug924=$V_BUG924 excluded)" + cleanup + trap - EXIT INT TERM + exit 0 + fi + echo "[xce-test-ha] FAIL: chaotic verdict — committed_divergence=$V_DIVERGENCE committed_job_loss=$V_JOBLOSS (bug854=$V_BUG854 bug924=$V_BUG924 excluded)" >&2 + cleanup + trap - EXIT INT TERM + exit 1 +fi + # ── Steps 7–10 looped: kill primary, verify, submit, restart, reconverge ───── # # Each round kills the CURRENT primary, then restarts the node downed in the diff --git a/.claude/skills/xce-test-ha/soak-verdict-selftest.sh b/.claude/skills/xce-test-ha/soak-verdict-selftest.sh new file mode 100644 index 00000000..1674e33a --- /dev/null +++ b/.claude/skills/xce-test-ha/soak-verdict-selftest.sh @@ -0,0 +1,159 @@ +#!/usr/bin/env bash +# +# soak-verdict-selftest.sh — deterministic unit test for the verdict parser in +# soak-verdict.sh. Feeds FIXTURE output (a committed `walcompare` divergence, an +# uncommitted-tail divergence, a bug-854 client-discovery failure, and bug-924 +# `fdatasync FAILED` log lines) and asserts the verdict: +# - counts the committed divergence and committed job-loss (real failures), and +# - EXCLUDES bug-854 (client cannot locate the primary) and bug-924 (overlayfs +# fdatasync) from the divergence / job-loss result. +# +# No cluster, no binaries — pure bash, runs anywhere in well under a second. This +# is the RED→GREEN proof for the verdict logic (task 1.2): the full live chaotic +# soak is the separate acceptance gate (task 7.2). +# +# Usage: bash soak-verdict-selftest.sh (also reachable as run-ha-soak.sh --selftest) +set -u + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=soak-verdict.sh +. "$HERE/soak-verdict.sh" || { + echo "SELFTEST FAIL: cannot source soak-verdict.sh (verdict library missing)" >&2 + exit 1 +} + +fails=0 +check() { # desc expected actual + if [ "$2" = "$3" ]; then + echo "ok - $1" + else + echo "FAIL - $1: expected [$2] got [$3]" + fails=$((fails + 1)) + fi +} + +# ── Fixtures (captured/synthetic soak output) ──────────────────────────────── + +# A real committed-prefix divergence: member 1 holds a different (term, pcrc) at +# committed LSN 9 (the bug-851 family signature). Task 1.1's machine line marks +# it class=committed → a FATAL, COUNTED divergence. +WALCMP_COMMITTED_DIV="$(cat <<'EOF' +== walcompare == +WALCMP member=0 lsn=9 term=3 pcrc=deadbeef status=present committed=1 +WALCMP member=1 lsn=9 term=4 pcrc=cafebabe status=present committed=1 +WALCMP member=2 lsn=9 term=3 pcrc=00000000 status=MISSING committed=1 +WALCMP_DIVERGENCE lsn=9 class=committed compared=3 present=2 +*** DIVERGENCE at committed LSN 9 (compared 3, present 2) *** +walcompare: FAIL — committed-prefix divergence at LSN 9 +EOF +)" + +# An uncommitted-TAIL divergence (above the committed boundary). NON-fatal — it +# must NOT be counted toward the committed-divergence verdict. +WALCMP_UNCOMMITTED_DIV="$(cat <<'EOF' +WALCMP member=0 lsn=11 term=5 pcrc=11111111 status=present committed=0 +WALCMP member=1 lsn=11 term=6 pcrc=22222222 status=present committed=0 +WALCMP_DIVERGENCE lsn=11 class=uncommitted compared=2 present=2 +walcompare: OK +EOF +)" + +# A clean committed prefix (no divergence). +WALCMP_CLEAN="$(cat <<'EOF' +WALCMP member=0 lsn=3 term=2 pcrc=000000ab status=present committed=1 +WALCMP member=1 lsn=3 term=2 pcrc=000000ab status=present committed=1 +walcompare: OK +EOF +)" + +# bug-924: overlayfs fdatasync EIO. Two ERROR lines from persist.c. +FDATASYNC_LOG="$(cat <<'EOF' +[INFO] persist: appended lsn=7 +[ERROR] persist: flusher fdatasync FAILED (errno=5 Input/output error) — durable_lsn held +[ERROR] persist: persist_flush fdatasync FAILED (errno=5 Input/output error) — durable_lsn +EOF +)" + +# A clean controller log (no fdatasync errors). +CLEAN_LOG="$(cat <<'EOF' +[INFO] HA FSM (raft): → CTRL_PRIMARY (term=4, ...) +[INFO] persist: appended lsn=9 durable +EOF +)" + +# expect-job output: one tracked job is genuinely missing on a reachable primary. +EXPECT_LOSS="$(cat <<'EOF' +expect-job: ok (job-1) +expect-job: job job-2 not found +EOF +)" + +# expect-job output: every tracked job present. +EXPECT_OK="$(cat <<'EOF' +expect-job: ok (job-1) +expect-job: ok (job-2) +EOF +)" + +# The client connected but the primary vanished mid-command → no expect-job +# results at all (a discovery race — bug-854 class, EXCLUDED, not a loss). +EXPECT_VANISHED="$(cat <<'EOF' +wait-primary: 30000 ms elapsed, no primary +EOF +)" + +# ── Assertions ─────────────────────────────────────────────────────────────── + +check "committed divergence is counted" \ + 1 "$(printf '%s' "$WALCMP_COMMITTED_DIV" | verdict_count_committed_divergence)" +check "uncommitted-tail divergence is EXCLUDED" \ + 0 "$(printf '%s' "$WALCMP_UNCOMMITTED_DIV" | verdict_count_committed_divergence)" +check "clean walcompare => 0 committed divergence" \ + 0 "$(printf '%s' "$WALCMP_CLEAN" | verdict_count_committed_divergence)" + +check "bug-924 fdatasync errors are counted" \ + 2 "$(printf '%s' "$FDATASYNC_LOG" | verdict_count_fdatasync_bug924)" +check "clean log => 0 bug-924 errors" \ + 0 "$(printf '%s' "$CLEAN_LOG" | verdict_count_fdatasync_bug924)" + +check "reachable + not-found => LOSS (real)" \ + LOSS "$(printf '%s' "$EXPECT_LOSS" | verdict_classify_verify 1)" +check "unreachable => BUG854 (excluded, not loss)" \ + BUG854 "$(printf '%s' "$EXPECT_LOSS" | verdict_classify_verify 0)" +check "reachable + all present => OK" \ + OK "$(printf '%s' "$EXPECT_OK" | verdict_classify_verify 1)" +check "reachable but no expect-job results => BUG854 (mid-command discovery race)" \ + BUG854 "$(printf '%s' "$EXPECT_VANISHED" | verdict_classify_verify 1)" + +# Verdict-level: a committed divergence is FAIL even amid bug-854/bug-924 noise. +v="$(verdict_emit chaotic 6 18 1 0 3 2)"; rc=$? +check "committed divergence => result=FAIL token" \ + "FAIL" "$(printf '%s' "$v" | grep -oE 'result=[A-Z]+' | cut -d= -f2)" +check "committed divergence => non-zero rc" 1 "$rc" + +# THE KEY ASSERTION: with zero real divergence and zero real loss, the verdict is +# PASS even though bug-854 (3) and bug-924 (2) both fired — i.e. the verdict's +# divergence/job-loss result is UNAFFECTED by bug-854/bug-924 (R8.4). +v="$(verdict_emit chaotic 6 18 0 0 3 2)"; rc=$? +check "div=0 loss=0 with bug854+bug924 noise => result=PASS token" \ + "PASS" "$(printf '%s' "$v" | grep -oE 'result=[A-Z]+' | cut -d= -f2)" +check "div=0 loss=0 with bug854+bug924 noise => zero rc" 0 "$rc" +check "verdict line carries the excluded bug854 count" \ + "excluded_bug854_discovery=3" "$(printf '%s' "$v" | grep -oE 'excluded_bug854_discovery=[0-9]+')" +check "verdict line carries the excluded bug924 count" \ + "excluded_bug924_fdatasync=2" "$(printf '%s' "$v" | grep -oE 'excluded_bug924_fdatasync=[0-9]+')" + +# A real committed job-loss is FAIL regardless of how few excluded faults fired. +v="$(verdict_emit chaotic 6 18 0 2 0 0)"; rc=$? +check "committed job loss => non-zero rc" 1 "$rc" +check "committed job loss => result=FAIL token" \ + "FAIL" "$(printf '%s' "$v" | grep -oE 'result=[A-Z]+' | cut -d= -f2)" + +# ── Verdict ────────────────────────────────────────────────────────────────── +echo +if [ "$fails" -eq 0 ]; then + echo "SELFTEST PASS" + exit 0 +fi +echo "SELFTEST FAIL ($fails failed)" +exit 1 diff --git a/.claude/skills/xce-test-ha/soak-verdict.sh b/.claude/skills/xce-test-ha/soak-verdict.sh new file mode 100644 index 00000000..c364965c --- /dev/null +++ b/.claude/skills/xce-test-ha/soak-verdict.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# +# soak-verdict.sh — pure (side-effect-free) verdict helpers for run-ha-soak.sh. +# +# These functions PARSE captured output and emit counts / a classification; they +# never touch the cluster, so they are unit-testable against fixture text +# (soak-verdict-selftest.sh). The chaotic soak sources this file and uses the +# helpers to build a machine-parseable verdict whose committed-divergence and +# committed-job-loss counts are UNAFFECTED by two known, out-of-scope defects: +# +# - bug-854 (libxce client primary-discovery): the client cannot LOCATE the +# serving primary. This is a discovery fault, NOT a server losing committed +# state — so a "could not reach the primary" outcome is counted separately +# and EXCLUDED from the job-loss result. +# - bug-924 (overlayfs `fdatasync FAILED (errno=5/EIO)`): an environment defect +# of the sandbox filesystem, not the controller. Its log lines are counted +# separately and EXCLUDED from the divergence/job-loss result (the acceptance +# soak, task 7.2, runs on a real fs where bug-924 does not occur). +# +# Source this file; do not execute it directly. The terminal verdict is emitted +# by verdict_emit(). + +# Count committed-class `walcompare` divergences in the text on stdin. +# Consumes task 1.1's machine line: +# WALCMP_DIVERGENCE lsn= class=committed compared= present=

+# Only class=committed lines are counted; an uncommitted-tail divergence is +# non-fatal (above the committed boundary) and is deliberately NOT counted. +verdict_count_committed_divergence() { + local n + n="$(grep -c -E '^WALCMP_DIVERGENCE .*class=committed' || true)" + echo "${n:-0}" +} + +# Count bug-924 overlayfs `fdatasync FAILED` ERROR lines in the log text on +# stdin (persist.c emits "persist: ... fdatasync FAILED (errno=...)"). Reported +# as an EXCLUDED count — it never feeds the divergence/job-loss verdict. +verdict_count_fdatasync_bug924() { + local n + n="$(grep -c -E 'fdatasync FAILED' || true)" + echo "${n:-0}" +} + +# Classify one verify (expect-job) attempt. Reads the expect-job command OUTPUT +# on stdin; $1 = primary_reachable (1 iff client-side discovery already located a +# serving primary). Echoes exactly one token: +# BUG854 — the client could not reach a primary, OR the primary vanished +# mid-command so NO expect-job result was produced (discovery fault, +# EXCLUDED — never counted as committed-job loss) +# LOSS — the primary was reachable but a tracked committed job is "not found" +# (a REAL server-side committed-job loss, bug-828 class) +# OK — the primary was reachable and every tracked job is present +# This is the seam that separates "the server lost a committed job" (real) from +# "the client could not locate the primary" (bug-854, excluded). +verdict_classify_verify() { + local reachable="$1" out + out="$(cat)" + if [ "$reachable" != "1" ]; then + echo BUG854 + return 0 + fi + # Reachable per discovery, but if the verify command yielded NO expect-job + # results at all, the client lost the primary mid-command (a discovery race) + # — classify as bug-854, not a loss. + if ! printf '%s' "$out" | grep -q "expect-job:"; then + echo BUG854 + return 0 + fi + if printf '%s' "$out" | grep -q "not found"; then + echo LOSS + return 0 + fi + echo OK +} + +# Emit the single machine-parseable verdict line. PASS iff committed_divergence +# == 0 AND committed_job_loss == 0; the excluded bug-854/bug-924 counts are +# REPORTED for visibility but do NOT affect the pass/fail decision. Returns 0 on +# PASS, 1 on FAIL. +# Args: mode rounds jobs committed_divergence committed_job_loss bug854 bug924 +verdict_emit() { + local mode="$1" rounds="$2" jobs="$3" div="$4" loss="$5" b854="$6" b924="$7" + local result="PASS" + if [ "$div" -ne 0 ] || [ "$loss" -ne 0 ]; then + result="FAIL" + fi + echo "[xce-test-ha] VERDICT mode=$mode rounds=$rounds jobs=$jobs" \ + "committed_divergence=$div committed_job_loss=$loss" \ + "excluded_bug854_discovery=$b854 excluded_bug924_fdatasync=$b924" \ + "result=$result" + [ "$result" = "PASS" ] +} diff --git a/.kiro/specs/raft-ha-core-safety/tasks.md b/.kiro/specs/raft-ha-core-safety/tasks.md index 0700cf9a..45339afc 100644 --- a/.kiro/specs/raft-ha-core-safety/tasks.md +++ b/.kiro/specs/raft-ha-core-safety/tasks.md @@ -13,7 +13,7 @@ - _Requirements: 7.1, 7.5_ - _Boundary: Deterministic Chaotic Harness_ -- [ ] 1.2 Add a chaotic soak mode with an isolated verdict +- [x] 1.2 Add a chaotic soak mode with an isolated verdict - Add an overlapping-kill mode to the soak driver (kills landing inside election windows, not only sequential kill→wait→rejoin); produce a verdict that counts `walcompare` divergences and committed-job loss while excluding client-discovery (bug-854) and overlayfs `fdatasync` (bug-924) failures from the divergence/job-loss result - Observable completion: the soak runs chaotic mode against real-fs release binaries and emits a verdict whose divergence/job-loss counts are unaffected by bug-854/bug-924 - _Requirements: 7.5, 8.4_