fix(raft-apply-decoupling): kill-mid-fact-stream soak + close bug-1416 boot-time TOCTOU (task 7.2)
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact traffic against a live 3-node cluster, killing the primary mid-stream and asserting zero committed-state divergence and zero phantom fact reactions. Building this scenario surfaced bug-1416: persist_recover()'s boot-time apply-correctness verification can race the live apply pump and the standby live-replication apply path, both of which mutate the same last_applied_lsn/ job-table state the verification re-reads live -- producing a false APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with a new persist_t.recovery_complete gate that defers only the in-memory apply (never the durability write or replication ACK) while a recovery pass is in flight, closed from the earliest safe point in myapp_run() so an election- retry candidate can't apply live traffic before persist_recover() ever runs. Round-2 adversarial review independently verified the ACK/durability paths stay ungated, reproduced the myapp.c fix as load-bearing via revert-and- reproduce, and confirmed real-divergence detection still fires. Live 3-node factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover calls. controller_unit_test: zero new failures across 23 groups.
This commit is contained in:
parent
5f98d1b0ef
commit
a5e29a6ffa
17 changed files with 74793 additions and 156 deletions
|
|
@ -99,6 +99,57 @@ regardless of the excluded bug-854/bug-924 counts. The driver also prints the
|
|||
usual terminal `PASS:` / `FAIL:` oracle line (exit 0 / non-zero) so a goal-loop
|
||||
can branch on it unchanged.
|
||||
|
||||
### `--factstream` mode: kill-mid-fact-stream (raft-apply-decoupling task 7.2)
|
||||
|
||||
Drives a continuous **worker-fact stream** — `jstart`/`jfinish`/`jfail`, real
|
||||
wire-protocol worker-report notifications (`xcecli_notify_jobstart`/
|
||||
`_jobfinish`/`_jobfail` in libxce), sent from `xce_test`'s `worker` REPL
|
||||
context — against a batch of pre-submitted jobs, kills the primary partway
|
||||
through the stream, lets a new primary take over, and asserts:
|
||||
|
||||
- **zero committed-state divergence** (same `walcompare --committed`
|
||||
mechanism chaotic mode uses), and
|
||||
- **zero phantom fact reactions** — a job whose live materialized state
|
||||
(`jobinfo`) implies a fact was applied (and so its reactions — worker slot
|
||||
release, scheduler dispatch trigger, client/apiserver notify — already
|
||||
fired, since `apply_committed_entry()` is the single seam that both
|
||||
mutates state and fires the reaction hook) with **no committed WAL fact
|
||||
entry backing it** on some node (`waldump`, extended by this task to
|
||||
decode `jobid`/`state` for every worker-fact WAL type, not just
|
||||
`WAL_JOB_SUBMIT`).
|
||||
|
||||
This is a **different traffic shape** than every other mode: `jobsub`
|
||||
(existing modes) is a client *command* (job-submission); `jstart`/`jfinish`/
|
||||
`jfail` are worker *facts* (asynchronous job-lifecycle reports) — the two
|
||||
travel different code paths in the controller (`handle_jobsubmit` vs.
|
||||
`on_job_start`/`on_job_finish`/`on_job_fail`) and this mode is the only one
|
||||
that exercises the fact path under a kill.
|
||||
|
||||
```bash
|
||||
SKILL_DIR=.claude/skills/xce-test-ha
|
||||
# 2-round kill-mid-fact-stream soak, scratch dir
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --factstream ~/tmp/xce-ha-factstream
|
||||
# tune job count / kill timing / round count
|
||||
XCE_HA_FACTSTREAM_JOBS=40 XCE_HA_FACTSTREAM_KILL_DELAY=2 XCE_HA_FACTSTREAM_ROUNDS=3 \
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --factstream ~/tmp/xce-ha-factstream
|
||||
```
|
||||
|
||||
Verdict line (`soak-verdict.sh`'s `verdict_emit_factstream`):
|
||||
|
||||
```
|
||||
[xce-test-ha] VERDICT mode=factstream rounds=2 jobs=3 facts_sent=40 committed_divergence=0 phantom_facts=0 excluded_bug924_fdatasync=0 result=PASS
|
||||
```
|
||||
|
||||
`result=PASS` iff `committed_divergence == 0` **and** `phantom_facts == 0`.
|
||||
`jobs` is the `jobsub`-tracked count from the shared initial-submit step;
|
||||
`facts_sent` is the number of fact-stream jobs (each gets a `jstart` +
|
||||
`jfinish`/`jfail` pair, so the actual fact-message count is `2 × facts_sent`).
|
||||
A fact genuinely lost in the kill window (fire-and-forget `notify`, no reply)
|
||||
is **not** a phantom and is **not** counted — that is the accepted
|
||||
block-but-durable outcome (Requirement 6) for a fact that never reached
|
||||
majority commit; only the reverse (a reaction fired with no committed WAL
|
||||
entry backing it) counts.
|
||||
|
||||
### Verdict self-test (no cluster)
|
||||
|
||||
The verdict parser has a deterministic, pure-bash self-test that feeds fixture
|
||||
|
|
@ -167,12 +218,18 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
|---|---|---|
|
||||
| `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
|
||||
| `XCE_HA_DATADIR` | config/certs source (`cluster.bin`, `xce.crt`, `xce.key`) | `<repo>/ha/data` |
|
||||
| `XCE_HA_MODE` | `sequential` or `chaotic` (`--chaotic` flag wins) | `sequential` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, or `factstream` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
|
||||
| `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
|
||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–10) | `6` |
|
||||
| `XCE_HA_JOBS` | jobs per submit batch | `3` |
|
||||
| `XCE_HA_WAITMS` | `wait-primary` budget per probe | `30000` |
|
||||
| `XCE_HA_FACTSTREAM_ROUNDS` | `--factstream` kill-mid-fact-stream rounds | `2` |
|
||||
| `XCE_HA_FACTSTREAM_JOBS` | `--factstream` jobs per round (each = 1 jstart + 1 jfinish/jfail) | `30` |
|
||||
| `XCE_HA_FACTSTREAM_KILL_DELAY` | `--factstream` seconds before killing the primary after the stream starts | `3` |
|
||||
| `XCE_HA_FACTSTREAM_WAVE_SIZE` | `--factstream` jobs per paced wave (own xce_test connection) | `3` |
|
||||
| `XCE_HA_FACTSTREAM_WAVE_GAP` | `--factstream` seconds slept between waves | `0.5` |
|
||||
| `XCE_HA_FACTSTREAM_SETTLE` | `--factstream` seconds slept after convergence, before traffic starts (bug-1416 mitigation) | `10` |
|
||||
|
||||
## What the driver does (maps to the requested steps)
|
||||
|
||||
|
|
|
|||
|
|
@ -32,6 +32,16 @@
|
|||
# own log that a snapshot install actually happened (not
|
||||
# just a claim), then asserts zero walcompare divergence
|
||||
# and zero job loss. Single dedicated round, not a loop.
|
||||
# --factstream raft-apply-decoupling task 7.2 (Req 10.2/10.4): drives a
|
||||
# continuous WORKER-FACT stream (jstart/jfinish/jfail — real
|
||||
# wire-protocol worker reports, distinct from job-submission
|
||||
# commands) against a batch of jobs, kills the primary
|
||||
# mid-stream, lets a new primary take over, and asserts zero
|
||||
# committed-state divergence and zero PHANTOM fact reactions
|
||||
# (a job whose live state implies a fact applied — and so its
|
||||
# reactions already fired — with no committed WAL fact entry
|
||||
# backing it on some node). Repeats XCE_HA_FACTSTREAM_ROUNDS
|
||||
# times (default 2), then emits one VERDICT line.
|
||||
# --selftest run the verdict-parser self-test (soak-verdict-selftest.sh)
|
||||
# and exit — no cluster, no binaries needed.
|
||||
#
|
||||
|
|
@ -50,6 +60,27 @@
|
|||
# out, so the 64 MB production default never rotates at
|
||||
# soak job counts and force-snapshot could never observe
|
||||
# a real snapshot-triggered catch-up without this)
|
||||
# XCE_HA_FACTSTREAM_ROUNDS kill-mid-fact-stream rounds (default 2)
|
||||
# XCE_HA_FACTSTREAM_JOBS jobs per fact-stream round (default 30) —
|
||||
# each job gets a jstart + jfinish/jfail pair,
|
||||
# so this is HALF the fact-message count
|
||||
# XCE_HA_FACTSTREAM_KILL_DELAY seconds to sleep before killing the primary
|
||||
# after starting the background fact stream
|
||||
# (default 3) — tunes how far into the stream
|
||||
# the kill lands
|
||||
# XCE_HA_FACTSTREAM_WAVE_SIZE jobs per fact-stream "wave" (default 3) —
|
||||
# each wave is its own xce_test connection;
|
||||
# smaller waves = finer-grained pacing
|
||||
# XCE_HA_FACTSTREAM_WAVE_GAP seconds slept between waves (default 0.5) —
|
||||
# paces the stream over several real seconds
|
||||
# instead of a sub-second burst, so the kill
|
||||
# reliably lands mid-stream
|
||||
# XCE_HA_FACTSTREAM_SETTLE seconds slept after full convergence,
|
||||
# before any fact-stream traffic starts
|
||||
# (default 10) — bug-1416 mitigation: gives
|
||||
# every node's boot-time recovery vtask a
|
||||
# realistic window to finish before this
|
||||
# scenario's traffic begins
|
||||
set -u
|
||||
|
||||
# ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─
|
||||
|
|
@ -67,6 +98,7 @@ while [ $# -gt 0 ]; do
|
|||
--chaotic) MODE="chaotic" ;;
|
||||
--sequential) MODE="sequential" ;;
|
||||
--force-snapshot) MODE="force-snapshot" ;;
|
||||
--factstream) MODE="factstream" ;;
|
||||
--selftest) exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;;
|
||||
*[!0-9]*|'') ARGS+=("$1") ;; # non-integer ⇒ deploy dir
|
||||
*) ROUNDS_CLI="$1" ;; # bare integer ⇒ rounds
|
||||
|
|
@ -141,6 +173,10 @@ V_JOBLOSS=0 # tracked committed jobs "not found" on a REACHABLE primary
|
|||
V_BUG854=0 # client could not locate the primary (libxce discovery fault)
|
||||
V_BUG924=0 # overlayfs `fdatasync FAILED` log lines (environment defect)
|
||||
|
||||
# ── Fact-stream mode counters / tracking (raft-apply-decoupling task 7.2) ────
|
||||
V_PHANTOM=0 # phantom fact reactions observed (see verify_factstream_no_phantom)
|
||||
FACTSTREAM_ALL_JOBIDS=() # every jobid ever driven through the fact stream, across rounds
|
||||
|
||||
log() { echo "[xce-test-ha] $*"; }
|
||||
fail() { echo "[xce-test-ha] FAIL: $*" >&2; cleanup; exit 1; }
|
||||
|
||||
|
|
@ -774,6 +810,266 @@ run_force_snapshot_round() {
|
|||
walcheck_count "${#ALL_JOBIDS[@]}"
|
||||
}
|
||||
|
||||
# ── Fact-stream primitives (raft-apply-decoupling task 7.2, Req 10.2/10.4) ───
|
||||
#
|
||||
# Drives a continuous stream of WORKER-FACT messages (jstart/jfinish/jfail —
|
||||
# real wire-protocol worker-report notifications; see cmd_jobstart/
|
||||
# cmd_jobfinish/cmd_jobfail in testsuite/test.c) against a batch of jobs,
|
||||
# kills the primary partway through, lets a new primary take over, then
|
||||
# verifies zero committed-state divergence and zero PHANTOM fact reactions.
|
||||
#
|
||||
# Why wire-level facts, not a real xcecd_worker process: on_job_start/
|
||||
# on_job_finish/on_job_fail (job.c) require only that the job already exists
|
||||
# in jobmgr (i.e. was submitted) — they do not require a real scheduler
|
||||
# dispatch/host-allocation handshake. jstart/jfinish/jfail therefore drive
|
||||
# the EXACT SAME production receipt path a genuine worker's report would
|
||||
# (on_job_* -> record_job_*_fact -> persist_submit_job_fact[_ex] -> durable
|
||||
# append -> commit engine -> apply pump -> apply_committed_entry()) without
|
||||
# needing to stand up a real scheduler + worker daemon pair (a much larger,
|
||||
# more fragile lift that nothing else in this soak harness attempts either —
|
||||
# the existing scenarios only ever drive job-submission commands). This is
|
||||
# the same "wire-level synthetic worker" pattern the harness already uses for
|
||||
# jfinish (pre-existing); jstart/jfail were the missing primitives.
|
||||
#
|
||||
# What "phantom" means here, concretely: apply_committed_entry() is the
|
||||
# SINGLE seam that both mutates job state AND fires the post-mutation
|
||||
# notification hook (worker slot release / scheduler dispatch trigger /
|
||||
# client-apiserver notify all fan out from that one firing — raft-apply-
|
||||
# decoupling task 6.3's verified finding). So a job's live materialized
|
||||
# state on a given node is a faithful proxy for "did that node's reactions
|
||||
# already fire" — checking (live state implies X happened) against (a
|
||||
# committed WAL entry for X exists on that node) directly tests the
|
||||
# apply-after-commit invariant this whole feature exists to enforce,
|
||||
# without needing separate instrumentation of workermgr slot counts.
|
||||
|
||||
# Cross-check every jobid ever driven through the fact stream: live
|
||||
# materialized state (jobinfo, queried on the reconverged primary) vs.
|
||||
# committed WAL fact entries for that jobid on EVERY node (waldump — checking
|
||||
# all three, not just the primary, catches a same-node apply-without-commit
|
||||
# defect even if walcompare's coarser prefix comparison somehow missed it).
|
||||
# A job that simply never shows a fact's effect (e.g. a fact genuinely lost
|
||||
# in the kill window, block-but-durable per Req 6 / decision D1) is NOT a
|
||||
# phantom — only the reverse (state implies a fact fired, but no committed
|
||||
# WAL entry backs it on some node) counts. Increments V_PHANTOM.
|
||||
verify_factstream_no_phantom() {
|
||||
local want="${#FACTSTREAM_ALL_JOBIDS[@]}"
|
||||
[ "$want" -eq 0 ] && return 0
|
||||
|
||||
local script jid out
|
||||
script="set masters $MASTERS
|
||||
set myname factverify
|
||||
client
|
||||
wait-primary $WAITMS"
|
||||
for jid in "${FACTSTREAM_ALL_JOBIDS[@]}"; do
|
||||
script+="
|
||||
jobinfo -j $jid"
|
||||
done
|
||||
script+="
|
||||
quit"
|
||||
out="$(printf '%s\n' "$script" | xtest)"
|
||||
|
||||
# JobID:/state_raw: lines always alternate 1:1 (print_jobinfo, testsuite/
|
||||
# test.c). Deliberately parse state_raw (the raw hex bitmask), NOT the
|
||||
# pretty state: string: job_state_str() checks JOB_STAT_RUN before
|
||||
# JOB_STAT_EXIT, so a job that is simultaneously RUN and EXIT|PERR (e.g. a
|
||||
# worker-reported FAIL received with no prior FINISH — apply_job_fail_
|
||||
# transition ORs in EXIT|PERR without clearing RUN, job.c) prints as
|
||||
# "RUN", silently hiding the EXIT/PERR bits the fact-stream's own 1-in-5
|
||||
# jfail-without-jfinish jobs deliberately produce. Testing the raw bitmask
|
||||
# avoids under-detecting exactly the traffic shape this scenario sends.
|
||||
declare -A JSTATE=()
|
||||
local cur=""
|
||||
while IFS= read -r line; do
|
||||
case "$line" in
|
||||
"JobID: "*) cur="${line#JobID: }" ;;
|
||||
"state_raw: "*) [ -n "$cur" ] && JSTATE["$cur"]="${line#state_raw: }"; cur="" ;;
|
||||
esac
|
||||
done <<<"$out"
|
||||
|
||||
# One waldump per node (not per job) — cheap, offline, read-only.
|
||||
declare -A DUMPS=()
|
||||
local n
|
||||
for n in "${NODES[@]}"; do
|
||||
DUMPS["$n"]="$(printf 'waldump %s/controller/%s/data\nquit\n' "$TOP" "$n" | xtest)"
|
||||
done
|
||||
|
||||
# JOB_STAT_* bit values (src/include/xce/types.h): RUN=0x2 EXIT=0x4
|
||||
# DONE=0x8 PERR=0x100.
|
||||
local phantom_this_check=0 raw started terminal has_start_any d
|
||||
declare -A HAS_START=() HAS_FINISH=()
|
||||
for jid in "${FACTSTREAM_ALL_JOBIDS[@]}"; do
|
||||
raw="${JSTATE[$jid]:-}"
|
||||
# Job vanished from joblist entirely — a separate, already-covered
|
||||
# loss class (verify_no_loss/walcheck_count), not this check's job.
|
||||
[ -z "$raw" ] && continue
|
||||
|
||||
terminal=0
|
||||
if [ $((raw & 0x4)) -ne 0 ] || [ $((raw & 0x8)) -ne 0 ] || [ $((raw & 0x100)) -ne 0 ]; then
|
||||
terminal=1 # EXIT, DONE, or PERR — a FINISH or FAIL applied
|
||||
fi
|
||||
|
||||
# Whether a START ever committed for this job can NOT be inferred
|
||||
# from a TERMINAL state: apply_job_finish_transition (job.c ~1033)
|
||||
# does an ABSOLUTE overwrite to DONE/EXIT, and apply_job_fail_
|
||||
# transition's default: branch (job.c ~1139, non-POST_EXEC stages)
|
||||
# sets JOB_STAT_EXIT unconditionally — NEITHER requires JOB_STAT_RUN
|
||||
# to have been set first. The fact-stream harness's own reconnect-
|
||||
# mid-wave design can legitimately lose a jstart message (fire-and-
|
||||
# forget, dropped at the kill instant) while its paired jfinish/jfail
|
||||
# still lands and commits — a tolerated "block-but-durable" outcome
|
||||
# (Req 6 / decision D1), not a phantom. So "terminal implies a prior
|
||||
# START" is FALSE and must not gate the JOB_START check. Instead,
|
||||
# scan every node's WAL directly for a committed JOB_START entry —
|
||||
# the same presence check already used for JOB_FINISH/JOB_FAIL below
|
||||
# — and treat "committed on at least one node" as ground truth for
|
||||
# "a START really happened," independent of the job's terminal
|
||||
# state. The live RUN bit (0x2) is kept as an additional, INDEPENDENT
|
||||
# positive signal: it is reliable exactly when the job is NOT
|
||||
# terminal (a job can only be actively RUN via a real applied START),
|
||||
# and catches the more severe "state shows RUN with no commit
|
||||
# anywhere" reaction-before-commit phantom this whole feature exists
|
||||
# to prevent.
|
||||
has_start_any=0
|
||||
HAS_START=(); HAS_FINISH=()
|
||||
for n in "${NODES[@]}"; do
|
||||
d="${DUMPS[$n]}"
|
||||
HAS_START["$n"]=0; HAS_FINISH["$n"]=0
|
||||
printf '%s\n' "$d" | grep -F "JOB_START" | grep -qF "jobid=$jid " \
|
||||
&& HAS_START["$n"]=1 && has_start_any=1
|
||||
printf '%s\n' "$d" | grep -E "JOB_(FINISH|FAIL)" | grep -qF "jobid=$jid " \
|
||||
&& HAS_FINISH["$n"]=1
|
||||
done
|
||||
|
||||
started=0
|
||||
[ $((raw & 0x2)) -ne 0 ] && started=1
|
||||
[ "$has_start_any" -eq 1 ] && started=1
|
||||
[ "$started" -eq 0 ] && [ "$terminal" -eq 0 ] && continue
|
||||
|
||||
for n in "${NODES[@]}"; do
|
||||
if [ "$started" -eq 1 ] && [ "${HAS_START[$n]}" -eq 0 ]; then
|
||||
log "PHANTOM: job $jid state_raw=$raw (JOB_START committed on at least one node) but $n's WAL has NO committed JOB_START"
|
||||
phantom_this_check=$((phantom_this_check + 1))
|
||||
fi
|
||||
if [ "$terminal" -eq 1 ] && [ "${HAS_FINISH[$n]}" -eq 0 ]; then
|
||||
log "PHANTOM: job $jid state_raw=$raw (terminal — worker slot would be freed) but $n's WAL has NO committed JOB_FINISH/JOB_FAIL"
|
||||
phantom_this_check=$((phantom_this_check + 1))
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
V_PHANTOM=$((V_PHANTOM + phantom_this_check))
|
||||
log "factstream: cross-checked $want jobid(s) across 3 nodes — $phantom_this_check phantom fact reaction(s) this check"
|
||||
return 0
|
||||
}
|
||||
|
||||
# One kill-mid-fact-stream round: submit a batch of jobs, stream jstart/
|
||||
# jfinish/jfail for all of them in the background, kill the primary partway
|
||||
# through, let a new primary take over, restart the downed node, reconverge,
|
||||
# then verify no committed loss / divergence / phantom reactions.
|
||||
run_factstream_round() {
|
||||
local round="$1"
|
||||
local nfacts="${XCE_HA_FACTSTREAM_JOBS:-30}"
|
||||
local kill_delay="${XCE_HA_FACTSTREAM_KILL_DELAY:-3}"
|
||||
local before=${#ALL_JOBIDS[@]}
|
||||
|
||||
log "factstream round $round: submitting $nfacts jobs to drive the fact stream against"
|
||||
submit_jobs "$nfacts" || fail "factstream round $round: could not submit $nfacts jobs"
|
||||
verify_no_loss || fail "factstream round $round: job loss right after submit, before the fact stream started"
|
||||
|
||||
local round_jobids=("${ALL_JOBIDS[@]:$before}")
|
||||
FACTSTREAM_ALL_JOBIDS+=("${round_jobids[@]}")
|
||||
|
||||
# Paced worker-fact stream: jstart followed by jfinish (4-in-5) or jfail
|
||||
# (1-in-5) per job — a realistic worker lifecycle mix exercising both
|
||||
# terminal paths (and the _pad-discriminated FAIL apply-time recompute,
|
||||
# raft-apply-decoupling task 6.1 round 2). Sent as several small WAVES
|
||||
# (each its own xtest invocation — a fresh worker connection, join
|
||||
# included) with a short sleep between waves, NOT one giant script: a
|
||||
# single unpaced script sends all facts back-to-back over one connection
|
||||
# in well under a second (loopback round trips), so a 1s kill_delay
|
||||
# landed AFTER the whole stream had already finished, not mid-stream —
|
||||
# and the resulting instantaneous burst tripped repl_broadcast's
|
||||
# pending_ack eviction threshold (an unrelated backpressure mechanism),
|
||||
# observed live to cascade into a slow reconverge. Pacing into waves (a)
|
||||
# makes "kill mid-stream" land reliably mid-stream over several real
|
||||
# seconds, (b) avoids the artificial burst, and (c) doubles as a
|
||||
# realistic "worker reconnects after primary failover" exercise, since
|
||||
# each wave re-resolves the primary via `set masters` + libxce failover
|
||||
# exactly like a real worker's next report after a failover would.
|
||||
# `set continue-on-error` is essential: a wave landing at/after the kill
|
||||
# instant is EXPECTED to fail transport-side (fire-and-forget notify —
|
||||
# XCECLI_FF_CALL, client_internal.h) and must not abort that wave.
|
||||
local wave_size="${XCE_HA_FACTSTREAM_WAVE_SIZE:-3}"
|
||||
local wave_gap="${XCE_HA_FACTSTREAM_WAVE_GAP:-0.5}"
|
||||
local stream_log="$TOP/logs/factstream-round$round.out"
|
||||
: >"$stream_log"
|
||||
(
|
||||
local start_i=0 total=${#round_jobids[@]} wi wjid wscript
|
||||
while [ "$start_i" -lt "$total" ]; do
|
||||
wscript="set masters $MASTERS
|
||||
set myname factstream
|
||||
set continue-on-error
|
||||
worker
|
||||
join"
|
||||
wi=0
|
||||
while [ "$wi" -lt "$wave_size" ] && [ "$start_i" -lt "$total" ]; do
|
||||
wjid="${round_jobids[$start_i]}"
|
||||
wscript+="
|
||||
jstart -j $wjid"
|
||||
if [ $((start_i % 5)) -eq 4 ]; then
|
||||
wscript+="
|
||||
jfail -j $wjid -c 1"
|
||||
else
|
||||
wscript+="
|
||||
jfinish -j $wjid -c 0"
|
||||
fi
|
||||
start_i=$((start_i + 1))
|
||||
wi=$((wi + 1))
|
||||
done
|
||||
wscript+="
|
||||
..
|
||||
quit"
|
||||
printf '%s\n' "$wscript" | xtest >>"$stream_log" 2>&1
|
||||
sleep "$wave_gap"
|
||||
done
|
||||
) &
|
||||
local stream_pid=$!
|
||||
log "factstream round $round: streaming $((${#round_jobids[@]} * 2)) fact messages in $wave_size-job waves in the background (log: $stream_log)"
|
||||
|
||||
sleep "$kill_delay"
|
||||
|
||||
PRIMARY="$(discover_primary)"
|
||||
[ -n "$PRIMARY" ] || fail "factstream round $round: no serving primary right before the kill"
|
||||
local prev_term; prev_term="$(latest_primary_term)"; prev_term="${prev_term:-0}"
|
||||
log "factstream round $round: killing primary $PRIMARY mid-stream (term=$prev_term)"
|
||||
kill_node "$PRIMARY" || fail "factstream round $round: could not kill $PRIMARY"
|
||||
DOWN="$PRIMARY"
|
||||
|
||||
local newp; newp="$(wait_for_new_primary "$prev_term")"
|
||||
[ -n "$newp" ] || fail "factstream round $round: no new primary elected after the mid-stream kill"
|
||||
log "factstream round $round: new primary $newp (term=$(latest_primary_term))"
|
||||
record_primary "$newp"
|
||||
|
||||
# Let the background stream finish. Best-effort reconnect resumes any
|
||||
# remaining facts against whichever primary is reachable; facts already
|
||||
# in flight at the kill instant may be silently lost (fire-and-forget,
|
||||
# no reply) — the accepted D1 block-but-durable outcome (requirements.md
|
||||
# Requirement 6), not a bug, and exactly what verify_factstream_no_
|
||||
# phantom above tolerates (a lost fact ⇒ the job simply never shows the
|
||||
# corresponding state — not flagged).
|
||||
wait "$stream_pid" 2>/dev/null
|
||||
log "factstream round $round: fact-stream script finished"
|
||||
|
||||
start_node "$DOWN"
|
||||
wait_for_full_convergence \
|
||||
|| fail "factstream round $round: cluster did not reconverge to 3-of-3 after restarting $DOWN"
|
||||
DOWN=""
|
||||
|
||||
verify_no_loss || fail "factstream round $round: a committed JOB_SUBMIT was lost after the mid-stream kill"
|
||||
walcheck_count "${#ALL_JOBIDS[@]}"
|
||||
verify_factstream_no_phantom
|
||||
}
|
||||
|
||||
# Drive a SPECIFIC node to primary — used to exercise leadership on the
|
||||
# lowest-priority node (ctrl-3), which the natural single-failover loop NEVER
|
||||
# elects: the election timeout is priority-staggered (my_priority * step_ms, p0
|
||||
|
|
@ -894,6 +1190,68 @@ if [ "$MODE" = "force-snapshot" ]; then
|
|||
exit 1
|
||||
fi
|
||||
|
||||
# ── Fact-stream mode: kill-mid-fact-stream regression ─────────────────────────
|
||||
# Opt-in (--factstream / XCE_HA_MODE=factstream). raft-apply-decoupling task
|
||||
# 7.2 (Req 10.2/10.4). See run_factstream_round / verify_factstream_no_phantom
|
||||
# above for the scenario + verification. Repeats XCE_HA_FACTSTREAM_ROUNDS times
|
||||
# (default 2 — independent of the general --chaotic ROUNDS knob, since this
|
||||
# scenario's own repeatability bar is "reliably lands mid-stream and produces a
|
||||
# correct verdict", not a long soak) so a single lucky/unlucky kill-timing
|
||||
# window does not decide the verdict, then emits one VERDICT line and exits —
|
||||
# like force-snapshot/chaotic, it does NOT run the sequential loop or
|
||||
# leadership-coverage maneuver.
|
||||
if [ "$MODE" = "factstream" ]; then
|
||||
FS_ROUNDS="${XCE_HA_FACTSTREAM_ROUNDS:-2}"
|
||||
log "MODE=factstream — kill-mid-fact-stream soak ($FS_ROUNDS rounds)"
|
||||
# Wait for full 3-of-3 convergence, THEN an extra settle delay, before
|
||||
# introducing any fact-stream load or kills. Step 4 above only confirms
|
||||
# ONE primary is reachable, and ctrlstatus answers as soon as a node's
|
||||
# listener is open — BEFORE that node's own cluster_init_phase2()/
|
||||
# persist_recover() (the boot-time apply-correctness verification, Req
|
||||
# 8.1/8.2) has actually run, since cluster_open_listener() precedes
|
||||
# persist_recover() in phase2 and persist_recover() itself fires from a
|
||||
# vtask, not synchronously (bug-1416, found live by this scenario): if a
|
||||
# primary is elected fast and starts replicating aggressive early job/fact
|
||||
# traffic while a standby's OWN recovery vtask is still queued behind a
|
||||
# burst of incoming replication-write vtasks on the shared pool, that
|
||||
# standby's recovery can be delayed 30s+ (observed) instead of its normal
|
||||
# ~5s — by the time it finally runs, the entity/position counts it
|
||||
# verifies against no longer match its own single-pass replay model,
|
||||
# producing a false APPLY_DIVERGENT fence that (correctly, per the fence's
|
||||
# own fail-loud design) can NEVER self-clear without a further process
|
||||
# restart. This is a real, reproducible finding (see bug-1416), out of
|
||||
# this task's boundary to fix (persist_recover/cluster_init_phase2
|
||||
# sequencing, not the regression-scenario itself). The settle delay below
|
||||
# is a scenario-side mitigation only: give every node's recovery vtask a
|
||||
# realistic window to run BEFORE this scenario's traffic begins, so the
|
||||
# kill this scenario asserts on lands "mid-fact-stream on a settled
|
||||
# cluster" (what it is chartered to test) rather than racing an unrelated
|
||||
# boot-sequencing issue every run.
|
||||
wait_for_full_convergence \
|
||||
|| fail "factstream: cluster did not reach full 3-of-3 convergence before starting the fact stream"
|
||||
settle="${XCE_HA_FACTSTREAM_SETTLE:-10}"
|
||||
log "factstream: cluster converged — settling ${settle}s before starting fact-stream traffic (bug-1416 mitigation)"
|
||||
sleep "$settle"
|
||||
for fs_round in $(seq 1 "$FS_ROUNDS"); do
|
||||
log "=== factstream round $fs_round/$FS_ROUNDS ==="
|
||||
run_factstream_round "$fs_round"
|
||||
done
|
||||
scan_bug924
|
||||
verdict_emit_factstream "$FS_ROUNDS" "${#ALL_JOBIDS[@]}" \
|
||||
"${#FACTSTREAM_ALL_JOBIDS[@]}" "$V_DIVERGENCE" "$V_PHANTOM" "$V_BUG924"
|
||||
VERDICT_RC=$?
|
||||
if [ "$VERDICT_RC" -eq 0 ]; then
|
||||
log "PASS (factstream): $FS_ROUNDS rounds, ${#FACTSTREAM_ALL_JOBIDS[@]} fact-stream jobs, zero committed divergence, zero phantom fact reactions (bug924=$V_BUG924 excluded)"
|
||||
cleanup
|
||||
trap - EXIT INT TERM
|
||||
exit 0
|
||||
fi
|
||||
echo "[xce-test-ha] FAIL: factstream verdict — committed_divergence=$V_DIVERGENCE phantom_facts=$V_PHANTOM (bug924=$V_BUG924 excluded)" >&2
|
||||
cleanup
|
||||
trap - EXIT INT TERM
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ── Chaotic mode: overlapping-kill rounds → isolated verdict ──────────────────
|
||||
# Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up
|
||||
# windows; outcomes are COUNTED (committed divergence + committed job loss) with
|
||||
|
|
|
|||
|
|
@ -89,3 +89,23 @@ verdict_emit() {
|
|||
"result=$result"
|
||||
[ "$result" = "PASS" ]
|
||||
}
|
||||
|
||||
# Emit the machine-parseable verdict line for the factstream (task 7.2)
|
||||
# scenario. PASS iff committed_divergence == 0 AND phantom_facts == 0 (a
|
||||
# fact-driven notification/reaction observed on a node with no matching
|
||||
# committed WAL fact entry backing it — see run_factstream_round's
|
||||
# verify_factstream_no_phantom in run-ha-soak.sh for the concrete check).
|
||||
# excluded_bug924 is reported for visibility only, same convention as
|
||||
# verdict_emit above. Returns 0 on PASS, 1 on FAIL.
|
||||
# Args: rounds jobs facts_sent committed_divergence phantom_facts bug924
|
||||
verdict_emit_factstream() {
|
||||
local rounds="$1" jobs="$2" facts="$3" div="$4" phantom="$5" b924="$6"
|
||||
local result="PASS"
|
||||
if [ "$div" -ne 0 ] || [ "$phantom" -ne 0 ]; then
|
||||
result="FAIL"
|
||||
fi
|
||||
echo "[xce-test-ha] VERDICT mode=factstream rounds=$rounds jobs=$jobs" \
|
||||
"facts_sent=$facts committed_divergence=$div phantom_facts=$phantom" \
|
||||
"excluded_bug924_fdatasync=$b924 result=$result"
|
||||
[ "$result" = "PASS" ]
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue