test(job-array-single-wal-record): large-array HA soak scenario
Extends the HA soak harness with a new --array-soak mode: small (N=2) and large (N=10000) array submission sub-rounds, each with one clean kill/failover, asserting one WAL record per submission regardless of array size, per-element addressability (status/cancel/completion) spread across low/mid/high indices post-failover, and a real millisecond-resolution replay-time bound comparing large-array vs small-array kill-to-addressable latency. waldump.c gains a WAL_JOB_ARRAY_SUBMIT case in wal_type_name() so the scenario can grep an exact type name for its live WAL-record-count delta check. Went through one remediation round: the replay-time measurement initially used whole-second resolution (collapsed to 0s/n-a on a real run) and never asserted on the comparison, giving zero actual evidence for the O(1)-replay-time requirement. Fixed with $EPOCHREALTIME (millisecond resolution -- this sandbox's BusyBox date silently mangles the more common `date +%s%3N` idiom) and a real bound assertion that hard-fails the run on a genuine O(N) regression, independently verified to have real teeth. Live-verified 3 times with genuinely discriminating numbers each time; final run: small=210ms, large=163ms -- large-array replay was not slower, live evidence of the feature's O(1) claim. Task 6.1 of job-array-single-wal-record (Requirements 4.3, 5.2, 5.3, 8.2, 8.3). Depends on and reuses task 6.0a's own soak verification.
This commit is contained in:
parent
16747aa96d
commit
0d8986a6a9
6 changed files with 550 additions and 2 deletions
|
|
@ -151,6 +151,57 @@ block-but-durable outcome (Requirement 6) for a fact that never reached
|
|||
majority commit; only the reverse (a reaction fired with no committed WAL
|
||||
entry backing it) counts.
|
||||
|
||||
### `--array-soak` mode: large-array replay/addressing scenario (job-array-single-wal-record task 6.1)
|
||||
|
||||
Submits a **small** (default 2-element) then a **large** (default 10,000-
|
||||
element) job array via `jobsub -a N`, and — for EACH, regardless of size —
|
||||
asserts:
|
||||
|
||||
- **exactly one `WAL_JOB_ARRAY_SUBMIT` record / one commit barrier** (`waldump`,
|
||||
which this task extended to name WAL type 11 `JOB_ARRAY_SUBMIT` for an exact
|
||||
count instead of an `UNKNOWN`-bucket proxy);
|
||||
- **kill-to-first-addressable failover replay time**, measured at millisecond
|
||||
resolution (bash's `$EPOCHREALTIME` builtin — deliberately not `date +%s%N`,
|
||||
since this environment's plain `date` can resolve to a BusyBox binary that
|
||||
silently passes an unrecognized `%N` through as a literal string instead of
|
||||
erroring), with a **hard pass/fail bound**: the large-array replay time must
|
||||
not exceed `max(t_small_ms * ratio_max, floor_ms) + margin_ms` (defaults
|
||||
`ratio_max=20`, `floor_ms=2000`, `margin_ms=3000` — see
|
||||
`verdict_check_arraysoak_replay_bound` in `soak-verdict.sh`). This is the
|
||||
Req 4.3 assertion (*"failover replay time governed by the record's size,
|
||||
not element count"*) — a genuine O(N)-scaling regression fails the run via
|
||||
`fail()`, it does not just show up as a displayed number; and
|
||||
- **per-element status query, cancel (`jobkill -s 9`), and completion
|
||||
(`jstart`+`jfinish`)** on three distinct LARGE-array elements spread across
|
||||
low/mid/high indices (not clustered near the start), after a real failover.
|
||||
|
||||
Like `--force-snapshot`, this is **one dedicated round, not a loop**, and it
|
||||
hard-fails (not count-and-tolerate) on any violated invariant — these are the
|
||||
feature's core correctness claims, not a racy/sometimes-expected outcome
|
||||
class.
|
||||
|
||||
```bash
|
||||
SKILL_DIR=.claude/skills/xce-test-ha
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --array-soak ~/tmp/xce-ha-arraysoak
|
||||
# tune array sizes / replay-time bound knobs
|
||||
XCE_HA_ARRAYSOAK_SMALL=3 XCE_HA_ARRAYSOAK_LARGE=20000 XCE_HA_ARRAYSOAK_REPLAY_RATIO_MAX=30 \
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --array-soak ~/tmp/xce-ha-arraysoak
|
||||
```
|
||||
|
||||
Verdict line (`soak-verdict.sh`'s `verdict_emit_arraysoak`):
|
||||
|
||||
```
|
||||
[xce-test-ha] VERDICT mode=array-soak n_small=2 t_small_ms=165 n_large=10000 t_large_ms=164 element_count_ratio=5000x replay_time_ratio_x100=99 replay_bound_ms=6300 replay_bound_verdict=OK result=PASS
|
||||
```
|
||||
|
||||
`result` is unconditionally `PASS` here — unlike chaotic/factstream's
|
||||
count-and-classify verdicts, every hard invariant this scenario checks
|
||||
(record count, replay-time bound, per-element addressing) is asserted with an
|
||||
immediate `fail()` inside `run_array_soak_round()`, so a violation aborts the
|
||||
run before this verdict line is ever reached. The line's job is to report the
|
||||
actual measurements (and the bound they were checked against) for visibility,
|
||||
not to gate pass/fail itself.
|
||||
|
||||
### Verdict self-test (no cluster)
|
||||
|
||||
The verdict parser has a deterministic, pure-bash self-test that feeds fixture
|
||||
|
|
@ -247,7 +298,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
|---|---|---|
|
||||
| `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
|
||||
| `XCE_HA_DATADIR` | config/certs source (`cluster.bin`, `xce.crt`, `xce.key`) | `<repo>/ha/data` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, or `factstream` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, or `array-soak` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
|
||||
| `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
|
||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–50) | `6` |
|
||||
|
|
@ -259,6 +310,11 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
| `XCE_HA_FACTSTREAM_WAVE_SIZE` | `--factstream` jobs per paced wave (own xce_test connection) | `3` |
|
||||
| `XCE_HA_FACTSTREAM_WAVE_GAP` | `--factstream` seconds slept between waves | `0.5` |
|
||||
| `XCE_HA_FACTSTREAM_SETTLE` | `--factstream` seconds slept after convergence, before traffic starts (bug-1416 mitigation) | `10` |
|
||||
| `XCE_HA_ARRAYSOAK_SMALL` | `--array-soak` element count of the "small" comparison array | `2` |
|
||||
| `XCE_HA_ARRAYSOAK_LARGE` | `--array-soak` element count of the "large" comparison array | `10000` |
|
||||
| `XCE_HA_ARRAYSOAK_REPLAY_RATIO_MAX` | `--array-soak` max tolerated large/small replay-time ratio before the floor/margin (Req 4.3 bound) | `20` |
|
||||
| `XCE_HA_ARRAYSOAK_REPLAY_FLOOR_MS` | `--array-soak` minimum bound floor in ms before margin is added | `2000` |
|
||||
| `XCE_HA_ARRAYSOAK_REPLAY_MARGIN_MS` | `--array-soak` ms added on top of the scaled/floor bound to absorb election-timing jitter | `3000` |
|
||||
|
||||
## What the driver does (maps to the requested steps)
|
||||
|
||||
|
|
|
|||
|
|
@ -42,6 +42,20 @@
|
|||
# reactions already fired — with no committed WAL fact entry
|
||||
# backing it on some node). Repeats XCE_HA_FACTSTREAM_ROUNDS
|
||||
# times (default 2), then emits one VERDICT line.
|
||||
# --array-soak job-array-single-wal-record task 6.1 (Req 4.3, 5.2, 5.3,
|
||||
# 8.2, 8.3): submits a small then a large job ARRAY
|
||||
# (`jobsub -a N`), asserting for EACH — regardless of array
|
||||
# size — exactly one WAL_JOB_ARRAY_SUBMIT record / one
|
||||
# commit barrier; measures kill-to-first-addressable
|
||||
# failover replay time for both and reports them so a
|
||||
# 10,000-element array's replay can be confirmed NOT to
|
||||
# scale with element count; and proves per-element status
|
||||
# query, cancel (jobkill -s 9), and completion (jstart+
|
||||
# jfinish) on several distinct LARGE-array elements after a
|
||||
# real failover. One dedicated round, not a loop — like
|
||||
# --force-snapshot, hard-fails (not count-and-tolerate) on
|
||||
# any violated invariant, since these are the feature's core
|
||||
# correctness claims, not a chaotic/racy outcome class.
|
||||
# --selftest run the verdict-parser self-test (soak-verdict-selftest.sh)
|
||||
# and exit — no cluster, no binaries needed.
|
||||
#
|
||||
|
|
@ -81,6 +95,28 @@
|
|||
# every node's boot-time recovery vtask a
|
||||
# realistic window to finish before this
|
||||
# scenario's traffic begins
|
||||
# XCE_HA_ARRAYSOAK_SMALL element count of the "small" comparison array
|
||||
# (default 2 — the smallest size that exercises
|
||||
# the new WAL_JOB_ARRAY_SUBMIT path at all;
|
||||
# array_size == 1 stays on the pre-existing
|
||||
# single-job path, task 2.2, so it cannot be used
|
||||
# for an apples-to-apples array-path comparison)
|
||||
# XCE_HA_ARRAYSOAK_LARGE element count of the "large" comparison array
|
||||
# (default 10000, matching Req 4.3's own example)
|
||||
# XCE_HA_ARRAYSOAK_REPLAY_RATIO_MAX Req 4.3 replay-time-bound gate: max
|
||||
# tolerated large-array/small-array kill-to-
|
||||
# addressable time ratio before the scaled floor/
|
||||
# margin (default 20 — generous vs. the ~1x a
|
||||
# real O(1) implementation should show, far
|
||||
# tighter than the ~5000x an O(N) regression at
|
||||
# the default sizes would produce)
|
||||
# XCE_HA_ARRAYSOAK_REPLAY_FLOOR_MS minimum bound floor in ms before
|
||||
# margin is added (default 2000) — keeps the
|
||||
# bound meaningful when t_small itself is a few
|
||||
# tens/hundreds of ms on a fast local cluster
|
||||
# XCE_HA_ARRAYSOAK_REPLAY_MARGIN_MS ms added on top of the scaled/floor
|
||||
# bound to absorb ordinary election-timing jitter
|
||||
# shared by both sub-rounds (default 3000)
|
||||
set -u
|
||||
|
||||
# ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─
|
||||
|
|
@ -99,6 +135,7 @@ while [ $# -gt 0 ]; do
|
|||
--sequential) MODE="sequential" ;;
|
||||
--force-snapshot) MODE="force-snapshot" ;;
|
||||
--factstream) MODE="factstream" ;;
|
||||
--array-soak) MODE="array-soak" ;;
|
||||
--selftest) exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;;
|
||||
*[!0-9]*|'') ARGS+=("$1") ;; # non-integer ⇒ deploy dir
|
||||
*) ROUNDS_CLI="$1" ;; # bare integer ⇒ rounds
|
||||
|
|
@ -1141,6 +1178,299 @@ quit"
|
|||
verify_factstream_no_phantom
|
||||
}
|
||||
|
||||
# ── Array-soak primitives (job-array-single-wal-record task 6.1) ────────────
|
||||
#
|
||||
# Proves, against a REAL 3-node cluster and a REAL failover — not just the
|
||||
# existing white-box unit tests (task 2.3/3.4 already prove WAL-record-count
|
||||
# and idempotent-expansion in isolation) — three of this feature's live
|
||||
# claims:
|
||||
# (a) exactly one WAL_JOB_ARRAY_SUBMIT record / one commit barrier per
|
||||
# array submission, regardless of array size (Req 1.1-1.3)
|
||||
# (b) failover replay time governed by the RECORD's size, not element count
|
||||
# (Req 4.3) — measured as wall-clock kill-to-first-addressable and
|
||||
# compared between a small and a large array, BOTH on the array-record
|
||||
# code path (array_size > 1; array_size == 1 stays on the pre-existing
|
||||
# single-job path per task 2.2, so it cannot serve as an apples-to-
|
||||
# apples comparison point for this specific claim)
|
||||
# (c) every element individually addressable — status query, cancel,
|
||||
# completion — after a failover (Req 5.2), proven against several
|
||||
# distinct elements of the LARGE array, not just index 0
|
||||
#
|
||||
# Per-element addressing needs no new xce_test command: `jobinfo`/
|
||||
# `expect-job`/`jobkill`'s `-j`/positional argument already parses the
|
||||
# bracketed array-id format (utils.c parse_jobid, REGEX_JOBARRAY_ID). The
|
||||
# base jobid `jobsub -a N` captures into $jobid is array_jobid()-stripped
|
||||
# (tag cleared before printing, per cmd_jobsub/test.c), so it prints WITHOUT
|
||||
# brackets (e.g. "12-3-1700000000-32768") — element i's addressable string is
|
||||
# that exact base string with "[i]" appended: parse_jobid reconstructs
|
||||
# base_jobid_with_tag + i from "<base>[i]", the identical arithmetic
|
||||
# restore_job_array() (job.c) itself uses for jobid = base_jobid + i. Traced
|
||||
# through utils.h's MAKE_JOBID/EXTRACT_ARRAY_* bit-layout macros to confirm
|
||||
# this is exact, not an approximation.
|
||||
|
||||
# Submit ONE array of size $1 via `jobsub -a N`. Echoes the captured base
|
||||
# jobid string (bracket-less) on success; returns 1 with nothing echoed on
|
||||
# failure. Deliberately does NOT append to ALL_JOBIDS — array elements are
|
||||
# addressed by base+index arithmetic here, not individually tracked the way
|
||||
# submit_jobs() tracks single-job submissions, which also keeps this
|
||||
# scenario's (possibly 10,000-element) array out of every OTHER helper's
|
||||
# O(len(ALL_JOBIDS)) sweep (verify_no_loss, walcheck, ...).
|
||||
submit_array() {
|
||||
local n="$1" out base
|
||||
out="$(printf 'set masters %s\nset myname arraysoak\nclient\nwait-primary %s\njobsub -u root -q normal -c /bin/true -a %s\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" "$n" | xtest)"
|
||||
base="$(printf '%s\n' "$out" | sed -n 's/^jobsubmit ok: JobId: \(.*\)$/\1/p' | head -1)"
|
||||
[ -n "$base" ] || return 1
|
||||
echo "$base"
|
||||
}
|
||||
|
||||
# Count WAL_JOB_ARRAY_SUBMIT entries physically present in node $1's on-disk
|
||||
# WAL right now. Relies on this task's own waldump.c change (wal_type_name()
|
||||
# now names type 11 "JOB_ARRAY_SUBMIT") for an EXACT count — not an
|
||||
# "UNKNOWN"-bucket proxy that would also catch a genuinely unrelated type.
|
||||
count_array_wal_records() {
|
||||
local node="$1" out
|
||||
out="$(printf 'waldump %s/controller/%s/data\nquit\n' "$TOP" "$node" | xtest)"
|
||||
printf '%s\n' "$out" | grep -c "JOB_ARRAY_SUBMIT"
|
||||
}
|
||||
|
||||
# Per-element jobid string for array element $2 (0-based) of base string $1.
|
||||
# See this section's header comment for why string concatenation is exact.
|
||||
array_element_id() { echo "$1[$2]"; }
|
||||
|
||||
# `jobinfo -j <id>` against the live primary. Prints the jobinfo text and
|
||||
# returns 0 on a hit; returns 1 (nothing printed) if not found/unreachable.
|
||||
# A direct single-job lookup (xcecli_jobinfo) — unlike expect-job's
|
||||
# O(current job count) joblist-scan-and-match, the right primitive for
|
||||
# checking individual elements of a large array without an O(N) list
|
||||
# round-trip per check. Retries twice (short sleep) to absorb the same
|
||||
# post-election client-retry-storm race verify_no_loss already tolerates.
|
||||
# $2 = wait-primary budget in ms (default $WAITMS).
|
||||
jobinfo_of() {
|
||||
local id="$1" waitms="${2:-$WAITMS}" out attempt
|
||||
for attempt in 1 2 3; do
|
||||
out="$(printf 'set masters %s\nset myname arraysoak\nclient\nwait-primary %s\njobinfo -j %s\nquit\n' \
|
||||
"$MASTERS" "$waitms" "$id" | xtest)"
|
||||
if printf '%s\n' "$out" | grep -q "^JobID: "; then
|
||||
printf '%s\n' "$out"
|
||||
return 0
|
||||
fi
|
||||
[ "$attempt" -lt 3 ] && sleep 0.5
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Millisecond-resolution wall-clock reading via bash's own $EPOCHREALTIME
|
||||
# (bash >= 5.0 builtin, "seconds.microseconds", always 6 fractional digits) —
|
||||
# deliberately NOT `date +%s%3N`/`date +%s%N`: this environment's plain
|
||||
# `date` can resolve to a BusyBox binary ahead of GNU coreutils on $PATH, and
|
||||
# BusyBox `date` silently passes an unrecognized `%N`/`%3N` through as a
|
||||
# LITERAL string instead of erroring (confirmed: `date +%s%3N` => `<epoch>%3N`,
|
||||
# rc=0) — that garbage would poison every downstream arithmetic use. Reading
|
||||
# $EPOCHREALTIME is a pure bash builtin with no subprocess/PATH dependency at
|
||||
# all, so it can't hit that trap. `10#` forces base-10 on the fractional
|
||||
# slice so a leading-zero fraction (e.g. "007123") isn't misread as octal.
|
||||
epoch_ms() {
|
||||
local t="$EPOCHREALTIME" sec frac
|
||||
sec="${t%%.*}"
|
||||
frac="${t#*.}"
|
||||
printf '%d' "$((sec * 1000 + 10#${frac:0:3}))"
|
||||
}
|
||||
|
||||
# Poll jobinfo_of every 0.5s (up to $2 seconds, default 90) until element $1
|
||||
# is addressable. Echoes elapsed MILLISECONDS (epoch_ms above) from this
|
||||
# function's own start to first success — the kill-to-first-addressable
|
||||
# failover replay time measurement (Req 4.3). Millisecond resolution matters
|
||||
# here: on a fast local cluster, whole-second (`date +%s`) measurements can
|
||||
# collapse both the small- and large-array sub-rounds to "0s", which reports
|
||||
# a real number but proves nothing (bug found in review — see
|
||||
# run_array_soak_round's replay-time-bound assertion, which needs a
|
||||
# genuinely discriminating measurement to gate on). The 90s outer timeout
|
||||
# budget itself is tracked via bash's own $SECONDS builtin (reset at this
|
||||
# function's start), NOT by diffing $EPOCHREALTIME pairs inside the loop —
|
||||
# per this project's own logged lesson (.wolf/cerebrum.md, 2026-07-07):
|
||||
# mixing a sub-second precision timestamp into per-iteration deadline
|
||||
# arithmetic is a source of self-inflicted timing bugs; a single
|
||||
# $EPOCHREALTIME pair (start here, end at the success branch) captured once
|
||||
# and diffed once is the correct shape. Returns 1 (nothing echoed) on
|
||||
# timeout.
|
||||
poll_addressable_elapsed() {
|
||||
local id="$1" budget="${2:-90}" start_ms
|
||||
SECONDS=0
|
||||
start_ms="$(epoch_ms)"
|
||||
while :; do
|
||||
jobinfo_of "$id" "$PROBE_WAITMS" >/dev/null 2>&1 && {
|
||||
echo "$(($(epoch_ms) - start_ms))"
|
||||
return 0
|
||||
}
|
||||
[ "$SECONDS" -ge "$budget" ] && return 1
|
||||
sleep 0.5
|
||||
done
|
||||
}
|
||||
|
||||
# Verify each element index in "$@" (after base string $1) is individually
|
||||
# addressable via jobinfo. Logs + returns 1 on the first miss.
|
||||
verify_array_elements() {
|
||||
local base="$1"; shift
|
||||
local idx id
|
||||
for idx in "$@"; do
|
||||
id="$(array_element_id "$base" "$idx")"
|
||||
if ! jobinfo_of "$id" >/dev/null; then
|
||||
log "array-soak: element idx=$idx ($id) NOT addressable"
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
# One dedicated round (not a loop): small array then large array, each
|
||||
# asserting one-record/one-barrier, a real kill/failover, and (for the large
|
||||
# array) per-element status/cancel/completion post-failover. Sets
|
||||
# ARRAYSOAK_{N,T}_{SMALL,LARGE} for the caller's verdict line. Hard-fails via
|
||||
# fail() on any violated invariant — see this section's header comment.
|
||||
run_array_soak_round() {
|
||||
local small="${XCE_HA_ARRAYSOAK_SMALL:-2}"
|
||||
local large="${XCE_HA_ARRAYSOAK_LARGE:-10000}"
|
||||
local rec_before rec_after delta base_small base_large t_small t_large
|
||||
local prev_term newp mid last
|
||||
|
||||
# ── Small sub-round ──────────────────────────────────────────────────
|
||||
log "array-soak: SMALL sub-round — submitting a $small-element array"
|
||||
PRIMARY="$(discover_primary)"; [ -n "$PRIMARY" ] || fail "array-soak: no serving primary before small-array submit"
|
||||
rec_before="$(count_array_wal_records "$PRIMARY")"
|
||||
base_small="$(submit_array "$small")" || fail "array-soak: small array ($small) submission failed"
|
||||
rec_after="$(count_array_wal_records "$PRIMARY")"
|
||||
delta=$((rec_after - rec_before))
|
||||
[ "$delta" -eq 1 ] || fail "array-soak: expected exactly 1 WAL_JOB_ARRAY_SUBMIT record for the $small-element array, observed $delta (base=$base_small)"
|
||||
log "array-soak: CONFIRMED exactly ONE WAL_JOB_ARRAY_SUBMIT record for a $small-element array (base=$base_small)"
|
||||
verify_array_elements "$base_small" 0 $((small - 1)) \
|
||||
|| fail "array-soak: small array elements not addressable immediately after commit (pre-failover)"
|
||||
|
||||
prev_term="$(latest_primary_term)"; prev_term="${prev_term:-0}"
|
||||
log "array-soak: killing primary $PRIMARY (term=$prev_term) to measure small-array failover replay time"
|
||||
kill_node "$PRIMARY" || fail "array-soak: could not kill $PRIMARY"
|
||||
DOWN="$PRIMARY"
|
||||
newp="$(wait_for_new_primary "$prev_term")" || fail "array-soak: no new primary elected after small-array kill"
|
||||
record_primary "$newp"
|
||||
t_small="$(poll_addressable_elapsed "$(array_element_id "$base_small" 0)" 90)" \
|
||||
|| fail "array-soak: small array element 0 never became addressable on new primary $newp within 90s"
|
||||
log "array-soak: small-array (N=$small) kill-to-addressable = ${t_small}ms (new primary=$newp)"
|
||||
verify_array_elements "$base_small" 0 $((small - 1)) \
|
||||
|| fail "array-soak: small array elements missing on new primary after failover"
|
||||
log "array-soak: all $small small-array elements confirmed addressable on new primary"
|
||||
|
||||
start_node "$DOWN"
|
||||
wait_for_full_convergence || fail "array-soak: cluster did not reconverge to 3-of-3 after the small-array sub-round"
|
||||
DOWN=""
|
||||
|
||||
# ── Large sub-round ──────────────────────────────────────────────────
|
||||
log "array-soak: LARGE sub-round — submitting a $large-element array"
|
||||
PRIMARY="$(discover_primary)"; [ -n "$PRIMARY" ] || fail "array-soak: no serving primary before large-array submit"
|
||||
rec_before="$(count_array_wal_records "$PRIMARY")"
|
||||
base_large="$(submit_array "$large")" || fail "array-soak: large array ($large) submission failed"
|
||||
rec_after="$(count_array_wal_records "$PRIMARY")"
|
||||
delta=$((rec_after - rec_before))
|
||||
[ "$delta" -eq 1 ] || fail "array-soak: expected exactly 1 WAL_JOB_ARRAY_SUBMIT record for the $large-element array, observed $delta (base=$base_large)"
|
||||
log "array-soak: CONFIRMED exactly ONE WAL_JOB_ARRAY_SUBMIT record for a $large-element array (base=$base_large) — same ONE record as the $small-element array above; record count is O(1) in array size"
|
||||
|
||||
mid=$((large / 2)); last=$((large - 1))
|
||||
verify_array_elements "$base_large" 0 "$mid" "$last" \
|
||||
|| fail "array-soak: large array elements not addressable immediately after commit (pre-failover)"
|
||||
|
||||
prev_term="$(latest_primary_term)"; prev_term="${prev_term:-0}"
|
||||
log "array-soak: killing primary $PRIMARY (term=$prev_term) to measure large-array failover replay time"
|
||||
kill_node "$PRIMARY" || fail "array-soak: could not kill $PRIMARY"
|
||||
DOWN="$PRIMARY"
|
||||
newp="$(wait_for_new_primary "$prev_term")" || fail "array-soak: no new primary elected after large-array kill"
|
||||
record_primary "$newp"
|
||||
t_large="$(poll_addressable_elapsed "$(array_element_id "$base_large" 0)" 90)" \
|
||||
|| fail "array-soak: large array element 0 never became addressable on new primary $newp within 90s"
|
||||
log "array-soak: large-array (N=$large) kill-to-addressable = ${t_large}ms (new primary=$newp)"
|
||||
|
||||
# Req 4.3 HARD assertion — replay time must be governed by record size,
|
||||
# not element count. This is a real gate, not just a reported number: a
|
||||
# genuine O(N)-scaling regression (e.g. large-array replay taking
|
||||
# multiple real seconds while small-array stays near-instant) must
|
||||
# fail() here, not just show up as a displayed ratio. See
|
||||
# verdict_check_arraysoak_replay_bound (soak-verdict.sh) for the bound
|
||||
# rationale; knobs are env-overridable for a slower/noisier host.
|
||||
local replay_ratio_max replay_floor_ms replay_margin_ms bound_check bound_ms bound_verdict
|
||||
replay_ratio_max="${XCE_HA_ARRAYSOAK_REPLAY_RATIO_MAX:-20}"
|
||||
replay_floor_ms="${XCE_HA_ARRAYSOAK_REPLAY_FLOOR_MS:-2000}"
|
||||
replay_margin_ms="${XCE_HA_ARRAYSOAK_REPLAY_MARGIN_MS:-3000}"
|
||||
bound_check="$(verdict_check_arraysoak_replay_bound \
|
||||
"$t_small" "$t_large" "$replay_ratio_max" "$replay_floor_ms" "$replay_margin_ms")"
|
||||
bound_ms="${bound_check% *}"
|
||||
bound_verdict="${bound_check#* }"
|
||||
log "array-soak: replay-time bound check — small=${t_small}ms large=${t_large}ms" \
|
||||
"bound=${bound_ms}ms (ratio_max=${replay_ratio_max} floor=${replay_floor_ms}ms margin=${replay_margin_ms}ms)" \
|
||||
"verdict=${bound_verdict}"
|
||||
[ "$bound_verdict" = "OK" ] || fail "array-soak: large-array (N=$large) failover replay time ${t_large}ms" \
|
||||
"exceeds bound ${bound_ms}ms derived from small-array (N=$small) replay time ${t_small}ms" \
|
||||
"(ratio_max=${replay_ratio_max} floor=${replay_floor_ms}ms margin=${replay_margin_ms}ms)" \
|
||||
"— replay time is NOT governed by record size alone (Req 4.3 violated); possible O(N) regression in array apply/replay"
|
||||
|
||||
verify_array_elements "$base_large" 0 "$mid" "$last" \
|
||||
|| fail "array-soak: large array elements missing on new primary after failover"
|
||||
log "array-soak: spot-checked elements 0, $mid, $last of the $large-element array all addressable on new primary $newp"
|
||||
|
||||
# Per-element status/cancel/completion (Req 5.2) — three DISTINCT,
|
||||
# untouched large-array elements spread across low/mid/high indices
|
||||
# (like the addressability spot-check above uses 0/mid/last), not
|
||||
# clustered at the start — a bug affecting only high-index elements
|
||||
# (e.g. an off-by-one in a chunking/offset calculation that only bites
|
||||
# past some element count) would NOT be caught by indices 1/2/3 alone.
|
||||
local idx_status idx_cancel idx_complete
|
||||
local id_status id_cancel id_complete out_cancel out_complete
|
||||
idx_status=1 # near start (distinct from idx 0's read-only checks)
|
||||
idx_cancel=$((mid + 1)) # near middle (distinct from the read-only mid check)
|
||||
idx_complete=$((last - 1)) # near end (distinct from the read-only last check)
|
||||
id_status="$(array_element_id "$base_large" "$idx_status")"
|
||||
id_cancel="$(array_element_id "$base_large" "$idx_cancel")"
|
||||
id_complete="$(array_element_id "$base_large" "$idx_complete")"
|
||||
|
||||
jobinfo_of "$id_status" | grep -q "^state: PEND" \
|
||||
|| fail "array-soak: status query on element idx=$idx_status ($id_status) did not show PEND state after failover"
|
||||
log "array-soak: status query OK on element idx=$idx_status ($id_status) — state PEND"
|
||||
|
||||
# Cancel: SIGKILL a not-yet-started element via jobkill — the real
|
||||
# per-element cancel mechanism for a PEND job with no worker attached
|
||||
# (job.c job_ctrl's not-started/siginfo/SIGKILL branch -> mark_job_fail).
|
||||
out_cancel="$(printf 'set masters %s\nset myname arraysoak\nclient\nwait-primary %s\njobkill -j %s -s 9\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" "$id_cancel" | xtest)"
|
||||
printf '%s\n' "$out_cancel" | grep -q "signaled" \
|
||||
|| fail "array-soak: jobkill on element idx=$idx_cancel ($id_cancel) did not report success"
|
||||
sleep 2 # fact-append + apply-pump settle (same margin class as factstream's own settle waits)
|
||||
jobinfo_of "$id_cancel" | grep -qE "^state: (EXIT|ERROR)" \
|
||||
|| fail "array-soak: element idx=$idx_cancel ($id_cancel) did not transition off PEND after jobkill -s 9 (cancel)"
|
||||
log "array-soak: cancel OK on element idx=$idx_cancel ($id_cancel) — state transitioned off PEND"
|
||||
|
||||
# Completion: jstart + jfinish — the same wire-level worker-fact
|
||||
# primitives --factstream already uses (job.c's on_job_start/
|
||||
# on_job_finish require only that the job exists in jobmgr, not a real
|
||||
# scheduler/worker handshake).
|
||||
out_complete="$(printf 'set masters %s\nset myname arraysoak\nworker\njoin\njstart -j %s\njfinish -j %s -c 0\nquit\n' \
|
||||
"$MASTERS" "$id_complete" "$id_complete" | xtest)"
|
||||
[ "$(printf '%s\n' "$out_complete" | grep -c "^ok$")" -eq 2 ] \
|
||||
|| fail "array-soak: jstart/jfinish on element idx=$idx_complete ($id_complete) did not both report ok"
|
||||
sleep 2
|
||||
jobinfo_of "$id_complete" | grep -q "^state: DONE" \
|
||||
|| fail "array-soak: element idx=$idx_complete ($id_complete) did not reach DONE after jstart+jfinish (completion)"
|
||||
log "array-soak: completion OK on element idx=$idx_complete ($id_complete) — state DONE"
|
||||
|
||||
start_node "$DOWN"
|
||||
wait_for_full_convergence || fail "array-soak: cluster did not reconverge to 3-of-3 after the large-array sub-round"
|
||||
DOWN=""
|
||||
walcheck_count "0"
|
||||
|
||||
ARRAYSOAK_N_SMALL="$small"
|
||||
ARRAYSOAK_T_SMALL="$t_small"
|
||||
ARRAYSOAK_N_LARGE="$large"
|
||||
ARRAYSOAK_T_LARGE="$t_large"
|
||||
ARRAYSOAK_BOUND_MS="$bound_ms"
|
||||
ARRAYSOAK_BOUND_VERDICT="$bound_verdict"
|
||||
}
|
||||
|
||||
# Drive a SPECIFIC node to primary — used to exercise leadership on the
|
||||
# lowest-priority node (ctrl-3), which the natural single-failover loop NEVER
|
||||
# elects: the election timeout is priority-staggered (my_priority * step_ms, p0
|
||||
|
|
@ -1323,6 +1653,32 @@ if [ "$MODE" = "factstream" ]; then
|
|||
exit 1
|
||||
fi
|
||||
|
||||
# ── Array-soak mode: job-array-single-wal-record task 6.1 ────────────────────
|
||||
# Opt-in (--array-soak / XCE_HA_MODE=array-soak). One dedicated round (not a
|
||||
# loop): submits a small then a large job array, asserts one WAL record/one
|
||||
# commit barrier for EACH regardless of size, measures kill-to-addressable
|
||||
# failover replay time for both AND hard-asserts the large-array time is
|
||||
# bounded relative to the small-array time (Req 4.3 — see
|
||||
# verdict_check_arraysoak_replay_bound in soak-verdict.sh), and proves
|
||||
# per-element status/cancel/completion addressing after a real failover on
|
||||
# several LARGE-array elements spread across low/mid/high indices. See
|
||||
# run_array_soak_round's own header comment for the full rationale. Like
|
||||
# force-snapshot, it does NOT run the sequential loop or leadership-coverage
|
||||
# maneuver, and hard-fails (via fail(), inside the round function) rather
|
||||
# than count-and-tolerate — these are the feature's core correctness claims,
|
||||
# not a racy/sometimes-expected outcome class.
|
||||
if [ "$MODE" = "array-soak" ]; then
|
||||
log "MODE=array-soak — large-array soak scenario (small=${XCE_HA_ARRAYSOAK_SMALL:-2} large=${XCE_HA_ARRAYSOAK_LARGE:-10000})"
|
||||
run_array_soak_round
|
||||
verdict_emit_arraysoak "$ARRAYSOAK_N_SMALL" "$ARRAYSOAK_T_SMALL" \
|
||||
"$ARRAYSOAK_N_LARGE" "$ARRAYSOAK_T_LARGE" \
|
||||
"$ARRAYSOAK_BOUND_MS" "$ARRAYSOAK_BOUND_VERDICT"
|
||||
log "PASS (array-soak): one WAL record/one commit barrier confirmed for both $ARRAYSOAK_N_SMALL- and $ARRAYSOAK_N_LARGE-element arrays; failover replay time small=${ARRAYSOAK_T_SMALL}ms large=${ARRAYSOAK_T_LARGE}ms (bound=${ARRAYSOAK_BOUND_MS}ms, Req 4.3 verdict=${ARRAYSOAK_BOUND_VERDICT}); per-element status/cancel/completion confirmed post-failover"
|
||||
cleanup
|
||||
trap - EXIT INT TERM
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Chaotic mode: overlapping-kill rounds → isolated verdict ──────────────────
|
||||
# Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up
|
||||
# windows; outcomes are COUNTED (committed divergence + committed job loss) with
|
||||
|
|
|
|||
|
|
@ -149,6 +149,58 @@ check "committed job loss => non-zero rc" 1 "$rc"
|
|||
check "committed job loss => result=FAIL token" \
|
||||
"FAIL" "$(printf '%s' "$v" | grep -oE 'result=[A-Z]+' | cut -d= -f2)"
|
||||
|
||||
# ── Array-soak replay-time bound (task 6.1 remediation round 1, Req 4.3) ─────
|
||||
# verdict_check_arraysoak_replay_bound(t_small_ms t_large_ms ratio_max
|
||||
# floor_ms margin_ms) => "<bound_ms> <OK|BOUND_EXCEEDED>". Using the
|
||||
# scenario's own defaults (ratio_max=20 floor_ms=2000 margin_ms=3000) as
|
||||
# fixtures so this selftest exercises the SAME math the live soak gates on.
|
||||
|
||||
# A healthy run: small=80ms, large=650ms. bound = max(80*20,2000)+3000 =
|
||||
# max(1600,2000)+3000 = 5000ms. 650 <= 5000 => OK. This is the shape task
|
||||
# 6.0a's own prior live runs actually saw (both sub-rounds fast, large not
|
||||
# meaningfully slower than small) — a real O(1) implementation.
|
||||
check "healthy small=80ms large=650ms => bound 5000ms" \
|
||||
"5000 OK" "$(verdict_check_arraysoak_replay_bound 80 650 20 2000 3000)"
|
||||
|
||||
# A genuine O(N) regression: small=50ms (near-instant), large=5000ms (a real
|
||||
# 5-second replay-time blowup — the exact scenario Finding 1's review called
|
||||
# out as silently passing under whole-second resolution). bound =
|
||||
# max(50*20,2000)+3000 = max(1000,2000)+3000 = 5000ms. 5000 is NOT > 5000,
|
||||
# so nudge large to 5001ms to cross the boundary and assert it trips.
|
||||
check "O(N) regression small=50ms large=5001ms => BOUND_EXCEEDED (bound 5000ms)" \
|
||||
"5000 BOUND_EXCEEDED" "$(verdict_check_arraysoak_replay_bound 50 5001 20 2000 3000)"
|
||||
|
||||
# Exact-boundary case: large-array time equal to the bound is still OK (only
|
||||
# STRICTLY greater trips it) — confirms the comparison is `>`, not `>=`.
|
||||
check "large exactly at bound => OK (boundary is inclusive)" \
|
||||
"5000 OK" "$(verdict_check_arraysoak_replay_bound 50 5000 20 2000 3000)"
|
||||
|
||||
# Zero/near-zero small-array time (the original Finding-1 whole-second
|
||||
# collapse case) must not make the bound degenerate — the floor still
|
||||
# applies, so a large-array time that is itself small stays OK.
|
||||
check "t_small=0ms (whole-second-collapse case) => floor still applies, OK" \
|
||||
"5000 OK" "$(verdict_check_arraysoak_replay_bound 0 1200 20 2000 3000)"
|
||||
check "t_small=0ms but large-array time itself blows past the floor+margin => BOUND_EXCEEDED" \
|
||||
"5000 BOUND_EXCEEDED" "$(verdict_check_arraysoak_replay_bound 0 5001 20 2000 3000)"
|
||||
|
||||
# verdict_emit_arraysoak: ratios + pass-through of the bound/verdict fields.
|
||||
# result is unconditionally PASS (the real gate is the fail() in
|
||||
# run_array_soak_round, upstream of this reporting-only function — see its
|
||||
# header comment) but the reported fields must reflect what was passed in.
|
||||
v="$(verdict_emit_arraysoak 2 300 10000 650 5000 OK)"
|
||||
check "arraysoak verdict line reports t_small_ms" \
|
||||
"t_small_ms=300" "$(printf '%s' "$v" | grep -oE 't_small_ms=[0-9]+')"
|
||||
check "arraysoak verdict line reports t_large_ms" \
|
||||
"t_large_ms=650" "$(printf '%s' "$v" | grep -oE 't_large_ms=[0-9]+')"
|
||||
check "arraysoak verdict line reports element_count_ratio" \
|
||||
"element_count_ratio=5000x" "$(printf '%s' "$v" | grep -oE 'element_count_ratio=[0-9]+x')"
|
||||
check "arraysoak verdict line passes through the replay bound" \
|
||||
"replay_bound_ms=5000" "$(printf '%s' "$v" | grep -oE 'replay_bound_ms=[0-9]+')"
|
||||
check "arraysoak verdict line passes through the replay bound verdict" \
|
||||
"replay_bound_verdict=OK" "$(printf '%s' "$v" | grep -oE 'replay_bound_verdict=[A-Z_]+')"
|
||||
check "arraysoak verdict result is always PASS (real gate is upstream fail())" \
|
||||
"PASS" "$(printf '%s' "$v" | grep -oE 'result=[A-Z]+' | cut -d= -f2)"
|
||||
|
||||
# ── Verdict ──────────────────────────────────────────────────────────────────
|
||||
echo
|
||||
if [ "$fails" -eq 0 ]; then
|
||||
|
|
|
|||
|
|
@ -109,3 +109,80 @@ verdict_emit_factstream() {
|
|||
"excluded_bug924_fdatasync=$b924 result=$result"
|
||||
[ "$result" = "PASS" ]
|
||||
}
|
||||
|
||||
# Compute the pass/fail replay-time bound for the array-soak scenario's
|
||||
# large-array kill-to-addressable time (Req 4.3: failover replay time
|
||||
# governed by the array record's size, not element count) and classify
|
||||
# t_large_ms against it. Pure/side-effect-free, unlike the rest of this
|
||||
# scenario's checks — the actual fail()/abort gate lives in
|
||||
# run_array_soak_round() (run-ha-soak.sh), which calls this helper and hard-
|
||||
# fails the run on a BOUND_EXCEEDED verdict, mirroring this scenario's other
|
||||
# hard-fail-not-count-and-tolerate invariants.
|
||||
#
|
||||
# The bound is deliberately RATIO-based with a floor and margin, not a bare
|
||||
# multiple of t_small: on a fast local single-node-per-process cluster
|
||||
# t_small itself can be a few tens/hundreds of ms, where a naive "N times
|
||||
# t_small" bound would be flaky-tight, and BOTH sub-rounds' measured time is
|
||||
# dominated by the same roughly-constant election/catch-up overhead, not
|
||||
# actual per-record replay cost — so a healthy run's t_large should sit close
|
||||
# to t_small regardless of array size. floor_ms sets a minimum meaningful
|
||||
# window so small jitter in t_small can't make the bound absurdly tight;
|
||||
# margin_ms absorbs ordinary election-timing jitter shared by both
|
||||
# sub-rounds; ratio_max is the real "how many times slower is tolerated"
|
||||
# gate — generous relative to the ~1x a genuine O(1) implementation should
|
||||
# show, but far tighter than the ~5000x growth (n_large/n_small at the
|
||||
# defaults) a real O(N) regression would produce, so a genuine regression
|
||||
# (e.g. several real seconds of added replay time) still trips it.
|
||||
#
|
||||
# Args: t_small_ms t_large_ms ratio_max floor_ms margin_ms
|
||||
# Echoes: "<bound_ms> <verdict>" where verdict is OK or BOUND_EXCEEDED.
|
||||
verdict_check_arraysoak_replay_bound() {
|
||||
local t_small="$1" t_large="$2" ratio_max="$3" floor_ms="$4" margin_ms="$5"
|
||||
local bound verdict
|
||||
bound=$((t_small * ratio_max))
|
||||
[ "$bound" -lt "$floor_ms" ] && bound="$floor_ms"
|
||||
bound=$((bound + margin_ms))
|
||||
verdict="OK"
|
||||
[ "$t_large" -gt "$bound" ] && verdict="BOUND_EXCEEDED"
|
||||
echo "$bound $verdict"
|
||||
}
|
||||
|
||||
# Emit the machine-parseable verdict line for the array-soak
|
||||
# (job-array-single-wal-record task 6.1) scenario. Unlike verdict_emit /
|
||||
# verdict_emit_factstream, this is NOT a count-and-classify verdict: every
|
||||
# hard invariant it reports on (exactly one WAL_JOB_ARRAY_SUBMIT record per
|
||||
# array submission regardless of size, every checked element addressable
|
||||
# post-failover, AND — as of this fix — the replay-time bound from
|
||||
# verdict_check_arraysoak_replay_bound above) is asserted with an immediate
|
||||
# fail() inside run_array_soak_round() in run-ha-soak.sh — a violation
|
||||
# aborts the run before this function is ever reached, mirroring
|
||||
# --force-snapshot's "the scenario proves nothing if the forcing mechanism
|
||||
# didn't fire" hard-fail convention rather than chaotic mode's
|
||||
# tolerate-and-count convention. So result is always PASS here; this
|
||||
# function's job is to report the ACTUAL replay-time measurement (Req 4.3)
|
||||
# and the bound it was checked against as a machine-parseable line, not to
|
||||
# gate pass/fail itself.
|
||||
#
|
||||
# Args: n_small t_small_ms n_large t_large_ms [bound_ms] [bound_verdict]
|
||||
# t_small_ms / t_large_ms are millisecond-resolution kill-to-first-
|
||||
# addressable measurements (see poll_addressable_elapsed in run-ha-soak.sh,
|
||||
# EPOCHREALTIME-based). bound_ms/bound_verdict are optional — when the
|
||||
# caller already computed them via verdict_check_arraysoak_replay_bound,
|
||||
# passing them through avoids recomputing and keeps the reported bound
|
||||
# identical to the one that actually gated the run.
|
||||
verdict_emit_arraysoak() {
|
||||
local n_small="$1" t_small="$2" n_large="$3" t_large="$4"
|
||||
local bound_ms="${5:-n/a}" bound_verdict="${6:-n/a}"
|
||||
local elem_ratio="n/a" time_ratio_x100="n/a"
|
||||
if [ "${n_small:-0}" -gt 0 ]; then
|
||||
elem_ratio="$((n_large / n_small))x"
|
||||
fi
|
||||
if [ "${t_small:-0}" -gt 0 ]; then
|
||||
time_ratio_x100="$(((t_large * 100) / t_small))"
|
||||
fi
|
||||
echo "[xce-test-ha] VERDICT mode=array-soak n_small=$n_small t_small_ms=$t_small" \
|
||||
"n_large=$n_large t_large_ms=$t_large element_count_ratio=$elem_ratio" \
|
||||
"replay_time_ratio_x100=$time_ratio_x100 replay_bound_ms=$bound_ms" \
|
||||
"replay_bound_verdict=$bound_verdict result=PASS"
|
||||
return 0
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue