test(job-array-single-wal-record): large-array HA soak scenario
Extends the HA soak harness with a new --array-soak mode: small (N=2) and large (N=10000) array submission sub-rounds, each with one clean kill/failover, asserting one WAL record per submission regardless of array size, per-element addressability (status/cancel/completion) spread across low/mid/high indices post-failover, and a real millisecond-resolution replay-time bound comparing large-array vs small-array kill-to-addressable latency. waldump.c gains a WAL_JOB_ARRAY_SUBMIT case in wal_type_name() so the scenario can grep an exact type name for its live WAL-record-count delta check. Went through one remediation round: the replay-time measurement initially used whole-second resolution (collapsed to 0s/n-a on a real run) and never asserted on the comparison, giving zero actual evidence for the O(1)-replay-time requirement. Fixed with $EPOCHREALTIME (millisecond resolution -- this sandbox's BusyBox date silently mangles the more common `date +%s%3N` idiom) and a real bound assertion that hard-fails the run on a genuine O(N) regression, independently verified to have real teeth. Live-verified 3 times with genuinely discriminating numbers each time; final run: small=210ms, large=163ms -- large-array replay was not slower, live evidence of the feature's O(1) claim. Task 6.1 of job-array-single-wal-record (Requirements 4.3, 5.2, 5.3, 8.2, 8.3). Depends on and reuses task 6.0a's own soak verification.
This commit is contained in:
parent
16747aa96d
commit
0d8986a6a9
6 changed files with 550 additions and 2 deletions
|
|
@ -151,6 +151,57 @@ block-but-durable outcome (Requirement 6) for a fact that never reached
|
|||
majority commit; only the reverse (a reaction fired with no committed WAL
|
||||
entry backing it) counts.
|
||||
|
||||
### `--array-soak` mode: large-array replay/addressing scenario (job-array-single-wal-record task 6.1)
|
||||
|
||||
Submits a **small** (default 2-element) then a **large** (default 10,000-
|
||||
element) job array via `jobsub -a N`, and — for EACH, regardless of size —
|
||||
asserts:
|
||||
|
||||
- **exactly one `WAL_JOB_ARRAY_SUBMIT` record / one commit barrier** (`waldump`,
|
||||
which this task extended to name WAL type 11 `JOB_ARRAY_SUBMIT` for an exact
|
||||
count instead of an `UNKNOWN`-bucket proxy);
|
||||
- **kill-to-first-addressable failover replay time**, measured at millisecond
|
||||
resolution (bash's `$EPOCHREALTIME` builtin — deliberately not `date +%s%N`,
|
||||
since this environment's plain `date` can resolve to a BusyBox binary that
|
||||
silently passes an unrecognized `%N` through as a literal string instead of
|
||||
erroring), with a **hard pass/fail bound**: the large-array replay time must
|
||||
not exceed `max(t_small_ms * ratio_max, floor_ms) + margin_ms` (defaults
|
||||
`ratio_max=20`, `floor_ms=2000`, `margin_ms=3000` — see
|
||||
`verdict_check_arraysoak_replay_bound` in `soak-verdict.sh`). This is the
|
||||
Req 4.3 assertion (*"failover replay time governed by the record's size,
|
||||
not element count"*) — a genuine O(N)-scaling regression fails the run via
|
||||
`fail()`, it does not just show up as a displayed number; and
|
||||
- **per-element status query, cancel (`jobkill -s 9`), and completion
|
||||
(`jstart`+`jfinish`)** on three distinct LARGE-array elements spread across
|
||||
low/mid/high indices (not clustered near the start), after a real failover.
|
||||
|
||||
Like `--force-snapshot`, this is **one dedicated round, not a loop**, and it
|
||||
hard-fails (not count-and-tolerate) on any violated invariant — these are the
|
||||
feature's core correctness claims, not a racy/sometimes-expected outcome
|
||||
class.
|
||||
|
||||
```bash
|
||||
SKILL_DIR=.claude/skills/xce-test-ha
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --array-soak ~/tmp/xce-ha-arraysoak
|
||||
# tune array sizes / replay-time bound knobs
|
||||
XCE_HA_ARRAYSOAK_SMALL=3 XCE_HA_ARRAYSOAK_LARGE=20000 XCE_HA_ARRAYSOAK_REPLAY_RATIO_MAX=30 \
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --array-soak ~/tmp/xce-ha-arraysoak
|
||||
```
|
||||
|
||||
Verdict line (`soak-verdict.sh`'s `verdict_emit_arraysoak`):
|
||||
|
||||
```
|
||||
[xce-test-ha] VERDICT mode=array-soak n_small=2 t_small_ms=165 n_large=10000 t_large_ms=164 element_count_ratio=5000x replay_time_ratio_x100=99 replay_bound_ms=6300 replay_bound_verdict=OK result=PASS
|
||||
```
|
||||
|
||||
`result` is unconditionally `PASS` here — unlike chaotic/factstream's
|
||||
count-and-classify verdicts, every hard invariant this scenario checks
|
||||
(record count, replay-time bound, per-element addressing) is asserted with an
|
||||
immediate `fail()` inside `run_array_soak_round()`, so a violation aborts the
|
||||
run before this verdict line is ever reached. The line's job is to report the
|
||||
actual measurements (and the bound they were checked against) for visibility,
|
||||
not to gate pass/fail itself.
|
||||
|
||||
### Verdict self-test (no cluster)
|
||||
|
||||
The verdict parser has a deterministic, pure-bash self-test that feeds fixture
|
||||
|
|
@ -247,7 +298,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
|---|---|---|
|
||||
| `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
|
||||
| `XCE_HA_DATADIR` | config/certs source (`cluster.bin`, `xce.crt`, `xce.key`) | `<repo>/ha/data` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, or `factstream` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, or `array-soak` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
|
||||
| `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
|
||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–50) | `6` |
|
||||
|
|
@ -259,6 +310,11 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
| `XCE_HA_FACTSTREAM_WAVE_SIZE` | `--factstream` jobs per paced wave (own xce_test connection) | `3` |
|
||||
| `XCE_HA_FACTSTREAM_WAVE_GAP` | `--factstream` seconds slept between waves | `0.5` |
|
||||
| `XCE_HA_FACTSTREAM_SETTLE` | `--factstream` seconds slept after convergence, before traffic starts (bug-1416 mitigation) | `10` |
|
||||
| `XCE_HA_ARRAYSOAK_SMALL` | `--array-soak` element count of the "small" comparison array | `2` |
|
||||
| `XCE_HA_ARRAYSOAK_LARGE` | `--array-soak` element count of the "large" comparison array | `10000` |
|
||||
| `XCE_HA_ARRAYSOAK_REPLAY_RATIO_MAX` | `--array-soak` max tolerated large/small replay-time ratio before the floor/margin (Req 4.3 bound) | `20` |
|
||||
| `XCE_HA_ARRAYSOAK_REPLAY_FLOOR_MS` | `--array-soak` minimum bound floor in ms before margin is added | `2000` |
|
||||
| `XCE_HA_ARRAYSOAK_REPLAY_MARGIN_MS` | `--array-soak` ms added on top of the scaled/floor bound to absorb election-timing jitter | `3000` |
|
||||
|
||||
## What the driver does (maps to the requested steps)
|
||||
|
||||
|
|
|
|||
|
|
@ -42,6 +42,20 @@
|
|||
# reactions already fired — with no committed WAL fact entry
|
||||
# backing it on some node). Repeats XCE_HA_FACTSTREAM_ROUNDS
|
||||
# times (default 2), then emits one VERDICT line.
|
||||
# --array-soak job-array-single-wal-record task 6.1 (Req 4.3, 5.2, 5.3,
|
||||
# 8.2, 8.3): submits a small then a large job ARRAY
|
||||
# (`jobsub -a N`), asserting for EACH — regardless of array
|
||||
# size — exactly one WAL_JOB_ARRAY_SUBMIT record / one
|
||||
# commit barrier; measures kill-to-first-addressable
|
||||
# failover replay time for both and reports them so a
|
||||
# 10,000-element array's replay can be confirmed NOT to
|
||||
# scale with element count; and proves per-element status
|
||||
# query, cancel (jobkill -s 9), and completion (jstart+
|
||||
# jfinish) on several distinct LARGE-array elements after a
|
||||
# real failover. One dedicated round, not a loop — like
|
||||
# --force-snapshot, hard-fails (not count-and-tolerate) on
|
||||
# any violated invariant, since these are the feature's core
|
||||
# correctness claims, not a chaotic/racy outcome class.
|
||||
# --selftest run the verdict-parser self-test (soak-verdict-selftest.sh)
|
||||
# and exit — no cluster, no binaries needed.
|
||||
#
|
||||
|
|
@ -81,6 +95,28 @@
|
|||
# every node's boot-time recovery vtask a
|
||||
# realistic window to finish before this
|
||||
# scenario's traffic begins
|
||||
# XCE_HA_ARRAYSOAK_SMALL element count of the "small" comparison array
|
||||
# (default 2 — the smallest size that exercises
|
||||
# the new WAL_JOB_ARRAY_SUBMIT path at all;
|
||||
# array_size == 1 stays on the pre-existing
|
||||
# single-job path, task 2.2, so it cannot be used
|
||||
# for an apples-to-apples array-path comparison)
|
||||
# XCE_HA_ARRAYSOAK_LARGE element count of the "large" comparison array
|
||||
# (default 10000, matching Req 4.3's own example)
|
||||
# XCE_HA_ARRAYSOAK_REPLAY_RATIO_MAX Req 4.3 replay-time-bound gate: max
|
||||
# tolerated large-array/small-array kill-to-
|
||||
# addressable time ratio before the scaled floor/
|
||||
# margin (default 20 — generous vs. the ~1x a
|
||||
# real O(1) implementation should show, far
|
||||
# tighter than the ~5000x an O(N) regression at
|
||||
# the default sizes would produce)
|
||||
# XCE_HA_ARRAYSOAK_REPLAY_FLOOR_MS minimum bound floor in ms before
|
||||
# margin is added (default 2000) — keeps the
|
||||
# bound meaningful when t_small itself is a few
|
||||
# tens/hundreds of ms on a fast local cluster
|
||||
# XCE_HA_ARRAYSOAK_REPLAY_MARGIN_MS ms added on top of the scaled/floor
|
||||
# bound to absorb ordinary election-timing jitter
|
||||
# shared by both sub-rounds (default 3000)
|
||||
set -u
|
||||
|
||||
# ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─
|
||||
|
|
@ -99,6 +135,7 @@ while [ $# -gt 0 ]; do
|
|||
--sequential) MODE="sequential" ;;
|
||||
--force-snapshot) MODE="force-snapshot" ;;
|
||||
--factstream) MODE="factstream" ;;
|
||||
--array-soak) MODE="array-soak" ;;
|
||||
--selftest) exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;;
|
||||
*[!0-9]*|'') ARGS+=("$1") ;; # non-integer ⇒ deploy dir
|
||||
*) ROUNDS_CLI="$1" ;; # bare integer ⇒ rounds
|
||||
|
|
@ -1141,6 +1178,299 @@ quit"
|
|||
verify_factstream_no_phantom
|
||||
}
|
||||
|
||||
# ── Array-soak primitives (job-array-single-wal-record task 6.1) ────────────
|
||||
#
|
||||
# Proves, against a REAL 3-node cluster and a REAL failover — not just the
|
||||
# existing white-box unit tests (task 2.3/3.4 already prove WAL-record-count
|
||||
# and idempotent-expansion in isolation) — three of this feature's live
|
||||
# claims:
|
||||
# (a) exactly one WAL_JOB_ARRAY_SUBMIT record / one commit barrier per
|
||||
# array submission, regardless of array size (Req 1.1-1.3)
|
||||
# (b) failover replay time governed by the RECORD's size, not element count
|
||||
# (Req 4.3) — measured as wall-clock kill-to-first-addressable and
|
||||
# compared between a small and a large array, BOTH on the array-record
|
||||
# code path (array_size > 1; array_size == 1 stays on the pre-existing
|
||||
# single-job path per task 2.2, so it cannot serve as an apples-to-
|
||||
# apples comparison point for this specific claim)
|
||||
# (c) every element individually addressable — status query, cancel,
|
||||
# completion — after a failover (Req 5.2), proven against several
|
||||
# distinct elements of the LARGE array, not just index 0
|
||||
#
|
||||
# Per-element addressing needs no new xce_test command: `jobinfo`/
|
||||
# `expect-job`/`jobkill`'s `-j`/positional argument already parses the
|
||||
# bracketed array-id format (utils.c parse_jobid, REGEX_JOBARRAY_ID). The
|
||||
# base jobid `jobsub -a N` captures into $jobid is array_jobid()-stripped
|
||||
# (tag cleared before printing, per cmd_jobsub/test.c), so it prints WITHOUT
|
||||
# brackets (e.g. "12-3-1700000000-32768") — element i's addressable string is
|
||||
# that exact base string with "[i]" appended: parse_jobid reconstructs
|
||||
# base_jobid_with_tag + i from "<base>[i]", the identical arithmetic
|
||||
# restore_job_array() (job.c) itself uses for jobid = base_jobid + i. Traced
|
||||
# through utils.h's MAKE_JOBID/EXTRACT_ARRAY_* bit-layout macros to confirm
|
||||
# this is exact, not an approximation.
|
||||
|
||||
# Submit ONE array of size $1 via `jobsub -a N`. Echoes the captured base
|
||||
# jobid string (bracket-less) on success; returns 1 with nothing echoed on
|
||||
# failure. Deliberately does NOT append to ALL_JOBIDS — array elements are
|
||||
# addressed by base+index arithmetic here, not individually tracked the way
|
||||
# submit_jobs() tracks single-job submissions, which also keeps this
|
||||
# scenario's (possibly 10,000-element) array out of every OTHER helper's
|
||||
# O(len(ALL_JOBIDS)) sweep (verify_no_loss, walcheck, ...).
|
||||
submit_array() {
|
||||
local n="$1" out base
|
||||
out="$(printf 'set masters %s\nset myname arraysoak\nclient\nwait-primary %s\njobsub -u root -q normal -c /bin/true -a %s\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" "$n" | xtest)"
|
||||
base="$(printf '%s\n' "$out" | sed -n 's/^jobsubmit ok: JobId: \(.*\)$/\1/p' | head -1)"
|
||||
[ -n "$base" ] || return 1
|
||||
echo "$base"
|
||||
}
|
||||
|
||||
# Count WAL_JOB_ARRAY_SUBMIT entries physically present in node $1's on-disk
|
||||
# WAL right now. Relies on this task's own waldump.c change (wal_type_name()
|
||||
# now names type 11 "JOB_ARRAY_SUBMIT") for an EXACT count — not an
|
||||
# "UNKNOWN"-bucket proxy that would also catch a genuinely unrelated type.
|
||||
count_array_wal_records() {
|
||||
local node="$1" out
|
||||
out="$(printf 'waldump %s/controller/%s/data\nquit\n' "$TOP" "$node" | xtest)"
|
||||
printf '%s\n' "$out" | grep -c "JOB_ARRAY_SUBMIT"
|
||||
}
|
||||
|
||||
# Per-element jobid string for array element $2 (0-based) of base string $1.
|
||||
# See this section's header comment for why string concatenation is exact.
|
||||
array_element_id() { echo "$1[$2]"; }
|
||||
|
||||
# `jobinfo -j <id>` against the live primary. Prints the jobinfo text and
|
||||
# returns 0 on a hit; returns 1 (nothing printed) if not found/unreachable.
|
||||
# A direct single-job lookup (xcecli_jobinfo) — unlike expect-job's
|
||||
# O(current job count) joblist-scan-and-match, the right primitive for
|
||||
# checking individual elements of a large array without an O(N) list
|
||||
# round-trip per check. Retries twice (short sleep) to absorb the same
|
||||
# post-election client-retry-storm race verify_no_loss already tolerates.
|
||||
# $2 = wait-primary budget in ms (default $WAITMS).
|
||||
jobinfo_of() {
|
||||
local id="$1" waitms="${2:-$WAITMS}" out attempt
|
||||
for attempt in 1 2 3; do
|
||||
out="$(printf 'set masters %s\nset myname arraysoak\nclient\nwait-primary %s\njobinfo -j %s\nquit\n' \
|
||||
"$MASTERS" "$waitms" "$id" | xtest)"
|
||||
if printf '%s\n' "$out" | grep -q "^JobID: "; then
|
||||
printf '%s\n' "$out"
|
||||
return 0
|
||||
fi
|
||||
[ "$attempt" -lt 3 ] && sleep 0.5
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Millisecond-resolution wall-clock reading via bash's own $EPOCHREALTIME
|
||||
# (bash >= 5.0 builtin, "seconds.microseconds", always 6 fractional digits) —
|
||||
# deliberately NOT `date +%s%3N`/`date +%s%N`: this environment's plain
|
||||
# `date` can resolve to a BusyBox binary ahead of GNU coreutils on $PATH, and
|
||||
# BusyBox `date` silently passes an unrecognized `%N`/`%3N` through as a
|
||||
# LITERAL string instead of erroring (confirmed: `date +%s%3N` => `<epoch>%3N`,
|
||||
# rc=0) — that garbage would poison every downstream arithmetic use. Reading
|
||||
# $EPOCHREALTIME is a pure bash builtin with no subprocess/PATH dependency at
|
||||
# all, so it can't hit that trap. `10#` forces base-10 on the fractional
|
||||
# slice so a leading-zero fraction (e.g. "007123") isn't misread as octal.
|
||||
epoch_ms() {
|
||||
local t="$EPOCHREALTIME" sec frac
|
||||
sec="${t%%.*}"
|
||||
frac="${t#*.}"
|
||||
printf '%d' "$((sec * 1000 + 10#${frac:0:3}))"
|
||||
}
|
||||
|
||||
# Poll jobinfo_of every 0.5s (up to $2 seconds, default 90) until element $1
|
||||
# is addressable. Echoes elapsed MILLISECONDS (epoch_ms above) from this
|
||||
# function's own start to first success — the kill-to-first-addressable
|
||||
# failover replay time measurement (Req 4.3). Millisecond resolution matters
|
||||
# here: on a fast local cluster, whole-second (`date +%s`) measurements can
|
||||
# collapse both the small- and large-array sub-rounds to "0s", which reports
|
||||
# a real number but proves nothing (bug found in review — see
|
||||
# run_array_soak_round's replay-time-bound assertion, which needs a
|
||||
# genuinely discriminating measurement to gate on). The 90s outer timeout
|
||||
# budget itself is tracked via bash's own $SECONDS builtin (reset at this
|
||||
# function's start), NOT by diffing $EPOCHREALTIME pairs inside the loop —
|
||||
# per this project's own logged lesson (.wolf/cerebrum.md, 2026-07-07):
|
||||
# mixing a sub-second precision timestamp into per-iteration deadline
|
||||
# arithmetic is a source of self-inflicted timing bugs; a single
|
||||
# $EPOCHREALTIME pair (start here, end at the success branch) captured once
|
||||
# and diffed once is the correct shape. Returns 1 (nothing echoed) on
|
||||
# timeout.
|
||||
poll_addressable_elapsed() {
|
||||
local id="$1" budget="${2:-90}" start_ms
|
||||
SECONDS=0
|
||||
start_ms="$(epoch_ms)"
|
||||
while :; do
|
||||
jobinfo_of "$id" "$PROBE_WAITMS" >/dev/null 2>&1 && {
|
||||
echo "$(($(epoch_ms) - start_ms))"
|
||||
return 0
|
||||
}
|
||||
[ "$SECONDS" -ge "$budget" ] && return 1
|
||||
sleep 0.5
|
||||
done
|
||||
}
|
||||
|
||||
# Verify each element index in "$@" (after base string $1) is individually
|
||||
# addressable via jobinfo. Logs + returns 1 on the first miss.
|
||||
verify_array_elements() {
|
||||
local base="$1"; shift
|
||||
local idx id
|
||||
for idx in "$@"; do
|
||||
id="$(array_element_id "$base" "$idx")"
|
||||
if ! jobinfo_of "$id" >/dev/null; then
|
||||
log "array-soak: element idx=$idx ($id) NOT addressable"
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
# One dedicated round (not a loop): small array then large array, each
|
||||
# asserting one-record/one-barrier, a real kill/failover, and (for the large
|
||||
# array) per-element status/cancel/completion post-failover. Sets
|
||||
# ARRAYSOAK_{N,T}_{SMALL,LARGE} for the caller's verdict line. Hard-fails via
|
||||
# fail() on any violated invariant — see this section's header comment.
|
||||
run_array_soak_round() {
|
||||
local small="${XCE_HA_ARRAYSOAK_SMALL:-2}"
|
||||
local large="${XCE_HA_ARRAYSOAK_LARGE:-10000}"
|
||||
local rec_before rec_after delta base_small base_large t_small t_large
|
||||
local prev_term newp mid last
|
||||
|
||||
# ── Small sub-round ──────────────────────────────────────────────────
|
||||
log "array-soak: SMALL sub-round — submitting a $small-element array"
|
||||
PRIMARY="$(discover_primary)"; [ -n "$PRIMARY" ] || fail "array-soak: no serving primary before small-array submit"
|
||||
rec_before="$(count_array_wal_records "$PRIMARY")"
|
||||
base_small="$(submit_array "$small")" || fail "array-soak: small array ($small) submission failed"
|
||||
rec_after="$(count_array_wal_records "$PRIMARY")"
|
||||
delta=$((rec_after - rec_before))
|
||||
[ "$delta" -eq 1 ] || fail "array-soak: expected exactly 1 WAL_JOB_ARRAY_SUBMIT record for the $small-element array, observed $delta (base=$base_small)"
|
||||
log "array-soak: CONFIRMED exactly ONE WAL_JOB_ARRAY_SUBMIT record for a $small-element array (base=$base_small)"
|
||||
verify_array_elements "$base_small" 0 $((small - 1)) \
|
||||
|| fail "array-soak: small array elements not addressable immediately after commit (pre-failover)"
|
||||
|
||||
prev_term="$(latest_primary_term)"; prev_term="${prev_term:-0}"
|
||||
log "array-soak: killing primary $PRIMARY (term=$prev_term) to measure small-array failover replay time"
|
||||
kill_node "$PRIMARY" || fail "array-soak: could not kill $PRIMARY"
|
||||
DOWN="$PRIMARY"
|
||||
newp="$(wait_for_new_primary "$prev_term")" || fail "array-soak: no new primary elected after small-array kill"
|
||||
record_primary "$newp"
|
||||
t_small="$(poll_addressable_elapsed "$(array_element_id "$base_small" 0)" 90)" \
|
||||
|| fail "array-soak: small array element 0 never became addressable on new primary $newp within 90s"
|
||||
log "array-soak: small-array (N=$small) kill-to-addressable = ${t_small}ms (new primary=$newp)"
|
||||
verify_array_elements "$base_small" 0 $((small - 1)) \
|
||||
|| fail "array-soak: small array elements missing on new primary after failover"
|
||||
log "array-soak: all $small small-array elements confirmed addressable on new primary"
|
||||
|
||||
start_node "$DOWN"
|
||||
wait_for_full_convergence || fail "array-soak: cluster did not reconverge to 3-of-3 after the small-array sub-round"
|
||||
DOWN=""
|
||||
|
||||
# ── Large sub-round ──────────────────────────────────────────────────
|
||||
log "array-soak: LARGE sub-round — submitting a $large-element array"
|
||||
PRIMARY="$(discover_primary)"; [ -n "$PRIMARY" ] || fail "array-soak: no serving primary before large-array submit"
|
||||
rec_before="$(count_array_wal_records "$PRIMARY")"
|
||||
base_large="$(submit_array "$large")" || fail "array-soak: large array ($large) submission failed"
|
||||
rec_after="$(count_array_wal_records "$PRIMARY")"
|
||||
delta=$((rec_after - rec_before))
|
||||
[ "$delta" -eq 1 ] || fail "array-soak: expected exactly 1 WAL_JOB_ARRAY_SUBMIT record for the $large-element array, observed $delta (base=$base_large)"
|
||||
log "array-soak: CONFIRMED exactly ONE WAL_JOB_ARRAY_SUBMIT record for a $large-element array (base=$base_large) — same ONE record as the $small-element array above; record count is O(1) in array size"
|
||||
|
||||
mid=$((large / 2)); last=$((large - 1))
|
||||
verify_array_elements "$base_large" 0 "$mid" "$last" \
|
||||
|| fail "array-soak: large array elements not addressable immediately after commit (pre-failover)"
|
||||
|
||||
prev_term="$(latest_primary_term)"; prev_term="${prev_term:-0}"
|
||||
log "array-soak: killing primary $PRIMARY (term=$prev_term) to measure large-array failover replay time"
|
||||
kill_node "$PRIMARY" || fail "array-soak: could not kill $PRIMARY"
|
||||
DOWN="$PRIMARY"
|
||||
newp="$(wait_for_new_primary "$prev_term")" || fail "array-soak: no new primary elected after large-array kill"
|
||||
record_primary "$newp"
|
||||
t_large="$(poll_addressable_elapsed "$(array_element_id "$base_large" 0)" 90)" \
|
||||
|| fail "array-soak: large array element 0 never became addressable on new primary $newp within 90s"
|
||||
log "array-soak: large-array (N=$large) kill-to-addressable = ${t_large}ms (new primary=$newp)"
|
||||
|
||||
# Req 4.3 HARD assertion — replay time must be governed by record size,
|
||||
# not element count. This is a real gate, not just a reported number: a
|
||||
# genuine O(N)-scaling regression (e.g. large-array replay taking
|
||||
# multiple real seconds while small-array stays near-instant) must
|
||||
# fail() here, not just show up as a displayed ratio. See
|
||||
# verdict_check_arraysoak_replay_bound (soak-verdict.sh) for the bound
|
||||
# rationale; knobs are env-overridable for a slower/noisier host.
|
||||
local replay_ratio_max replay_floor_ms replay_margin_ms bound_check bound_ms bound_verdict
|
||||
replay_ratio_max="${XCE_HA_ARRAYSOAK_REPLAY_RATIO_MAX:-20}"
|
||||
replay_floor_ms="${XCE_HA_ARRAYSOAK_REPLAY_FLOOR_MS:-2000}"
|
||||
replay_margin_ms="${XCE_HA_ARRAYSOAK_REPLAY_MARGIN_MS:-3000}"
|
||||
bound_check="$(verdict_check_arraysoak_replay_bound \
|
||||
"$t_small" "$t_large" "$replay_ratio_max" "$replay_floor_ms" "$replay_margin_ms")"
|
||||
bound_ms="${bound_check% *}"
|
||||
bound_verdict="${bound_check#* }"
|
||||
log "array-soak: replay-time bound check — small=${t_small}ms large=${t_large}ms" \
|
||||
"bound=${bound_ms}ms (ratio_max=${replay_ratio_max} floor=${replay_floor_ms}ms margin=${replay_margin_ms}ms)" \
|
||||
"verdict=${bound_verdict}"
|
||||
[ "$bound_verdict" = "OK" ] || fail "array-soak: large-array (N=$large) failover replay time ${t_large}ms" \
|
||||
"exceeds bound ${bound_ms}ms derived from small-array (N=$small) replay time ${t_small}ms" \
|
||||
"(ratio_max=${replay_ratio_max} floor=${replay_floor_ms}ms margin=${replay_margin_ms}ms)" \
|
||||
"— replay time is NOT governed by record size alone (Req 4.3 violated); possible O(N) regression in array apply/replay"
|
||||
|
||||
verify_array_elements "$base_large" 0 "$mid" "$last" \
|
||||
|| fail "array-soak: large array elements missing on new primary after failover"
|
||||
log "array-soak: spot-checked elements 0, $mid, $last of the $large-element array all addressable on new primary $newp"
|
||||
|
||||
# Per-element status/cancel/completion (Req 5.2) — three DISTINCT,
|
||||
# untouched large-array elements spread across low/mid/high indices
|
||||
# (like the addressability spot-check above uses 0/mid/last), not
|
||||
# clustered at the start — a bug affecting only high-index elements
|
||||
# (e.g. an off-by-one in a chunking/offset calculation that only bites
|
||||
# past some element count) would NOT be caught by indices 1/2/3 alone.
|
||||
local idx_status idx_cancel idx_complete
|
||||
local id_status id_cancel id_complete out_cancel out_complete
|
||||
idx_status=1 # near start (distinct from idx 0's read-only checks)
|
||||
idx_cancel=$((mid + 1)) # near middle (distinct from the read-only mid check)
|
||||
idx_complete=$((last - 1)) # near end (distinct from the read-only last check)
|
||||
id_status="$(array_element_id "$base_large" "$idx_status")"
|
||||
id_cancel="$(array_element_id "$base_large" "$idx_cancel")"
|
||||
id_complete="$(array_element_id "$base_large" "$idx_complete")"
|
||||
|
||||
jobinfo_of "$id_status" | grep -q "^state: PEND" \
|
||||
|| fail "array-soak: status query on element idx=$idx_status ($id_status) did not show PEND state after failover"
|
||||
log "array-soak: status query OK on element idx=$idx_status ($id_status) — state PEND"
|
||||
|
||||
# Cancel: SIGKILL a not-yet-started element via jobkill — the real
|
||||
# per-element cancel mechanism for a PEND job with no worker attached
|
||||
# (job.c job_ctrl's not-started/siginfo/SIGKILL branch -> mark_job_fail).
|
||||
out_cancel="$(printf 'set masters %s\nset myname arraysoak\nclient\nwait-primary %s\njobkill -j %s -s 9\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" "$id_cancel" | xtest)"
|
||||
printf '%s\n' "$out_cancel" | grep -q "signaled" \
|
||||
|| fail "array-soak: jobkill on element idx=$idx_cancel ($id_cancel) did not report success"
|
||||
sleep 2 # fact-append + apply-pump settle (same margin class as factstream's own settle waits)
|
||||
jobinfo_of "$id_cancel" | grep -qE "^state: (EXIT|ERROR)" \
|
||||
|| fail "array-soak: element idx=$idx_cancel ($id_cancel) did not transition off PEND after jobkill -s 9 (cancel)"
|
||||
log "array-soak: cancel OK on element idx=$idx_cancel ($id_cancel) — state transitioned off PEND"
|
||||
|
||||
# Completion: jstart + jfinish — the same wire-level worker-fact
|
||||
# primitives --factstream already uses (job.c's on_job_start/
|
||||
# on_job_finish require only that the job exists in jobmgr, not a real
|
||||
# scheduler/worker handshake).
|
||||
out_complete="$(printf 'set masters %s\nset myname arraysoak\nworker\njoin\njstart -j %s\njfinish -j %s -c 0\nquit\n' \
|
||||
"$MASTERS" "$id_complete" "$id_complete" | xtest)"
|
||||
[ "$(printf '%s\n' "$out_complete" | grep -c "^ok$")" -eq 2 ] \
|
||||
|| fail "array-soak: jstart/jfinish on element idx=$idx_complete ($id_complete) did not both report ok"
|
||||
sleep 2
|
||||
jobinfo_of "$id_complete" | grep -q "^state: DONE" \
|
||||
|| fail "array-soak: element idx=$idx_complete ($id_complete) did not reach DONE after jstart+jfinish (completion)"
|
||||
log "array-soak: completion OK on element idx=$idx_complete ($id_complete) — state DONE"
|
||||
|
||||
start_node "$DOWN"
|
||||
wait_for_full_convergence || fail "array-soak: cluster did not reconverge to 3-of-3 after the large-array sub-round"
|
||||
DOWN=""
|
||||
walcheck_count "0"
|
||||
|
||||
ARRAYSOAK_N_SMALL="$small"
|
||||
ARRAYSOAK_T_SMALL="$t_small"
|
||||
ARRAYSOAK_N_LARGE="$large"
|
||||
ARRAYSOAK_T_LARGE="$t_large"
|
||||
ARRAYSOAK_BOUND_MS="$bound_ms"
|
||||
ARRAYSOAK_BOUND_VERDICT="$bound_verdict"
|
||||
}
|
||||
|
||||
# Drive a SPECIFIC node to primary — used to exercise leadership on the
|
||||
# lowest-priority node (ctrl-3), which the natural single-failover loop NEVER
|
||||
# elects: the election timeout is priority-staggered (my_priority * step_ms, p0
|
||||
|
|
@ -1323,6 +1653,32 @@ if [ "$MODE" = "factstream" ]; then
|
|||
exit 1
|
||||
fi
|
||||
|
||||
# ── Array-soak mode: job-array-single-wal-record task 6.1 ────────────────────
|
||||
# Opt-in (--array-soak / XCE_HA_MODE=array-soak). One dedicated round (not a
|
||||
# loop): submits a small then a large job array, asserts one WAL record/one
|
||||
# commit barrier for EACH regardless of size, measures kill-to-addressable
|
||||
# failover replay time for both AND hard-asserts the large-array time is
|
||||
# bounded relative to the small-array time (Req 4.3 — see
|
||||
# verdict_check_arraysoak_replay_bound in soak-verdict.sh), and proves
|
||||
# per-element status/cancel/completion addressing after a real failover on
|
||||
# several LARGE-array elements spread across low/mid/high indices. See
|
||||
# run_array_soak_round's own header comment for the full rationale. Like
|
||||
# force-snapshot, it does NOT run the sequential loop or leadership-coverage
|
||||
# maneuver, and hard-fails (via fail(), inside the round function) rather
|
||||
# than count-and-tolerate — these are the feature's core correctness claims,
|
||||
# not a racy/sometimes-expected outcome class.
|
||||
if [ "$MODE" = "array-soak" ]; then
|
||||
log "MODE=array-soak — large-array soak scenario (small=${XCE_HA_ARRAYSOAK_SMALL:-2} large=${XCE_HA_ARRAYSOAK_LARGE:-10000})"
|
||||
run_array_soak_round
|
||||
verdict_emit_arraysoak "$ARRAYSOAK_N_SMALL" "$ARRAYSOAK_T_SMALL" \
|
||||
"$ARRAYSOAK_N_LARGE" "$ARRAYSOAK_T_LARGE" \
|
||||
"$ARRAYSOAK_BOUND_MS" "$ARRAYSOAK_BOUND_VERDICT"
|
||||
log "PASS (array-soak): one WAL record/one commit barrier confirmed for both $ARRAYSOAK_N_SMALL- and $ARRAYSOAK_N_LARGE-element arrays; failover replay time small=${ARRAYSOAK_T_SMALL}ms large=${ARRAYSOAK_T_LARGE}ms (bound=${ARRAYSOAK_BOUND_MS}ms, Req 4.3 verdict=${ARRAYSOAK_BOUND_VERDICT}); per-element status/cancel/completion confirmed post-failover"
|
||||
cleanup
|
||||
trap - EXIT INT TERM
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Chaotic mode: overlapping-kill rounds → isolated verdict ──────────────────
|
||||
# Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up
|
||||
# windows; outcomes are COUNTED (committed divergence + committed job loss) with
|
||||
|
|
|
|||
|
|
@ -149,6 +149,58 @@ check "committed job loss => non-zero rc" 1 "$rc"
|
|||
check "committed job loss => result=FAIL token" \
|
||||
"FAIL" "$(printf '%s' "$v" | grep -oE 'result=[A-Z]+' | cut -d= -f2)"
|
||||
|
||||
# ── Array-soak replay-time bound (task 6.1 remediation round 1, Req 4.3) ─────
|
||||
# verdict_check_arraysoak_replay_bound(t_small_ms t_large_ms ratio_max
|
||||
# floor_ms margin_ms) => "<bound_ms> <OK|BOUND_EXCEEDED>". Using the
|
||||
# scenario's own defaults (ratio_max=20 floor_ms=2000 margin_ms=3000) as
|
||||
# fixtures so this selftest exercises the SAME math the live soak gates on.
|
||||
|
||||
# A healthy run: small=80ms, large=650ms. bound = max(80*20,2000)+3000 =
|
||||
# max(1600,2000)+3000 = 5000ms. 650 <= 5000 => OK. This is the shape task
|
||||
# 6.0a's own prior live runs actually saw (both sub-rounds fast, large not
|
||||
# meaningfully slower than small) — a real O(1) implementation.
|
||||
check "healthy small=80ms large=650ms => bound 5000ms" \
|
||||
"5000 OK" "$(verdict_check_arraysoak_replay_bound 80 650 20 2000 3000)"
|
||||
|
||||
# A genuine O(N) regression: small=50ms (near-instant), large=5000ms (a real
|
||||
# 5-second replay-time blowup — the exact scenario Finding 1's review called
|
||||
# out as silently passing under whole-second resolution). bound =
|
||||
# max(50*20,2000)+3000 = max(1000,2000)+3000 = 5000ms. 5000 is NOT > 5000,
|
||||
# so nudge large to 5001ms to cross the boundary and assert it trips.
|
||||
check "O(N) regression small=50ms large=5001ms => BOUND_EXCEEDED (bound 5000ms)" \
|
||||
"5000 BOUND_EXCEEDED" "$(verdict_check_arraysoak_replay_bound 50 5001 20 2000 3000)"
|
||||
|
||||
# Exact-boundary case: large-array time equal to the bound is still OK (only
|
||||
# STRICTLY greater trips it) — confirms the comparison is `>`, not `>=`.
|
||||
check "large exactly at bound => OK (boundary is inclusive)" \
|
||||
"5000 OK" "$(verdict_check_arraysoak_replay_bound 50 5000 20 2000 3000)"
|
||||
|
||||
# Zero/near-zero small-array time (the original Finding-1 whole-second
|
||||
# collapse case) must not make the bound degenerate — the floor still
|
||||
# applies, so a large-array time that is itself small stays OK.
|
||||
check "t_small=0ms (whole-second-collapse case) => floor still applies, OK" \
|
||||
"5000 OK" "$(verdict_check_arraysoak_replay_bound 0 1200 20 2000 3000)"
|
||||
check "t_small=0ms but large-array time itself blows past the floor+margin => BOUND_EXCEEDED" \
|
||||
"5000 BOUND_EXCEEDED" "$(verdict_check_arraysoak_replay_bound 0 5001 20 2000 3000)"
|
||||
|
||||
# verdict_emit_arraysoak: ratios + pass-through of the bound/verdict fields.
|
||||
# result is unconditionally PASS (the real gate is the fail() in
|
||||
# run_array_soak_round, upstream of this reporting-only function — see its
|
||||
# header comment) but the reported fields must reflect what was passed in.
|
||||
v="$(verdict_emit_arraysoak 2 300 10000 650 5000 OK)"
|
||||
check "arraysoak verdict line reports t_small_ms" \
|
||||
"t_small_ms=300" "$(printf '%s' "$v" | grep -oE 't_small_ms=[0-9]+')"
|
||||
check "arraysoak verdict line reports t_large_ms" \
|
||||
"t_large_ms=650" "$(printf '%s' "$v" | grep -oE 't_large_ms=[0-9]+')"
|
||||
check "arraysoak verdict line reports element_count_ratio" \
|
||||
"element_count_ratio=5000x" "$(printf '%s' "$v" | grep -oE 'element_count_ratio=[0-9]+x')"
|
||||
check "arraysoak verdict line passes through the replay bound" \
|
||||
"replay_bound_ms=5000" "$(printf '%s' "$v" | grep -oE 'replay_bound_ms=[0-9]+')"
|
||||
check "arraysoak verdict line passes through the replay bound verdict" \
|
||||
"replay_bound_verdict=OK" "$(printf '%s' "$v" | grep -oE 'replay_bound_verdict=[A-Z_]+')"
|
||||
check "arraysoak verdict result is always PASS (real gate is upstream fail())" \
|
||||
"PASS" "$(printf '%s' "$v" | grep -oE 'result=[A-Z]+' | cut -d= -f2)"
|
||||
|
||||
# ── Verdict ──────────────────────────────────────────────────────────────────
|
||||
echo
|
||||
if [ "$fails" -eq 0 ]; then
|
||||
|
|
|
|||
|
|
@ -109,3 +109,80 @@ verdict_emit_factstream() {
|
|||
"excluded_bug924_fdatasync=$b924 result=$result"
|
||||
[ "$result" = "PASS" ]
|
||||
}
|
||||
|
||||
# Compute the pass/fail replay-time bound for the array-soak scenario's
|
||||
# large-array kill-to-addressable time (Req 4.3: failover replay time
|
||||
# governed by the array record's size, not element count) and classify
|
||||
# t_large_ms against it. Pure/side-effect-free, unlike the rest of this
|
||||
# scenario's checks — the actual fail()/abort gate lives in
|
||||
# run_array_soak_round() (run-ha-soak.sh), which calls this helper and hard-
|
||||
# fails the run on a BOUND_EXCEEDED verdict, mirroring this scenario's other
|
||||
# hard-fail-not-count-and-tolerate invariants.
|
||||
#
|
||||
# The bound is deliberately RATIO-based with a floor and margin, not a bare
|
||||
# multiple of t_small: on a fast local single-node-per-process cluster
|
||||
# t_small itself can be a few tens/hundreds of ms, where a naive "N times
|
||||
# t_small" bound would be flaky-tight, and BOTH sub-rounds' measured time is
|
||||
# dominated by the same roughly-constant election/catch-up overhead, not
|
||||
# actual per-record replay cost — so a healthy run's t_large should sit close
|
||||
# to t_small regardless of array size. floor_ms sets a minimum meaningful
|
||||
# window so small jitter in t_small can't make the bound absurdly tight;
|
||||
# margin_ms absorbs ordinary election-timing jitter shared by both
|
||||
# sub-rounds; ratio_max is the real "how many times slower is tolerated"
|
||||
# gate — generous relative to the ~1x a genuine O(1) implementation should
|
||||
# show, but far tighter than the ~5000x growth (n_large/n_small at the
|
||||
# defaults) a real O(N) regression would produce, so a genuine regression
|
||||
# (e.g. several real seconds of added replay time) still trips it.
|
||||
#
|
||||
# Args: t_small_ms t_large_ms ratio_max floor_ms margin_ms
|
||||
# Echoes: "<bound_ms> <verdict>" where verdict is OK or BOUND_EXCEEDED.
|
||||
verdict_check_arraysoak_replay_bound() {
|
||||
local t_small="$1" t_large="$2" ratio_max="$3" floor_ms="$4" margin_ms="$5"
|
||||
local bound verdict
|
||||
bound=$((t_small * ratio_max))
|
||||
[ "$bound" -lt "$floor_ms" ] && bound="$floor_ms"
|
||||
bound=$((bound + margin_ms))
|
||||
verdict="OK"
|
||||
[ "$t_large" -gt "$bound" ] && verdict="BOUND_EXCEEDED"
|
||||
echo "$bound $verdict"
|
||||
}
|
||||
|
||||
# Emit the machine-parseable verdict line for the array-soak
|
||||
# (job-array-single-wal-record task 6.1) scenario. Unlike verdict_emit /
|
||||
# verdict_emit_factstream, this is NOT a count-and-classify verdict: every
|
||||
# hard invariant it reports on (exactly one WAL_JOB_ARRAY_SUBMIT record per
|
||||
# array submission regardless of size, every checked element addressable
|
||||
# post-failover, AND — as of this fix — the replay-time bound from
|
||||
# verdict_check_arraysoak_replay_bound above) is asserted with an immediate
|
||||
# fail() inside run_array_soak_round() in run-ha-soak.sh — a violation
|
||||
# aborts the run before this function is ever reached, mirroring
|
||||
# --force-snapshot's "the scenario proves nothing if the forcing mechanism
|
||||
# didn't fire" hard-fail convention rather than chaotic mode's
|
||||
# tolerate-and-count convention. So result is always PASS here; this
|
||||
# function's job is to report the ACTUAL replay-time measurement (Req 4.3)
|
||||
# and the bound it was checked against as a machine-parseable line, not to
|
||||
# gate pass/fail itself.
|
||||
#
|
||||
# Args: n_small t_small_ms n_large t_large_ms [bound_ms] [bound_verdict]
|
||||
# t_small_ms / t_large_ms are millisecond-resolution kill-to-first-
|
||||
# addressable measurements (see poll_addressable_elapsed in run-ha-soak.sh,
|
||||
# EPOCHREALTIME-based). bound_ms/bound_verdict are optional — when the
|
||||
# caller already computed them via verdict_check_arraysoak_replay_bound,
|
||||
# passing them through avoids recomputing and keeps the reported bound
|
||||
# identical to the one that actually gated the run.
|
||||
verdict_emit_arraysoak() {
|
||||
local n_small="$1" t_small="$2" n_large="$3" t_large="$4"
|
||||
local bound_ms="${5:-n/a}" bound_verdict="${6:-n/a}"
|
||||
local elem_ratio="n/a" time_ratio_x100="n/a"
|
||||
if [ "${n_small:-0}" -gt 0 ]; then
|
||||
elem_ratio="$((n_large / n_small))x"
|
||||
fi
|
||||
if [ "${t_small:-0}" -gt 0 ]; then
|
||||
time_ratio_x100="$(((t_large * 100) / t_small))"
|
||||
fi
|
||||
echo "[xce-test-ha] VERDICT mode=array-soak n_small=$n_small t_small_ms=$t_small" \
|
||||
"n_large=$n_large t_large_ms=$t_large element_count_ratio=$elem_ratio" \
|
||||
"replay_time_ratio_x100=$time_ratio_x100 replay_bound_ms=$bound_ms" \
|
||||
"replay_bound_verdict=$bound_verdict result=PASS"
|
||||
return 0
|
||||
}
|
||||
|
|
|
|||
|
|
@ -166,7 +166,7 @@
|
|||
- _Requirements: 3.1, 3.2, 4.1, 9.4_
|
||||
- _Boundary: Array Apply Expansion (`persist.c`'s replay-side job-count accounting) — discovered during task 6.1's soak (2026-07-15), inserted per explicit user decision; blocks 6.1/6.2/6.3, all of which require a cluster that survives a restart after an array submission_
|
||||
|
||||
- [ ] 6.1 Large-array soak scenario
|
||||
- [x] 6.1 Large-array soak scenario
|
||||
- Extend the existing HA soak harness with a large-array submission scenario asserting: one WAL record and one commit barrier regardless of array size, failover replay time governed by the record's size rather than element count, and every array element individually addressable (status, cancel, completion) after a failover
|
||||
- Observable: a soak run reporting the one-record/one-barrier assertion, a replay-time measurement independent of array size, and successful per-element status checks post-failover
|
||||
- _Requirements: 4.3, 5.2, 5.3, 8.2, 8.3_
|
||||
|
|
@ -205,3 +205,4 @@
|
|||
- **Task 5.4**: Go/CGO receive path — `rawJobArrayToModels()` (pure-Go N-way expansion), `cJobArrayToModels()` (CGO adapter sibling to `cJobToModel`), `dispatchJobArray()` (dedup via the SAME `store.Jobs.Has()` the ordinary `Data_data_jobs` path writes through, one confirmation log only on ≥1 newly-dispatched element), new `case C.Data_data_jobArray:` in `handleDataset`. Elements route through `OnJobUpdated` (not `OnJobCreated`), matching the ordinary single-job push's own routing — verified consistent, not a divergence. CGO boundary needed zero changes — `C.xce_job_submit_t` and `C.Data_data_jobArray` were already exposed/generated from prior tasks. A structural TOCTOU exists between the `Has()` dedup check and the eventual store write (two separate lock acquisitions, no compare-and-set primitive) but is narrower than task 5.3's accepted C-side gap (no intervening async hop) and strictly safer than the pre-existing baseline (the ordinary `Data_data_jobs` path has NO dedup check at all today) — not fixed, judged acceptable. `cJobArrayToModels()` has zero direct unit coverage at the CGO-struct level, but this exactly mirrors the pre-existing, repo-wide precedent for `cJobToModel()` itself (also never unit-tested with a real C struct) — reviewer independently reconstructed a full `CGO_ENABLED=1` build+link+test run against the real generated headers to confirm correctness beyond a compile-only check. `store.ArrayJobs` (array-level rollup aggregate) deliberately left unpopulated — confirmed genuinely out of this spec's scope per requirements.md's Boundary Context, which explicitly defers whole-array bulk query/cancel to a future spec; zero code anywhere in the codebase populates it today, including this diff.
|
||||
- **Task 5.5**: Cross-daemon integration test, test-only (zero net production diff, confirmed on `request.c` and every task 5.1-5.4 file). Drives real `xce_encode()`/`xce_decode()` wire bytes through the REAL production dispatch chain on both daemons — `extract_data()` → `scheduler_req_handler()` → `sched_handle_dataset()` → `sched_on_job_array()` (C, via `#include "../src/handler.c"` in the test) and `go_push_handler()` → `handleDataset()` → `cJobArrayToModels()`/`dispatchJobArray()` (Go/CGO) — skipping only `vtask_entry()`'s ~10 lines of `static` glue, which the test mirrors verbatim (independently verified line-by-line accurate on the one behavior that mattered: `if (callnext)` treating any nonzero `extract_data()` return, including a negative errno, as "proceed"). Redelivery is tested via genuinely independent fresh decodes of separately-encoded bytes, not a cheaper decode-once-call-twice shortcut. Bug-catching capability independently proven twice (implementer, then reviewer): temporarily reverting task 5.3b's fix makes both the C and Go tests fail meaningfully; restoring returns both to green. Go/cgo toolchain constraint discovered: `import "C"` is disallowed in any `_test.go` file, forcing a two-file split (`wire_dispatch_support.go` for cgo-preamble helpers, `wire_dispatch_test.go` for the actual tests) — logged as bug-1440/1441/1442. One low-severity, non-blocking nit (not fixed): the C test file's own header comment misdescribes which assertion fails first under the RED-phase revert (claims the `callnext != 0` check; actually a downstream `>= 0` check) — cosmetic only, the test's real bug-catching property is independently confirmed accurate.
|
||||
- **Task 6.0a**: Fixed `replay_wal()`'s job-count tally (persist.c ~3902) to add a `WAL_JOB_ARRAY_SUBMIT` branch that tallies `apply_rc` (the genuine per-record element count `apply_committed_entry()`'s array case already returns) when `apply_rc > 0`. Independently traced the FULL return-value space of `restore_job_array()`/`apply_committed_entry()`'s array case (6 distinct cases: 3 negative pre-mutation failures, `-EEXIST`→`apply_rc=0` idempotent no-op, the task-3.1 bulk-insert partial-failure case (stays negative, correctly still NOT tallied — a pre-existing, out-of-boundary undercount that still surfaces as a detectable anomaly rather than masking one), and positive `created` on genuine success) and confirmed every positive `apply_rc` this fix tallies is PROVABLY equal to real jobmgr growth for that record — the fix cannot overcount, and therefore cannot weaken `persist_verify_apply_correctness()`'s ability to catch a genuine divergence. Verified end-to-end via a from-scratch live 3-node HA soak re-run (`--array-soak`, both by the implementer and independently by the reviewer): zero `APPLY_DIVERGENT` across all three of the original bug's trigger paths (restart-mid-small-array, standby election-path recovery fallback, restart-mid-N=10000-array), one-WAL-record/one-commit-barrier confirmed for both N=2 and N=10000, per-element post-failover status/cancel/completion confirmed. This soak run also directly satisfies task 6.1's own Observable — its results should be reused rather than re-run from scratch. **New non-blocking finding (pre-existing, NOT introduced by this task)**: `controller_unit_test`'s single outer `UNITY_BEGIN()` is silently reset by ~19 of its called test-group functions' OWN nested `UNITY_BEGIN()/UNITY_END()` calls, making the process exit code / meson `Ok:/Fail:` summary line unreliable for detecting a regression in any group running before the last such nested call — both the implementer and reviewer worked around this by grepping raw `testlog.txt` for `:FAIL:` lines directly rather than trusting the summary, and this should be the standard practice going forward for this binary until a dedicated follow-up fixes the harness itself.
|
||||
- **Task 6.1**: Extended `.claude/skills/xce-test-ha/run-ha-soak.sh`/`soak-verdict.sh` with a new `--array-soak` mode (small N=2 + large N=10000 sub-rounds, one clean kill/failover per sub-round — deliberately single-kill, not chaotic, matching this task's correctness-not-chaos scope; chaos-under-array-submission is task 6.2's separate job). `waldump.c` gained a `WAL_JOB_ARRAY_SUBMIT` case in `wal_type_name()` so the scenario can grep an exact type name for its live before/after WAL-record-count delta check. **Round 1 REJECTED**: the replay-time measurement used whole-second (`date +%s`) resolution — both sub-rounds' timings collapsed to `0s`/`n/a` on a real run — and, more fundamentally, nothing ever asserted on the comparison (`verdict_emit_arraysoak()` always returned PASS regardless of measured values), leaving Requirement 4.3 ("replay time governed by record size, not element count") with zero actual evidence and unable to catch a genuine O(N) regression. **Round 2 fixed**: switched to `$EPOCHREALTIME` (millisecond resolution; discovered and worked around a genuine environment trap — this sandbox's BusyBox `date` precedes GNU coreutils on `$PATH` and silently passes `%N`/`%3N` through as a literal string suffix rather than expanding or erroring, which would have poisoned any `date +%s%3N`-based approach silently); added `verdict_check_arraysoak_replay_bound()` with a real `bound = max(t_small*ratio_max, floor_ms) + margin_ms` assertion that `run_array_soak_round()` hard-`fail()`s on when exceeded — independently verified to have real teeth (both the implementer's and reviewer's own hand-picked regression cases correctly trip `BOUND_EXCEEDED`). Live-reproduced 3 times total (once round 1, twice round 2) with genuinely discriminating millisecond numbers each time (final: small=210ms, large=163ms — large array replay was NOT slower, live evidence of the feature's O(1) claim). Also fixed opportunistically: status/cancel/completion checks spread across low/mid/high indices (previously clustered at 1,2,3) matching the already-good basic-addressability spread; selftest extended 17→28 cases covering the new bound-check logic; `SKILL.md` documented. "One commit barrier" remains an inference from the WAL-record-count delta (not independently observed live) — defensible given task 2.1/2.3's unit-tested 1:1 append↔continuation coupling, but flagged as an inference, not a direct check, should a future task want to close that gap.
|
||||
|
|
|
|||
|
|
@ -28,6 +28,12 @@ static const char* wal_type_name(uint32_t t)
|
|||
{
|
||||
switch (t) {
|
||||
case WAL_JOB_SUBMIT: return "JOB_SUBMIT";
|
||||
case WAL_JOB_ARRAY_SUBMIT: return "JOB_ARRAY_SUBMIT"; /* job-array-single-
|
||||
* wal-record task 6.1: name the type so the HA soak's WAL-record-count
|
||||
* assertion (exactly one JOB_ARRAY_SUBMIT per array submission,
|
||||
* regardless of array size) can grep an exact type name instead of
|
||||
* the generic "UNKNOWN" bucket, which would also catch any future,
|
||||
* genuinely-unrecognized type. */
|
||||
case WAL_JOB_START: return "JOB_START";
|
||||
case WAL_JOB_FINISH: return "JOB_FINISH";
|
||||
case WAL_JOB_FAIL: return "JOB_FAIL";
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue