Extends the HA soak harness with a new --array-soak mode: small (N=2)
and large (N=10000) array submission sub-rounds, each with one clean
kill/failover, asserting one WAL record per submission regardless of
array size, per-element addressability (status/cancel/completion)
spread across low/mid/high indices post-failover, and a real
millisecond-resolution replay-time bound comparing large-array vs
small-array kill-to-addressable latency.
waldump.c gains a WAL_JOB_ARRAY_SUBMIT case in wal_type_name() so the
scenario can grep an exact type name for its live WAL-record-count
delta check.
Went through one remediation round: the replay-time measurement
initially used whole-second resolution (collapsed to 0s/n-a on a real
run) and never asserted on the comparison, giving zero actual evidence
for the O(1)-replay-time requirement. Fixed with $EPOCHREALTIME
(millisecond resolution -- this sandbox's BusyBox date silently
mangles the more common `date +%s%3N` idiom) and a real bound
assertion that hard-fails the run on a genuine O(N) regression,
independently verified to have real teeth.
Live-verified 3 times with genuinely discriminating numbers each time;
final run: small=210ms, large=163ms -- large-array replay was not
slower, live evidence of the feature's O(1) claim.
Task 6.1 of job-array-single-wal-record (Requirements 4.3, 5.2, 5.3,
8.2, 8.3). Depends on and reuses task 6.0a's own soak verification.
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.
Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.
Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
Diffs member WALs by (lsn -> term, pcrc) over the committed prefix, exits non-zero
on any committed-prefix divergence with the diverging LSN + per-member entry
identities. Reuses the canonical waldump decoder via a new per-entry visitor seam
(no second WAL parser). Detects same-LSN/diff-term and same-LSN/diff-content
divergences (the bug-851 class) that an LSN-range check would mask.
