fix(xce-test-ha): source soak divergence boundary from live committedLsn, guard against a stale zero (task 7.2)

The chaotic soak's walcheck()/walcheck_count() previously forced a
static `1 + 2*njobs` boundary onto walcompare, which doesn't account
for per-election NOOP WAL entries and overshoots the real tip --
producing spurious committed_divergence counts on an otherwise-clean
cluster (verified via waldump: byte-identical, gap-free WALs across
all 3 nodes despite a reported divergence=7).

Replaces the static estimate with the LIVE, authoritative committedLsn
(via the existing probe_primary client probe), applied as an explicit
--committed boundary only when the cluster is genuinely converged
(cluster_committed_converged) -- falling back to walcompare's own
auto-derive (min CRC-intact tip) when not, so a legitimately-lagging
just-restarted standby never gets a false "MISSING" verdict mid-chaos.
Pure auto-derive alone can't catch a member that permanently lost a
committed entry (its own low tip just becomes the shared boundary),
so this restores exact-boundary detection at every genuinely-settled
instant while keeping the auto-derive fallback for in-flight kills.

Also fixes a live-caught edge case in the same mechanism: a transient
ctrlstatus race can report committedLsn=0 for the primary right after
reconvergence; live_committed_boundary() now rejects a bare "0" as
untrustworthy and falls back to auto-derive, instead of handing
walcompare a vacuous --committed 0 boundary that would check nothing.

Selftest 17/17. Three independent fresh 6-round chaotic soaks (own
run + reviewer's reproduction + post-guard-fix re-run) all PASS:
committed_divergence=0, committed_job_loss=0, 6/6 elections, 21/21
jobs present throughout. No production C code touched.
This commit is contained in:
Rongsong Shen 2026-07-01 23:47:12 +00:00
parent 7eefe769a3
commit a525091b02
2 changed files with 100 additions and 16 deletions

View file

@ -201,7 +201,7 @@
- _Requirements: 7.4_
- _Depends: 6.2, 5.2_
- [ ] 7.2 Acceptance chaotic-failover soak gate
- [x] 7.2 Acceptance chaotic-failover soak gate
- Run the chaotic soak (≥6 rounds, overlapping kills) against real-fs release binaries and assert zero `walcompare` divergence, zero committed-job loss, and 3-of-3 reconvergence plus a new primary after every kill round, with bug-854/bug-924 isolated from the verdict
- Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip
- _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_