run-ha-soak.sh start_node previously sent each controller's stdout+stderr to
/dev/null, hiding all crash output. Capture it per-node (logs/<node>.stderr).
That immediately revealed the true 7.2 6-round blocker.
Findings (research.md §"Task 7.2 on ~/tmp"):
- On the user's non-overlay ~/tmp, a 2-round chaotic soak COMPLETES and 6-round
rounds 1-3 each elect a higher-term primary — 5.2 + 5.3 hold live, job safety
holds (zero committed-job loss; committed prefix byte-identical across members).
- The 6-round stall is a RELEASE-BUILD SEGFAULT: a controller segfaults seconds
after being restarted into the churning cluster (recovery->failover/PreVote),
dropping the cluster to 1-of-3 so the survivor can't reach quorum. Captured
stderr: "<pid> Segmentation fault (core dumped) ... --myname ctrl-3".
- NOT consensus, NOT the in_catch_up deadlock (5.3 fixes that, unit-proven), NOT
bug-924 (a separate fdatasync wedge hit one earlier run only). ASAN-invisible
heisenbug; no gdb / no usable core in the sandbox, so the backtrace needs a
debug+core+gdb env.
Adds task 5.4 (root-cause+fix the chaotic-restart segfault) and bug-1281;
re-attributes the 7.2 block from bug-924 to the segfault. Also notes the soak's
divergence over-count (boundary 1+2*njobs vs primary committedLsn) as a separate
Phase-1 accuracy fix. Consensus work (Phase-2/5.2/5.3) is unit-complete and live-
confirmed for every round that completes.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
5.3 fix (bf88681) lands and is unit-validated (test_ha_core_safety 20 Tests /
0 Failures). Re-ran the 7.2 chaotic soak with all Phase-2+5.2+5.3 fixes: it
still stalls at round 4, but bisecting the frozen logs shows the cause is the
bug-924 WAL-fdatasync wedge, NOT consensus — a survivor's persist flusher spins
on `fdatasync FAILED (errno=22 EINVAL)` and stops answering PreVotes, so the
round times out (presents as an election stall, is a durability-layer wedge).
A file-level repro of the controller's exact WAL pattern (open + posix_fallocate
64MB + write + fdatasync) SUCCEEDS on every available ext4/tmpfs path, so it is
the live overlay-backed WAL fd under chaotic restart — the documented bug-924,
a Non-Goal of this spec (owned by ha-commit-durability, excluded from the
verdict).
Re-attributes the round-4 stall (the first §"Task 7.2" note guessed in_catch_up;
5.3 fixes that real latent eligibility deadlock at unit level but it is not the
soak blocker). 7.2 is now BLOCKED ON ENVIRONMENT, not a consensus NO-GO; the
consensus work this spec owns (Phase-2 payload integrity, 5.2, 5.3) is
unit-complete. research.md §"Task 5.3 + 7.2 re-run"; bug-1272 updated.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
First acceptance chaotic soak (6 rounds, real-fs ext4, release binaries rebuilt
at HEAD 01419e1 with Phase-2 payload fix + Phase-5 5.2 suppression). Outcome:
PASS (job safety + landed fixes):
- Zero committed-job loss (all tracked jobs present on the new primary each round).
- Committed prefix [1..11] byte-identical across all 3 members — Phase-2 payload-loss
signature absent. All counted walcompare divergences are present=2 catch-up-LAG
MISSING tails of restarted-uncaught followers (tips 19/15/11), not corruption.
- Rounds 1-3 elect a strictly higher-term primary; 5.2 spurious-failover suppression
confirmed working (round-4 genuine-loss takes the failover path, not suppression).
NO-GO (R8.3 liveness):
- Round-4 election STALL after 3 overlapping kills: the up-to-date leader (tip 19)
is killed, both survivors are lagging (15, 11) but each holds the full committed
prefix; their PreVote rounds repeatedly time out with no grant quorum, so no
primary is elected. Distinct from the 5.2 flap. Contributors: freshness
mutual-denial, in_catch_up-denies-all-votes (Req 2.5, obs 1240) with no live
primary to catch up from, and/or unestablished peer vote connections.
Per the chaotic-harness contingency: recorded full classification in research.md
§"Task 7.2"; added task 5.3 (lagging-but-safe survivor must be electable under
overlapping churn); 7.2 marked _Blocked_ on 5.3; logged bug-1272. Frozen evidence
at /home/claude/tmp/ha-7p2-frozen.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
A recovered standby whose control channel flaps (transient RDHUP) while a
current-term primary is alive treated each flap as "primary died" and launched
an unwinnable PreVote failover (peers see the live primary and deny, the round
times out, the channel re-establishes, repeat ~15s) — so the node prevote-spun
forever instead of rejoining, wedging the chaotic soak (research.md §"Phase 4/5").
Add a freshness guard at the RDHUP failover-initiation path
(ha_on_primary_ctrl_fd_rdhup): when CTRL_STANDBY and ha_heartbeat_is_fresh,
stay CTRL_STANDBY, re-arm the liveness watchdog (production), and await the
primary's reconnect instead of campaigning — the same guard
ha_failover_stagger_fn already applies on the deferred PreVote path. Genuine
loss (stale heartbeat) falls through to failover unchanged; no term bump / no
role change, so Election Safety is untouched.
Keeping the standby attached lets catch-up complete, which clears in_catch_up
and lets votes flow again — so this one guard also resolves the downstream
catching-up-node vote-deny deadlock (obs 1240).
Flips task 5.1's CORE RED to GREEN: test_ha_core_safety 17 Tests / 0 Failures
(spurious launches=0/5; genuine-loss final_term=8; Election Safety A=3/B=0 →
1 primary). Zero failing suites binary-wide; LSan unchanged vs baseline.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Add three Phase-5 cases to test_ha_core_safety.c pinning the spurious-failover
root cause (research.md §"Phase 4/5"):
- CORE RED test_5_1_spurious_failover_suppressed_while_primary_alive: a
control-channel flap (transient RDHUP) while a current-term primary is alive
(fresh heartbeat) must NOT launch a PreVote failover. FAILs on baseline
(Expected 0 Was 5) — the RDHUP-initiation path (ha_on_primary_ctrl_fd_rdhup
-> ha_begin_failover -> ha_start_prevote_async synchronous branch) has no
ha_heartbeat_is_fresh guard, unlike the deferred ha_failover_stagger_fn path.
GREEN when task 5.2 adds that guard.
- LIVENESS test_5_1_genuine_loss_elects_and_advances_term: genuine loss (stale
heartbeat) still launches a real failover; a failed round re-arms; rapid
rounds advance the term. Guards 5.2 against over-suppression (Req 6.1/6.2/6.3).
- SAFETY test_5_1_election_safety_one_primary_per_term: two candidates contest
the same term; the real handle_request_vote grants at most one vote per term
(A=3/B=0), so at most one committing primary per term. Falsifiable Election
Safety check mapped to raft/raft.tla (Req 4.1/4.3).
Deterministic (no sleeps; clock reads only), c==NULL unit-test mode, clean
ha_fini teardown. test_ha_core_safety suite: 17 Tests / 1 Failure (the
intended RED); all 2.x/4.x/seam cases still pass; no new ASAN leaks.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Phase-4/5 investigation found the live stall is a recovered standby with a
divergent uncommitted tail (former-primary shape) that never reconverges:
loops on ae_reject_logmismatch, its control channel flaps ~15s, each flap
spawns an unwinnable PreVote failover while a live primary exists.
- Phase 4 (Standby Reconvergence): lead with divergent-tail truncate+catch-up
(make the AE backtrack + follower-conflict-truncate fire for a former-primary
tail, not just a short suffix), keep the applied/log_len healthy gate.
- Phase 5 (Election Liveness): retarget from 'missing election-restart' (retry
works) to spurious-failover suppression — reconnect+catch-up on transient
RDHUP, don't storm unwinnable PreVotes while a current-term primary is
reachable. 5.2 now _Depends 4.2_.
Root-cause source fix: ha_stream_wal_to_standby reads the entry body via
stream_read_entry_body (bounded retry max 4, no sleep); on persistent short
pread/malloc-fail it fails loud + ABORTS the stream (flush valid prefix,
drop unreadable entry, -EIO) instead of shipping PLEN=0. Incomplete stream
re-driven by ACK-driven catch-up. With the 2.2 guard this closes the live
cross-term payload-loss divergence. 2.3 CORE RED -> GREEN (fail-loud: no
empty persisted; + retry-recover test ships full 64). Term-blind structural
retirement deferred to 6.2 (empty-over-full safety enforced by 2.2 guard +
pinned). 12/12 core suite; siblings green; no regression introduced.
Phase 2 complete.
Minimal XCE_TESTING fault-injection seam in ha_stream_wal_to_standby
(NULL/production-inert) forces a short body read on a cross-term entry.
CORE RED proves the streamer ships PLEN=0 for a full (source_len=64)
cross-term entry: instrumentation source_len=64 SEND_len=0 RECV_len=0 ->
loss is SEND-side (answers task 1.3's open send-vs-transitive question).
2 honest pins (term-blind route guarded by 2.2; below-tip fence). 11
tests/1 RED; production xce_controller links clean without the seam.
Case (a) empty-then-full = genuine RED (payload-blind guard at persist.c:822
no-ops the corrective write, Expected 64 Was 0). Cases (b) full-then-empty
and (c) identical-rewrite = baseline-passing pins that constrain the 2.2
fix asymmetry (empty must never overwrite stored full; identical re-stream
stays idempotent). 7 tests/1 RED/no ASAN.
Release build block-buffers --logfile (flush on signal only), so log-grep
primary/convergence detection false-aborted. Replace all log-grep detection
with client probes (xce_test ctrlstatus): probe_primary/node_probe/
cluster_committed_converged. Both modes use shared helpers; 1.2 verdict
self-test 17/17 intact; bug-854 isolation preserved. Live release-binary
runs reach the verdict (chaotic re-confirms committed_divergence=3 / job
loss 0; sequential PASS). No production code touched. Phase 1 complete.
New test_ha_core_safety.c with deterministic gate/probe seams (no sleeps,
drain-before-fini per bug-1224): payload-controllable WAL writer, on-disk
(term,pcrc,len) readback probe, walcompare observer, apply-gap + kill/
election injection for phases 2.3/4.1/5.1. Smoke-RED (Expected 64 Was 0)
proves the payload-blind (term,lsn)-only guard at persist.c:822 drops a
corrective full-payload write at a committed LSN. 5 tests/1 RED/4 seam
PASS, no ASAN errors.
Reproduced committed divergence at LSN 9 (real-fs release soak): all 3
members term=3, minority ctrl-1 pcrc=0/PLEN=0 vs majority PLEN=3560 =
PAYLOAD loss, term correct. Bisect: arrived via the reconcile seam under
cross-term catch-up (stored term 3, leader term 4 / bug-1067 path), NOT
the term-blind bypass. Real defect = cross-term catch-up payload loss in
ha_stream_wal_to_standby (silent empty ship on short pread/malloc-fail).
Independently verified against frozen WALs. Secondary: release-build log
buffering breaks soak primary-detection.
Per-member (lsn,term,pcrc,present) report + committed-vs-uncommitted
divergence classification via walcompare_report()/--report. Soak signal
(OK/exit-code) preserved. 4 new unit tests, all GREEN.
