run-ha-soak.sh start_node previously sent each controller's stdout+stderr to
/dev/null, hiding all crash output. Capture it per-node (logs/<node>.stderr).
That immediately revealed the true 7.2 6-round blocker.
Findings (research.md §"Task 7.2 on ~/tmp"):
- On the user's non-overlay ~/tmp, a 2-round chaotic soak COMPLETES and 6-round
rounds 1-3 each elect a higher-term primary — 5.2 + 5.3 hold live, job safety
holds (zero committed-job loss; committed prefix byte-identical across members).
- The 6-round stall is a RELEASE-BUILD SEGFAULT: a controller segfaults seconds
after being restarted into the churning cluster (recovery->failover/PreVote),
dropping the cluster to 1-of-3 so the survivor can't reach quorum. Captured
stderr: "<pid> Segmentation fault (core dumped) ... --myname ctrl-3".
- NOT consensus, NOT the in_catch_up deadlock (5.3 fixes that, unit-proven), NOT
bug-924 (a separate fdatasync wedge hit one earlier run only). ASAN-invisible
heisenbug; no gdb / no usable core in the sandbox, so the backtrace needs a
debug+core+gdb env.
Adds task 5.4 (root-cause+fix the chaotic-restart segfault) and bug-1281;
re-attributes the 7.2 block from bug-924 to the segfault. Also notes the soak's
divergence over-count (boundary 1+2*njobs vs primary committedLsn) as a separate
Phase-1 accuracy fix. Consensus work (Phase-2/5.2/5.3) is unit-complete and live-
confirmed for every round that completes.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Rongsong Shen
75243fbb86