test(raft-ha-core-safety): capture controller stderr in soak; root-cause 7.2 6-round blocker to a restart SEGFAULT (bug-1281, task 5.4)

run-ha-soak.sh start_node previously sent each controller's stdout+stderr to
/dev/null, hiding all crash output. Capture it per-node (logs/<node>.stderr).
That immediately revealed the true 7.2 6-round blocker.

Findings (research.md §"Task 7.2 on ~/tmp"):
- On the user's non-overlay ~/tmp, a 2-round chaotic soak COMPLETES and 6-round
  rounds 1-3 each elect a higher-term primary — 5.2 + 5.3 hold live, job safety
  holds (zero committed-job loss; committed prefix byte-identical across members).
- The 6-round stall is a RELEASE-BUILD SEGFAULT: a controller segfaults seconds
  after being restarted into the churning cluster (recovery->failover/PreVote),
  dropping the cluster to 1-of-3 so the survivor can't reach quorum. Captured
  stderr: "<pid> Segmentation fault (core dumped) ... --myname ctrl-3".
- NOT consensus, NOT the in_catch_up deadlock (5.3 fixes that, unit-proven), NOT
  bug-924 (a separate fdatasync wedge hit one earlier run only). ASAN-invisible
  heisenbug; no gdb / no usable core in the sandbox, so the backtrace needs a
  debug+core+gdb env.

Adds task 5.4 (root-cause+fix the chaotic-restart segfault) and bug-1281;
re-attributes the 7.2 block from bug-924 to the segfault. Also notes the soak's
divergence over-count (boundary 1+2*njobs vs primary committedLsn) as a separate
Phase-1 accuracy fix. Consensus work (Phase-2/5.2/5.3) is unit-complete and live-
confirmed for every round that completes.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
This commit is contained in:
Rongsong Shen 2026-06-30 03:33:25 +00:00
parent 5d72133216
commit 75243fbb86
4 changed files with 988 additions and 865 deletions

View file

@ -147,7 +147,14 @@ trap cleanup EXIT INT TERM
# found"), not the leak-polluted process exit code.
xtest() { ASAN_OPTIONS=detect_leaks=0 timeout 60 "$XT" 2>&1; }
start_node() { bash "$TOP/scripts/start.sh" "$1" -vvv >/dev/null 2>&1 & }
start_node() {
# Capture each controller's stdout+stderr to a per-node file so a SILENT crash
# (segfault/abort to stderr — the --logfile only holds tlog INFO/DEBUG, not a
# libc/ASAN crash message) leaves a forensic trace. Appended (>>) so a
# restarted node's later crash does not clobber the earlier instance's output.
mkdir -p "$TOP/logs"
bash "$TOP/scripts/start.sh" "$1" -vvv >>"$TOP/logs/$1.stderr" 2>&1 &
}
# ── Primary / convergence detection: CLIENT PROBE (release-build-log-immune) ──
# The RELEASE controller block-buffers its --logfile and flushes only on a signal