forked from lei.chen/myxce
test(raft-ha-core-safety): capture controller stderr in soak; root-cause 7.2 6-round blocker to a restart SEGFAULT (bug-1281, task 5.4)
run-ha-soak.sh start_node previously sent each controller's stdout+stderr to /dev/null, hiding all crash output. Capture it per-node (logs/<node>.stderr). That immediately revealed the true 7.2 6-round blocker. Findings (research.md §"Task 7.2 on ~/tmp"): - On the user's non-overlay ~/tmp, a 2-round chaotic soak COMPLETES and 6-round rounds 1-3 each elect a higher-term primary — 5.2 + 5.3 hold live, job safety holds (zero committed-job loss; committed prefix byte-identical across members). - The 6-round stall is a RELEASE-BUILD SEGFAULT: a controller segfaults seconds after being restarted into the churning cluster (recovery->failover/PreVote), dropping the cluster to 1-of-3 so the survivor can't reach quorum. Captured stderr: "<pid> Segmentation fault (core dumped) ... --myname ctrl-3". - NOT consensus, NOT the in_catch_up deadlock (5.3 fixes that, unit-proven), NOT bug-924 (a separate fdatasync wedge hit one earlier run only). ASAN-invisible heisenbug; no gdb / no usable core in the sandbox, so the backtrace needs a debug+core+gdb env. Adds task 5.4 (root-cause+fix the chaotic-restart segfault) and bug-1281; re-attributes the 7.2 block from bug-924 to the segfault. Also notes the soak's divergence over-count (boundary 1+2*njobs vs primary committedLsn) as a separate Phase-1 accuracy fix. Consensus work (Phase-2/5.2/5.3) is unit-complete and live- confirmed for every round that completes. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
This commit is contained in:
parent
5d72133216
commit
75243fbb86
4 changed files with 988 additions and 865 deletions
|
|
@ -147,7 +147,14 @@ trap cleanup EXIT INT TERM
|
|||
# found"), not the leak-polluted process exit code.
|
||||
xtest() { ASAN_OPTIONS=detect_leaks=0 timeout 60 "$XT" 2>&1; }
|
||||
|
||||
start_node() { bash "$TOP/scripts/start.sh" "$1" -vvv >/dev/null 2>&1 & }
|
||||
start_node() {
|
||||
# Capture each controller's stdout+stderr to a per-node file so a SILENT crash
|
||||
# (segfault/abort to stderr — the --logfile only holds tlog INFO/DEBUG, not a
|
||||
# libc/ASAN crash message) leaves a forensic trace. Appended (>>) so a
|
||||
# restarted node's later crash does not clobber the earlier instance's output.
|
||||
mkdir -p "$TOP/logs"
|
||||
bash "$TOP/scripts/start.sh" "$1" -vvv >>"$TOP/logs/$1.stderr" 2>&1 &
|
||||
}
|
||||
|
||||
# ── Primary / convergence detection: CLIENT PROBE (release-build-log-immune) ──
|
||||
# The RELEASE controller block-buffers its --logfile and flushes only on a signal
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue