test(raft-ha-core-safety): capture controller stderr in soak; root-cause 7.2 6-round blocker to a restart SEGFAULT (bug-1281, task 5.4)

run-ha-soak.sh start_node previously sent each controller's stdout+stderr to
/dev/null, hiding all crash output. Capture it per-node (logs/<node>.stderr).
That immediately revealed the true 7.2 6-round blocker.

Findings (research.md §"Task 7.2 on ~/tmp"):
- On the user's non-overlay ~/tmp, a 2-round chaotic soak COMPLETES and 6-round
  rounds 1-3 each elect a higher-term primary — 5.2 + 5.3 hold live, job safety
  holds (zero committed-job loss; committed prefix byte-identical across members).
- The 6-round stall is a RELEASE-BUILD SEGFAULT: a controller segfaults seconds
  after being restarted into the churning cluster (recovery->failover/PreVote),
  dropping the cluster to 1-of-3 so the survivor can't reach quorum. Captured
  stderr: "<pid> Segmentation fault (core dumped) ... --myname ctrl-3".
- NOT consensus, NOT the in_catch_up deadlock (5.3 fixes that, unit-proven), NOT
  bug-924 (a separate fdatasync wedge hit one earlier run only). ASAN-invisible
  heisenbug; no gdb / no usable core in the sandbox, so the backtrace needs a
  debug+core+gdb env.

Adds task 5.4 (root-cause+fix the chaotic-restart segfault) and bug-1281;
re-attributes the 7.2 block from bug-924 to the segfault. Also notes the soak's
divergence over-count (boundary 1+2*njobs vs primary committedLsn) as a separate
Phase-1 accuracy fix. Consensus work (Phase-2/5.2/5.3) is unit-complete and live-
confirmed for every round that completes.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC

.claude/skills/xce-test-ha/run-ha-soak.sh

@@ -147,7 +147,14 @@ trap cleanup EXIT INT TERM
 # found"), not the leak-polluted process exit code.
 xtest() { ASAN_OPTIONS=detect_leaks=0 timeout 60 "$XT" 2>&1; }
 
-start_node() { bash "$TOP/scripts/start.sh" "$1" -vvv >/dev/null 2>&1 & }
+start_node() {
+    # Capture each controller's stdout+stderr to a per-node file so a SILENT crash
+    # (segfault/abort to stderr — the --logfile only holds tlog INFO/DEBUG, not a
+    # libc/ASAN crash message) leaves a forensic trace.  Appended (>>) so a
+    # restarted node's later crash does not clobber the earlier instance's output.
+    mkdir -p "$TOP/logs"
+    bash "$TOP/scripts/start.sh" "$1" -vvv >>"$TOP/logs/$1.stderr" 2>&1 &
+}
 
 # ── Primary / convergence detection: CLIENT PROBE (release-build-log-immune) ──
 # The RELEASE controller block-buffers its --logfile and flushes only on a signal

.kiro/specs/raft-ha-core-safety/research.md

@@ -350,3 +350,54 @@ real-fs host/CI where the controller WAL `fdatasync` succeeds, OR the soak harne
 detect the fdatasync-wedge and exclude it from the liveness verdict (a Phase-1 harness
 improvement), OR bug-924 is resolved by `ha-commit-durability`. The consensus fixes do
 not need re-work for this; it is an environment/durability-layer blocker.
+
+---
+
+# Task 7.2 on ~/tmp (non-overlay) — DEFINITIVE 6-round blocker: a controller restart SEGFAULT
+
+Re-ran the chaotic soak on the user-provided non-overlay path `~/tmp` (= `/home/claude/tmp`)
+with the release binaries (Phase-2 + 5.2 + 5.3) and a harness change to **capture each
+controller's stdout+stderr** (`start_node` previously sent it to `/dev/null`, hiding all
+crash output — the key diagnostic gap).
+
+## What is now PROVEN
+- **Elections work under chaotic churn.** A 2-round chaotic soak COMPLETES (both rounds
+  elect a higher-term primary). In 6-round runs, rounds 1–3 each elect (terms climb
+  4→5→6). 5.2 + 5.3 hold live. Job safety holds (zero committed-job loss every run).
+- **The 6-round blocker is a RELEASE-BUILD SEGFAULT on restart**, captured at last in the
+  per-node stderr: `start.sh: line 14: <pid> Segmentation fault (core dumped) … xce_controller
+  --myname ctrl-3 …`. A controller **segfaults a few seconds after being restarted into the
+  churning cluster** (during recovery→failover/PreVote). One node down → cluster at 1-of-3 →
+  the lone survivor cannot reach quorum → "NO new primary within window." The harness's other
+  stderr lines are the expected `kill -9` ("Killed"); only the restarted node shows SEGV.
+- **This is NOT** the in_catch_up eligibility deadlock (5.3 fixes that, unit-proven), **NOT**
+  bug-924 fdatasync (that wedged a node in ONE earlier run but is absent here — zero
+  `fdatasync FAILED`), and **NOT** the consensus logic this spec owns. It is a restart/rejoin
+  stability crash.
+
+## Why it could not be root-caused in this sandbox
+- The crash is a **heisenbug**: it does NOT reproduce under the ASAN controller build (all
+  ASAN-run stderr shows only harness `kill -9`, no SEGV / no ASAN report) — memory-layout /
+  timing dependent.
+- No `gdb` and no usable core: `core_pattern` pipes to a non-running `systemd-coredump`, so
+  the "core dumped" core is discarded; `coredumpctl`/`gdb` are absent from the env.
+- Root-causing requires a debug-non-ASAN build with core dumps + gdb (or a host where the
+  systemd-coredump pipeline is live), i.e. a tooling environment this sandbox lacks.
+
+## Harness accuracy note (separate, in-scope)
+The soak counts `committed_divergence` via `walcompare --report --committed $((1 + 2*njobs))`
+(run-ha-soak.sh:543/663). That boundary is the OPTIMISTIC job-count estimate, not the true
+majority-committed LSN, so a lagging follower's not-yet-replicated tail (uncommitted on that
+member) is mis-counted as committed divergence. The completed 2-round soak reported
+`committed_divergence=3 committed_job_loss=0` — and a frozen-WAL `walcompare` at the true
+boundary shows the committed prefix byte-identical (the 3 are catch-up-lag MISSING tails, not
+corruption). `ctrlstatus` already reports `committedLsn`; the boundary should use it. (Even
+fixed, the 6-round verdict stays blocked by the segfault above.)
+
+## 7.2 status (corrected): blocked by a restart SEGFAULT, not consensus, not bug-924
+Consensus correctness (Phase-2 payload integrity, 5.2 spurious-failover suppression, 5.3
+catching-up-survivor electability) is unit-complete and confirmed live for every round that
+completes. The clean ≥6-round acceptance verdict is blocked by the restart segfault — added
+follow-up **task 5.4** (root-cause+fix the chaotic-restart crash; needs core+gdb / debug env)
+and **bug-1281**. The harness `committedLsn`-boundary fix is a separate Phase-1 accuracy
+improvement.

.kiro/specs/raft-ha-core-safety/tasks.md

@@ -120,6 +120,14 @@
   - _Depends: 5.2_
   - _Note: UNIT clause DONE (RED→GREEN, commit bf88681; `test_ha_core_safety` 20 Tests / 0 Failures). The 6-round-soak clause shares 7.2's environment block — the soak re-run still stalls at round 4, but the cause is the bug-924 WAL-fdatasync wedge (a survivor jams its persist flusher on `fdatasync`→EINVAL and stops answering PreVotes), NOT the eligibility deadlock this task fixes. See `research.md` §"Task 5.3 + 7.2 re-run". The eligibility fix is a real latent-defect fix proven at unit level; its live confirmation rides on 7.2's env resolution._
 
+- [ ] 5.4 Root-cause + fix the chaotic-restart controller SEGFAULT (NEW — the real 7.2 6-round blocker)
+  - _Root cause (task 7.2 on ~/tmp, `research.md` §"Task 7.2 on ~/tmp"; bug-1281): with per-node stderr finally captured (start_node had discarded it), a release `xce_controller` **Segmentation faults a few seconds after being restarted** into the churning cluster (during recovery→failover/PreVote). One restarted node down → cluster at 1-of-3 → no quorum → round-N "election stall." This is NOT consensus, NOT the in_catch_up deadlock (5.3 fixes that), NOT bug-924. Elections + job safety hold for every round that completes (2-round soak completes; 6-round rounds 1–3 elect)._
+  - Investigate: reproduce under a debug-non-ASAN build with core dumps enabled + gdb (the crash is ASAN-INVISIBLE — heisenbug, memory-layout/timing dependent — and the sandbox has no gdb and discards the core via a non-running systemd-coredump). Get a backtrace; classify whether the crashing site is in this spec's surface (election/5.2/5.3) or a recovery/rejoin/persist path (route upstream if so). Add a RED that reproduces the crash deterministically before fixing.
+  - Observable completion: the chaotic-restart crash is root-caused to a concrete site with a backtrace, fixed (or routed to its owning spec), and a restarted node no longer segfaults across a ≥6-round chaotic soak
+  - _Requirements: 5.1, 6.2, 7.4, 8.3_
+  - _Boundary: Standby Reconvergence, Election Liveness_
+  - _Depends: 5.3_
+
 - [ ] 6. Integration: single-seam invariant end-to-end
 - [ ] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end
   - Integration check that the live leader replication AND the cross-term catch-up streaming path transmit each entry's full `(term, payload)` to the receiver: no `term==0` on the wire from a live leader (defensive), and — the real concern — no non-empty entry ever arrives with `PLEN=0`; the walAck catch-up continuation feeds only the seam ingest path
@@ -145,8 +153,8 @@
   - Run the chaotic soak (≥6 rounds, overlapping kills) against real-fs release binaries and assert zero `walcompare` divergence, zero committed-job loss, and 3-of-3 reconvergence plus a new primary after every kill round, with bug-854/bug-924 isolated from the verdict
   - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip
   - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_
-  - _Depends: 1.5, 6.2, 5.2, 5.3_
-  - _Blocked: ENVIRONMENT (bug-924), not a consensus NO-GO. Two soak runs (HEAD 01419e1 then bf88681 with 5.3) both stall at round 4 — but the cause is the **bug-924 WAL-fdatasync wedge**: a survivor's persist flusher jams on `fdatasync`→EINVAL and stops answering PreVotes, so the round times out (it PRESENTS as an election stall but is a durability-layer wedge). Job safety + Phase-2 prefix integrity hold every run (zero job loss; committed prefix byte-identical; counted divergences are catch-up-lag MISSING tails, not corruption); 5.2 + 5.3 confirmed working. bug-924 is a Non-Goal of this spec (owned by `ha-commit-durability`, excluded from the verdict). A clean verdict needs a real-fs host/CI where the controller WAL `fdatasync` succeeds, OR a Phase-1 harness change to detect+exclude the fdatasync-wedge from the liveness verdict. Full classification in `research.md` §"Task 5.3 + 7.2 re-run"; bug-1272._
+  - _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_
+  - _Blocked (CORRECTED): a controller restart SEGFAULT (bug-1281, task 5.4), NOT consensus and NOT bug-924. With per-node stderr finally captured, the round-N stall is shown to be a release `xce_controller` that **segfaults seconds after restart** into the churning cluster — dropping the cluster to 1-of-3 so the survivor can't reach quorum. (An earlier run ALSO hit a separate bug-924 `fdatasync` wedge; that is a Non-Goal, but the consistent 6-round blocker is the segfault.) Elections + job safety hold for every round that completes (2-round soak COMPLETES; 6-round rounds 1–3 elect; committed prefix byte-identical; 5.2+5.3 working live). The segfault is ASAN-invisible and the sandbox has no gdb / discards cores → root-cause needs a debug+core+gdb env. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs the primary's `committedLsn`). Full detail in `research.md` §"Task 7.2 on ~/tmp"; bug-1281. Re-run 7.2 after 5.4._
 
 - [ ] 7.3 Hypothesis-validation closeout
   - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure