docs(raft-ha-core-safety): unify 7.2 round-N blocker via gdb — one restart-stability bug (SIGSEGV / iomgr accept-hang)

Used the now-available gdb (wrapped the release controller under gdb -batch via
XCE_HA_BIN) to expose the round-4 blocker's second face. Findings:

- The release-build SIGSEGV and a gdb-run accept-loop HANG are the SAME restart-
  time I/O-subsystem corruption: under gdb the memory bug is perturbed away from
  a crash and instead wedges the inbound-accept path (ctrl-3 stops accepting at
  03:42:03 while still firing its own election timer), so two restarted survivors
  can never exchange PreVotes -> permanent round-N stall.
- This is a restart-STABILITY bug, NOT a consensus-logic gap: PreVote/eligibility/
  freshness (incl. 5.3) are correct; the restarted node is simply I/O-dead. It
  predates this session (baseline 21e26a1 failed round 3-4).
- Heisenbug: invisible under ASAN and under gdb; no usable core (read-only
  core_pattern -> non-running systemd-coredump). Needs a non-ASAN DEBUG (-O0 -g)
  build under gdb, or a host with a live coredump pipeline, to backtrace.

Updates research.md, refines task 5.4 (the unified restart-stability bug +
the debug-build path), updates bug-1281. Consensus correctness (Phase-2/5.2/5.3)
is unaffected and validated for every round that completes (2-round soak PASSES;
6-round rounds 1-3 elect; zero job loss; committed prefix byte-identical).

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC

.kiro/specs/raft-ha-core-safety/research.md

@@ -401,3 +401,42 @@ completes. The clean ≥6-round acceptance verdict is blocked by the restart seg
 follow-up **task 5.4** (root-cause+fix the chaotic-restart crash; needs core+gdb / debug env)
 and **bug-1281**. The harness `committedLsn`-boundary fix is a separate Phase-1 accuracy
 improvement.
+
+---
+
+# Task 7.2/5.4 — UNIFIED root cause via gdb: round-N stall is a restart-stability bug (I/O wedge / SIGSEGV), not a consensus-logic gap
+
+Wrapped the release `xce_controller` under `gdb -batch` (handles RT signals; dumps all-thread
+bt on SIGSEGV) via `XCE_HA_BIN=<gdb-wrapper>` and re-ran the chaotic soak on `~/tmp`.
+
+## Decisive findings
+- **Two round-4 faces, ONE root cause.** Plain release: a restarted node **SIGSEGVs**
+  (captured `Segmentation fault (core dumped) … --myname ctrl-3`). Under gdb the crash does NOT
+  fire (ptrace perturbs the memory bug); instead the same restarted node's **inbound-accept /
+  iomgr path WEDGES** — ctrl-3's last `cluster.c:737 server sock accepted` is 03:42:03, yet it
+  keeps firing its own election timer (`pre_vote_start`) to 03:51:59. It campaigns but can
+  neither accept peers' PreVote connections nor get answered; ctrl-2's outbound PreVote connects
+  hang on the dead listener (zero connect-fail logs). The two restarted survivors can't exchange
+  votes → permanent round-4 stall. Release SEGV and gdb accept-hang = the same restart-time
+  corruption manifesting differently.
+- **Restart-STABILITY bug, not a consensus-logic gap.** PreVote/eligibility/freshness (incl. the
+  5.3 fix) are correct; the node is simply I/O-dead after restart. Predates this session —
+  baseline `21e26a1` failed round 3–4 (requirements.md symptom 3). 5.2/5.3 sit in the
+  restart/failover path so 5.4 must rule them out during the backtrace, but they did not
+  introduce the instability.
+- **Cannot backtrace in this sandbox.** Heisenbug: invisible under ASAN (no SEGV) AND under gdb
+  (becomes a hang, not a catchable signal); `core_pattern` is read-only (pipes to a non-running
+  systemd-coredump) so the release core is discarded. Needs a non-ASAN DEBUG build (-O0 -g) run
+  directly under gdb so the SIGSEGV is caught with symbols, or a host with a live coredump pipe.
+
+## Net for 7.2
+Elections + job safety are sound for every round that COMPLETES (2-round soak PASSES; 6-round
+rounds 1–3 elect higher-term primaries; committed prefix byte-identical; zero job loss). The
+≥6-round acceptance verdict is blocked by the restart-stability bug (bug-1281 / task 5.4),
+reachable at round 4 (kill the up-to-date node → two restarted survivors). Spec-owned consensus
+correctness (Phase-2/5.2/5.3) is unaffected.
+
+## Harness wins
+`start_node` now captures per-node stdout+stderr (committed 75243fb) — this exposed the SEGV.
+The gdb-wrapper (`XCE_HA_BIN=<wrapper>` → `gdb -batch … --args <release ctrl>`) is the reusable
+technique to catch a restarted node's fault once a debug+core env is available.

.kiro/specs/raft-ha-core-safety/tasks.md

@@ -121,8 +121,8 @@
   - _Note: UNIT clause DONE (RED→GREEN, commit bf88681; `test_ha_core_safety` 20 Tests / 0 Failures). The 6-round-soak clause shares 7.2's environment block — the soak re-run still stalls at round 4, but the cause is the bug-924 WAL-fdatasync wedge (a survivor jams its persist flusher on `fdatasync`→EINVAL and stops answering PreVotes), NOT the eligibility deadlock this task fixes. See `research.md` §"Task 5.3 + 7.2 re-run". The eligibility fix is a real latent-defect fix proven at unit level; its live confirmation rides on 7.2's env resolution._
 
 - [ ] 5.4 Root-cause + fix the chaotic-restart controller SEGFAULT (NEW — the real 7.2 6-round blocker)
-  - _Root cause (task 7.2 on ~/tmp, `research.md` §"Task 7.2 on ~/tmp"; bug-1281): with per-node stderr finally captured (start_node had discarded it), a release `xce_controller` **Segmentation faults a few seconds after being restarted** into the churning cluster (during recovery→failover/PreVote). One restarted node down → cluster at 1-of-3 → no quorum → round-N "election stall." This is NOT consensus, NOT the in_catch_up deadlock (5.3 fixes that), NOT bug-924. Elections + job safety hold for every round that completes (2-round soak completes; 6-round rounds 1–3 elect)._
+  - _Root cause (task 7.2 on ~/tmp + gdb, `research.md` §"Task 7.2/5.4 — UNIFIED root cause"; bug-1281): a restart-STABILITY bug, UNIFIED via gdb. A restarted `xce_controller` becomes unable to participate at round 4 (kill the up-to-date node → two restarted survivors). Release build **SIGSEGVs** ("Segmentation fault (core dumped) --myname ctrl-3"); under gdb the SAME corruption manifests as a **wedged inbound-accept/iomgr path** (last accept logged while the election timer keeps firing) so the two survivors can't exchange PreVotes → permanent round-N stall. NOT a consensus-logic gap (PreVote/eligibility/freshness incl. 5.3 are correct; the node is I/O-dead after restart), NOT the in_catch_up deadlock, NOT bug-924. Predates this session (baseline 21e26a1 failed round 3–4). Elections + job safety hold for every round that completes (2-round soak PASSES; 6-round rounds 1–3 elect)._
-  - Investigate: reproduce under a debug-non-ASAN build with core dumps enabled + gdb (the crash is ASAN-INVISIBLE — heisenbug, memory-layout/timing dependent — and the sandbox has no gdb and discards the core via a non-running systemd-coredump). Get a backtrace; classify whether the crashing site is in this spec's surface (election/5.2/5.3) or a recovery/rejoin/persist path (route upstream if so). Add a RED that reproduces the crash deterministically before fixing.
+  - Investigate: the bug is a HEISENBUG — invisible under ASAN (no SEGV) AND under gdb (becomes an accept-hang, not a catchable signal), and the sandbox has no usable core (read-only core_pattern → non-running systemd-coredump). Reproduce under a **non-ASAN DEBUG build (-O0 -g) run directly under gdb** so the SIGSEGV is caught with symbols, OR a host with a live coredump pipeline. Get a backtrace; classify whether the crashing site is this spec's surface (election/5.2/5.3) or a recovery/rejoin/persist/iomgr path (route upstream if so). Diagnostic enablers already landed: per-node stderr capture (commit 75243fb) + the gdb-wrapper technique (`XCE_HA_BIN=<wrapper>` → `gdb -batch … --args <release ctrl>`). Add a RED that reproduces the crash deterministically before fixing.
   - Observable completion: the chaotic-restart crash is root-caused to a concrete site with a backtrace, fixed (or routed to its owning spec), and a restarted node no longer segfaults across a ≥6-round chaotic soak
   - _Requirements: 5.1, 6.2, 7.4, 8.3_
   - _Boundary: Standby Reconvergence, Election Liveness_

.wolf/buglog.json

@@ -21140,27 +21140,31 @@
     {
       "id": "bug-1281",
       "timestamp": "2026-06-30T03:25:00Z",
-      "error_message": "xce_controller (release) Segmentation fault (core dumped) a few seconds after being restarted into a chaotic-churn HA cluster (during recovery->failover/PreVote); drops cluster below quorum -> round-N election stall in the 7.2 soak",
+      "error_message": "Restart-stability bug: a restarted xce_controller becomes unable to participate at chaotic round 4. Release build SIGSEGVs ('Segmentation fault (core dumped) --myname ctrl-3'); under gdb the SAME corruption manifests as a WEDGED inbound-accept/iomgr path (last accept 03:42:03 while the election timer keeps firing) so two restarted survivors can't exchange PreVotes -> permanent round-N election stall.",
       "file": "src/controller (restart/recovery/election path — exact site unknown, no backtrace)",
-      "root_cause": "UNKNOWN — heisenbug. Captured via per-node stderr (start_node previously discarded it). Does NOT reproduce under ASAN (memory-layout/timing dependent); no gdb and no usable core in the sandbox (core_pattern pipes to a non-running systemd-coredump). Distinct from the in_catch_up deadlock (bug-1272, fixed by 5.3) and the bug-924 fdatasync wedge (separate transient).",
+      "root_cause": "UNIFIED via gdb: one restart-time I/O-subsystem corruption manifesting as a SIGSEGV (release) OR an accept-loop hang (under gdb, which perturbs the memory bug away from a crash). NOT a consensus-logic gap — PreVote/eligibility/freshness (incl. 5.3) are correct; the node is I/O-dead after restart. Predates this session (baseline 21e26a1 failed round 3-4). Heisenbug: invisible under ASAN AND gdb; no usable core (read-only core_pattern -> non-running systemd-coredump).",
-      "fix": "NOT FIXED. Tracked as raft-ha-core-safety task 5.4 — needs a debug-non-ASAN build with core dumps + gdb (or a host with live coredump pipeline) to get a backtrace. Blocks the 7.2 ≥6-round acceptance soak. Consensus correctness (Phase-2/5.2/5.3) is unaffected and unit-validated.",
+      "fix": "NOT FIXED (task 5.4). Needs a non-ASAN DEBUG build (-O0 -g) run directly under gdb so the SIGSEGV is caught with symbols, or a host with a live coredump pipeline. 5.4 must also rule out 5.2/5.3 (they sit in the restart/failover path). Diagnostic enablers landed: per-node stderr capture (75243fb) + the gdb-wrapper technique.",
       "tags": [
-        "segfault",
+        "7.2",
+        "iomgr",
+        "gdb",
+        "election",
         "crash",
+        "5.4",
         "restart",
         "recovery",
-        "election",
+        "heisenbug",
-        "chaotic-soak",
+        "accept-hang",
-        "7.2",
+        "restart-stability",
-        "5.4",
+        "segfault",
-        "heisenbug"
+        "chaotic-soak"
       ],
       "related_bugs": [
         "bug-1272",
         "bug-924"
       ],
-      "occurrences": 1,
+      "occurrences": 3,
-      "last_seen": "2026-06-30T03:25:00Z"
+      "last_seen": "2026-06-30T04:00:00Z"
     },
     {
       "id": "bug-1278",
@@ -21193,6 +21197,22 @@
       "related_bugs": [],
       "occurrences": 1,
       "last_seen": "2026-06-30T03:33:06.273Z"
+    },
+    {
+      "id": "bug-1280",
+      "timestamp": "2026-06-30T03:57:32.784Z",
+      "error_message": "Incorrect value in code",
+      "file": ".kiro/specs/raft-ha-core-safety/tasks.md",
+      "root_cause": "Had \"Task 7.2 on ~/tmp\"",
+      "fix": "Changed to \"Task 7.2/5.4 — UNIFIED root cause\"",
+      "tags": [
+        "auto-detected",
+        "wrong-value",
+        "md"
+      ],
+      "related_bugs": [],
+      "occurrences": 1,
+      "last_seen": "2026-06-30T03:57:32.785Z"
     }
   ]
 }