docs(raft-ha-core-safety): unify 7.2 round-N blocker via gdb — one restart-stability bug (SIGSEGV / iomgr accept-hang)

Used the now-available gdb (wrapped the release controller under gdb -batch via
XCE_HA_BIN) to expose the round-4 blocker's second face. Findings:

- The release-build SIGSEGV and a gdb-run accept-loop HANG are the SAME restart-
  time I/O-subsystem corruption: under gdb the memory bug is perturbed away from
  a crash and instead wedges the inbound-accept path (ctrl-3 stops accepting at
  03:42:03 while still firing its own election timer), so two restarted survivors
  can never exchange PreVotes -> permanent round-N stall.
- This is a restart-STABILITY bug, NOT a consensus-logic gap: PreVote/eligibility/
  freshness (incl. 5.3) are correct; the restarted node is simply I/O-dead. It
  predates this session (baseline 21e26a1 failed round 3-4).
- Heisenbug: invisible under ASAN and under gdb; no usable core (read-only
  core_pattern -> non-running systemd-coredump). Needs a non-ASAN DEBUG (-O0 -g)
  build under gdb, or a host with a live coredump pipeline, to backtrace.

Updates research.md, refines task 5.4 (the unified restart-stability bug +
the debug-build path), updates bug-1281. Consensus correctness (Phase-2/5.2/5.3)
is unaffected and validated for every round that completes (2-round soak PASSES;
6-round rounds 1-3 elect; zero job loss; committed prefix byte-identical).

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
This commit is contained in:
Rongsong Shen 2026-06-30 03:57:51 +00:00
parent 75243fbb86
commit e80abd5cc4
3 changed files with 72 additions and 13 deletions

View file

@ -21140,27 +21140,31 @@
{
"id": "bug-1281",
"timestamp": "2026-06-30T03:25:00Z",
"error_message": "xce_controller (release) Segmentation fault (core dumped) a few seconds after being restarted into a chaotic-churn HA cluster (during recovery->failover/PreVote); drops cluster below quorum -> round-N election stall in the 7.2 soak",
"error_message": "Restart-stability bug: a restarted xce_controller becomes unable to participate at chaotic round 4. Release build SIGSEGVs ('Segmentation fault (core dumped) --myname ctrl-3'); under gdb the SAME corruption manifests as a WEDGED inbound-accept/iomgr path (last accept 03:42:03 while the election timer keeps firing) so two restarted survivors can't exchange PreVotes -> permanent round-N election stall.",
"file": "src/controller (restart/recovery/election path — exact site unknown, no backtrace)",
"root_cause": "UNKNOWN — heisenbug. Captured via per-node stderr (start_node previously discarded it). Does NOT reproduce under ASAN (memory-layout/timing dependent); no gdb and no usable core in the sandbox (core_pattern pipes to a non-running systemd-coredump). Distinct from the in_catch_up deadlock (bug-1272, fixed by 5.3) and the bug-924 fdatasync wedge (separate transient).",
"fix": "NOT FIXED. Tracked as raft-ha-core-safety task 5.4 — needs a debug-non-ASAN build with core dumps + gdb (or a host with live coredump pipeline) to get a backtrace. Blocks the 7.2 ≥6-round acceptance soak. Consensus correctness (Phase-2/5.2/5.3) is unaffected and unit-validated.",
"root_cause": "UNIFIED via gdb: one restart-time I/O-subsystem corruption manifesting as a SIGSEGV (release) OR an accept-loop hang (under gdb, which perturbs the memory bug away from a crash). NOT a consensus-logic gap — PreVote/eligibility/freshness (incl. 5.3) are correct; the node is I/O-dead after restart. Predates this session (baseline 21e26a1 failed round 3-4). Heisenbug: invisible under ASAN AND gdb; no usable core (read-only core_pattern -> non-running systemd-coredump).",
"fix": "NOT FIXED (task 5.4). Needs a non-ASAN DEBUG build (-O0 -g) run directly under gdb so the SIGSEGV is caught with symbols, or a host with a live coredump pipeline. 5.4 must also rule out 5.2/5.3 (they sit in the restart/failover path). Diagnostic enablers landed: per-node stderr capture (75243fb) + the gdb-wrapper technique.",
"tags": [
"segfault",
"7.2",
"iomgr",
"gdb",
"election",
"crash",
"5.4",
"restart",
"recovery",
"election",
"chaotic-soak",
"7.2",
"5.4",
"heisenbug"
"heisenbug",
"accept-hang",
"restart-stability",
"segfault",
"chaotic-soak"
],
"related_bugs": [
"bug-1272",
"bug-924"
],
"occurrences": 1,
"last_seen": "2026-06-30T03:25:00Z"
"occurrences": 3,
"last_seen": "2026-06-30T04:00:00Z"
},
{
"id": "bug-1278",
@ -21193,6 +21197,22 @@
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-30T03:33:06.273Z"
},
{
"id": "bug-1280",
"timestamp": "2026-06-30T03:57:32.784Z",
"error_message": "Incorrect value in code",
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
"root_cause": "Had \"Task 7.2 on ~/tmp\"",
"fix": "Changed to \"Task 7.2/5.4 — UNIFIED root cause\"",
"tags": [
"auto-detected",
"wrong-value",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-30T03:57:32.785Z"
}
]
}