diff --git a/.kiro/specs/raft-ha-core-safety/research.md b/.kiro/specs/raft-ha-core-safety/research.md index ebae0729..c60da8b9 100644 --- a/.kiro/specs/raft-ha-core-safety/research.md +++ b/.kiro/specs/raft-ha-core-safety/research.md @@ -401,3 +401,42 @@ completes. The clean ≥6-round acceptance verdict is blocked by the restart seg follow-up **task 5.4** (root-cause+fix the chaotic-restart crash; needs core+gdb / debug env) and **bug-1281**. The harness `committedLsn`-boundary fix is a separate Phase-1 accuracy improvement. + +--- + +# Task 7.2/5.4 — UNIFIED root cause via gdb: round-N stall is a restart-stability bug (I/O wedge / SIGSEGV), not a consensus-logic gap + +Wrapped the release `xce_controller` under `gdb -batch` (handles RT signals; dumps all-thread +bt on SIGSEGV) via `XCE_HA_BIN=` and re-ran the chaotic soak on `~/tmp`. + +## Decisive findings +- **Two round-4 faces, ONE root cause.** Plain release: a restarted node **SIGSEGVs** + (captured `Segmentation fault (core dumped) … --myname ctrl-3`). Under gdb the crash does NOT + fire (ptrace perturbs the memory bug); instead the same restarted node's **inbound-accept / + iomgr path WEDGES** — ctrl-3's last `cluster.c:737 server sock accepted` is 03:42:03, yet it + keeps firing its own election timer (`pre_vote_start`) to 03:51:59. It campaigns but can + neither accept peers' PreVote connections nor get answered; ctrl-2's outbound PreVote connects + hang on the dead listener (zero connect-fail logs). The two restarted survivors can't exchange + votes → permanent round-4 stall. Release SEGV and gdb accept-hang = the same restart-time + corruption manifesting differently. +- **Restart-STABILITY bug, not a consensus-logic gap.** PreVote/eligibility/freshness (incl. the + 5.3 fix) are correct; the node is simply I/O-dead after restart. Predates this session — + baseline `21e26a1` failed round 3–4 (requirements.md symptom 3). 5.2/5.3 sit in the + restart/failover path so 5.4 must rule them out during the backtrace, but they did not + introduce the instability. +- **Cannot backtrace in this sandbox.** Heisenbug: invisible under ASAN (no SEGV) AND under gdb + (becomes a hang, not a catchable signal); `core_pattern` is read-only (pipes to a non-running + systemd-coredump) so the release core is discarded. Needs a non-ASAN DEBUG build (-O0 -g) run + directly under gdb so the SIGSEGV is caught with symbols, or a host with a live coredump pipe. + +## Net for 7.2 +Elections + job safety are sound for every round that COMPLETES (2-round soak PASSES; 6-round +rounds 1–3 elect higher-term primaries; committed prefix byte-identical; zero job loss). The +≥6-round acceptance verdict is blocked by the restart-stability bug (bug-1281 / task 5.4), +reachable at round 4 (kill the up-to-date node → two restarted survivors). Spec-owned consensus +correctness (Phase-2/5.2/5.3) is unaffected. + +## Harness wins +`start_node` now captures per-node stdout+stderr (committed 75243fb) — this exposed the SEGV. +The gdb-wrapper (`XCE_HA_BIN=` → `gdb -batch … --args `) is the reusable +technique to catch a restarted node's fault once a debug+core env is available. diff --git a/.kiro/specs/raft-ha-core-safety/tasks.md b/.kiro/specs/raft-ha-core-safety/tasks.md index 2105a2bb..875b15cc 100644 --- a/.kiro/specs/raft-ha-core-safety/tasks.md +++ b/.kiro/specs/raft-ha-core-safety/tasks.md @@ -121,8 +121,8 @@ - _Note: UNIT clause DONE (RED→GREEN, commit bf88681; `test_ha_core_safety` 20 Tests / 0 Failures). The 6-round-soak clause shares 7.2's environment block — the soak re-run still stalls at round 4, but the cause is the bug-924 WAL-fdatasync wedge (a survivor jams its persist flusher on `fdatasync`→EINVAL and stops answering PreVotes), NOT the eligibility deadlock this task fixes. See `research.md` §"Task 5.3 + 7.2 re-run". The eligibility fix is a real latent-defect fix proven at unit level; its live confirmation rides on 7.2's env resolution._ - [ ] 5.4 Root-cause + fix the chaotic-restart controller SEGFAULT (NEW — the real 7.2 6-round blocker) - - _Root cause (task 7.2 on ~/tmp, `research.md` §"Task 7.2 on ~/tmp"; bug-1281): with per-node stderr finally captured (start_node had discarded it), a release `xce_controller` **Segmentation faults a few seconds after being restarted** into the churning cluster (during recovery→failover/PreVote). One restarted node down → cluster at 1-of-3 → no quorum → round-N "election stall." This is NOT consensus, NOT the in_catch_up deadlock (5.3 fixes that), NOT bug-924. Elections + job safety hold for every round that completes (2-round soak completes; 6-round rounds 1–3 elect)._ - - Investigate: reproduce under a debug-non-ASAN build with core dumps enabled + gdb (the crash is ASAN-INVISIBLE — heisenbug, memory-layout/timing dependent — and the sandbox has no gdb and discards the core via a non-running systemd-coredump). Get a backtrace; classify whether the crashing site is in this spec's surface (election/5.2/5.3) or a recovery/rejoin/persist path (route upstream if so). Add a RED that reproduces the crash deterministically before fixing. + - _Root cause (task 7.2 on ~/tmp + gdb, `research.md` §"Task 7.2/5.4 — UNIFIED root cause"; bug-1281): a restart-STABILITY bug, UNIFIED via gdb. A restarted `xce_controller` becomes unable to participate at round 4 (kill the up-to-date node → two restarted survivors). Release build **SIGSEGVs** ("Segmentation fault (core dumped) --myname ctrl-3"); under gdb the SAME corruption manifests as a **wedged inbound-accept/iomgr path** (last accept logged while the election timer keeps firing) so the two survivors can't exchange PreVotes → permanent round-N stall. NOT a consensus-logic gap (PreVote/eligibility/freshness incl. 5.3 are correct; the node is I/O-dead after restart), NOT the in_catch_up deadlock, NOT bug-924. Predates this session (baseline 21e26a1 failed round 3–4). Elections + job safety hold for every round that completes (2-round soak PASSES; 6-round rounds 1–3 elect)._ + - Investigate: the bug is a HEISENBUG — invisible under ASAN (no SEGV) AND under gdb (becomes an accept-hang, not a catchable signal), and the sandbox has no usable core (read-only core_pattern → non-running systemd-coredump). Reproduce under a **non-ASAN DEBUG build (-O0 -g) run directly under gdb** so the SIGSEGV is caught with symbols, OR a host with a live coredump pipeline. Get a backtrace; classify whether the crashing site is this spec's surface (election/5.2/5.3) or a recovery/rejoin/persist/iomgr path (route upstream if so). Diagnostic enablers already landed: per-node stderr capture (commit 75243fb) + the gdb-wrapper technique (`XCE_HA_BIN=` → `gdb -batch … --args `). Add a RED that reproduces the crash deterministically before fixing. - Observable completion: the chaotic-restart crash is root-caused to a concrete site with a backtrace, fixed (or routed to its owning spec), and a restarted node no longer segfaults across a ≥6-round chaotic soak - _Requirements: 5.1, 6.2, 7.4, 8.3_ - _Boundary: Standby Reconvergence, Election Liveness_ diff --git a/.wolf/buglog.json b/.wolf/buglog.json index f8f12220..d96a9e5b 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -21140,27 +21140,31 @@ { "id": "bug-1281", "timestamp": "2026-06-30T03:25:00Z", - "error_message": "xce_controller (release) Segmentation fault (core dumped) a few seconds after being restarted into a chaotic-churn HA cluster (during recovery->failover/PreVote); drops cluster below quorum -> round-N election stall in the 7.2 soak", + "error_message": "Restart-stability bug: a restarted xce_controller becomes unable to participate at chaotic round 4. Release build SIGSEGVs ('Segmentation fault (core dumped) --myname ctrl-3'); under gdb the SAME corruption manifests as a WEDGED inbound-accept/iomgr path (last accept 03:42:03 while the election timer keeps firing) so two restarted survivors can't exchange PreVotes -> permanent round-N election stall.", "file": "src/controller (restart/recovery/election path — exact site unknown, no backtrace)", - "root_cause": "UNKNOWN — heisenbug. Captured via per-node stderr (start_node previously discarded it). Does NOT reproduce under ASAN (memory-layout/timing dependent); no gdb and no usable core in the sandbox (core_pattern pipes to a non-running systemd-coredump). Distinct from the in_catch_up deadlock (bug-1272, fixed by 5.3) and the bug-924 fdatasync wedge (separate transient).", - "fix": "NOT FIXED. Tracked as raft-ha-core-safety task 5.4 — needs a debug-non-ASAN build with core dumps + gdb (or a host with live coredump pipeline) to get a backtrace. Blocks the 7.2 ≥6-round acceptance soak. Consensus correctness (Phase-2/5.2/5.3) is unaffected and unit-validated.", + "root_cause": "UNIFIED via gdb: one restart-time I/O-subsystem corruption manifesting as a SIGSEGV (release) OR an accept-loop hang (under gdb, which perturbs the memory bug away from a crash). NOT a consensus-logic gap — PreVote/eligibility/freshness (incl. 5.3) are correct; the node is I/O-dead after restart. Predates this session (baseline 21e26a1 failed round 3-4). Heisenbug: invisible under ASAN AND gdb; no usable core (read-only core_pattern -> non-running systemd-coredump).", + "fix": "NOT FIXED (task 5.4). Needs a non-ASAN DEBUG build (-O0 -g) run directly under gdb so the SIGSEGV is caught with symbols, or a host with a live coredump pipeline. 5.4 must also rule out 5.2/5.3 (they sit in the restart/failover path). Diagnostic enablers landed: per-node stderr capture (75243fb) + the gdb-wrapper technique.", "tags": [ - "segfault", + "7.2", + "iomgr", + "gdb", + "election", "crash", + "5.4", "restart", "recovery", - "election", - "chaotic-soak", - "7.2", - "5.4", - "heisenbug" + "heisenbug", + "accept-hang", + "restart-stability", + "segfault", + "chaotic-soak" ], "related_bugs": [ "bug-1272", "bug-924" ], - "occurrences": 1, - "last_seen": "2026-06-30T03:25:00Z" + "occurrences": 3, + "last_seen": "2026-06-30T04:00:00Z" }, { "id": "bug-1278", @@ -21193,6 +21197,22 @@ "related_bugs": [], "occurrences": 1, "last_seen": "2026-06-30T03:33:06.273Z" + }, + { + "id": "bug-1280", + "timestamp": "2026-06-30T03:57:32.784Z", + "error_message": "Incorrect value in code", + "file": ".kiro/specs/raft-ha-core-safety/tasks.md", + "root_cause": "Had \"Task 7.2 on ~/tmp\"", + "fix": "Changed to \"Task 7.2/5.4 — UNIFIED root cause\"", + "tags": [ + "auto-detected", + "wrong-value", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T03:57:32.785Z" } ] } \ No newline at end of file