From 5d7213321696781a37a6f7d36566aeb6cff13340 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 30 Jun 2026 02:56:00 +0000 Subject: [PATCH] docs(raft-ha-core-safety): task 5.3 done (unit) + 7.2 re-attributed to bug-924 env block MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 5.3 fix (bf88681) lands and is unit-validated (test_ha_core_safety 20 Tests / 0 Failures). Re-ran the 7.2 chaotic soak with all Phase-2+5.2+5.3 fixes: it still stalls at round 4, but bisecting the frozen logs shows the cause is the bug-924 WAL-fdatasync wedge, NOT consensus — a survivor's persist flusher spins on `fdatasync FAILED (errno=22 EINVAL)` and stops answering PreVotes, so the round times out (presents as an election stall, is a durability-layer wedge). A file-level repro of the controller's exact WAL pattern (open + posix_fallocate 64MB + write + fdatasync) SUCCEEDS on every available ext4/tmpfs path, so it is the live overlay-backed WAL fd under chaotic restart — the documented bug-924, a Non-Goal of this spec (owned by ha-commit-durability, excluded from the verdict). Re-attributes the round-4 stall (the first §"Task 7.2" note guessed in_catch_up; 5.3 fixes that real latent eligibility deadlock at unit level but it is not the soak blocker). 7.2 is now BLOCKED ON ENVIRONMENT, not a consensus NO-GO; the consensus work this spec owns (Phase-2 payload integrity, 5.2, 5.3) is unit-complete. research.md §"Task 5.3 + 7.2 re-run"; bug-1272 updated. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC --- .kiro/specs/raft-ha-core-safety/research.md | 46 +++++++++++ .kiro/specs/raft-ha-core-safety/tasks.md | 5 +- .wolf/buglog.json | 85 +++++++++++++++++++-- 3 files changed, 126 insertions(+), 10 deletions(-) diff --git a/.kiro/specs/raft-ha-core-safety/research.md b/.kiro/specs/raft-ha-core-safety/research.md index 5e22505..d83243a 100644 --- a/.kiro/specs/raft-ha-core-safety/research.md +++ b/.kiro/specs/raft-ha-core-safety/research.md @@ -304,3 +304,49 @@ design's chaotic-harness Contingency, this is a recorded new finding, not closur added follow-up **task 5.3** (overlapping-churn election stall — a lagging-but-safe survivor must be electable; the catching-up/freshness vote gates must not deadlock when the primary is gone). 7.2 stays blocked on 5.3. Re-run 7.2 after 5.3 lands. + +--- + +# Task 5.3 + 7.2 re-run — CORRECTED root cause: the soak stall is bug-924, not consensus + +After landing task 5.3 (commit bf88681 — gate the `in_catch_up` PreVote deny on a +LIVE primary), the chaotic soak was re-run on real-fs release binaries +(`/home/claude/tmp/ha-7p2b`, controller rebuilt 02:44 with all Phase-2+5.2+5.3 fixes). + +## Result: round-4 stalls AGAIN — but the cause is the persist fdatasync wedge (bug-924), NOT election logic +- Rounds 1–3 elect a higher-term primary each round (term 4→5→6); job safety holds. +- Round 4 "NO new primary within window" — and ctrl-2's PreVote rounds get **zero + replies** (timeout, not LOST), because the *other* survivor **ctrl-3 is wedged**: + its persist flusher spins on `flusher fallback fdatasync FAILED (errno=22 Invalid + argument)` from 02:45:49 onward and stops answering PreVotes. ctrl-3's process is + alive but non-responsive — jammed in the durability layer, not the election layer. +- This is the **bug-924** WAL-fdatasync-on-overlay artifact (memory obs 1476/1477): + the WAL segment's `fdatasync` returns EINVAL in this sandbox. A file-level repro of + the exact controller pattern (`open O_RDWR|O_CREAT|O_TRUNC` → `posix_fallocate(64MB)` + → write → `fdatasync`) SUCCEEDS on `/home/claude/tmp`, `/home/rshen/projects/xce` + (both ext4) and `/dev/shm` — so it is the controller's live overlay-backed WAL fd + under chaotic restart, not a plain fs limitation. bug-924 is a **Non-Goal of this + spec** (design §Non-Goals / §Out of Boundary: "the overlayfs `fdatasync` + environment defect (bug-924) — excluded from the acceptance verdict, not fixed + here"); durability sync is owned by `ha-commit-durability`. + +## Re-attribution of the round-4 stall (corrects the first §"Task 7.2" entry) +The first soak's round-4 stall was attributed to the `in_catch_up` mutual-deny. That +IS a real latent defect — 5.3 reproduces it deterministically at unit level (a +catching-up survivor with no live primary denied a fresh PreVote) and fixes it — but +it is NOT what blocks the soak. The soak's actual round-4 blocker in BOTH runs is the +bug-924 fdatasync wedge that makes a survivor non-responsive. 5.3 is kept as a valid +correctness fix (unit RED→GREEN, `test_ha_core_safety` 20 Tests / 0 Failures); its +6-round-soak observable clause shares 7.2's environment block. + +## 7.2 status: BLOCKED ON ENVIRONMENT (bug-924), not a consensus NO-GO +The consensus work this spec owns is unit-complete and validated: Phase-2 payload +integrity (committed prefix byte-identical across members), 5.2 spurious-failover +suppression, 5.3 catching-up-survivor electability. The live ≥6-round acceptance soak +cannot reach a clean verdict in this sandbox because the controller's WAL `fdatasync` +fails (bug-924) and wedges nodes mid-soak — the same env block that left +`ha-term-safety`'s live soak permanently BLOCKED. 7.2 (and 5.3's soak clause) require a +real-fs host/CI where the controller WAL `fdatasync` succeeds, OR the soak harness must +detect the fdatasync-wedge and exclude it from the liveness verdict (a Phase-1 harness +improvement), OR bug-924 is resolved by `ha-commit-durability`. The consensus fixes do +not need re-work for this; it is an environment/durability-layer blocker. diff --git a/.kiro/specs/raft-ha-core-safety/tasks.md b/.kiro/specs/raft-ha-core-safety/tasks.md index 262ee75..9a9176d 100644 --- a/.kiro/specs/raft-ha-core-safety/tasks.md +++ b/.kiro/specs/raft-ha-core-safety/tasks.md @@ -110,7 +110,7 @@ - _Boundary: Election Liveness_ - _Depends: 4.2_ -- [ ] 5.3 Lagging-but-safe survivor must be electable under overlapping churn (NEW — surfaced by the 7.2 soak) +- [x] 5.3 Lagging-but-safe survivor must be electable under overlapping churn (NEW — surfaced by the 7.2 soak) - _Root cause (task 7.2 first soak, `research.md` §"Task 7.2"): distinct from the 5.2 flap. After ≥3 overlapping kills the only fully-up-to-date node is killed and the two survivors are both restarted-and-lagging (e.g. tips 15 and 11 after the leader at 19 dies). Their PreVote rounds repeatedly time out with no grant quorum — prospective terms frozen, never bumped — so no primary is elected, even though the fresher survivor holds the FULL committed prefix (Leader Completeness satisfied) and is a SAFE candidate. Observed contributors: freshness mutual-denial between two lagging survivors, the in_catch_up-denies-all-votes gate (Req 2.5; obs 1240) with no live primary left to catch up FROM, and/or peer-to-peer vote connections never established between two nodes that each only held a control channel to the now-dead primary._ - RED: deterministically construct the round-4 shape (kill the up-to-date leader; two survivors each holding the full committed prefix but at different lagging tips, both flagged in_catch_up, no live primary) and assert the freshest survivor reaches CTRL_PRIMARY at a higher term within the failover window (no permanent PreVote-timeout loop). Reuse the 1.4 kill/election + apply-gap seams + the 5.1 PreVote dispatch seam. - Fix: the catching-up/freshness vote gates must not deadlock when the primary is gone — a survivor that holds the majority-agreed committed prefix must be able to grant/win a PreVote once no live primary remains (drop in_catch_up for vote-eligibility when the primary is unreachable AND the candidate is ≥ the voter's committed floor; ensure the peer vote connection is established during failover). Preserve Election Safety (≤1 primary/term), Leader Completeness (never elect a candidate below the committed floor), and the 5.2 spurious-failover suppression. @@ -118,6 +118,7 @@ - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ - _Boundary: Election Liveness_ - _Depends: 5.2_ + - _Note: UNIT clause DONE (RED→GREEN, commit bf88681; `test_ha_core_safety` 20 Tests / 0 Failures). The 6-round-soak clause shares 7.2's environment block — the soak re-run still stalls at round 4, but the cause is the bug-924 WAL-fdatasync wedge (a survivor jams its persist flusher on `fdatasync`→EINVAL and stops answering PreVotes), NOT the eligibility deadlock this task fixes. See `research.md` §"Task 5.3 + 7.2 re-run". The eligibility fix is a real latent-defect fix proven at unit level; its live confirmation rides on 7.2's env resolution._ - [ ] 6. Integration: single-seam invariant end-to-end - [ ] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end @@ -145,7 +146,7 @@ - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_ - _Depends: 1.5, 6.2, 5.2, 5.3_ - - _Blocked: first soak run (HEAD 01419e1, Phase-2+5 fixes) is NO-GO at round 4 — job safety + Phase-2 prefix integrity hold (zero job loss; committed prefix [1..11] identical; all counted divergences are catch-up-lag MISSING tails, not corruption) and 5.2 spurious-failover suppression is confirmed working, but the cluster STALLS electing a new primary after 3 overlapping kills (two lagging-but-safe survivors mutually fail PreVote). Root cause + full classification in `research.md` §"Task 7.2"; fix owned by NEW task 5.3. Re-run 7.2 after 5.3 lands._ + - _Blocked: ENVIRONMENT (bug-924), not a consensus NO-GO. Two soak runs (HEAD 01419e1 then bf88681 with 5.3) both stall at round 4 — but the cause is the **bug-924 WAL-fdatasync wedge**: a survivor's persist flusher jams on `fdatasync`→EINVAL and stops answering PreVotes, so the round times out (it PRESENTS as an election stall but is a durability-layer wedge). Job safety + Phase-2 prefix integrity hold every run (zero job loss; committed prefix byte-identical; counted divergences are catch-up-lag MISSING tails, not corruption); 5.2 + 5.3 confirmed working. bug-924 is a Non-Goal of this spec (owned by `ha-commit-durability`, excluded from the verdict). A clean verdict needs a real-fs host/CI where the controller WAL `fdatasync` succeeds, OR a Phase-1 harness change to detect+exclude the fdatasync-wedge from the liveness verdict. Full classification in `research.md` §"Task 5.3 + 7.2 re-run"; bug-1272._ - [ ] 7.3 Hypothesis-validation closeout - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 9009114..ae89091 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -21048,25 +21048,94 @@ "error_message": "raft-ha-core-safety 7.2 chaotic soak: round-4 'NO new primary within window (possible election stall)' after 3 overlapping kills", "file": "src/controller/ha.c (election/PreVote path)", "root_cause": "After 3 overlapping kills the only fully-up-to-date node is killed; the two survivors are both restarted-and-lagging (tips 15 and 11; leader was 19). Both hold the full committed prefix [1..11] (safe candidates) but their PreVote rounds repeatedly time out with no grant quorum (prospective terms frozen, never bumped). Distinct from the 5.2 spurious-failover flap (that is fixed and confirmed). Contributors: freshness mutual-denial between lagging survivors, in_catch_up-denies-all-votes (Req 2.5) with no live primary to catch up from, and/or peer vote connection not established between two nodes that only held a control channel to the dead primary.", - "fix": "NOT YET FIXED. Tracked as raft-ha-core-safety task 5.3: a lagging-but-safe survivor (holds majority-agreed committed prefix) must be electable when no live primary remains; relax in_catch_up/freshness vote gates for vote-eligibility once the primary is unreachable, preserving Election Safety + Leader Completeness + the 5.2 suppression. 7.2 blocked on 5.3.", + "fix": "FIXED at unit level by task 5.3 (commit bf88681): gate the in_catch_up PreVote deny in raft_cb_vote_eligible on a LIVE primary (primary_ctrl_fd>=0 && ha_heartbeat_is_fresh), so a catching-up survivor with a dead primary can grant/win a fresh PreVote (logOk preserves Leader Completeness). CORRECTION: this in_catch_up deadlock is a real latent defect (unit RED\u2192GREEN) but is NOT the soak's round-4 blocker. The soak stall is bug-924 (WAL fdatasync\u2192EINVAL wedges a survivor so it stops answering PreVotes; presents as an election stall). 7.2 live acceptance is blocked on that env defect (out of scope; owned by ha-commit-durability), not on consensus.", "tags": [ - "raft", - "ha", - "election", - "liveness", "prevote", - "in_catch_up", + "ha", + "liveness", "chaotic-soak", + "environment", + "election", + "bug-924", + "corrected", "7.2", - "5.3" + "fdatasync", + "raft", + "5.3", + "in_catch_up" ], "related_bugs": [ + "bug-924", "bug-851", "bug-1042", "bug-1060" ], + "occurrences": 2, + "last_seen": "2026-06-30T02:55:00Z" + }, + { + "id": "bug-1273", + "timestamp": "2026-06-30T02:41:41.017Z", + "error_message": "Missing error handling in unknown", + "file": "src/controller/testsuite/test_ha_core_safety.c", + "root_cause": "Code path had no error handling \u2014 exceptions would propagate uncaught", + "fix": "Added try/catch block | Also: /* Task 5.3 lagging-but-safe survivor electability under ove; * CORE RED \u2014 a catching-up survivor with NO live primary m", + "tags": [ + "auto-detected", + "error-handling", + "c" + ], + "related_bugs": [], + "occurrences": 2, + "last_seen": "2026-06-30T02:41:53.665Z" + }, + { + "id": "bug-1274", + "timestamp": "2026-06-30T02:42:59.119Z", + "error_message": "Significant refactor of ", + "file": "src/controller/ha.c", + "root_cause": "2 lines replaced/restructured", + "fix": "Rewrote 3\u219220 lines (2 removed)", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], "occurrences": 1, - "last_seen": "2026-06-30T02:25:00Z" + "last_seen": "2026-06-30T02:42:59.119Z" + }, + { + "id": "bug-1275", + "timestamp": "2026-06-30T02:55:23.808Z", + "error_message": "Function not marked async", + "file": ".kiro/specs/raft-ha-core-safety/tasks.md", + "root_cause": "Function uses await but wasn't declared async", + "fix": "Added async modifier", + "tags": [ + "auto-detected", + "async-fix", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T02:55:23.808Z" + }, + { + "id": "bug-1276", + "timestamp": "2026-06-30T02:55:37.693Z", + "error_message": "Incorrect value in code", + "file": ".kiro/specs/raft-ha-core-safety/tasks.md", + "root_cause": "Had `research.md`", + "fix": "Changed to 's persist flusher jams on `", + "tags": [ + "auto-detected", + "wrong-value", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T02:55:37.693Z" } ] } \ No newline at end of file