From 5be165312ad3e8ede06f68fb63a0abe026d54bb4 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 30 Jun 2026 13:39:55 +0000 Subject: [PATCH] docs(raft-ha-core-safety): 5.7 verified (election resonance fixed); round-4 re-roots to bug-1298 UAF (task 5.8) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 5.7 election-timeout resonance FIXED + soak-confirmed: 6-round chaotic soak now elects with bounded terms (rounds 1-3 at terms 4/5/6, not 25-30+), zero job loss. The election-liveness layer is done. Round-4 stall re-roots to a NON-election crash: ctrl-3 ASAN heap-use-after-free in endpoint teardown — io_worker (endpoint_killer, cluster.c:813) frees an endpoint while the vtask dispatcher reads ep->busy (endpoint_is_task_blocked, netio.c:1008). ctrl-3 dies -> cluster drops to 1/3 -> no quorum -> no primary. Same async-io dispatch surface as bug-1282/1197; UPSTREAM-owned (raft-drive-async / libxce). Logged bug-1298; tracked as task 5.8. --- .kiro/specs/raft-ha-core-safety/research.md | 72 +++++++++ .kiro/specs/raft-ha-core-safety/tasks.md | 17 +- .wolf/buglog.json | 164 ++++++++++++++++++++ 3 files changed, 249 insertions(+), 4 deletions(-) diff --git a/.kiro/specs/raft-ha-core-safety/research.md b/.kiro/specs/raft-ha-core-safety/research.md index 1df66997..62354582 100644 --- a/.kiro/specs/raft-ha-core-safety/research.md +++ b/.kiro/specs/raft-ha-core-safety/research.md @@ -647,3 +647,75 @@ two survivors desynchronize and one's term stops racing the other's reply. Prese — the freshest survivor is granted but its term out-races the grant. New follow-up. Consensus SAFETY intact throughout (no two leaders; DropStaleResponse correct; zero job loss every completed round). Purely election LIVENESS convergence under sustained overlapping-kill churn. + +--- + +## Task 5.7 fix VERIFIED + round-4 blocker re-rooted (2026-06-30) + +**5.7 (election-timeout resonance) — FIXED, commit 74784b2.** + +The 5.6 note guessed the 5.7 cause as "term re-escalates faster than the grant +can land." Reading the code pinned the *mechanism* precisely: the per-CAMPAIGN +election watchdog (`raft_start_election` → `on_schedule_timeout` → +`raft_cb_schedule_timeout`, ha.c) was armed with a **FIXED** `election_timeout_ms` +— **no randomization**. Only the *startup-discovery* path (`ha_wait_for_heartbeat_or_timeout`) +jittered; the recurring campaign watchdog did not. raft.c:157 passes the bare +`s->election_timeout_ms` straight through. + +A fixed deadline + a roughly-constant cold-connection grant RTT **RESONATE**: if +ctrl-3's grant consistently lands a hair *after* a constant deadline, every +campaign loses by the *same* margin forever → the campaign times out, the term +re-escalates, and the now-stale grant is dropped by `DropStaleResponse` +(raft.c:453, which is *correct* Raft). Terms climb 25→30+. The bug was never +"RTT genuinely > 5 s"; it was that the deterministic cadence locks the grant +permanently just outside every window. + +**Fix (Ongaro thesis §3.4 — randomized election timeout, re-drawn each campaign):** +new `ha_jittered_election_delay(ha, base)` returns `base + rand[0, base]` (the +canonical [T, 2T] interval), wired into `raft_cb_schedule_timeout` so the watchdog +is re-randomized per campaign. Done in the HA adapter so the pure, sim-tested raft +library stays deterministic. Independent of `election_jitter_max_ms` (which tunes +only the sub-second startup/stagger jitter) so it spreads even at jmax=200; never +below the floor so genuine-loss failover is not sped up past `election_timeout_ms`. + +Unit: `test_5_7_campaign_timeout_is_randomized` (CORE) + `test_5_7_floor_preserved_and_zero_safe` +(PIN). RED verified by stubbing the jitter to 0 → the randomization test fails with +the resonance message while the PIN stays green. `test_ha_core_safety` 24 Tests / 0 +Failures. (Aside: the full aggregate `controller_unit_test` binary cannot reach the +ha_core_safety suite because an EARLIER suite, `test_ha_vep_teardown`, SIGSEGVs in +`iomgr_fini`→`SSL_CTX_free` under openssl 3.6 — unrelated to this change; run the +suite in isolation with `-t test_ha_core_safety`.) + +**Soak confirmation:** a fresh ASAN 6-round chaotic soak now elects with **BOUNDED +terms** — rounds 1→3 elect ctrl-3/ctrl-2/ctrl-1 at terms **4→5→6** (not 25→30+). +The unbounded escalation is gone; zero job loss on every completed round (3/6/9/12 +tracked jobs all present on each new primary). The election-liveness layer is DONE. + +**Round-4 blocker RE-ROOTS to a non-election crash (task 5.8 / bug-1298).** +Round 4 still stalls, but no longer for any election reason. ctrl-3 **ABORTs** on +an ASAN heap-use-after-free: + +``` +UAF READ size 4 endpoint_is_task_blocked netio.c:1008 (vtask dispatcher T3) + vtask_eligible vtaskmgr.c:255 ← dataq_deq_first_match ← vdispatcher_entry +FREED BY (T2) endpoint_killer cluster.c:813 (io_worker) + handle_endpoint_error cluster.c:108 ← handle_error ← io_worker +ALLOC endpoint_get netio.c:748 ← handle_connection cluster.c:741 +``` + +io_worker (T2) tears down a connection on error and `free()`s the endpoint via +`endpoint_killer` while the vtask dispatcher (T3) concurrently reads `ep->busy` on +that just-freed endpoint during its dataq eligibility scan. The endpoint is freed +while still referenced by a pending vtask; nothing drains its queued vtasks before +the free and no refcount keeps it alive across the dispatcher read. ctrl-3 dies → +the cluster drops to 1/3 (the round-4 survivors are the just-restarted ctrl-2 and a +now-dead ctrl-3) → no quorum → "NO new primary within window." ctrl-2's logs show +only PreVote-round timeouts (`prospective term=6`, never wins) — the textbook +symptom of a lost quorum peer, not an election defect. + +Same netio surface and routing precedent as task 5.4/bug-1282: an async-io dispatch +concurrency bug, **UPSTREAM-owned** by raft-drive-async / core libxce. Likely a +latent UAF that the bug-1282 lockless `ep->busy` read unmasked (the dispatcher now +reaches the read without contending on `ep->lock`). Logged as **bug-1298**; tracked +as task 5.8. Consensus safety (Phase-2/5.2/5.3/5.5/5.6/5.7) is sound; elections and +job safety hold for every round that completes. diff --git a/.kiro/specs/raft-ha-core-safety/tasks.md b/.kiro/specs/raft-ha-core-safety/tasks.md index 64b58d7e..753eeb05 100644 --- a/.kiro/specs/raft-ha-core-safety/tasks.md +++ b/.kiro/specs/raft-ha-core-safety/tasks.md @@ -146,14 +146,23 @@ - _Depends: 5.5_ - _PARTIAL FIX (commit 81cb09a): the RDHUP-entry now drops an RDHUP while role==CTRL_ELECTING or pre_vote_in_progress (don't preempt an in-flight election). Unit RED→GREEN (22 Tests/0 Failures); reduces the term churn. But the soak still stalls at round 4 → a deeper 5th layer (task 5.7): the freshest survivor is GRANTED but its term out-races the grant (vote_reply_stale_drop). Detail in `research.md` §"Task 5.6 fix VERIFIED"._ -- [ ] 5.7 Stop the vote-reply term-lag / unbounded term escalation under cold-connection churn (NEW — the 5th layer under 5.6) - - _Root cause (`research.md` §"Task 5.6 fix VERIFIED"): with 5.6 in, round-4 ctrl-2 (freshest, 15@t5) wins PreVote, enters the real election, and ctrl-3 GRANTS it (vote_granted) — ctrl-3 votes FOR ctrl-2, not leapfrogging. But ctrl-2 never tallies a majority: every grant arrives as vote_reply_stale_drop (reply term < ctrl-2's current term). ctrl-2's term escalates via repeated election timeouts faster than ctrl-3 can reply, so each grant is one term stale → dropped → timeout → term++ → self-perpetuating. A just-restarted ctrl-2 has COLD peer connections (each real-vote opens a fresh endpoint_connect+SSL) and races connect attempts to the dead ctrl-1, so the grant RTT exceeds the window before ctrl-2 re-escalates. election_timeout_ms=5000, jitter=200ms._ - - Fix direction: (a) do not bump the election term while a vote round-trip for the current term is still outstanding (let the grant land before re-escalating); and/or (b) warm/reuse the peer vote connection so the grant RTT < election_timeout (avoid per-vote connection setup); and/or (c) widen election-timeout jitter so the two survivors desynchronize. Preserve Election Safety (DropStaleResponse is correct Raft — fix the escalation cadence, not the drop), genuine-loss liveness, 5.2 suppression, and the 5.6 no-preempt guard. RED: a just-restarted freshest survivor + a willing grantor with a cold connection; assert it wins within a bounded number of election terms (term does not escalate unbounded). - - Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (bounded term, no escalation stall) with all prior fixes in place — the green 7.2 gate +- [x] 5.7 Stop the vote-reply term-lag / unbounded term escalation under cold-connection churn (NEW — the 5th layer under 5.6) + - _FIXED + VERIFIED (commit 74784b2, `research.md` §"Task 5.7 fix VERIFIED"). Actual root cause was narrower than the 5.6 note guessed: the per-CAMPAIGN election watchdog was armed with a FIXED `election_timeout_ms` (`raft_start_election`→`on_schedule_timeout`→`raft_cb_schedule_timeout`); only the startup-discovery path jittered. A fixed deadline + a roughly-constant cold-connection grant RTT RESONATE — the single needed grant lands a hair past every deadline forever, the campaign times out, the term re-escalates, and the now-stale grant is dropped (DropStaleResponse correctly). Fix (Ongaro §3.4): re-randomize the watchdog per campaign to the canonical [T, 2T] interval, injected in the HA adapter so the pure (sim-tested) raft library stays deterministic; independent of `election_jitter_max_ms`; never below the floor. New `ha_jittered_election_delay()` + 2 unit tests (RED verified — stubbing jitter to 0 fails `test_5_7_campaign_timeout_is_randomized` with the resonance message; PIN guards floor + modulo-by-zero). `test_ha_core_safety` 24 Tests / 0 Failures. SOAK-CONFIRMED: a fresh 6-round chaotic soak now elects with BOUNDED terms (rounds 1→3 at terms 4→5→6, NOT 25→30+); zero job loss across completed rounds. The election-liveness layer is DONE._ + - _Root cause (`research.md` §"Task 5.6 fix VERIFIED"): with 5.6 in, round-4 ctrl-2 (freshest, 15@t5) wins PreVote, enters the real election, and ctrl-3 GRANTS it (vote_granted) — ctrl-3 votes FOR ctrl-2, not leapfrogging. But ctrl-2 never tallies a majority: every grant arrives as vote_reply_stale_drop (reply term < ctrl-2's current term). ctrl-2's term escalates via repeated election timeouts faster than ctrl-3 can reply, so each grant is one term stale → dropped → timeout → term++ → self-perpetuating. election_timeout_ms=5000, jitter=200ms._ + - Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (bounded term, no escalation stall) with all prior fixes in place — the green 7.2 gate ✅ (election layer met; 7.2 now blocked on a SEPARATE non-election crash, task 5.8) - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ - _Boundary: Election Liveness_ - _Depends: 5.6_ +- [ ] 5.8 Endpoint teardown-vs-dispatch heap-use-after-free crash (NEW — the round-4 7.2 blocker after 5.7, UPSTREAM-owned) + - _Root cause (`research.md` §"Task 5.7 fix VERIFIED" + bug-1298): with 5.7's election fix in, the 6-round chaotic soak's round-4 stall RE-ROOTS away from elections entirely. ctrl-3 ABORTs on an ASAN heap-use-after-free: the vtask dispatcher (T3) reads `ep->busy` via `endpoint_is_task_blocked` (netio.c:1008, in `vtask_eligible`→`dataq_deq_first_match`) on an endpoint that the io_worker (T2) just `free()`d via `endpoint_killer` (cluster.c:813, from `handle_endpoint_error`→`zarray_del`). The endpoint is freed while still referenced by a pending vtask in the dataq — no drain of its queued vtasks before the free, no refcount across the dispatcher read. ctrl-3 dies → cluster drops to 1/3 → no quorum → "no new primary within window." NOT an election bug; elections + job safety hold for every round that completes (rounds 1-3: zero job loss, bounded terms)._ + - **Ownership = UPSTREAM raft-drive-async / core libxce async-io dispatch** (vtaskmgr/dataq/netio/cluster teardown) — same surface and same routing precedent as task 5.4/bug-1282 (this spec's Non-Goals exclude the async-io layer). Same family as bug-1282 (ABBA, ep->busy made lockless) and the bug-1197 dispatch_inflight precedent; the bug-1282 lockless busy-read likely unmasked this latent UAF by letting the dispatcher reach the read sooner._ + - Fix direction (upstream): drain/cancel an endpoint's queued vtasks from the dataq BEFORE `endpoint_killer` frees it, OR hold a refcount/epoch on `endpoint_t` across the dispatcher eligibility read so the free defers until no dispatcher references remain (mirror bug-1197 `dispatch_inflight`). Must keep the bug-1282 lockless busy read and not reintroduce the ABBA ep->lock/q->lock inversion. + - Observable completion: the 6-round chaotic soak runs all rounds with NO controller crash — the green 7.2 gate + - _Requirements: 6.1, 6.2, 6.3_ + - _Boundary: Async-IO dispatch (UPSTREAM)_ + - _Depends: 5.7_ + - [ ] 6. Integration: single-seam invariant end-to-end - [ ] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end - Integration check that the live leader replication AND the cross-term catch-up streaming path transmit each entry's full `(term, payload)` to the receiver: no `term==0` on the wire from a live leader (defensive), and — the real concern — no non-empty entry ever arrives with `PLEN=0`; the walAck catch-up continuation feeds only the seam ingest path diff --git a/.wolf/buglog.json b/.wolf/buglog.json index ac1af7f7..d68ec8d9 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -21394,6 +21394,170 @@ "related_bugs": [], "occurrences": 1, "last_seen": "2026-06-30T09:15:12.861Z" + }, + { + "id": "bug-1291", + "timestamp": "2026-06-30T09:32:54.664Z", + "error_message": "Missing error handling in unknown", + "file": ".kiro/specs/raft-ha-core-safety/tasks.md", + "root_cause": "Code path had no error handling — exceptions would propagate uncaught", + "fix": "Added try/catch block", + "tags": [ + "auto-detected", + "error-handling", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T09:32:54.664Z" + }, + { + "id": "bug-1292", + "timestamp": "2026-06-30T09:33:08.918Z", + "error_message": "Incorrect value in code", + "file": ".kiro/specs/raft-ha-core-safety/tasks.md", + "root_cause": "Had `1+2*njobs`", + "fix": "Changed to `c = ep->owner`", + "tags": [ + "auto-detected", + "wrong-value", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T09:33:08.918Z" + }, + { + "id": "bug-1293", + "timestamp": "2026-06-30T11:15:05.373Z", + "error_message": "Significant refactor of ", + "file": "src/controller/ha.c", + "root_cause": "5 lines replaced/restructured", + "fix": "Rewrote 11→19 lines (5 removed) | Also: /* ha-term-safety task 1.2: advertise our Raft (la; * — identical sourcing to the real-vote send path ", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 2, + "last_seen": "2026-06-30T11:15:18.266Z" + }, + { + "id": "bug-1294", + "timestamp": "2026-06-30T11:42:35.428Z", + "error_message": "Missing error handling in unknown", + "file": "src/controller/testsuite/test_ha_core_safety.c", + "root_cause": "Code path had no error handling — exceptions would propagate uncaught", + "fix": "Added try/catch block", + "tags": [ + "auto-detected", + "error-handling", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T11:42:35.428Z" + }, + { + "id": "bug-1295", + "timestamp": "2026-06-30T13:19:42.706Z", + "error_message": "Missing guard clause", + "file": "src/controller/ha.c", + "root_cause": "No early return/throw for edge case: jmax > 0", + "fix": "Added guard clause: if (jmax > 0)", + "tags": [ + "auto-detected", + "guard-clause", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T13:19:42.706Z" + }, + { + "id": "bug-1296", + "timestamp": "2026-06-30T13:21:00.000Z", + "error_message": "10-round chaotic soak FAIL at round 7: all 3 nodes in RDHUP→PreVote→timeout→repeat livelock — zero pre_vote_reply_in events (not denials, silence). ctrl-3 sent 20+ PreVote rounds with NO reply received; ctrl-1 and ctrl-2 same loop. Rounds 1-6 PASS clean (all elections + job-verification succeeded); round 7 kills ctrl-2 (term=9 primary), restarts ctrl-3 (no convergence wait — chaotic). 2 surviving nodes (ctrl-1, ctrl-3) both start PreVote; neither processes the other's request → mutual timeout → RDHUP fires again → repeat forever.", + "file": "src/controller/ha.c", + "root_cause": "TASK 5.6 dueling-candidate PreVote cross-denial — DISTINCT from the earlier PreVote-WINS + real-election split-vote (which had both nodes winning PreVote then deadlocking in RequestVote). Here PreVote itself starves: each node's PreVote handler likely drops/ignores inbound PreVote requests while in its own PreVote round (no reply sent, not even denial). The task 5.6 RDHUP guard (commit 25e741d+) only prevents restarting failover while in CTRL_ELECTING — it does NOT fix the core problem where two nodes enter simultaneous PreVote rounds and each starves the other's reply. The priority stagger (bug-606) helps when 3 nodes are alive (one grants to break symmetry), but with only 2 survivors (the third restarting without convergence in chaotic mode), both deny each other and the stagger is insufficient.", + "fix": "NOT YET FIXED. Two approaches: (A) In the PreVote request handler, if we are in our own PreVote round and the incoming candidate has a FRESHER log (higher lastLogIndex/lastLogTerm), grant the PreVote and abort our own round (let the more-complete node lead). (B) Add randomized election-timeout jitter on the failover re-election path (currently only applied at cold-start via ha_wait_for_heartbeat_or_timeout) so the two survivors de-synchronize and one grants before starting its own round. Approach (A) is preferred for determinism.", + "tags": [ + "raft-ha-core-safety", + "election-liveness", + "prevote-cross-denial", + "dueling-candidates", + "task-5.6", + "chaotic-soak", + "round-7-stall" + ], + "related_bugs": [ + "bug-1281", + "bug-1282", + "bug-1286", + "bug-1292" + ], + "occurrences": 1, + "last_seen": "2026-06-30T13:21:00.000Z" + }, + { + "id": "bug-1297", + "timestamp": "2026-06-30T13:28:01.170Z", + "error_message": "Missing guard clause", + "file": "src/controller/ha.c", + "root_cause": "No early return/throw for edge case: jmax > 0", + "fix": "Added guard clause: if (jmax > 0)", + "tags": [ + "auto-detected", + "guard-clause", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T13:28:01.170Z" + }, + { + "id": "bug-1298", + "timestamp": "2026-06-30T13:36:00-07:00", + "error_message": "AddressSanitizer: heap-use-after-free READ size 4 at endpoint_is_task_blocked netio.c:1008 (vtask dispatcher T3) — endpoint freed by endpoint_killer cluster.c:813 (io_worker T2); ctrl-3 ABORTs mid-chaotic-soak round 3→4, dropping the cluster to 1/3 (no quorum) so round 4 'NO new primary within window'", + "file": "src/lib/netio.c / src/lib/vtaskmgr.c / src/controller/cluster.c", + "root_cause": "Teardown-vs-dispatch race: io_worker (T2) tears down a connection on error — handle_endpoint_error → zarray_del → endpoint_killer free()s the endpoint_t — while the vtask dispatcher (T3) is concurrently scanning the dataq (dataq_deq_first_match → vtask_eligible → endpoint_is_task_blocked) and reads ep->busy on that just-freed endpoint. The endpoint is freed while still referenced by a pending vtask in the dataq; nothing drains/guards the ep's queued vtasks before the free, and no refcount keeps it alive across the dispatcher read. Same netio surface as bug-1282 (ABBA, ep->busy made lockless) and the bug-1197 dispatch_inflight precedent; the bug-1282 lockless read removed the (incidental) lock contention so the dispatcher reaches the read faster, exposing the latent UAF.", + "fix": "NOT FIXED — proposed: drain/cancel an endpoint's queued vtasks from the dataq BEFORE endpoint_killer frees it, or hold a refcount/epoch on endpoint_t across the dispatcher eligibility read so the free is deferred until no dispatcher references remain (mirror bug-1197 dispatch_inflight). Must preserve the bug-1282 lockless busy read and not reintroduce the ABBA ep->lock/q->lock inversion.", + "tags": [ + "netio", + "vtaskmgr", + "use-after-free", + "asan", + "endpoint-teardown", + "concurrency", + "raft-ha-core-safety", + "soak", + "quorum", + "bug-1282-family" + ], + "related_bugs": [ + "bug-1282", + "bug-1197", + "bug-1286" + ], + "occurrences": 1, + "last_seen": "2026-06-30T13:36:00-07:00" + }, + { + "id": "bug-1299", + "timestamp": "2026-06-30T13:39:09.194Z", + "error_message": "Function not marked async", + "file": ".kiro/specs/raft-ha-core-safety/tasks.md", + "root_cause": "Function uses await but wasn't declared async", + "fix": "Added async modifier", + "tags": [ + "auto-detected", + "async-fix", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T13:39:09.194Z" } ] } \ No newline at end of file