docs(raft-ha-core-safety): 5.7 verified (election resonance fixed); round-4 re-roots to bug-1298 UAF (task 5.8)
5.7 election-timeout resonance FIXED + soak-confirmed: 6-round chaotic soak now elects with bounded terms (rounds 1-3 at terms 4/5/6, not 25-30+), zero job loss. The election-liveness layer is done. Round-4 stall re-roots to a NON-election crash: ctrl-3 ASAN heap-use-after-free in endpoint teardown — io_worker (endpoint_killer, cluster.c:813) frees an endpoint while the vtask dispatcher reads ep->busy (endpoint_is_task_blocked, netio.c:1008). ctrl-3 dies -> cluster drops to 1/3 -> no quorum -> no primary. Same async-io dispatch surface as bug-1282/1197; UPSTREAM-owned (raft-drive-async / libxce). Logged bug-1298; tracked as task 5.8.
This commit is contained in:
parent
74784b2b18
commit
5be165312a
3 changed files with 249 additions and 4 deletions
|
|
@ -21394,6 +21394,170 @@
|
|||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T09:15:12.861Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1291",
|
||||
"timestamp": "2026-06-30T09:32:54.664Z",
|
||||
"error_message": "Missing error handling in unknown",
|
||||
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
|
||||
"root_cause": "Code path had no error handling — exceptions would propagate uncaught",
|
||||
"fix": "Added try/catch block",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"error-handling",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T09:32:54.664Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1292",
|
||||
"timestamp": "2026-06-30T09:33:08.918Z",
|
||||
"error_message": "Incorrect value in code",
|
||||
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
|
||||
"root_cause": "Had `1+2*njobs`",
|
||||
"fix": "Changed to `c = ep->owner`",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"wrong-value",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T09:33:08.918Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1293",
|
||||
"timestamp": "2026-06-30T11:15:05.373Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "5 lines replaced/restructured",
|
||||
"fix": "Rewrote 11→19 lines (5 removed) | Also: /* ha-term-safety task 1.2: advertise our Raft (la; * — identical sourcing to the real-vote send path ",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-06-30T11:15:18.266Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1294",
|
||||
"timestamp": "2026-06-30T11:42:35.428Z",
|
||||
"error_message": "Missing error handling in unknown",
|
||||
"file": "src/controller/testsuite/test_ha_core_safety.c",
|
||||
"root_cause": "Code path had no error handling — exceptions would propagate uncaught",
|
||||
"fix": "Added try/catch block",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"error-handling",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T11:42:35.428Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1295",
|
||||
"timestamp": "2026-06-30T13:19:42.706Z",
|
||||
"error_message": "Missing guard clause",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "No early return/throw for edge case: jmax > 0",
|
||||
"fix": "Added guard clause: if (jmax > 0)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"guard-clause",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T13:19:42.706Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1296",
|
||||
"timestamp": "2026-06-30T13:21:00.000Z",
|
||||
"error_message": "10-round chaotic soak FAIL at round 7: all 3 nodes in RDHUP→PreVote→timeout→repeat livelock — zero pre_vote_reply_in events (not denials, silence). ctrl-3 sent 20+ PreVote rounds with NO reply received; ctrl-1 and ctrl-2 same loop. Rounds 1-6 PASS clean (all elections + job-verification succeeded); round 7 kills ctrl-2 (term=9 primary), restarts ctrl-3 (no convergence wait — chaotic). 2 surviving nodes (ctrl-1, ctrl-3) both start PreVote; neither processes the other's request → mutual timeout → RDHUP fires again → repeat forever.",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "TASK 5.6 dueling-candidate PreVote cross-denial — DISTINCT from the earlier PreVote-WINS + real-election split-vote (which had both nodes winning PreVote then deadlocking in RequestVote). Here PreVote itself starves: each node's PreVote handler likely drops/ignores inbound PreVote requests while in its own PreVote round (no reply sent, not even denial). The task 5.6 RDHUP guard (commit 25e741d+) only prevents restarting failover while in CTRL_ELECTING — it does NOT fix the core problem where two nodes enter simultaneous PreVote rounds and each starves the other's reply. The priority stagger (bug-606) helps when 3 nodes are alive (one grants to break symmetry), but with only 2 survivors (the third restarting without convergence in chaotic mode), both deny each other and the stagger is insufficient.",
|
||||
"fix": "NOT YET FIXED. Two approaches: (A) In the PreVote request handler, if we are in our own PreVote round and the incoming candidate has a FRESHER log (higher lastLogIndex/lastLogTerm), grant the PreVote and abort our own round (let the more-complete node lead). (B) Add randomized election-timeout jitter on the failover re-election path (currently only applied at cold-start via ha_wait_for_heartbeat_or_timeout) so the two survivors de-synchronize and one grants before starting its own round. Approach (A) is preferred for determinism.",
|
||||
"tags": [
|
||||
"raft-ha-core-safety",
|
||||
"election-liveness",
|
||||
"prevote-cross-denial",
|
||||
"dueling-candidates",
|
||||
"task-5.6",
|
||||
"chaotic-soak",
|
||||
"round-7-stall"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1281",
|
||||
"bug-1282",
|
||||
"bug-1286",
|
||||
"bug-1292"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T13:21:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1297",
|
||||
"timestamp": "2026-06-30T13:28:01.170Z",
|
||||
"error_message": "Missing guard clause",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "No early return/throw for edge case: jmax > 0",
|
||||
"fix": "Added guard clause: if (jmax > 0)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"guard-clause",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T13:28:01.170Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1298",
|
||||
"timestamp": "2026-06-30T13:36:00-07:00",
|
||||
"error_message": "AddressSanitizer: heap-use-after-free READ size 4 at endpoint_is_task_blocked netio.c:1008 (vtask dispatcher T3) — endpoint freed by endpoint_killer cluster.c:813 (io_worker T2); ctrl-3 ABORTs mid-chaotic-soak round 3→4, dropping the cluster to 1/3 (no quorum) so round 4 'NO new primary within window'",
|
||||
"file": "src/lib/netio.c / src/lib/vtaskmgr.c / src/controller/cluster.c",
|
||||
"root_cause": "Teardown-vs-dispatch race: io_worker (T2) tears down a connection on error — handle_endpoint_error → zarray_del → endpoint_killer free()s the endpoint_t — while the vtask dispatcher (T3) is concurrently scanning the dataq (dataq_deq_first_match → vtask_eligible → endpoint_is_task_blocked) and reads ep->busy on that just-freed endpoint. The endpoint is freed while still referenced by a pending vtask in the dataq; nothing drains/guards the ep's queued vtasks before the free, and no refcount keeps it alive across the dispatcher read. Same netio surface as bug-1282 (ABBA, ep->busy made lockless) and the bug-1197 dispatch_inflight precedent; the bug-1282 lockless read removed the (incidental) lock contention so the dispatcher reaches the read faster, exposing the latent UAF.",
|
||||
"fix": "NOT FIXED — proposed: drain/cancel an endpoint's queued vtasks from the dataq BEFORE endpoint_killer frees it, or hold a refcount/epoch on endpoint_t across the dispatcher eligibility read so the free is deferred until no dispatcher references remain (mirror bug-1197 dispatch_inflight). Must preserve the bug-1282 lockless busy read and not reintroduce the ABBA ep->lock/q->lock inversion.",
|
||||
"tags": [
|
||||
"netio",
|
||||
"vtaskmgr",
|
||||
"use-after-free",
|
||||
"asan",
|
||||
"endpoint-teardown",
|
||||
"concurrency",
|
||||
"raft-ha-core-safety",
|
||||
"soak",
|
||||
"quorum",
|
||||
"bug-1282-family"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1282",
|
||||
"bug-1197",
|
||||
"bug-1286"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T13:36:00-07:00"
|
||||
},
|
||||
{
|
||||
"id": "bug-1299",
|
||||
"timestamp": "2026-06-30T13:39:09.194Z",
|
||||
"error_message": "Function not marked async",
|
||||
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
|
||||
"root_cause": "Function uses await but wasn't declared async",
|
||||
"fix": "Added async modifier",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"async-fix",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T13:39:09.194Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue