lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(ha-commit-durability): sharpen bug-1306 root cause — xce_run_hooks clobbers commit -ETIMEDOUT

Browse source 
10-round soak waldump+log bisect: the chaotic committed_job_loss is NOT the
catch-up/entity-snapshot prefix gap. All 3 WALs gapless/CRC-intact/full
payloads; lost job only on ctrl-1@LSN30 term9 vs term-10 NOOP on the majority.
The durable-majority gate worked (commit_submit -> -ETIMEDOUT), but
xce_run_hooks (hook.c:134-157) returns only the LAST-registered hook's rc, so a
later JOB AFTER_EVENT hook (sched/apiserver) returning 0 clobbered persist's
-ETIMEDOUT -> handle_jobsubmit replied jobsubmit ok for an uncommitted entry ->
term-10 leader legally overwrote it. Fix per raft-apply-decoupling item #1:
move the WAL append + commit barrier out of the hook into an explicit
handle_jobsubmit-driven raft step whose rc is read directly (down payment on
the full apply-after-commit inversion). Entity-snapshot gap is a separate,
still-real defect (not this soak's mechanism).
This commit is contained in:
Rongsong Shen 2026-07-01 00:21:23 +00:00
parent 08bdaba3dc
commit 6bb9d81225
2 changed files with 37 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

24
.wolf/buglog.json
View file

@ -21660,19 +21660,23 @@
"id": "bug-1306",
"timestamp": "2026-06-30T14:31:00-07:00",
"error_message": "6-round chaotic-failover soak FAIL: committed_job_loss=4 committed_divergence=7 (rounds 4-6). Committed job 0-0-1782829181-1 permanently absent on reachable primary from round 4 on, across 3 elected primaries; rounds 1-3 clean.",
"file": ".kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md (persist.c/config_repl.c catch-up streaming)",
"root_cause": "PRE-EXISTING failover catch-up gap (bug-795/828/851 family): a just-restarted node rejoins mid-churn and is elected primary WITHOUT backfilling the full committed prefix (empty-rejoin catch-up streams [watermark..committed], no snapshot backfill) -> permanent gap that propagates as peers catch up FROM the gapped node. First OBSERVED here because the 5.8 endpoint-refcount fix (bug-1298) let a chaotic soak run past round 4 for the first time.",
"fix": "NOT FIXED \u2014 EXTERNAL spec ha-commit-durability (+ raft term/log-safety); raft-ha-core-safety task 5.9. NOT a regression: zero WAL/commit/catch-up code changed this session (only ep-refcount + 5.7 election-timeout); ASAN clean; rounds 1-3 loss-free; rounds 4-6 never reached before (no prior baseline).",
"file": "src/lib/hook.c (xce_run_hooks); job.c:603 gate is correct",
"root_cause": "SHARPENED 2026-06-30 (waldump+log bisect of the 10-round soak): NOT the catch-up prefix-gap/entity-snapshot mechanism. xce_run_hooks (src/lib/hook.c:134-157) returns only the LAST-registered hook's rc (rc overwritten each loop iter; break is BEFORE_EVENT-only). The durable-majority gate works: a term-9 leader's job at LSN 30 timed out commit (peers mid-restart, repl_wait_majority_ack -> commit_submit -> -ETIMEDOUT), persist_job_hook_fun returned -ETIMEDOUT, but a later-registered JOB AFTER_EVENT hook (sched/apiserver) returned 0 and CLOBBERED it -> handle_jobsubmit saw hook_rc==0 -> replied 'jobsubmit ok' for an uncommitted entry -> term-10 leader legally overwrote LSN 30 (term-10 NOOP) -> committed-job-loss. Violates the Commit Durability invariant (reply-success => majority-durable). Evidence: all 3 WALs gapless/CRC-intact/full payloads; lost job only on ctrl-1@LSN30 term9 vs term10 NOOP on majority; ctrl-1 log has the WAIT-deadline-expired WARNs and NO 'rejecting job' line.",
"fix": "PROPOSED (not yet applied): make xce_run_hooks fold to a sticky-negative rc on AFTER_EVENT (run all hooks, preserve the first/any rc<0; BEFORE_EVENT keeps veto/break). RED: two AFTER_EVENT hooks, first returns -ETIMEDOUT, second 0 -> assert xce_run_hooks returns -ETIMEDOUT. Then re-run the chaotic soak: committed_job_loss must be 0. SEPARATE still-real gap: entity-bearing snapshots for trimmed-prefix rejoin (not this soak's mechanism).",
"tags": [
"data-loss",
"failover",
"committed-job-loss",
"catch-up-gap",
"ha-commit-durability",
"soak",
"pre-existing",
"not-a-regression",
"task-5.9"
"soak",
"task-5.9",
"committed-job-loss",
"commit-durability",
"term-boundary-divergence",
"catch-up-gap",
"failover",
"hook-rc-clobber",
"ack-before-majority",
"data-loss"
],
"related_bugs": [
"bug-795",
@ -21681,7 +21685,7 @@
"bug-1298"
],
"occurrences": 1,
"last_seen": "2026-06-30T14:31:00-07:00"
"last_seen": "2026-06-30T14:35:00Z"
},
{
"id": "bug-1307",