---
id: "bug-1306"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "unknown"
timestamp: "2026-06-30T14:31:00-07:00"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/lib/hook.c (xce_run_hooks); job.c:603 gate is correct"]
tags: ["ha-commit-durability", "pre-existing", "not-a-regression", "soak", "task-5.9", "committed-job-loss", "commit-durability", "term-boundary-divergence", "catch-up-gap", "failover", "hook-rc-clobber", "ack-before-majority", "data-loss"]
related: ["bug-795", "bug-828", "bug-851", "bug-1298"]
legacy: true
---

# bug-1306: 6-round chaotic-failover soak FAIL: committed_job_loss=4 committed_divergence=7 (rounds 4-6)

## Description
6-round chaotic-failover soak FAIL: committed_job_loss=4 committed_divergence=7 (rounds 4-6). Committed job 0-0-1782829181-1 permanently absent on reachable primary from round 4 on, across 3 elected primaries; rounds 1-3 clean.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
SHARPENED 2026-06-30 (waldump+log bisect of the 10-round soak): NOT the catch-up prefix-gap/entity-snapshot mechanism. xce_run_hooks (src/lib/hook.c:134-157) returns only the LAST-registered hook's rc (rc overwritten each loop iter; break is BEFORE_EVENT-only). The durable-majority gate works: a term-9 leader's job at LSN 30 timed out commit (peers mid-restart, repl_wait_majority_ack -> commit_submit -> -ETIMEDOUT), persist_job_hook_fun returned -ETIMEDOUT, but a later-registered JOB AFTER_EVENT hook (sched/apiserver) returned 0 and CLOBBERED it -> handle_jobsubmit saw hook_rc==0 -> replied 'jobsubmit ok' for an uncommitted entry -> term-10 leader legally overwrote LSN 30 (term-10 NOOP) -> committed-job-loss. Violates the Commit Durability invariant (reply-success => majority-durable). Evidence: all 3 WALs gapless/CRC-intact/full payloads; lost job only on ctrl-1@LSN30 term9 vs term10 NOOP on majority; ctrl-1 log has the WAIT-deadline-expired WARNs and NO 'rejecting job' line.

## Fix
FIXED 2026-06-30 (raft-apply-decoupling item #1 down payment). Extracted the WAL append + majority-commit barrier out of persist_job_hook_fun's WAIT branch into an explicit persist_submit_wal_and_commit(p, jlist); handle_jobsubmit now calls it directly and reads the commit rc DIRECTLY (rejects on <0 before notifying), so the -ETIMEDOUT can no longer be clobbered by a later notify hook's 0 in xce_run_hooks. The JOB_NEW AFTER hook is now notification-only. PROVEN: 10-round chaotic soak committed_job_loss 5 -> 0 (same election code, A/B isolates the durability fix). Unit: +test_job_hook_new_is_notify_only; persist_hooks 19/0, e2e_hooks 3/0, commit_barrier_reply 22/0, commit_durability 38/0. Remaining soak FAIL is committed_divergence (transient catch-up lag, present=2) + a separate round-4 election escalation tail (5.7 class) — neither is job loss.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-795, bug-828, bug-851, bug-1298