10-round soak waldump+log bisect: the chaotic committed_job_loss is NOT the catch-up/entity-snapshot prefix gap. All 3 WALs gapless/CRC-intact/full payloads; lost job only on ctrl-1@LSN30 term9 vs term-10 NOOP on the majority. The durable-majority gate worked (commit_submit -> -ETIMEDOUT), but xce_run_hooks (hook.c:134-157) returns only the LAST-registered hook's rc, so a later JOB AFTER_EVENT hook (sched/apiserver) returning 0 clobbered persist's -ETIMEDOUT -> handle_jobsubmit replied jobsubmit ok for an uncommitted entry -> term-10 leader legally overwrote it. Fix per raft-apply-decoupling item #1: move the WAL append + commit barrier out of the hook into an explicit handle_jobsubmit-driven raft step whose rc is read directly (down payment on the full apply-after-commit inversion). Entity-snapshot gap is a separate, still-real defect (not this soak's mechanism). |
||
|---|---|---|
| .. | ||
| repro-851 | ||
| brief.md | ||
| design.md | ||
| INVESTIGATION-failover-job-loss.md | ||
| requirements.md | ||
| research.md | ||
| spec.json | ||
| tasks.md | ||