--- id: "bug-1418" status: "fixed" status_updated: "2026-09-24" status_note: "9b173c9a + bug-1421 + raft-apply-decoupling" component: "controller" severity: "hang" timestamp: "2026-07-09T10:30:00.000Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 5 files: ["src/controller/ha.c"] tags: ["ha", "raft", "apply-divergence", "promotion-gate", "chaotic-failover", "liveness", "term-desync", "resolved-via-upstream"] related: ["bug-1417", "bug-1419", "bug-1421", "bug-1416"] fix_commits: ["9b173c9a"] legacy: true --- # bug-1418: Final resolution pass over bug-1418 (all 4 prior occurrences): the two contributing factors from… ## Description Final resolution pass over bug-1418 (all 4 prior occurrences): the two contributing factors from the 2026-07-07T18:20 analysis (term desync + apply-divergence promotion-gate liveness deadlock) are both now resolved, not by the originally-proposed gate relaxation. ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction _Not recorded (migrated from buglog.json)._ ## Root cause Re-audited both factors against the CURRENT code (post raft-apply-decoupling 32/32, post bug-1421): (1) TERM DESYNC — ha_start_prevote_async (ha.c:6715) already computes its prospective term from ha->raft.term + 1, not the stale ha->term the original analysis described; ha.c:1009-1010 explicitly syncs ha->term from the raft term on every relevant transition. This half was already fixed independently of bug-1418's own tracked fix. (2) APPLY-DIVERGENCE LIVENESS DEADLOCK — the promotion gate (ha.c raft_cb_become_leader ~line 1130, and promotion_fence_timeout_fn ~line 876) unconditionally refuses promotion while persist_apply_verdict()==APPLY_DIVERGENT, by deliberate design (raft-apply-decoupling task 5.3, Req 8.4: a divergent local materialization must never be trusted to self-clear via any live/structural condition, including durable_floor>=cluster_hwm — only a fresh persist_recover() replay may clear it). The buglog's own previously-proposed fix (relax the gate to self-apply when durable_floor>=cluster_hwm) would have violated this intentional invariant and risked serving genuinely divergent state to clients. Instead, raft-apply-decoupling's tasks 5.1 (checkpoint_lsn mislabeled from WAL tip instead of the logical applied cursor, causing a false APPLY_DIVERGENT on an ordinary crash-during-checkpoint) and 7.2/bug-1416 (a boot-time TOCTOU where a returning node's persist_recover() replay-verify raced the live apply pump / live-replication apply path, producing a false APPLY_DIVERGENT on the restart/catch-up path specifically — the exact scenario bug-1418's residual occurrence hit: 'apply pump lagged behind WAL tip during kill') closed the false-positive detection paths that were producing spurious APPLY_DIVERGENT verdicts in the first place. A genuine (non-false-positive) divergence still correctly fences forever by design. ## Fix [Consolidated 2026-08-15: this ID previously had 4 duplicate entries in buglog.json (a pgrep-guard test-harness fix, a partial term-desync analysis, and a stale mid-investigation 'NOT FIXED' snapshot -- all superseded by this final resolution pass, which already accounted for all 3 in its own analysis). Collapsed to this single entry; the other 3 removed.] No code change to the promotion gate itself — it is intentionally correct and must stay unconditional. The actual fix is the combination of already-shipped upstream work: bug-1421 (AE-reject term-discovery blindspot, this session) + raft-apply-decoupling tasks 5.1/5.3/7.2 (false-APPLY_DIVERGENT elimination, closed prior to this session). Validated: two 40-round soaks today (sequential 123 jobs zero-loss all-3-nodes-led; chaotic 123 jobs zero-committed-divergence zero-job-loss) — an order of magnitude more election churn than the original 10-round soak that first surfaced bug-1418 — produced zero apply-divergent promotion refusals and zero liveness deadlocks. Further corroborated by the 30-round sequential+chaotic regression run immediately after this entry. ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-1417, bug-1419, bug-1421, bug-1416