--- id: "bug-1432" status: "fixed" status_updated: "2026-09-24" status_note: "" component: "controller" severity: "unknown" timestamp: "2026-07-10T10:15:00Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 2 files: ["src/controller/ha.c"] tags: ["raft", "heartbeat", "promotion", "delaymgr", "chaotic-soak", "latency"] related: ["bug-1431"] legacy: true --- # bug-1432: chaotic HA soak still FAIL (committed_job_loss=8) after bug-1431 force-flush fix; jobs above the… ## Description chaotic HA soak still FAIL (committed_job_loss=8) after bug-1431 force-flush fix; jobs above the conservative 2-voter safe_committed_lsn floor stayed invisible via expect-job for ~3.1s post-promotion, matching the primary heartbeat interval (3000ms) almost exactly ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction _Not recorded (migrated from buglog.json)._ ## Root cause ha_start_primary_heartbeat armed the periodic heartbeat via delay_task_ms(interval_ms, -1, primary_heartbeat_fn, c), whose FIRST invocation fires only after interval_ms elapses (standard periodic-timer semantics) -- despite the functions own long-standing comment claiming to start the heartbeat immediately. A freshly-promoted leader therefore waited a full heartbeat interval before ever sending a NOP to a reconnecting/catching-up standby, so it could not learn that standbys true applied_lsn (via the heartbeat-reply stateReport, ha_process_standby_state) for up to 3s. That knowledge gates when catch-up WAL streaming starts and, in turn, the per-entry majority-ack that advances committed_lsn past the elections conservative safe_committed_lsn floor (jobs above that floor were legitimately majority-committed via the OLD primary before it died, per Leader Completeness, but the new leader cannot expose them without fresh quorum corroboration -- a correct Raft Figure-8 safety requirement, not a bug in itself). ## Fix ha_start_primary_heartbeat now calls primary_heartbeat_fn(c) synchronously once, immediately, before arming the periodic delay_task_ms timer -- matching the functions own documented intent. primary_heartbeat_fn only performs non-blocking async sends (xce_ha_heartbeat_req), so this is safe inline with no thread-role change. Single fix point covers all 3 promotion call sites (raft_cb_become_leader, ha_promote_to_primary, and the third raft-driven-decision-loop site) since all route through ha_start_primary_heartbeat. Closed the gap: 10-round chaotic soak went from committed_job_loss=8 to committed_job_loss=0, committed_divergence=0, result=PASS; 10-round sequential soak + leadership-coverage also PASS (36 jobs, zero loss). ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-1431