--- id: "bug-828" status: "fixed" status_updated: "2026-09-24" status_note: "91b9a445, b77d4122" component: "controller" severity: "unknown" timestamp: "2026-06-20T10:19:56.678Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 3 files: ["src/controller/ha.c (vote_reply_dispatch)", "src/controller/persist.c (persist_wal_write_replicated)"] tags: ["async-fix", "md", "data-loss", "failover", "committed-job-loss", "wal", "catch-up", "snapshot", "ha-commit-durability", "bug-795", "architecture", "NOT-FIXED", "safety", "safe-committed-lsn", "vote-reply", "leader-completeness", "ha-repl-hardening", "PARTIAL-FIX", "c"] related: ["bug-795", "bug-851"] fix_commits: ["91b9a445", "b77d4122"] legacy: true --- # bug-828: Clean single failover under-computes safe_committed_lsn: new leader logs 'safe_committed_lsn=1 ...… ## Description Clean single failover under-computes safe_committed_lsn: new leader logs 'safe_committed_lsn=1 ... (from 1 granted voters)' (self only) and truncates committed entries down to its own boot floor. Post-election safe-commit-from-votes plumbing was DEAD in production. ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction _Not recorded (migrated from buglog.json)._ ## Root cause vote_reply_decode_and_dispatch decoded the granting voter's advertised peerLastLogLsn (vr_lsn at ha.c ~4254) but vote_reply_dispatch DISCARDED it ('(void)peer_lsn; raft_on_vote_reply(...)'). raft_on_vote_reply only bumps s->votes_granted; it never records the peer LSN. The only writer of ctx->votes[idx].vote_granted/peer_lsn (handle_request_vote_reply) was, in production, called ONLY from the connect/error failure paths with (granted=0, lsn=0); the real-grant caller was gated behind a test-only hook (xce_vote_reply_hook, never set in production). So ctx->votes[*] stayed calloc'd zero -> ha_compute_safe_committed_lsn saw n_granted=1 (self) -> safe_committed_lsn = self_lsn. A standby promoting with its own durable_floor=1 then persist_truncate_wal_above(1) destroyed committed LSN 2..5. ## Fix PRIMARY: in vote_reply_dispatch production branch, route the decoded reply into the election context via a NEW record-only seam ha_record_vote_reply(ha,ctx,idx,term,granted,tag,lsn) BEFORE raft_on_vote_reply. Factored ha_record_vote_reply_locked out of handle_request_vote_reply: it records ctx->votes[idx] under ctx->lock with the idempotency guard but does NOT call ha_elect_evaluate_and_signal -> no second/racing legacy promotion path (raft_on_vote_reply remains the SOLE production promotion driver; the two quorum tallies (raft s->votes_granted / legacy ctx->votes_granted) and their guards (s->election_decided / ctx->done_posted) stay independent and idempotent). idx == vctx->peer_idx == the local ctx->votes[] index (absolute->local map done in raft_cb_send_vote_request), exactly the index ha_compute_safe_committed_lsn + the test hook use. SECONDARY: persist_wal_write_replicated now advances durable_lsn to the synced tip immediately after the inline fdatasync (mirror persist_flush; monotonic max; fail-loud rc-gated) so a standby advertises its own synced durable floor without waiting for the 10ms cadence flusher. VERIFIED: unit test_safe_commit_reflects_granting_voter_lsn RED (safe=1 'from 1 granted voters') -> GREEN (safe=5 'from 3 granted voters'); 177/177 controller_unit_test pass. NOTE: the LIVE clean_failover_preservation harness still FAILS, but the failure is PRE-EXISTING on baseline 14c41cb (verified by stash+rerun: baseline runs also diverge / lose jobs) — it is the SEPARATE bug-795/828 replication+catch-up gap (standbys' WALs physically contain only LSN 1..2, durable_floor=1; committed entries 2..5 never reach the standby WAL; primary advanced committed_lsn to 5 without a standby durable at 5). The fix DID change the live fingerprint from 'from 1 granted voters' to 'from 2 granted voters', proving the plumbing is now alive. safe-commit-from-votes is correct; the residual live job loss is out of scope (needs the catch-up/snapshot-backfill fix tracked under ha-commit-durability). ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-795, bug-851