---
id: "bug-777"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "data-loss"
timestamp: "2026-06-19T13:00:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/controller/ha.c (raft_cb_become_leader promotion fence release) + persist.c (recovery_truncated verdict stickiness); design ha-commit-durability Req 2.2/2.6 + Boot Policy Decision Table"]
tags: ["ha", "convergence", "promotion-fence", "recovery-truncated", "torn-tail", "SIGKILL", "availability", "election-wedge", "ha-commit-durability", "ha-convergence-recovery", "live-E2E", "false-positive-data-loss"]
related: ["bug-661", "bug-339"]
fix_commits: ["64ab210"]
legacy: true
---

# bug-777: 3-node rolling-SIGKILL soak round 3: 'no new primary elected within 40s'; soak reports 'DATA LOSS…

## Description
3-node rolling-SIGKILL soak round 3: 'no new primary elected within 40s'; soak reports 'DATA LOSS 5/5 jobs vanished'. Both surviving nodes recovered RECOVER_TRUNCATED after SIGKILL (torn WAL tail) and stay permanently fenced/non-promotable even though durable_floor==cluster_hwm (they hold ALL committed data).

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
A SIGKILL tears the WAL tail; on restart replay_wal hits a CRC-break/short-read at the tail -> RECOVER_TRUNCATED (sticky flag). The promotion-fence release condition requires `!recovery_truncated`, but the truncated flag is NEVER cleared by catch-up — so a node whose intact durable prefix (durable_floor) already reaches/exceeds the cluster committed high-water (durable_floor>=hwm, i.e. holds ALL committed entries; the torn tail is entirely UNCOMMITTED) stays permanently non-promotable. The design assumed 'TRUNCATED => provably below HWM', which is FALSE for a torn TAIL above the committed prefix. When all survivors are in this state (after multiple rolling failovers), none can promote => election/availability WEDGE. Observed: ctrl-1 recovery_truncated=1 durable_floor=13 cluster_hwm=13, PreVote-DENY peers reason=catch_up, indefinitely.

## Fix
DATA-LOSS dimension RESOLVED by the bug-778 fix (commit 64ab210): the cold-restart loss was the CRC-chain clobber; with the chain intact, cold restart recovers all committed jobs (soak 5/5, joblist 6/6 after cold restart). The AVAILABILITY-WEDGE dimension was a CONSEQUENCE of the spurious RECOVER_TRUNCATED from the broken chain — no longer triggered (nodes now recover CLEAN; round-3 re-elects normally). RESIDUAL HARDENING (latent, not currently triggered): the promotion fence keeps recovery_truncated sticky even when durable_floor>=HWM, so a GENUINE torn-tail that leaves a node RECOVER_TRUNCATED while holding all committed data could still over-fence. Optional follow-up: clear the truncated fence when durable_floor>=cluster_hwm (the torn tail is provably uncommitted). Not blocking — the systematic trigger is gone.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-661, bug-339