--- id: "bug-848" status: "fixed" status_updated: "2026-09-24" status_note: "" component: "controller" severity: "unknown" timestamp: "2026-06-20T14:40:02.211Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 3 files: ["src/controller/cluster.c (cluster_init_phase2", "after persist_recover) + src/controller/ha.c (ha_init applied_lsn=0; state report sr->applied_lsn)"] tags: ["c", "ha", "raft", "recovery", "catchup", "applied_lsn", "data-loss", "root-trigger", "critical"] related: ["bug-847", "bug-828", "bug-795"] legacy: true --- # bug-848: On a PRESERVED-DATA restart (no wipe — user confirmed), a controller recovers its WAL [1..7] from… ## Description On a PRESERVED-DATA restart (no wipe — user confirmed), a controller recovers its WAL [1..7] from disk but rejoins advertising applied_lsn=0, so the primary re-streams the WHOLE log [1..committed] and O_TRUNC-overwrites the node's already-correct WAL. Logs: 'HA: learned committed_lsn=10 ... (was 0)'. THE root trigger that forces the full re-stream (then bug-847 empties it). ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction _Not recorded (migrated from buglog.json)._ ## Root cause ha->applied_lsn is NEVER reconstructed from the recovered WAL on boot. ha_init zeros it (ha.c:1367); persist_recover (cluster.c:1352) replays the WAL and advances persist->current_lsn/durable_lsn, but nothing copies that recovered position into ha->applied_lsn (persist_t and ha_state_t are separate; the only setter besides ha_init is the SNAPSHOT receive path ha.c:1771 = checkpoint_lsn). So the standby state report advertises sr->applied_lsn = ha->applied_lsn = 0 (ha.c:1787) and the primary's ha_evaluate_repl_action streams [0+1 .. committed]. ASYMMETRY proving it's a bug: ELECTION advertise uses ha_candidate_advertise_lsn = persist_durable_floor (the RECOVERED position, correct), but CATCH-UP advertise uses the un-recovered applied_lsn=0. The system knows its durable tip for voting but not for catch-up. Defeats Raft incremental catch-up for every restarted node; also the upstream cause of the earlier missing-prefix cases (full re-stream from a trimmed primary -> prefix gap). ## Fix FIXED 2026-06-20 (HA-layer). Added ha_sync_applied_lsn_from_recovery(ha,p) (ha.c/ha.h) = monotonic store of persist_durable_floor(p) into ha->applied_lsn; wired into cluster_init_phase2 right after persist_recover, plus a diagnostic INFO log of recovered durable_floor/current_lsn/applied_lsn. TDD: test_recovered_member_advertises_real_applied_lsn (test_persist_recovery.c) write->recover->sync asserts applied_lsn==durable_floor>0 and monotonic. controller_unit_test 6/6 + adjacent HA 8/8, daemon builds. CAVEAT: live ctrl-1 also showed current_lsn=1 (seg O_TRUNC'd from LSN 1) implying persist_recover may not have loaded the WAL on that restart at all; if so this HA-layer fix is necessary but not sufficient. The new INFO log reveals on the next live restart whether persist_recover is effective (durable_floor>0) or not (bug-849 candidate: recovery not running/loading on the rejoin path). ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-847, bug-828, bug-795