From a91275be16ea418281ccb92848a3cf00179b89c6 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 25 Aug 2026 16:31:48 +0000 Subject: [PATCH] chore(wolf): checkpoint buglog.json (1037 bugs) --- .wolf/buglog.json | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 35833896..838228b4 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -19028,6 +19028,40 @@ ], "occurrences": 2, "last_seen": "2026-08-25T16:00:00Z" + }, + { + "id": "bug-1696", + "timestamp": "2026-08-25T16:20:00Z", + "error_message": "Chaotic HA soak (post-bug-1695-fix) reports committed_job_loss on every round; waldump on ctrl-1's live data dir shows the ENTIRE on-disk WAL truncated to a single NOOP entry (entries: 1 submits: 0) despite 21+ jobs previously verified committed; no snapshot exists; log shows ha_compute_safe_committed_lsn=0 immediately before raft_cb_become_leader promotes ctrl-1 to CTRL_PRIMARY at term=2 on its SECOND boot.", + "file": "src/controller/ha.c", + "root_cause": "raft_cb_become_leader() computes the promotion WAL-truncation floor (line ~1658-1662: trunc_floor = ha_promotion_truncate_floor(safe_lsn, persist_durable_floor(&c->persist)); persist_truncate_wal_above(&c->persist, trunc_floor)) and evaluates the whole promotion-eligibility gate (durable_floor/truncated/apply_verdict, ~line 1699-1823) BEFORE cluster_init_phase2() -- the ONLY call that invokes persist_recover() for this process's first-ever boot -- which does not run until is_first_boot handling at line ~1873-1899, further down in the SAME function. persist_durable_floor()/persist_recovery_truncated()/persist_apply_verdict() all read fields persist_recover() is responsible for populating from the real on-disk WAL; before that call, a freshly persist_init()'d struct reports durable_lsn=0 (bug-1445's comment at line 1901 already flagged this exact 'gate evaluated blind before recovery' hazard for ONE flag -- persist_apply_resource_fenced -- with a narrow re-check-after-phase2 patch, but the far more consequential durable_floor/truncation-floor computation was never covered by that fix). With durable_floor=0 (bogus, pre-recovery) and safe_lsn=0 (every voter, including self, ALSO advertised its LSN via persist_durable_floor() before its own recovery ran -- myapp.c:324-329, ha_candidate_advertise_lsn), ha_promotion_truncate_floor(0,0)=0 and persist_truncate_wal_above(p, 0) physically ftruncates/unlinks the real on-disk WAL segments down to LSN 0/1 -- destroying committed job data that persist_recover(), had it run FIRST, would have correctly reported as durable and protected via the bug-1060 floor guard. This scenario (a node RESTARTS with real durable data on disk, then immediately WINS its own promotion on that same boot, before its own recovery has run) was unreachable before bug-1695's fix -- previously a restarted node's stale-but-nonzero persisted term made it always skip its own election and defer to a peer, so it never hit is_first_boot-during-become_leader with real data at stake. bug-1695 correctly restored real re-election on restart and, in doing so, newly exposed this pre-existing boot-ordering flaw. A genuine Leader Completeness / bug-1060-guard-defeat, not a test-harness artifact: confirmed via direct waldump on-disk evidence plus source-level trace of the read-before-recover ordering.", + "fix": "PENDING -- dispatching a dedicated fix: reorder raft_cb_become_leader() so cluster_init_phase2()/persist_recover() (is_first_boot path, currently ~line 1873-1899) runs BEFORE the safe_committed_lsn/promotion-truncation-floor computation and the promotion-eligibility gate (currently ~line 1637-1823), so persist_durable_floor()/persist_recovery_truncated()/persist_apply_verdict() always reflect this boot's REAL recovered WAL state before any truncation or eligibility decision reads them. Must NOT touch the separate become_follower cluster_init_phase2 call site (~line 2101-2135) or the bug-1445 apply_resource_fenced re-check (keep as a defense-in-depth no-op once reordered). Requires new RED regression test: a node with real durable committed WAL data, restarted, winning its own promotion on that same restart, must retain all committed data post-promotion (not truncate to an empty/near-empty floor). Prove with full controller unit suite + a live chaotic HA soak A/B (pre-fix reproduces committed_job_loss>0, post-fix committed_job_loss=0) before closing.", + "tags": [ + "controller", + "ha", + "raft", + "election", + "wal-truncation", + "leader-completeness", + "data-safety", + "bug-1060-regression-class", + "boot-ordering-race", + "persist_recover", + "cluster_init_phase2", + "found-via-soak", + "critical", + "bug-1695-adjacent", + "unfixed" + ], + "related_bugs": [ + "bug-1695", + "bug-1060", + "bug-1445", + "bug-828", + "bug-1487" + ], + "occurrences": 1, + "last_seen": "2026-08-25T16:20:00Z" } ] }