Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
14 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1478 | fixed | 2026-09-24 | controller | unknown | 2026-07-28T09:00:00Z | 2026-09-19T09:00:00Z | 2 |
|
|
|
true |
bug-1478: config-upgrade-snapshot task 8.9 live soak (run-ha-soak.sh --config-upgrade-wal-trim…
Description
config-upgrade-snapshot task 8.9 live soak (run-ha-soak.sh --config-upgrade-wal-trim ~/tmp/xce-ha-waltrim): bug-1474's floor-preservation fix confirmed working (floor segment seg-5 correctly preserved, all below-floor segments correctly trimmed) but the round still FAILED: "cluster did not reconverge to 3-of-3 after restarting ctrl-1 -- the preserved floor segment did not actually suffice for real incremental catch-up". ctrl-1 (TARGET, killed+restarted 3x) settles into a PERMANENT, zero-progress loop: its own raft AppendEntries exchanges cleanly reach match=22 with zero ae_reject events (heartbeat probe at prevIdx=22,n=0 -> success=1 match=22, repeating every ~3s forever), while ctrl-2 (primary) simultaneously and forever logs 'HA repl: standby[0] lsn=21 < committed_lsn=22 -> stream WAL batches' / 'ha_stream_wal_to_standby: streamed WAL [21+1..22] to standby' every heartbeat cycle with zero effect -- an off-by-one disagreement (21 vs 22) between the two sides that never reconciles.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
ROOT-CAUSED with byte/log-precise evidence; NOT a recurrence of bug-1474 (independently re-confirmed working in this same run -- floor preservation and below-floor trim both correct), bug-1475 (independently re-confirmed working -- ctrl-1's own raft log_len/term-index bookkeeping across the checkpoint boundary is clean, zero ae_reject_logmismatch anywhere in its log, unlike bug-1475's own signature), or bug-1477 (ctrl-1 clearly DID wire its commit engine every boot -- multiple 'commit engine wired'/persist_recover lines confirmed, unlike bug-1477's ctrl-3 which never ran its own election at all). This is a THIRD, previously-undiscovered gap in the same 'WAL_CHECKPOINT breaks LSN co-tracking invariants' family already flagged (not fixed at its root) as bug-1466.
Ground truth via xce_test waldump on the preserved deploy dir (~/tmp/xce-ha-waltrim): ctrl-2's WAL shows lsn=21 as the WAL_CONFIG_MARKER (type UNKNOWN/40 bytes, matching ctrl-2's own log 'Config change committed (marker_lsn=21)') and lsn=22 as a WAL_CHECKPOINT (0-byte payload) -- confirming persist_write_checkpoint_marker (persist.c:1747, hdr.lsn = p->current_lsn++) fired immediately after the marker committed and consumed the very next LSN slot, exactly as apply_committed_entry's WAL_CONFIG_MARKER case (persist_checkpoint_marker_triggered) is designed to do (config-upgrade-snapshot task 5). Critically, ctrl-2's OWN commit_advance_locked (commit.c, the SOLE authoritative committed_lsn writer) never logs an advance past 21 anywhere in the run ('commit: committed_lsn 20 -> 21' is the LAST such line) -- the real, majority-gated persist.committed_lsn correctly and permanently stays at 21, since no standby can ever legitimately ack a local-only, non-replicated checkpoint entry.
Yet ha_dispatch_repl_action_for_standby's own log line reports 'committed_lsn=22' from the very next heartbeat cycle onward. Traced to repl_ack_decode_and_dispatch's stateReportMsg branch (ha.c): last_written = c->persist.current_lsn - 1 (the RAW physical WAL tip, which unconditionally counts EVERY locally-appended entry including a local-only WAL_CHECKPOINT) is combined via target_lsn = max(committed_lsn, last_written) and THIS value -- not the real committed_lsn -- is what gets passed into ha_dispatch_repl_action_for_standby's committed_lsn parameter and printed in its log line. Once ctrl-2 wrote its checkpoint, last_written jumped to 22 and target_lsn followed, permanently outrunning the real committed_lsn by exactly one. Since a checkpoint's own consumed LSN slot is, by design, NEVER streamed (ha_stream_wal_to_standby) or applied (persist_wal_write_replicated / persist_wal_async both explicitly reject type==WAL_CHECKPOINT) by any standby, no standby's applied_lsn can EVER legitimately reach 22 -- ha_evaluate_repl_action's standby_lsn < committed_lsn check can never become false, so HA_REPL_CONVERGED is permanently unreachable for any standby once a local checkpoint has interleaved ahead of the next real committed advance. ha_stream_wal_to_standby is correctly re-invoked every heartbeat (per its own logic) but finds nothing new in [22,22] to send every single time -- a deterministic, permanent, zero-progress busy-loop, exactly matching this run's observed symptom (and ctrl-1's own applied_lsn -- correctly capped at 21 by ha_advance_applied_lsn/ha_apply_term_entry_via_library's own last-contiguously-held-LSN semantic, mirrored by ha_handle_snapshot_chunk's snapshot-install path setting applied_lsn=checkpoint_lsn never checkpoint_lsn+1 -- can never satisfy it).
The target_lsn = max(committed_lsn, last_written) fallback's OWN doc comment states its intended purpose narrowly: 'so standbys can catch up even when committed_lsn is stuck at 0 (no replicator-registered standbys to advance it)' -- a cluster-bootstrap convenience. But the code applies the max() UNCONDITIONALLY, not only when committed_lsn==0 -- broader than its own documented justification. This mismatch was latent and harmless until config-upgrade-snapshot task 5 introduced a LIVE, mid-operation local-only WAL entry (the marker-triggered checkpoint) that advances current_lsn without a corresponding real committed advance; bug-1466 already flagged the SAME persist_write_checkpoint_marker current_lsn bump as breaking raft's own log_len co-tracking invariant (affecting vote-freshness comparisons) -- this is a THIRD, distinct consequence of that same architectural gap, this time poisoning the replication-target computation instead. git blame confirms the target_lsn code (commit 06135954f, 2026-06-11) predates config-upgrade-snapshot's first commit by roughly two weeks -- a pre-existing HA replication-decision defect that config-upgrade-snapshot's task 5 merely exposed (same routing pattern as bug-1471/1472/1474/1477 before it), not something task 8.9 introduced.
Fix
FIXED (2026-07-28, narrowly-scoped TDD fix, in-scope to land here per explicit high-confidence + narrow-scope authorization -- unlike bug-1474's angle (a) and bug-1477, which were left for the HA-core spec owner): extracted the target_lsn computation out of repl_ack_decode_and_dispatch into a new small, pure, side-effect-free, directly-unit-testable function ha_compute_repl_target_lsn(committed_lsn, last_written) (declared in ha.h, defined in ha.c right after ha_process_standby_state). The fix restores the fallback to EXACTLY its own documented intent: last_written is used ONLY when committed_lsn == 0 (genuine bootstrap, nothing ever committed) -- any other case returns committed_lsn unchanged, never leaking a trailing local-only checkpoint's consumed LSN into the convergence decision. repl_ack_decode_and_dispatch's call site now reads target_lsn = ha_compute_repl_target_lsn(committed_lsn, last_written), replacing the old unconditional max(). No other function touched -- ha_evaluate_repl_action, ha_process_standby_state, ha_dispatch_repl_action_for_standby, ha_stream_wal_to_standby, persist_write_checkpoint_marker, and commit.c's majority_candidate_locked/commit_advance_locked are all confirmed correct and untouched (the bug was purely in what value got PASSED to them, not in their own logic).
TDD: four new unit tests added to src/controller/testsuite/test_ha_role.c (all four call the new pure function directly, no capnp/network scaffolding needed): test_compute_repl_target_lsn_bootstrap_fallback (committed_lsn=0, last_written=42 -> 42, confirms the intended bootstrap behavior is preserved), test_compute_repl_target_lsn_zero_zero (0,0 -> 0, fresh-cluster edge case), test_compute_repl_target_lsn_ignores_trailing_checkpoint_lsn (this bug's EXACT reproduction shape: committed_lsn=21, last_written=22 -> must return 21, not 22 -- confirmed RED under the old unconditional-max code via a temporary revert, 'Expected 21 Was 22', then GREEN after restoring the fix), test_compute_repl_target_lsn_committed_already_ahead (100,50 -> 100, normal steady-state sanity check). GREEN: full controller_unit_test suite = 1040 Tests, 0 Failures, 0 Ignored, run twice clean (ASAN_OPTIONS=detect_leaks=0, nix devshell, ninja -C src/xce.build controller/controller_unit_test). RED confirmed independently: reverting only the new function's body to the old unconditional-max logic (leaving the new tests in place) reproduced exactly 1 failure at the predicted assertion ('Expected 21 Was 22'), then re-applying the fix restored 0 failures -- proving the test genuinely exercises this bug and the fix genuinely closes it, not a tautological test.
NOT yet re-verified via a fresh live run-ha-soak.sh --config-upgrade-wal-trim run (recommended next step, mirroring bug-1474's own verification precedent, before considering config-upgrade-snapshot task 8.9 unblocked) -- this fix was validated at the unit level only within this investigation's scope.
[UPDATE 2026-07-28, kiro-impl config-upgrade-snapshot task 8.9 verification] The initial fix landed only ONE of TWO call sites duplicating this formula: ha.c repl_ack_decode_and_dispatch (fixed, via the new ha_compute_repl_target_lsn) and config_repl.c catchup_derive_target (a SEPARATE, independently-duplicated inline copy of the exact same max(committed_lsn,last_written) formula, used by ha_walack_on_ack/catchup_lane_fn -- the ACK-driven catch-up continuation path, not the heartbeat/stateReport path the first fix covered). Rerunning the live --config-upgrade-wal-trim soak after the first fix STILL FAILED ("cluster did not reconverge to 3-of-3... preserved floor segment did not actually suffice for real incremental catch-up") -- confirmed via log correlation that ctrl-2 (primary) was stuck forever at ha_dispatch_repl_action_for_standby committed_lsn=22 while catchup_lane_fn's own INFO line independently reported target=22 (the OLD, unfixed formula) even though the heartbeat path's committed_lsn logging correctly showed 21. Root-caused to catchup_derive_target (config_repl.c:2617-2627, its own comment literally said "re-derive the replication target exactly as ha.c:2771-2776 / D6" -- a deliberate, now-stale duplication). Fixed by making catchup_derive_target call the SAME ha_compute_repl_target_lsn pure function instead of re-deriving the formula inline, so the two paths can never disagree again. Applying just this second-call-site fix then caused a NEW symptom: controller_unit_test hung indefinitely (0.8%% CPU, blocked not spinning) inside test_convergence_verification.c's test_walack_behind_target_schedules_one_continuation. Isolated via a controlled revert-and-rerun (confirmed clean 1040/1040 with the OLD formula restored, confirming the SECOND fix -- not something else -- was the trigger). Root cause of the hang: three pre-existing tests in that file ("behind", "single", "respawn") hardcoded committed_lsn=4 with appended_lsn=9, deliberately exercising the OLD buggy max(4,9)=9 behavior as their expected target -- with the fix, the correct target becomes 4, so acked_lsn=5/6 (used by the later tests) no longer legitimately reads as "still behind", and the FIRST test's own hardcoded assertion fails. Because this test file's harness (cv_setup/cv_teardown) does not run teardown from a signal-safe/finally-equivalent path, a failed TEST_ASSERT's Unity longjmp skips cv_teardown() entirely, leaking that test's live 8-thread commit_engine still bound to the soon-to-be-memset g_cv global -- the NEXT test's cv_setup() memset-ing g_cv out from under those still-running threads is what actually produced the observed permanent hang, not a defect in the production fix itself. Fixed by updating all three tests' committed_lsn from 4 to 9 (matching appended_lsn, so target=9 under EITHER formula) -- this decouples them from the bug-1478 edge case entirely (which already has dedicated, correct pure-function coverage in test_ha_role.c), preserving each test's TRUE original intent (continuation scheduling / single-flight / respawn-after-complete mechanics) with a minimal, surgical change. Stale docstrings referencing the old "max(committed_lsn, last_written)" formula were also corrected to reference ha_compute_repl_target_lsn. GREEN: full controller_unit_test suite = 1040 Tests, 0 Failures, run clean 2x after all fixes (plus the isolation-revert run confirming the pre-fix baseline was also clean). Live-verified: run-ha-soak.sh --config-upgrade-wal-trim now reports a clean PASS (18 jobs, zero job loss, ctrl-1 reconverged 3-of-3 using the preserved floor segment for real incremental catch-up) -- config-upgrade-snapshot task 8.9 unblocked.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1466, bug-1474, bug-1475, bug-1477