Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
19 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | fix_commits | legacy | |||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1474 | fixed | 2026-09-27 | c3a17b19: root cause of angle (a) -- standby_endpoints loops bounded by count skipped entries past zarray_del holes (lost heartbeats, spurious re-dial); now scan caps |
controller | unknown | 2026-07-25T20:50:00Z | 2026-09-19T09:00:00Z | 3 |
|
|
|
|
true |
bug-1474: config-upgrade-snapshot task 8.9's new live --config-upgrade-wal-trim soak (run-ha-soak.sh…
Description
config-upgrade-snapshot task 8.9's new live --config-upgrade-wal-trim soak (run-ha-soak.sh, run_config_upgrade_wal_trim_round): the marker-triggered checkpoint's WAL-segment trim (persist_checkpoint_marker_triggered -> trim_wal_segments -> repl_trim_lsn, bug-1147's guard) strands a genuinely mid-catch-up standby's still-needed WAL segment -- but NOT because bug-1147's guard logic itself is wrong. Reproduced identically on 2/2 independent live 3-node runs: a killed, still-down standby's disconnect-guard (last_disconnected_lsn) is silently DESTROYED before the trim ever runs, because an UNRELATED, already-connected peer's redundant reconnect attempt reuses the dead standby's just-freed replicator slot.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
Confirmed via direct log evidence on 2 independent live-soak runs (deploy dirs /home/claude/tmp/xce-ha-waltrim and /home/claude/tmp/xce-ha-waltrim2, both --config-upgrade-wal-trim). Sequence, identical both times: (1) An election promotes surviving VOTE endpoints to standby replication channels (raft_cb_become_leader: 'promoted vote ep[i] -> standby channel', repl_add_standby registers them at replicator slots 0/1 -- e.g. run 1: ep[0]=ctrl-1 -> slot 0, ep[1]=ctrl-3 -> slot 1). (2) The test kills a non-primary TARGET standby (ctrl-1) while it is caught up to a known WAL floor. The primary's err_handler (vote_ep_err_handler, correctly wired) detects the RDHUP and calls repl_remove_standby, freeing slot 0 and setting last_disconnected_lsn = ctrl-1's real acked_lsn (confirmed matching the test's own captured floor segment, e.g. seg-5 in run 1, seg-c in run 2) -- bug-1147's guard is set CORRECTLY at this point. (3) Seconds later (matching the ~9s default heartbeat interval), primary_heartbeat_fn's periodic 'reconnect to any peer missing from standby_endpoints' sweep (ha.c ~9219) runs. It correctly identifies ctrl-1 (peer[0]) as missing (reconnect attempt fails with ECONNREFUSED, confirmed in both runs' logs: 'HA: connect to peer[0] (localhost:9991) failed: -111') -- but it ALSO, incorrectly, identifies ctrl-3 (peer[2]) as missing, even though ctrl-3 has been continuously connected the whole time via its slot-1 promoted-vote-channel connection (confirmed acking WAL entries without any gap throughout, via ctrl-2.log's 'commit: ack standby_idx=1' sequence). This triggers a redundant outbound reconnect to ctrl-3 (on_standby_connected fires: 'HA: outbound standby connection to peer[2] established' + 'HA: repl registered late-joiner standby peer[2] (slot 0, localhost:9993)'). (4) repl_add_standby's slot allocation picks the FIRST EMPTY slot by lowest index (repl.c ~85-110) with no regard for whether that slot still holds an unexpired bug-1147 disconnect guard for a DIFFERENT peer -- it picks slot 0 (ctrl-1's just-freed, guard-holding slot, since it is lower-indexed than slot 1) and unconditionally memsets it to zero, WIPING OUT ctrl-1's last_disconnected_lsn guard entirely. (5) When the marker-triggered checkpoint's trim runs shortly after, repl_trim_lsn (repl.c ~650-683) sees slot 0 now occupied by a live, fully-caught-up ctrl-3 duplicate connection (contributing no meaningful floor-lowering) and slot 1 also fully caught up -- with NO surviving trace that ctrl-1 (still down, still needing its old floor) was ever disconnected. floor collapses to the active segment itself, and every segment below it -- including the one ctrl-1 genuinely still needs -- is deleted. This is NOT a bug in bug-1147's guard logic (repl_trim_lsn/trim_wal_segments are working exactly as designed against the replicator state they are handed); it is an upstream defect in how a duplicate/late-joiner reconnect for an ALREADY-connected peer gets allocated a replicator slot. Root cause of WHY the 'peer already has a standby endpoint' check (ha.c ~9225-9249) fails to recognize ctrl-3's existing promoted-vote-channel connection is not fully pinned to a single line -- standby_ep_ctx_t is deliberately layout-compatible with vote_ep_ctx_t (confirmed by direct struct read, ha.c:8590-8603, both start with ha/c/ctx/peer_idx in the same order, so the blind cast the check performs is NOT a type-confusion bug) -- but vote_ep_ctx_t.peer_idx appears to be populated under inconsistent conventions across its several creation call sites in ha.c (e.g. line 542 sets peer_idx=votes_idx [relative, votes-array index] while a comment at line ~6120 asserts 'vctx->peer_idx is the ABSOLUTE cluster index' for a different call site) -- a mismatch between the relative votes_idx_for_i the check computes and whichever convention the SPECIFIC promoted vote_ep_ctx_t for ectx->votes[i].ep was actually built under would make the sctx->peer_idx == votes_idx_for_i comparison fail deterministically for promoted-vote-channel entries, matching the 100% reproduction rate observed. This needs further tracing by whoever owns ha.c's HA connection-management code to identify the exact mismatched call site.
Fix
[UPDATE 2026-08-16] Angle-a FIXED (TDD RED->GREEN, root-caused precisely -- not the original peer_idx-convention hypothesis, a different, concrete bug in an adjacent function). Traced every vote_ep_ctx_t.peer_idx assignment site as recommended: raft_cb_send_vote_request (real vote, ha.c) correctly sets peer_idx=votes_idx (RELATIVE) and is the ONLY function that stores its endpoint into ectx->votes[] (the array raft_cb_become_leader's promotion loop reads) -- confirmed by full read, no ectx->votes[].ep assignment exists anywhere else. raft_cb_send_pre_vote_request deliberately sets peer_idx=ABSOLUTE (correct, documented, needed by the PreVote tally) but NEVER stores into ectx->votes[] either -- so PreVote endpoints structurally can never reach promotion, ruling out the original hypothesis (PreVote-vs-real-vote convention mismatch reaching the reconnect-sweep check). ha->my_priority is set once at init and never reassigned, ruling out a self-idx-drift hypothesis too. The reconnect-sweep check itself (primary_heartbeat_fn, ~line 9787) already correctly discriminates promoted-vote-ep (relative) vs standby-ep (absolute) via sctx->ctx != NULL, unchanged since 2026-06-01 (predates this bug's 2026-07-25 discovery, confirmed via git log -L) -- it was never the bug. The REAL bug: on_standby_connected's own same-peer duplicate-eviction check (the function's OWN doc comment already named this exact hazard -- 'two live outbound state-push channels to one peer') compared cur_sctx->peer_idx == sctx->peer_idx DIRECTLY, with no relative/absolute discrimination at all -- unlike the adjacent, correct reconnect-sweep check ~500 lines away. sctx (the newly-connecting standby) always carries the ABSOLUTE index; if the existing entry for the same physical peer is a promoted vote ep (RELATIVE), the raw comparison can never match, so the stale promoted-vote channel is never evicted -- both channels survive, reproducing the buglog's exact live evidence (heartbeats/WAL fan out to a stale duplicate; the genuinely-connected promoted channel's own bug-1147 disconnect-guard slot gets silently reused by the SECOND, spurious connect attempt for the SAME live peer). Fix: apply the identical votes_idx_for_i conversion the reconnect-sweep check already uses, in on_standby_connected's eviction loop, discriminating on cur_sctx->ctx != NULL exactly as that check does. Regression test (test_ha_liveness.c's test_reconnect_resolves_dual_channel_promoted_vote_ep): stale entry is a genuine promoted-vote-ep test double (vote_ep_ctx_build_for_test, relative peer_idx) for the same physical peer a fresh standby reconnect targets (absolute peer_idx) -- RED under the pre-fix code (both channels survive, zero evictions -- the existing sibling test, test_reconnect_resolves_dual_channel_same_peer, only ever exercised the standby-vs-standby case, both absolute, which is why this mixed case slipped through originally), GREEN under the fix (exactly one channel survives, one eviction). Full controller_unit_test suite: 1100 tests, 2 pre-existing unrelated failures (bug-1561), no regressions. NOT independently re-verified via a live multi-node chaotic soak (the class of test that originally discovered this bug) -- the unit test directly constructs and proves the exact mechanism the live soak's log evidence showed, but a live re-run of config-upgrade-snapshot task 8.9's --config-upgrade-wal-trim scenario would be the strongest additional confirmation if this is revisited.
[Original entry -- angle-b, already fixed 2026-07-26]
NOT FIXED -- explicitly out of config-upgrade-snapshot task 8.9's boundary (task 8.9 is a regression test for the pre-existing, unmodified bug-1147 guard under the NEW marker-triggered trigger path, not a fix for this separately-discovered HA connection-management defect). Two independent, complementary fix angles for whoever owns ha.c/repl.c's HA connection layer: (a) fix the 'peer already has a standby endpoint' check (ha.c ~9225-9249) so it correctly recognizes an existing promoted-vote-channel connection for a peer, eliminating the spurious duplicate/late-joiner reconnect entirely (the more targeted fix, addresses the root cause) -- likely requires auditing every vote_ep_ctx_t.peer_idx assignment site (ha.c lines 542, 697, 6956, and whichever one specifically constructs ectx->votes[i].ep's ep->data) against the relative-votes_idx convention votes_idx_for_i expects; (b) independently harden repl_add_standby's slot allocation (repl.c) to never silently clobber a slot whose last_disconnected_lsn guard has not yet expired per snap_interval_s -- e.g. skip a guard-holding slot in the first-empty-slot scan (falling through to a genuinely empty slot, or failing loud if none exists) -- this is a defense-in-depth hardening of bug-1147's own retention mechanism against ANY future caller that might reuse a slot prematurely, not just this specific duplicate-connection path. Recommend (b) regardless of whether (a) is fixed, since it directly protects the invariant Requirement 7.4 cares about. The run_config_upgrade_wal_trim_round test infrastructure (run-ha-soak.sh, config-upgrade-snapshot task 8.9) is CORRECT and reproducibly catches this defect via direct on-disk WAL segment inspection -- re-run run-ha-soak.sh --config-upgrade-wal-trim after either fix lands to confirm a clean PASS; task 8.9 itself should stay incomplete/blocked on this bug per its own scope boundary (mirrors task 8.6's bug-1471/1472 precedent).
[UPDATE 2026-07-26] FIXED (fix angle (b), the repl.c defense-in-depth hardening): repl_add_standby's slot-allocation scan (repl.c) now does a two-pass search instead of first-empty-slot-by-index. Pass 1 prefers a genuinely virgin slot (ep==NULL && last_disconnected_lsn==0). Pass 2 (fallback, only when no virgin slot exists) reuses the guard-holding slot (ep==NULL && last_disconnected_lsn!=0) with the OLDEST last_disconnect_time -- a deliberate policy: that guard is closest to its own natural repl_expire_disconnect_slots expiry, so reusing it discards the LEAST remaining WAL-retention protection of any guard-holding slot. With MAX_STANDBYS==3, the exhausted case (both non-primary slots simultaneously guard-holding, no virgin slot free) is handled by this same fallback rather than -ENOSPC -- refusing a legitimate new/reconnecting peer would be worse for cluster availability than accepting an early guard loss in an already-degraded state; the pre-existing -ENOSPC path (more distinct live peers than MAX_STANDBYS slots) is untouched, reserved for that genuinely-impossible case only.
TDD: two new unit tests added to src/controller/testsuite/test_repl_unit.c -- test_repl_add_standby_prefers_virgin_slot_over_guard (RED under the old code: asserted a new unrelated peer must not land in a lower-indexed guard-holding slot when a virgin slot exists, and that the guard's last_disconnected_lsn must survive untouched) and test_repl_add_standby_exhausted_guards_prefers_oldest_disconnect (RED under the old code: asserted the exhausted-fallback deliberately picks the OLDER-disconnected guard slot, not merely the lowest-indexed one -- the test deliberately makes 'oldest-disconnected' and 'lowest-indexed' point at DIFFERENT slots so an accidental first-guard-by-index fallthrough would still fail it). Both new tests failed pre-fix, pass post-fix.
GREEN: full controller_unit_test suite = 1030 tests (1028 known-good baseline + 2 new), 0 failures, no regressions (ASAN_OPTIONS=detect_leaks=0, nix develop build via ninja -C src/xce.build controller/controller_unit_test).
repl_remove_standby, repl_expire_disconnect_slots, repl_trim_lsn, and trim_wal_segments were NOT modified -- all confirmed correct and untouched, per this fix's scope.
This fix does NOT address fix angle (a) -- the separate ha.c heartbeat-reconnect-sweep defect (vote_ep_ctx_t.peer_idx convention mismatch hypothesis, ha.c ~9219-9254 / ~8687-8708) that triggers the spurious duplicate reconnect in the first place -- that remains a separate, not-fully-root-caused issue, out of this fix's scope. Recommend re-running run-ha-soak.sh --config-upgrade-wal-trim (config-upgrade-snapshot task 8.9) to confirm this defense-in-depth closes the observed WAL-segment-loss symptom even without (a) landing.
[UPDATE 2026-07-26, post-review remediation] Independent review REJECTED the initial fix on one finding: the Pass 2 exhausted-fallback branch silently destroyed the reused slot's last_disconnected_lsn/last_disconnect_time via memset with zero logging, undercutting the fix's own stated goal of making the previously-silent WAL-retention-guard destruction visible/bounded -- this file already has an established convention for exactly this class of event (repl_broadcast's WARN("repl_broadcast: no standbys registered...") and WARN("repl_broadcast: standby %s:%u evicted...")). Remediated: repl_add_standby now tracks whether Pass 1 found a virgin slot (used_fallback_slot) and, only on the genuine Pass 2 fallback path, logs WARN("repl_add_standby: reusing guard-holding slot %d (host=%s:%u, last_disconnected_lsn=%lu) -- no virgin slot available, WAL-retention protection discarded", ...) immediately before the memset, capturing the discarded slot's host/port/last_disconnected_lsn while they're still readable. No new test required (a log line has no observable return-value/state effect for a test to assert on) -- verified instead by confirming the WARN string is present in the built binary (strings controller_unit_test | grep 'reusing guard-holding slot') and that both existing bug-1474 tests (test_repl_add_standby_prefers_virgin_slot_over_guard, test_repl_add_standby_exhausted_guards_prefers_oldest_disconnect -- the latter specifically exercises the Pass 2 path this WARN sits on) still pass unchanged. GREEN: full controller_unit_test suite = 1033 tests (1030 baseline + bug-1472's 3 concurrently-landed tests), 0 failures. NOTE: a transient stale-object-file build artifact was observed once in this shared build directory during verification (a link picked up an out-of-date worker.c.o from a moment when a concurrent sibling agent's build was in flight, showing 2 spurious failures unrelated to this change) -- resolved by touching the affected sources to force recompilation; not a defect in this fix, noted here in case it recurs for a future concurrent multi-agent session in this worktree.
[UPDATE 2026-09-16, live re-observation during controller-status task 8.6's live verification, NOT independently fixed here -- out of that task's boundary] Directly observed fix angle (a)'s symptom again, live, on a fresh 3-node cluster (/tmp/xce-deploy-ctrlstatus86): killed standby ctrl-2 (localhost:9992) only; ctrl-3 (localhost:9993) was never killed and its own log shows zero RDHUP/disconnect/reconnect lines for the entire session. ~2s after the ctrl-2 kill, ctrl-1's log shows ha_connect_finish_peer: HA: peer[2] (localhost) not connected -- initiating outbound connect for ctrl-3 (peer[2]) -- a peer that was continuously, genuinely connected the whole time -- followed by on_standby_connected: HA: repl registered late-joiner standby peer[2] (slot 2, localhost:9993), consuming what would otherwise have been a virgin slot. Zero repl_remove_standby/evict/duplicate log lines appear anywhere in the entire run, so the angle-a on_standby_connected eviction-check fix (2026-08-16) either did not fire for this specific reconnect path or the original promoted-vote-channel slot for ctrl-3 was left live alongside the new one. Direct downstream effect: when the genuinely-killed ctrl-2 reconnected ~1 minute later, repl_add_standby's Pass 1 found 'no virgin slot available' (logged) and fell back to Pass 2, reusing ctrl-2's OWN prior guard-holding slot rather than landing in a fresh slot -- i.e. this spurious ctrl-3 duplicate-connect is what consumed the third MAX_STANDBYS slot in this run. Not investigated further (root-causing ha.c's reconnect-sweep peer-liveness check is explicitly out of controller-status task 8.6's boundary, which owns only ctrlstatus_fill's read-side status derivation, not connection-management) -- reported as a CONCERN alongside task 8.6's own fix (bug-1819) rather than silently worked around. task 8.6's ctrlstatus_fill fix tolerates this correctly either way (single-slot reuse or a genuine duplicate-live-slot pair both resolve to the right status), so it did not block that task, but fix angle (a) itself remains open.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1147, bug-1471, bug-1472, bug-1819