Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
18 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1479 | fixed | 2026-09-24 | controller | leak | 2026-07-28T13:25:00Z | 2026-09-19T09:00:00Z | 3 |
|
|
|
true |
bug-1479: Live config-upgrade-snapshot task 8.6 soak (run-ha-soak.sh --config-upgrade…
Description
Live config-upgrade-snapshot task 8.6 soak (run-ha-soak.sh --config-upgrade ~/tmp/xce-ha-configupgrade2), run on top of bug-1477's fix (checkpoint-identity, config-identity, and host-admin-status-survival assertions ALL PASSED for the first time), FAILED later in the force-elect-rotation step: 5 consecutive job-submit attempts on {ctrl-1 (primary), ctrl-2 (standby)} all got 0/3 acked ("waiting for write-quorum"), then hard-failed. Investigated the ORIGINAL hypothesis (a leaked VTASK_LIVE_BLOCK/commit-admission slot from an earlier burst of 17 commit timeouts, lsn=15..31, rc=-110) and REFUTED it: commit_resolve()/wait_release_locked() correctly release the admission slot on every timeout (confirmed 17 'async continuation deadline expired' + 17 'commit failed/timed out' WARN pairs, 1:1, in ctrl-1.log) — no admission-pool leak exists. The REAL mechanism is unrelated to admission control entirely: a permanent Raft AppendEntries log-conflict wedge on standby ctrl-2, compounded by a NACK-handling gap that never triggers catch-up.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
ROOT-CAUSED with byte-precise waldump evidence (xce_test's built-in waldump REPL command against the preserved deploy dir ~/tmp/xce-ha-configupgrade2), cross-correlated with both ctrl-1's and ctrl-2's logs line-by-line. During the soak's chaotic round-3 kill/restart churn, THREE leadership changes happened in rapid succession (term 3 -> 4 -> 5) within about 20 seconds (13:18:57-13:19:19): ctrl-2 briefly won term=3 and appended its own arm-own-term-gate NOOP, was killed before that entry could be widely replicated/committed, a short-lived term=4 leader (never directly observed in either surviving node's own become-leader lines, but proven to have existed by its WAL fingerprint) won next and wrote ITS OWN own-term NOOP at the SAME log index (14), and finally ctrl-1 won term=5 and became the lasting primary. xce_test waldump on ctrl-1's canonical (surviving, majority) WAL shows: lsn=12 CHECKPOINT term=2 (the marker-triggered checkpoint from config-upgrade-snapshot task 5, matching bug-1475/1477/1478's exact family), lsn=13 NOOP term=4 (the short-lived term-4 leader's noop), lsn=14 NOOP term=5 (ctrl-1's own winning noop). xce_test waldump on ctrl-2's own local pre-restart WAL shows: lsn=12 CHECKPOINT term=2 (agrees with ctrl-1), lsn=13 NOOP term=3 (ctrl-2's OWN aborted, never-widely-replicated noop from its deposed term-3 stint) -- a genuine, CONFLICTING entry at the SAME index (13) with a DIFFERENT term (3 vs the canonical 4). This is an entirely NORMAL, expected Raft outcome of rapid leader churn (Raft's own §5.3 log-matching property explicitly anticipates followers accumulating stale, uncommitted tail entries from deposed leaders) -- ctrl-2 correctly rejects ctrl-1's AppendEntries at prevLogIndex=13 with logOk's standard conflicting-term check (confirmed via ha_consume_append_entries's own trace: prevIdx=13 n=1 -> success=0 match=0, RAFT[p1] term=5 event=ae_reject_logmismatch, repeating for EVERY subsequent new entry ctrl-1 tries to send: prevIdx=14,15,16,...29, one per newly-committed lsn, all rejected, spanning 13:19:19 through 13:19:52). The BUG is what happens (or rather, does not happen) next, confirmed via two independent, compounding gaps: (1) ctrl-1 NEVER backs off and retries with a DECREMENTED prevLogIndex to find the point of log agreement (the standard Raft nextIndex-conflict-resolution protocol, which would let ctrl-2 truncate its conflicting lsn=13 tail and adopt the canonical term=4 entry) -- repl_broadcast (repl.c:295-380) is a pure fire-and-forget per-new-entry send with zero synchronous handling of the reply; it always advances prevIdx forward to (next-new-lsn - 1), never backward. (2) repl_ack_decode_and_dispatch's WAL-ACK reply-dispatch branch in ha.c (~6631-6642) handles a NACK (success=0) by checking ONLY for a higher observed term (raft_on_higher_term_observed, correctly added for the DIFFERENT bug-1421 stale-term scenario) and then explicitly WARN("...success=0 — ignoring") -- the comment says exactly what the code does: the rejection is discarded. Critically, repl_on_ack (repl.c:250-271) is the ONLY code path that decrements standby_conn_t.pending_ack_count, and it is called ONLY from the success branch -- so a standby that is stuck rejecting (for ANY reason) has pending_ack_count monotonically climb by 1 on every subsequent broadcast attempt with ZERO corresponding decrements, until repl_broadcast's own MAX_PENDING_ACKS=16 defensive eviction (repl.c:325-334, from ha-config-bootstrap task 7.2, commit 3969064e, 2026-05-28 -- a full month before config-upgrade-snapshot's first commit) fires and marks the standby in_catch_up=1 at 13:19:53 (confirmed: 'repl_broadcast: standby localhost:9992 evicted (pending_ack_count=17 > 16) — triggering catch-up reconnect'). This eviction correctly EXCLUDES the standby from repl_active_standby_count (hence from ha_has_write_quorum's majority calculation) -- but nothing in the codebase actually DRIVES a catch-up stream or snapshot-install for it afterward: ctrl-2's log shows ZERO entry-carrying AppendEntries for the remaining ~60 seconds of the run (13:19:53 through the log's end at 13:20:53) -- only bare heartbeat probes (prevIdx=13,n=0) that trivially succeed since a presence-only probe at the standby's own true log_len=13 doesn't hit the term-conflict check. With ctrl-2 permanently excluded (in_catch_up=1, never un-evicted) and ctrl-3 permanently failing to reconnect (separate, unrelated ECONNREFUSED loop, likely a real chaos-kill in this soak round), ctrl-1 has connected=0 standbys forever ('HA quorum lost: connected=0 required=1', logged once at the eviction and never restored) -- every subsequent job submission either times out waiting for write-quorum (0/3 acked, matching the force-elect step's failure verbatim) or gets an immediate -XCE_ERR_NOT_READY (-10013) fast-reject via ha_dispatch_gate's quorum check. This is a GENUINE, SERIOUS, and apparently PERMANENT availability defect: ANY standby whose log happens to diverge by even ONE entry at a single index during leader-election churn (an entirely normal, expected Raft occurrence, not itself a bug) can become irrecoverably stuck and permanently excluded from quorum, with no self-healing path -- in a 3-node cluster this is one bad-luck election sequence away from TOTAL, PERMANENT write-quorum loss if it hits 2 of 3 standbys. Same family as bug-1466 (open, 'WAL_CHECKPOINT breaks LSN/term co-tracking', NOT fixed at its root) and its now-3 fixed siblings (bug-1475, bug-1477, bug-1478) -- this is the family's 5th distinct member, but the trigger here (an ordinary post-election term-conflicted log tail, NOT specifically a checkpoint-boundary term-derivation issue) is narrower than bug-1466's checkpoint-specific framing and points instead at a genuinely MISSING piece of core Raft log-repair protocol (conflict-driven nextIndex backoff + NACK-triggered catch-up), which appears to have never been implemented for this replication path at all -- not a regression, a gap. git blame confirms both implicated mechanisms (repl_broadcast's pending_ack_count eviction, commit 3969064e, 2026-05-28; the NACK-handling 'ignoring' branch in repl_ack_decode_and_dispatch, present since at least bug-1421's fix, itself predating config-upgrade-snapshot) are pre-existing, core HA replication code untouched by any config-upgrade-snapshot task -- config-upgrade-snapshot's task 5 (marker-triggered per-member local checkpoint) merely adds one more entry into the WAL stream around the point where this pre-existing gap gets exercised; it does not create the gap itself.
Fix
FIXED AND VERIFIED LIVE (2026-07-29, see UPDATE 2 below for the final closing fix) -- two compounding gaps were closed after the initial investigation: (1) the standby's own reject path sent no NACK reply at all (ha_send_wal_ack_reply hardcoded success=1 unconditionally), fixed via ha_send_wal_ack_reply_ex with success=0 on reject; (2) even with NACKs flowing, ha_stream_wal_to_standby's catch-up range was hard-capped at committed_lsn, which could never advance in this exact scenario (self-referential deadlock) -- fixed via ha_compute_repl_stream_ceiling_lsn widening the ceiling to the leader's own log tail, matching raft.tla's real AppendEntries semantics. Verified via three successive live run-ha-soak.sh --config-upgrade re-runs tracking each iteration (v1/v2 still failed, v3 with both fixes together PASSED end-to-end: 3/3 jobs acked on both force-elect rounds, zero materialized-state or config divergence). 1053/1053 controller_unit_test PASS. Full investigation history preserved below for context (including the earlier partial-fix iterations and root-causing).
[Original investigation, 2026-07-28]
NOT FIXED -- explicitly out of scope for a single safe pass, per this investigation's own conclusion (matches kiro-debug's guidance against forcing a fix when the true defect requires a protocol design decision, not a narrow patch). At least TWO separate, non-trivial design questions need to be resolved by whoever owns ha.c/repl.c's replication-catch-up protocol before a fix can be written safely: (a) should an ae_reject_logmismatch NACK immediately trigger repl_start_catchup (moving the standby to catch-up mode on the FIRST conflict, rather than silently accumulating up to 16 failed sends first)? This is the more targeted, root-cause-adjacent fix. (b) independently, once a standby IS marked in_catch_up (whether via this path or the existing MAX_PENDING_ACKS eviction), what mechanism is SUPPOSED to actually drive its catch-up (stream from join_applied_lsn, or snapshot-install if too far behind) and clear in_catch_up via repl_complete_catchup? Evidence here shows NOTHING does, for at least this eviction path -- config_repl.c's ACK-driven catchup_lane_fn/ha_walack_on_ack continuation (the only other repl_complete_catchup call site found) is itself ACK-driven, so it can never engage for a standby that never sends a genuine ACK in the first place; whether IT (or a periodic sweep independent of ACKs) is supposed to notice an in_catch_up standby and proactively start streaming it needs its own trace by the owning team. A correct fix likely needs both: NACK-triggered catch-up entry (closing gap (a)) AND a proactive (not purely ACK-reactive) catch-up driver that streams from join_applied_lsn or falls back to snapshot-install for any in_catch_up standby (closing gap (b)) -- implementing only one without the other would leave the defect only partially closed. Recommend routing to whoever owns ha.c/repl.c's HA replication/catch-up protocol (the same 'needs-owner' bucket as bug-1474's still-open fix angle (a) and bug-1472-remainder) for a proper design pass, THEN a TDD fix + live re-verification via run-ha-soak.sh --config-upgrade on a fresh deploy dir. Preserved evidence: ~/tmp/xce-ha-configupgrade2 (do not delete) -- xce_test waldump <data_dir> (via the xce_test REPL: printf 'waldump <path>\nquit\n' | ASAN_OPTIONS=detect_leaks=0 timeout 60 <xce.build>/controller/xce_test) reproduces the byte-precise lsn=13 term-3-vs-term-4 conflict evidence directly from the preserved WAL segments at any time.
UPDATE (2026-07-29, this session): a fix WAS committed (uncommitted in the 'config' worktree at investigation time) implementing exactly gap (a) from this entry's own recommendation -- ha_raft_on_repl_nack() (ha.c/ha.h) routes a WAL-ACK-reply NACK into the raft library's nextIndex backoff and returns a corrected resume LSN; repl_ack_decode_and_dispatch's NACK branch calls it and immediately repl_start_catchup() with the corrected LSN; repl_get_catchup_join_lsn() (repl.c/repl.h) lets the heartbeat-driven ha_dispatch_repl_action_for_standby use the corrected position; repl_complete_catchup() now also resets pending_ack_count. 1048/1048 controller_unit_test PASS. HOWEVER: a fresh live-soak re-verification (run-ha-soak.sh --config-upgrade, deploy preserved at ~/tmp/xce_ha_bug1479_verify, NOT deleted) reproduced the ORIGINAL symptom verbatim -- force-elect ctrl-2 round: 5/5 submit_jobs attempts on {ctrl-1,ctrl-2} got 0/3 acked, hard FAIL. Root-caused: the fix's NACK-routing branch in repl_ack_decode_and_dispatch (ha.c ~6680-6720) is CORRECT but UNREACHABLE in production, because the standby's OWN reject path never sends a NACK (or any reply) in the first place. ha_apply_wal_catchup_entry has three reject paths (ha.c ~3230-3232 via ha_apply_term_entry_via_library for the term-carrying single/batch case -- the one actually hit live; ~3320-3321 for a batch sub-entry apply failure; ~3373-3374 for the legacy single-entry path) that all explicitly do 'no ACK' on a logOk reject/forward-gap. ha_send_wal_ack_reply (ha.c:3084-3106) is the ONLY function anywhere in the controller that ever constructs a WalAckReply, and it hardcodes success=1 unconditionally -- there is NO production code path that ever sends success=0. Live evidence: ctrl-2's own log shows repeating RAFT[p1] event=ae_reject_logmismatch / ha_consume_append_entries success=0 match=0 at growing prevIdx (40,41,...54) while stuck at match=20 forever; ctrl-1's log has ZERO 'WAL NACK from standby' or ha_raft_on_repl_nack lines the entire time -- the primary never learns of the rejection at all, so it never backs off nextIndex or re-arms catch-up, and committed_lsn stays frozen (the only alive standby can never re-sync, and the heartbeat-driven catch-up's own target is gated on the same frozen committed_lsn). STILL NOT FULLY FIXED. New follow-up spec .kiro/specs/raft-adapter-conformance/ created to (1) fix this concrete missing-NACK-reply defect at the three reject sites and (2) perform a broader raft.tla action-by-action conformance pass over the controller's Raft adapter (ha.c/repl.c/config_repl.c) -- distinct from raft-sim-conformance, which is scoped to src/lib/raft/raft_sim.c (library-only, in-process) and explicitly does not cover this adapter code.
UPDATE 2 (2026-07-29, this session): fully FIXED and verified live end-to-end. Two more compounding gaps found and closed after the NACK-send fix (UPDATE 1) landed: (1) ha_apply_term_entry_via_library's logOk-reject path (ha.c, the single live-reachable reject site) sent NO reply at all on rejection -- ha_send_wal_ack_reply hardcoded success=1 unconditionally, so the primary-side ha_raft_on_repl_nack (correct since UPDATE 1) was unreachable dead code. Fixed by adding ha_send_wal_ack_reply_ex(ha, ep, c, lsn, success) and calling it with success=0 on reject, plus a new ha_wal_ack_send_hook (ha.h, XCE_TESTING) so a unit test can observe the NACK without a live endpoint (test_bug1479_ae_reject_sends_nack_not_silence, test_config_repl.c). Confirmed live in a 2nd soak run: ctrl-1's log showed real 'WAL NACK from standby ... success=0 -- marking catch-up' lines for the first time. (2) Even with (1) working, that 2nd soak run STILL failed at the identical force-elect ctrl-2 point: ha_stream_wal_to_standby's catch-up range was hard-bounded at committed_lsn, and committed_lsn can never advance in this exact scenario because advancing it requires an ACK from the very standby that's stuck catching up (2-of-3 alive, this standby is the only path to a new majority) -- a self-referential deadlock. Fixed by adding a new pure function ha_compute_repl_stream_ceiling_lsn(committed_lsn, last_written, checkpoint_lsn) (ha.c/ha.h), wired into ha_dispatch_repl_action_for_standby to widen the STREAM_WAL ceiling (and the ha_evaluate_repl_action convergence check that gates it) to the leader's own log tail instead of just committed_lsn -- matching raft.tla's real AppendEntries semantics (leaderCommit is advisory to the follower, never a cap on what gets sent) -- while still guarding against bug-1478's old trap (never widening to a trailing local-only checkpoint's own unstreamable LSN slot). 4 new pure-function tests in test_ha_role.c (test_compute_repl_stream_ceiling_lsn_*), RED-confirmed against a stubbed committed_lsn-only version before the real fix was restored. Full verification: 1053/1053 controller_unit_test PASS twice clean. THREE successive live run-ha-soak.sh --config-upgrade re-runs tracked each fix: v1 (NACK-send only) still FAILED at force-elect ctrl-2 (0/3 acked); v2 confirmed NACKs now reaching the primary but still FAILED (frozen committed_lsn=14, ha_stream_wal_to_standby target never advanced); v3 (both fixes together) PASSED end-to-end: all 3 force-elect rounds succeeded (ctrl-2: 3/3 jobs acked, term=6; ctrl-3: 3/3 jobs acked, term=7), final verdict 'PASS (config-upgrade): 12 jobs, zero materialized-state divergence, zero configuration divergence, host admin-status confirmed across all members'. Evidence preserved at ~/tmp/xce_ha_bug1479_verify, ~/tmp/xce_ha_bug1479_verify_v2, ~/tmp/xce_ha_bug1479_verify_v3 (all not deleted). config-upgrade-snapshot task 8.6 unblocked. The raft-adapter-conformance spec created mid-investigation had its concrete seed defect (missing NACK reply) fixed here; the broader raft.tla action-by-action conformance audit it also scopes remains valid, separate future work.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1466, bug-1471, bug-1472, bug-1474, bug-1475, bug-1477, bug-1478, bug-1421