Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
13 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1477 | fixed | 2026-09-24 | controller | unknown | 2026-07-28T08:25:00Z | 2026-09-19T09:00:00Z | 2 |
|
|
|
true |
bug-1477: Live config-upgrade-snapshot task 8.6 soak (run-ha-soak.sh --config-upgrade…
Description
Live config-upgrade-snapshot task 8.6 soak (run-ha-soak.sh --config-upgrade ~/tmp/xce-ha-configupgrade), run on top of bug-1475's already-fixed and already-built-in commit ce8df990 (confirmed via binary mtime 08:16:27 postdating ce8df990 04:48:53Z and b08e392e 08:13:54Z, plus strings matching TERM_INDEX_UNSET/persist_durable_floor), FAILED: "ctrl-3 has no on-disk checkpoint after the upgrade". ctrl-1/ctrl-2 both have data/snap/snap-000000000000000b.capnp (51161 bytes, LSN 11 hex = the WAL_CONFIG_MARKER's own lsn); ctrl-3's data/snap/ is completely empty. ctrl-3's log shows a PERMANENT loop (repeats every ~3s/heartbeat for the entire ~3-minute run, never progressing): a bare heartbeat probe at prevIdx=11 succeeds (ae_already_have, match=11), but every walCatchupEntry at prevIdx=12/13/14 is rejected forever with ae_reject_logmismatch. Unlike bug-1475's own signature, this is NOT a transient race that self-heals once durable_lsn catches up -- it never recovers for the rest of the run.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
ROOT-CAUSED with byte/log-precise evidence, NOT config-upgrade-snapshot's own defect. ctrl-3 (my_priority=2, the highest-priority-number/most-deferred candidate in the 3-node cluster, per myapp_run's startup-stagger nanosleep) never ran its own raft candidacy for the ENTIRE test: zero occurrences of raft_cb_become_follower/raft_cb_become_leader/"-> CTRL_STANDBY" anywhere in ctrl-3.log. Instead, by the time its stagger delay elapsed, ctrl-1 (priority 0) had already voted/established itself, so myapp_run's own "leader already known" check (myapp.c ~343-390: voted_for_name set, or raft.term>0, or primary_ctrl_fd>=0) was true, and ctrl-3 took the goto skip_election fast path, entirely bypassing ha_start_election_async and therefore every raft-library election callback for the rest of its process lifetime (it only ever GRANTS votes to others afterward -- confirmed via repeated vote_granted/pre_vote_granted RAFT[p2] events with no corresponding become_follower/become_leader). The skip_election fast path (myapp.c ~415-425, bug-849, commit 20e59d0b, 2026-06-21 -- three-plus weeks before config-upgrade-snapshot's first commit 9e68335a) makes exactly ONE, correctly-guarded attempt at cluster_init_phase2(&c) (checks prc, only sets c.ha.phase2_done=1 on success) -- and at 08:18:26,930, that ONE attempt fired before c->myconf->cluster_config was populated (ctrl-3 had no local config file and had not yet received/activated a push), so cluster_init_phase2 returned -EINVAL immediately (cluster.c:1349-1358, before doing ANY of its real work) and the WARN 'cluster_init_phase2 on standby rejoin failed (rc=-22) -- WAL not recovered; falling back to election-path recovery' fired. phase2_done correctly stayed 0 -- but there is NO OTHER retry mechanism anywhere in the codebase for a node that (a) took the skip_election fast path and (b) never subsequently runs its own contested election: the only other three call sites that could retry (raft_cb_become_follower ha.c:2000-2012, raft_cb_become_leader ha.c:1772-1783, ha_elect_evaluate_and_signal's CTRL_PRIMARY/CTRL_STANDBY branches ha.c:5609-5629/5699-5710) never fire for ctrl-3 because it never engages the raft callback machinery at all. (Separately confirmed those three call sites ALSO unconditionally set ha->phase2_done=1 after calling cluster_init_phase2(c) with NO check of its return value -- unlike myapp.c's own careful guard -- a second, adjacent defect of the same class that did not trigger THIS failure only because ctrl-3 never reached them, but would identically strand any node whose FIRST raft-driven role transition happens to race ahead of config load.) Consequence, confirmed via grep: ctrl-3.log has ZERO occurrences of "commit engine wired", "persist_recover", "xce_config_init", or "create_lost_and_found" for the entire run (vs. multiple occurrences on ctrl-1/ctrl-2, once per boot/restart/re-election) -- cluster_init_phase2 (and therefore commit_engine_init, cluster.c:1484-1496) NEVER completed on ctrl-3. persist.c:796-799 documents on_durable_advance (the sole wake hook driving persist_apply_pump_run) as registered ONLY inside commit_engine_init -- so without it, persist_apply_pump_run/apply_committed_entry NEVER runs on ctrl-3, for ANY committed LSN, for the whole process lifetime. This means persist_checkpoint_marker_triggered (config-upgrade-snapshot task 5's own WAL_CONFIG_MARKER-triggered checkpoint, apply_committed_entry's WAL_CONFIG_MARKER case) never fires on ctrl-3 -- explaining precisely why its data/snap/ is empty while ctrl-1/ctrl-2 (whose commit engines DID wire) both hold the identical snap-000000000000000b.capnp. ctrl-3's config nonetheless activates cosmetically at 08:18:48,572 via a SEPARATE, independent code path (handle_repl_config_push -> load_primary_config -> config_activate_committed, an RPC push handler that does not touch the commit engine or apply pump at all) -- masking the underlying defect and making the soak's materialized-state/checkpoint assertions the first thing ever to surface it. bug-1475's three-part fix (config_repl.c's tracker-advance-across-WAL_CHECKPOINT, ha.c's JIT persist_durable_floor resync of ha->raft.log_len, persist.c's TERM_INDEX_UNSET sentinel) is necessary and correctly targets its own class of bug (a receiving standby's log_len bookkeeping merely LAGGING behind a checkpoint it DID independently write) but structurally cannot cover this failure: bug-1475's own fix explicitly documents and depends on the invariant 'a standby that independently applied the SAME committed WAL_CONFIG_MARKER writes its OWN local checkpoint at the identical LSN (deterministic per-member trigger)' -- ctrl-3 violates that precondition outright (it never writes the checkpoint AT ALL, not merely late), so the primary's catch-up stream's prev_log_index=12 (the checkpoint's own lsn, per bug-1475's now-correct sender tracking) checks against an entry ctrl-3 genuinely, permanently does not have. ae_reject_logmismatch is therefore a CORRECT rejection of ctrl-3's true log state, not a stale/racy one -- and since a WAL_CHECKPOINT entry is local-only/non-replicated BY DESIGN (persist_write_checkpoint_marker's own doc comment; persist_wal_write_replicated/persist_wal_async both explicitly reject type==WAL_CHECKPOINT), ctrl-3 can NEVER receive lsn=12 over the wire, and can never independently write it either (cluster_init_phase2/commit_engine_init never retries) -- a genuinely permanent, not merely time-bounded, wedge, exactly matching the soak's observed behavior (unlike bug-1475's transient race, this repeated with zero progress for the entire ~3-minute run). git blame confirms both implicated pre-existing defects (myapp.c's skip_election fast path, commit 20e59d0b bug-849 2026-06-21; ha.c's raft-callback phase2_done latch structure, commit 8f0d2d55 task 6.2 raft-library) predate config-upgrade-snapshot's first commit (9e68335a) by roughly a month -- this is a pre-existing HA bootstrap/election gap, NOT something config-upgrade-snapshot's task 5 introduced. Task 5 merely exposed it by creating a new cross-member structural dependency (every member independently checkpointing at a synchronized marker-boundary LSN) that a node whose commit engine/apply pump never wired silently violates. Before config-upgrade-snapshot, the same underlying gap would have left such a node's materialized state (jobmgr/qmgr/workermgr) permanently stale/un-applied with a much less visible failure signature (no crash, no wedge -- just silently wrong query results if that node were ever promoted), plausibly unnoticed by prior soaks that do not assert per-member materialized-state identity.
Fix
FIXED (2026-07-28). Root cause: myapp_run's skip_election passive path, and 3 raft-callback call sites (raft_cb_become_leader, raft_cb_become_follower, ha_elect_evaluate_and_signal x2), all latched ha->phase2_done=1 UNCONDITIONALLY right after calling cluster_init_phase2 (or its ha_phase2_hook test seam), discarding its return value entirely -- so a single failed attempt (e.g. racing config availability, cluster_init_phase2 returning -EINVAL because cluster_config wasn't loaded yet) permanently stranded the member: commit_engine_init never ran, so apply_committed_entry never applied ANY committed WAL entry for the rest of that process's life, and -- for a node on the passive skip_election path specifically -- NOTHING else would ever retry it (the other 3 call sites only fire via this node's own raft-library state transitions, which never happen for a passive standby whose primary keeps heartbeating normally).
Fix: extracted a shared, idempotent retry mechanism -- ha_phase2_retry_fn (the blocking retry itself, capturing cluster_init_phase2/ha_phase2_hook's real return value, latching phase2_done ONLY on rc==0) plus ha_schedule_phase2_retry (a delaymgr-scheduled, SELF-RESCHEDULING trigger at heartbeat_interval_ms cadence -- unlike upgrade_materialization_retry/config_activate_retry, which are re-triggered externally by recurring commit-advance events, nothing recurring would otherwise ever re-trigger this for a passive skip_election node, so this one self-arms). Wired into all 5 call sites: myapp.c's skip_election block now schedules a retry instead of just WARNing on failure; raft_cb_become_follower and ha_elect_evaluate_and_signal's STANDBY branch likewise (safe to stay passive while retrying); raft_cb_become_leader and ha_elect_evaluate_and_signal's PRIMARY branch additionally REFUSE promotion on failure in raft_cb_become_leader specifically (mirroring the sibling upgrade_materialization_fenced refusal in the same function -- a promoted primary with an unwired commit engine would be far worse than a delayed promotion); the legacy ha_elect_evaluate_and_signal PRIMARY branch does NOT refuse (its ctx->done semaphore is already posted to the caller before this point in that function, so refusing would leave an inconsistent signal -- it proceeds to CTRL_PRIMARY and lets the self-rescheduling retry wire the commit engine shortly after instead).
TDD: 4 new tests in test_config_repl.c (test_ha_phase2_retry_fn_noop_when_already_done, test_ha_phase2_retry_fn_does_not_latch_on_failure_and_succeeds_on_retry -- the core regression, RED confirmed by temporarily reverting the rc-gating back to an unconditional latch and observing the exact predicted failure -- test_schedule_phase2_retry_noop_when_already_done, test_schedule_phase2_retry_noop_when_delaymgr_not_running). GREEN: full controller_unit_test suite = 1044 tests, 0 failures, clean across multiple runs (including the RED/GREEN round-trip).
Live-verified: re-ran run-ha-soak.sh --config-upgrade (the exact scenario that originally found this bug) -- ctrl-2's log shows the retry mechanism firing for real (4 failed cluster_init_phase2 attempts at ~3s intervals, rc=-22, then success: "commit engine now wired (bug-1477)"), and the soak's own checkpoint-identity, config-identity, and host-admin-status-survival assertions (the exact things bug-1477 blocked) all PASS. The soak run subsequently hit a DIFFERENT, seemingly unrelated failure later in its own force-elect-rotation step (repeated commit timeouts, rc=-110, on ctrl-1-as-primary) -- flagged separately, NOT part of this fix's scope; this soak scenario has never previously reached that step (earlier runs always failed earlier, at bug-1471/1472's checkpoint divergence or this bug's own catch-up wedge), so it may be a pre-existing, newly-exposed issue rather than a regression from this fix -- needs its own dedicated investigation.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1475, bug-1466, bug-849