Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
9.9 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | fix_commits | legacy | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1469 | fixed | 2026-09-27 | 154a169f: reconcile_config_queues on activation creates added queues without restart |
controller | unknown | 2026-07-25T13:05:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
|
true |
bug-1469: config-upgrade-snapshot task 8.6's live soak (--config-upgrade, run-ha-soak.sh) FAILED on its…
Description
config-upgrade-snapshot task 8.6's live soak (--config-upgrade, run-ha-soak.sh) FAILED on its first real run: after a config upgrade committed successfully and genuinely interleaved with a 3-round chaotic kill/restart sequence (all 6 jobs survived, upgrade confirmed committed), the post-convergence materialized-state identity check failed -- snapdiff reported ctrl-1's latest checkpoint (snap-000000000000000b.capnp, LSN 11) as 51569 bytes vs ctrl-2/ctrl-3's identical 51621-byte file. This is a live, empirical reproduction of bug-857's originally-hypothesized failure mode (config upgrade leaves members with divergent materialized state), which bug-857's own buglog entry explicitly flagged as 'NOT empirically reproduced yet' and task 8.5 could only address at the unit level (test_bug857_regression.c, a DIFFERENT and already-fixed mechanism -- schedule_checkpoint's WAL-threshold trigger racing the marker boundary). This is a third, distinct mechanism, first observed here.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
Byte-level diff of the two checkpoint files (both LSN 11) shows the string 'soakupgrade' (the deliberately-added 5th queue's name, part of task 8.6's upgrade-target config) appears ONCE in ctrl-1's file (inside the embedded raw active-config-bytes blob, persist.c's read_active_config_bytes -- always a fresh disk re-read) but TWICE in ctrl-2/ctrl-3's files -- the second occurrence, present only on ctrl-2/ctrl-3, sits in a queue-ENTITY list section (immediately followed by the next queue's name 'lost_and_found', a runtime enumeration, not raw config text). So ctrl-1's checkpoint correctly embeds the NEW config bytes but its runtime QUEUE-ENTITY table still reflects the OLD (pre-upgrade) queue set. Traced to the code: do_upgrade_config_with_bytes (the handler that runs on whichever node ORIGINATES/commits the upgrade -- ctrl-1 was primary in term 2 when this upgrade was submitted, confirmed via its own log line 'do_upgrade_config_with_bytes: Config upgraded to tag=... (acked=2 standbys)') saves the new bytes to disk (conf_save_config) and bumps c->myconf->config_generation, but NEVER calls config_activate_committed()/load_primary_config() to actually re-parse the new bytes INTO c->myconf -- so the very struct the checkpoint writer presumably walks to emit the queue-entity list keeps the OLD parsed queue set. apply_committed_entry's WAL_CONFIG_MARKER case (persist.c) only calls persist_checkpoint_marker_triggered() then upgrade_materialization_verify() (ha.c) -- and upgrade_materialization_verify does NOT reload/activate config either, it only (a) checks whether some PRIOR checkpoint already referenced this marker (cfgref_present/tag/sha, circularly self-referential to the checkpoint it is meant to gate) and (b) hash-verifies the LIVE ON-DISK bytes -- neither check touches c->myconf's in-memory queue table, so the Activation Fence reports 'VERIFIED' (confirmed in ctrl-1's log, 'upgrade materialization VERIFIED... materialization complete') even though c->myconf itself was never refreshed. config_activate_committed (which DOES call load_primary_config, the only place that actually re-parses queues into c->myconf) is invoked only from handle_repl_config_push (a STANDBY receiving an out-of-band config-push) and from a member's own boot/recovery path -- confirmed empirically: ctrl-1's log shows ZERO config_activate_committed calls in its original (primary, term-2) process instance, but TWO calls in its restarted instance (12:59:16 and 12:59:39, after persist_recover reported APPLY_CLEAN at last_applied_lsn=11, i.e. loading the already-frozen stale checkpoint rather than replaying). Consequence: the node that ORIGINATES a config upgrade never refreshes its own in-memory queue table as part of that commit -- its checkpoint stays consistent with the OLD queue set until some LATER, independent event (restart+recovery, or an incoming config-push) happens to call config_activate_committed. Task 8.6's scenario kills the originating primary shortly after it commits (chaotic multi-kill interleaved with the upgrade, by design), so that self-correction never gets a chance to run, and no FURTHER WAL activity occurred to trigger a follow-up checkpoint that could have picked up a later-corrected queue table -- the divergent checkpoint file is permanently frozen.
Fix
PARTIALLY FIXED, then RE-INVESTIGATED (2026-07-25) -- correcting this entry's own earlier premature 'FIXED' claim. Two real findings:
(1) GENUINE FIX, LANDED: apply_committed_entry's WAL_CONFIG_MARKER case (persist.c) was calling config_activate_committed(c) AFTER persist_checkpoint_marker_triggered() (the checkpoint write), not before -- meaning even when activation ran, it ran too late to affect the checkpoint it was meant to fix. Reordered to run BEFORE the checkpoint call (still guarded on c->myconf != NULL, still positioned before upgrade_materialization_verify per task 6.2's own sequencing need). Verified: full controller_unit_test suite 1025/1025 PASS, 0 failures, no crash (including the previously-SEGV-prone test_marker_apply_via_live_pump_checkpoints_at_marker_own_lsn and the bug-1469 regression test_wal_config_marker_apply_activates_originating_member_config).
(2) DEEPER GAP FOUND, NOT FIXED -- genuinely out of task 8.6's scope: re-running the live --config-upgrade soak AFTER the reordering fix landed produced the IDENTICAL divergence signature (same ~52-byte checkpoint delta, same duplicated-vs-single 'soakupgrade' queue name pattern) on a DIFFERENT node this time (still one node showing stale queue state). Root-caused via direct code trace: config_activate_committed() -> load_primary_config() only re-parses bytes into myconf->cluster_config (config_repl.c:614-714) -- it NEVER touches c->qmgr, the RUNTIME queue table collect_queue_frames() (persist.c:1895, the checkpoint's queue-entity-list writer) actually walks. The ONLY code that populates c->qmgr from cluster_config->queues[] is build_queues() (queue.c:28), called EXACTLY ONCE per process lifetime from xce_config_init's HOOK_AFTER_EVENT branch (cluster.c, fired during cluster_init_phase2 at process boot/restart) -- and build_queues() is UNCONDITIONAL/blind-create (no existence check before calling create_queue/makeQueue for every queue in the config), so it is UNSAFE to simply re-call on a live cluster with existing queue objects (would attempt to recreate already-existing queues, risking duplicate-key/corruption semantics that were never designed for a live re-provisioning call). Consequence: NO MEMBER -- neither the config-upgrade originator NOR a replicating standby NOR any node's own live marker-apply of any kind -- has a working mechanism to add a NEWLY-INTRODUCED queue to its runtime qmgr without a full process restart. The apparent 'originating member vs standby' asymmetry in this bug's original report was COINCIDENTAL, not causal: whichever node's restart (inevitable during a chaotic multi-kill sequence, unrelated to which node 'originated' the upgrade) happened to occur AFTER the new config bytes were already on disk got the new queue for free via its own full boot sequence (xce_config_init -> build_queues); a node whose restart happened to land BEFORE that point, or that never restarted during the scenario at all, does not -- regardless of the fix in (1). This directly contradicts design.md's own stated goal for this spec ('No controller restart is required for a configuration upgrade after this change') for the specific case of a config upgrade that ADDS a queue: today it demonstrably STILL requires a coincidental restart to take effect on any given already-running member. requirements.md's own Design Invariant #1 ('Queues are config-provisioned... update-only restore is safe') is only true for the SNAPSHOT-INSTALL restore path (a fresh/behind joiner materializing a full checkpoint, restore_queue_state, queue.c:208) -- it was never validated against the LIVE no-restart marker-apply path this spec's Requirement 7/Introduction promises, and does not hold there.
NOT fixed as part of config-upgrade-snapshot task 8.6: doing so properly requires new, carefully-designed production logic (a live-safe, idempotent 'sync qmgr from the freshly committed config' function -- e.g. mirroring restore_host_state's exact find-or-create-only shape, worker.c:477-536, adapted for queue_t/c->qmgr) PLUS a deliberate design decision this task is not positioned to make safely under its own time/scope budget: what happens to a queue REMOVED by an upgrade (delete the queue_t? re-home its jobs to lost_and_found -- bug-857's own originally-hypothesized-but-never-implemented mechanism? refuse the upgrade while jobs remain queued?) -- a decision requiring its own requirements/design pass, not an implementation detail. Flagging squarely for whoever owns tasks 1-7 (persist.c/config_repl.c/queue.c materialization logic) as a genuine, previously-undiscovered gap in this spec's core 'no controller restart required' promise, confirmed via TWO independent live soak runs (before and after the ordering fix) reproducing the identical byte-level signature.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-857, bug-1466