Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
8.8 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1471 | fixed | 2026-09-24 | controller | unknown | 2026-07-25T14:45:00Z | 2026-09-19T09:00:00Z | 3 |
|
|
|
true |
bug-1471: config-upgrade-snapshot task 8.6 live --config-upgrade soak: after avoiding bug-1469's…
Description
config-upgrade-snapshot task 8.6 live --config-upgrade soak: after avoiding bug-1469's queue-creation gap (switched the upgrade-target config change from ADDING a queue to updating an existing queue's description text -- see bug-1469's updated entry), the materialized-state identity check (snapdiff of each member's latest checkpoint file, all at the same LSN) still fails, but now with a MUCH smaller divergence: exactly ONE byte differs (offset 42879/51161 bytes total) between ctrl-1 and ctrl-2/ctrl-3 (ctrl-2 and ctrl-3 are byte-IDENTICAL to each other). ctrl-1 has 0x50 at that offset; ctrl-2/ctrl-3 have 0x00.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
ROOT-CAUSED AND DISTINGUISHED (2026-07-25, follow-up investigation): the two prior hypotheses are now resolved -- Hypothesis 2 (uninitialized/non-zeroed capnp padding) is REFUTED, Hypothesis 1 (genuine content discrepancy) is CONFIRMED, and the true mechanism is a pre-existing, out-of-spec-boundary bug, NOT part of config-upgrade-snapshot. Evidence: (1) Wrote a byte-precise decoder in Python that walks the checkpoint's [host_count][{payload_len; capnp bytes}] host-frame section (same structural approach as snapverify.c task 8.4) and decodes each frame using the exact wire shape write_HostAdminState produces (HostAdminState_word_count=1, pointer_count=1: one 8-byte data word holding status via capn_write32 at data-section offset 0, one 8-byte pointer word for name Text). Decoding BOTH /tmp/xce-ha-configupgrade and /tmp/xce-ha-configupgrade-v2's preserved checkpoints for all 3 members shows: host[1] = 'cfgupgrade-host', ctrl-1 status=0x00500000, ctrl-2/ctrl-3 status=0x00000000 -- byte offset 42879 is byte index 2 of the little-endian status DATA FIELD itself (data-section file offset 42877-42880), NOT the 4-byte padding word that follows it (file offset 42881-42884), which is 0x00000000 IDENTICALLY across all 3 members in both reproductions. (2) 0x00500000 = XCE_WORKER_CLOSED (0x00100000) | XCE_WORKER_CLOSED_BY_ADMIN (0x00400000) -- exactly the bits worker.c:779 sets for HOST_CLOSE inside worker_ctrl(). This is semantically meaningful admin-closed status, not heap garbage. (3) Independently confirmed capn_init_malloc's segment allocator (create_local in the pinned c-capnproto build, /nix/store/.../capnpc-1.0.1/lib/libCapnC_Runtime.so, disassembled directly) calls calloc(1, size) -- zero-initializing -- so Hypothesis 2 was never physically possible for this library, consistent with the empirically-always-zero padding bytes. (4) Traced the actual mechanism: worker_ctrl() (worker.c:732-794) sets w->host.status on whichever member directly processes the hostctrl RPC (ctrl-1, primary at the time), then fires HOOK_DETAIL_WORKER_UPDATE, which persist.c:5722 maps to WAL_WORKER_CTRL and replicates. On EVERY member applying that WAL entry (standbys AND a member's own replay-from-WAL/restart path alike), persist.c:3724's case WAL_WORKER_CTRL calls worker_from_capnp() (worker.c:827-856) UNCONDITIONALLY, regardless of hook_detail (JOIN vs UPDATE vs DEAD all share this one apply path). worker_from_capnp's own idempotency guard (worker.c:847-853) does key_find() and returns -EEXIST if a worker with that name ALREADY EXISTS -- WITHOUT applying any of the just-decoded fields, including status. persist.c:3732-3733 then treats -EEXIST as 'idempotent re-apply, no hook' and silently returns 0, discarding the decoded status update entirely. Since cfgupgrade-host was JOINED (a prior, separate WAL_WORKER_CTRL/HOOK_DETAIL_WORKER_JOIN entry) BEFORE it was admin-closed, by the time the UPDATE entry replicates/replays, the worker already exists on every standby -- so the close never lands there. This is NOT specific to config upgrades, checkpoints, or any config-upgrade-snapshot code: it reproduces for ANY hostctrl -c/hostctrl -o issued against an already-joined host on ANY XCE HA cluster, with zero dependency on a config upgrade ever being submitted. git log -L on worker_from_capnp confirms it was introduced 2026-05-08 in commit 9b6a91a8 ('feat: implement restore_job/worker_from_capnp for WAL recovery'), ~2.5 months before config-upgrade-snapshot's spec work began (first spec commit visible ~2026-06-24); no config-upgrade-snapshot task touches this apply path (the spec's own worker.c addition, restore_host_state from commit 5b1600a8, is a wholly separate function used only by the SNAPSHOT-INSTALL restore path, task 4, never by the live WAL_WORKER_CTRL apply switch). collect_host_frames/capnp_encode_host_admin_state (this spec's task 2.1 code) are functioning exactly as designed -- they faithfully serialize whatever w->host.status currently holds; the bug is that w->host.status itself is wrong on ctrl-2/ctrl-3, entirely upstream of any checkpoint/materialization logic. Existing test coverage (test_apply_committed_entry.c:test_ace_worker_ctrl_join_restores_and_notifies) only exercises the fresh-JOIN case; no test exercises WAL_WORKER_CTRL UPDATE against an already-existing worker, confirming this is a genuine untested gap, not a previously-handled case. Filed separately as bug-1472 (the true root cause) since it is a distinct, real, pre-existing mechanism entirely outside config-upgrade-snapshot's boundary -- see that entry for the fix-owner routing. This entry (bug-1471) remains the observed SYMPTOM (checkpoint byte divergence) for task 8.6's soak; the underlying CAUSE is bug-1472.
Fix
FIXED -- this was the observed SYMPTOM (checkpoint host-frame byte divergence) of bug-1472's root cause (worker_from_capnp()'s WAL_WORKER_CTRL apply path silently discarding UPDATE payloads against an already-existing worker). bug-1472 was fixed 2026-07-25 (TDD RED->GREEN): worker_from_capnp() now decodes hook_detail before applying its existing-worker idempotency guard, and HOOK_DETAIL_WORKER_UPDATE replaces the existing worker's XCE_WORKER_ADMIN_MASK bits with the decoded payload instead of silently no-op'ing on -EEXIST. This directly closes bug-1471's exact repro (an admin-close status change failing to replicate to standbys, producing the byte-42879 checkpoint divergence). Note: HOOK_DETAIL_WORKER_DEAD was deliberately left unfixed in the same change -- tracked separately as bug-1472-remainder (needs a design decision, not yet resolved).
[Original entry] NOT FIXED, AND DELIBERATELY NOT ATTEMPTED IN THIS SPEC -- confirmed genuinely OUT OF config-upgrade-snapshot's scope (see root_cause above). The real fix belongs in worker.c's worker_from_capnp()/persist.c's WAL_WORKER_CTRL apply case (case WAL_WORKER_CTRL, persist.c:3724), which needs to distinguish JOIN semantics (create-if-missing, EEXIST is a legitimate no-op) from UPDATE/DEAD semantics (an already-existing worker is the EXPECTED case, and the decoded fields -- at minimum host.status -- must still be applied to it, not discarded) -- e.g. an update-in-place path mirroring how restore_host_state (task 3, worker.c, config-upgrade-snapshot's OWN precedent) ORs admin bits into an existing worker rather than treating existing-ness as terminal. That is a genuine, real, previously undiscovered defect (filed as bug-1472) but is squarely core HA worker-control replication, predating this spec by ~2.5 months and untouched by any of its tasks -- fixing it here would be scope creep into a different subsystem's correctness contract, not a config-upgrade-snapshot materialization fix. Task 8.6's soak will keep observing this exact symptom (host-frame byte divergence whenever its scenario admin-closes a host before the final checkpoint) until bug-1472 is fixed by whoever owns worker.c/persist.c's general worker-control replication path. Recommend task 8.6 either (a) accept this as a known, separately-tracked limitation and adjust the soak's own PASS criteria to tolerate it (e.g. skip the admin-close-a-host step, or snapdiff-exclude the host-admin-status byte range) until bug-1472 lands, or (b) block final PASS on bug-1472's fix -- an orchestrator/owner decision, not this investigation's to make.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1469, bug-1472