Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
7.9 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | fix_commits | legacy | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1472 | fixed | 2026-09-24 | controller | unknown | 2026-07-25T16:35:00Z | 2026-09-19T09:00:00Z | 2 |
|
|
|
|
true |
bug-1472: Host admin-control status changes (hostctrl -c / hostctrl -o, i.e
Description
Host admin-control status changes (hostctrl -c / hostctrl -o, i.e. HOST_CLOSE/HOST_OPEN via worker_ctrl()) never propagate to standby members' in-memory worker state once the target host has already been joined -- discovered while root-causing bug-1471's single-byte checkpoint divergence, but the defect itself is general and has nothing to do with checkpoints, config upgrades, or the config-upgrade-snapshot spec.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
worker_from_capnp() (worker.c:827-856) is invoked UNCONDITIONALLY by persist.c:3731 for every WAL_WORKER_CTRL entry, regardless of its hook_detail discriminator (JOIN, UPDATE, or DEAD all share this one apply path -- see persist.c:3766-3783's switch, which only controls which post-apply NOTIFICATION hook fires, not whether the mutation itself is applied). worker_from_capnp's own idempotency guard (worker.c:847-853): if key_find() on c->workermgr finds a worker with the decoded name ALREADY PRESENT, it returns -EEXIST immediately, WITHOUT applying any decoded field (including host.status) to the existing worker object. persist.c:3732-3733 treats any -EEXIST return as 'idempotent re-apply (replay / resync) -- no hook' and silently returns 0. This EEXIST short-circuit is correct for its originally-intended purpose (a duplicate/replayed JOIN entry for a worker that's already been created should indeed be a no-op), but WAL_WORKER_CTRL is also the opcode for HOOK_DETAIL_WORKER_UPDATE (worker.c:732-794's worker_ctrl(), which handles HOST_OPEN/ HOST_CLOSE admin commands, sets w->host.status, then fires exactly this same HOOK_AFTER_EVENT/HOOK_DETAIL_WORKER_UPDATE/WAL_WORKER_CTRL path at worker.c:786-787) and HOOK_DETAIL_WORKER_DEAD. For UPDATE/DEAD events, the target worker EXISTING is the NORMAL, EXPECTED case (you can only update or kill an already-joined worker) -- yet the same 'already exists -> skip entirely' logic applies, so the decoded status change is silently discarded on every member OTHER than the one that directly processed the original RPC. Confirmed via git blame: worker_from_capnp was introduced 2026-05-08 (commit 9b6a91a8, 'feat: implement restore_job/worker_from_capnp for WAL recovery'), predating config-upgrade-snapshot's spec work by ~2.5 months; no config-upgrade-snapshot task touches this function or the WAL_WORKER_CTRL apply case. Empirically confirmed via byte-precise capnp decode of two independent live-soak checkpoint sets (/tmp/xce-ha-configupgrade{,-v2}): ctrl-1 (which processed a hostctrl -c against a previously-joined synthetic host) has host.status=0x00500000 (XCE_WORKER_CLOSED|XCE_WORKER_CLOSED_BY_ADMIN) in its checkpoint's HostAdminState frame; ctrl-2/ctrl-3 (standbys) both show host.status=0x00000000 for the identically-named host entry -- proving the close never reached their live c->workermgr entries. Because a standby's OWN restart+full-WAL-replay hits the identical code path (persist.c's replay call site also routes through apply_committed_entry's same WAL_WORKER_CTRL case), a member that fully replays from LSN 1 would ALSO reconstruct the JOIN first (creating the worker) and then skip the later UPDATE (already exists) -- so this is not even a restart-repairable staleness; the divergence appears to be permanent for that member until directly patched or until some other mechanism re-applies the admin action. General severity: this affects ANY admin host-control action (hostctrl -c/hostctrl -o) against an already-joined host, unconditionally, cluster-wide -- not just this soak's synthetic cfgupgrade-host scenario.
Fix
FIXED 2026-07-25 (TDD: RED test first, confirmed failing against the pre-fix code, then GREEN). worker_from_capnp() (src/controller/worker.c) now decodes the WAL_WORKER_CTRL entry's hook_detail discriminator (the same 4 leading bytes it previously skipped without inspecting) BEFORE applying its existing-worker idempotency guard. When the target worker already exists: HOOK_DETAIL_WORKER_JOIN keeps the original -EEXIST no-op (a duplicate JOIN replay is genuinely idempotent -- unchanged, regression-guarded). HOOK_DETAIL_WORKER_UPDATE now REPLACES (not ORs -- worker_ctrl()'s HOST_OPEN uses assignment-style &= ~(...) to CLEAR admin bits, so OR would leave a re-opened worker stuck closed) the existing worker's XCE_WORKER_ADMIN_MASK bits with the decoded payload's admin-masked bits, under the worker's own write lock (mirrors restore_host_state's masking discipline, worker.c:487, with replace instead of OR semantics), then returns 0 so persist.c's WAL_WORKER_CTRL apply case (unchanged -- its existing '0 on any successful apply, -EEXIST only for genuine JOIN no-op' contract already handled this generically) falls through to fire the correct HOOK_DETAIL_WORKER_UPDATE notification instead of silently discarding the status change. HOOK_DETAIL_WORKER_DEAD is DELIBERATELY LEFT UNFIXED (still -EEXIST/no-op) -- investigated, not skipped: mark_worker_unavail()'s mutation (worker.c:661-679) is NOT confined to XCE_WORKER_ADMIN_MASK (sets XCE_WORKER_UNAVAIL + dead_since + resets hostattrs_t to 'unknown' + zeroes resource-value slots via set_host_slots), and critically, the live per-host resource-value array xce_worker_t.resvals (a zarray_t, what set_host_slots actually mutates and what scheduling reads) is NOT part of the Worker capnp wire schema at all -- only the separate resvals_to_sched/nresvals_to_sched fields are (confirmed by reading the generated decode_Worker/encode_Worker in controller.capnp.c and the schema in controller.capnp). A naive status-replace for DEAD would be an incomplete fix (would not zero standbys'/replayers' resource slots on a dead-worker event, a real correctness gap for POST-PROMOTION scheduling since the newly-promoted primary would read stale, non-zeroed slots) and needs its own investigation into whether/how to extend the wire schema -- filed separately as bug-1472-remainder below rather than rushed. Fix verified via 2 new + regression tests in test_apply_committed_entry.c (test_ace_worker_ctrl_update_applies_to_existing_worker, test_ace_worker_ctrl_update_replaces_not_ors_admin_bits, test_ace_worker_ctrl_join_duplicate_existing_is_noop -- the last one a regression guard proving JOIN's existing no-op behavior is unchanged) plus the full controller_unit_test suite: baseline (this worktree, immediately pre-fix) 1030 Tests 0 Failures, post-fix 1033 Tests 0 Failures (net +3, exactly the new tests, zero regressions). persist.c was NOT touched -- confirmed its WAL_WORKER_CTRL apply case (persist.c:3724-3786) already treats any non-negative worker_from_capnp() return as a successful apply generically (re-looks-up the worker by name and fires the hook_detail-keyed notification unconditionally), so returning 0 from the new UPDATE branch required no caller-side change.
[UPDATE 2026-08-16] Tag hygiene: this entry's own UPDATE-case fix landed 2026-07-25 (commit 7547a8f1) but its tags were never updated off 'not-fixed'/'needs-owner'. The DEAD-case gap this entry deferred is bug-1472-remainder, separately FIXED and landed today in commit 45400c3a. Both halves of bug-1472 are now fixed and committed; closing out.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1471, bug-1472-remainder