Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
6.1 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | fix_commits | legacy | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1444 | fixed | 2026-09-24 | controller | unknown | 2026-07-16T02:48:18Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
|
true |
bug-1444: persist_recover: APPLY_DIVERGENT -- materialized state does not match the committed log tip…
Description
persist_recover: APPLY_DIVERGENT -- materialized state does not match the committed log tip (expected last_applied_lsn=5 new_job_count=3); a hidden apply defect is present (ha-commit-durability recover_verdict=RECOVER_CLEAN) / persist_recover: UNRECOVERABLE -- post-replay apply-correctness verification is DIVERGENT ... Staying ALIVE and PERMANENTLY non-promotable (refusing to serve divergent state) and RETRYING catch-up indefinitely -- reproduced live during job-array-single-wal-record task 6.1's HA soak scenario the FIRST time a node is killed and restarted after ANY WAL_JOB_ARRAY_SUBMIT record has ever been committed to its local WAL (e.g. after a 2-element array submission).
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
replay_wal()'s post-replay apply-correctness expectation counter only tallies WAL_JOB_SUBMIT entries toward new_job_submits_out: if (hdr.type == WAL_JOB_SUBMIT && new_job_submits_out && apply_rc == 1) (*new_job_submits_out)++; (persist.c ~3902-3904). It has NO corresponding branch for WAL_JOB_ARRAY_SUBMIT (the new record type job-array-single-wal-record/task 1.1 introduced), even though apply_committed_entry()'s WAL_JOB_ARRAY_SUBMIT case (persist.c ~2809-2851, restore_job_array()) constructs N new job_t entries per record, not 1. So expected_new_job_count silently undercounts by N for every array record replayed during boot -- while persist_verify_apply_correctness() (persist.c ~3982-3988) measures actual_new_jobs via a REAL jobmgr entry-count delta (count_jobmgr_entries before/after), which correctly reflects the N elements. The mismatch (actual_new_jobs != expected_new_job_count) trips APPLY_DIVERGENT on every single restart of every node whose local WAL contains a WAL_JOB_ARRAY_SUBMIT entry above its checkpoint -- deterministic and 100% reproducible, not a race or resource-exhaustion condition. persist_recover_loud_fence_if_unrecoverable then classifies this as UNRECOVERABLE, permanently non-promotable-fencing that node for the rest of its process life (own boot-time verification re-runs identically on every subsequent restart too, since the miscounting bug reproduces every time). This is a NEW, separate gap from this spec's three self-flagged residual items (task 3.1 bulk-insert partial-visibility / bug-1443, task 4.2 boot-replay-doesn't-respect-apply-failures gap, task 5.3b's small per-commit leak) -- it lives in a DIFFERENT, pre-existing mechanism (raft-apply-decoupling task 5.1's post-replay apply-correctness verification / ha-commit-durability's Req 8.1-8.4), which this spec's own task list never touched to account for the new WAL_JOB_ARRAY_SUBMIT type. Practical severity: in any cluster where a job array has EVER been submitted, every ordinary node restart (routine maintenance, crash-restart, rolling upgrade) permanently non-promotable-fences that node; over enough restarts across a 3-node cluster this can leave ZERO nodes eligible for promotion -- a serious HA-availability regression, not a cosmetic log line. Discovered live via job-array-single-wal-record task 6.1's HA soak harness (.claude/skills/xce-test-ha/run-ha-soak.sh, new --array-soak mode) -- specifically NOT reproducible by this spec's existing white-box unit tests (task 2.3/3.4), which never drive a real persist_recover()/replay_wal() boot cycle after committing an array record.
Fix
FIXED and VERIFIED (2026-07-15/16, commit 16747aa, task 6.0a of job-array-single-wal-record) -- independently root-caused by a separate implementation pass on the same branch, inserted as a blocking task ahead of 6.1/6.2/6.3 per explicit user sign-off after the exact same bug was live-reproduced during this task's own --array-soak run. Fix matches the shape predicted below almost exactly: replay_wal()'s tally gained a WAL_JOB_ARRAY_SUBMIT branch that adds apply_rc (the genuine per-record new-element count apply_committed_entry()'s array case already returns, not the flat/undercounted +1 the WAL_JOB_SUBMIT branch uses) to *new_job_submits_out when apply_rc > 0, mirroring the existing WAL_JOB_SUBMIT pattern. Verified via: (a) new unit tests in test_apply_correctness_verification.c with an independently-reproduced RED-before-fix/GREEN-after-fix cycle; (b) two from-scratch live 3-node HA soak re-runs (--array-soak), one by the fixing pass's implementer and one by its independent reviewer, both showing zero APPLY_DIVERGENT across all three of this bug's trigger paths (restart-mid-small-array, standby election-path recovery fallback, restart-mid-large-array) with one-WAL-record/one-commit-barrier and per-element status/cancel/completion all confirmed post-failover. Full detail in .kiro/specs/job-array-single-wal-record/tasks.md's Implementation Notes, task 6.0a entry. This buglog entry was originally filed (independently, before its author discovered the fix already existed on the branch) with fix=NOT FIXED; corrected here once the duplication was noticed, since the underlying finding is accurate and well-characterized and deserves its own tracked entry (task 6.0a's fix was documented in tasks.md but never separately logged here) -- kept as a distinct entry from bug-1443 (a different, still-open gap in the same feature) rather than merged or deleted.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1443