Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
8.1 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | gitea_issue | gitea_synced_status | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1678 | fixed | 2026-09-24 | controller | unknown | 2026-08-24T13:55:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true | 43 | fixed |
bug-1678: Gitea issue #43 ("重启daemon后作业调度/信息不对"): after restarting ALL daemons (controller, scheduler…
Description
Gitea issue #43 ("重启daemon后作业调度/信息不对"): after restarting ALL daemons (controller, scheduler, worker, apiserver), previously-RUN jobs come back with STATUS=RUN correctly recovered but EXEC_HOST permanently blank ("-"), even though the underlying job is genuinely still running. CONFIRMED LIVE on branch scheduler-framework, corrected finding: an earlier verification pass on this same issue (2026-08-24, same session) restarted only controller+scheduler+worker and left the apiserver running, and did NOT reproduce the symptom -- that was a false negative, caused by the still-running apiserver simply continuing to display its stale-but-correct in-memory job cache from before the restart, never refreshed. Repeating the test with the apiserver ALSO restarted (matching the original report's "restart all daemons" exactly) reproduces the bug immediately and durably: 3 RUN jobs with EXEC_HOST=1*worker-1 before the restart; killed all 4 daemons; ran start-all.sh; EXEC_HOST was blank on every poll (15 polls over 30s, first poll ~6s after apiserver came back up) while STATUS stayed correctly RUN throughout. Controller log corroborates the mechanism: ctrl-1.log's recovery trace shows apply_job_update_transition logging 'duplicate UPDATE to state 0x2 -- no-op' for every job during WAL replay -- consistent with the WAL_JOB_DISPATCH fact applying SILENTLY first (PEND->RUN, no log line on the APPLIED path), immediately followed by each job's WAL_JOB_START fact (worker-confirmed start, also state=RUN) correctly detecting the now-redundant transition and logging the no-op. Neither replay step ever touches job.allocs.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
dispatch_a_job_() (job.c:1343-1372) sets j->job.allocs directly in memory (lines 1346-1353, the host/resource assignment the scheduler decided) and transitions j->job.state = JOB_STAT_RUN (line 1355), but the durable record of this transition -- persist_submit_job_fact(&c->persist, WAL_JOB_DISPATCH, alloc->jobid, JOB_STAT_RUN) at line 1369-1370 -- only carries {jobid, state} (persist_submit_job_fact()'s own signature, persist.c:6690-6694, has no allocation parameter at all; its payload type wal_jobstate_t is exactly {jobid, state}). The resource allocation itself is never written to the WAL anywhere. On a full-process restart, job objects are reconstructed purely from replay: restore_job() (job.c:288-351) decodes a job's ONLY WAL_JOB_SUBMIT entry (written at initial submission, strictly before dispatch -- confirmed via persist_job_hook_fun's own comment that WAL_JOB_SUBMIT durability is written once, before any job object exists -- so its capnp payload's allocs field is genuinely empty/NULL at encode time). apply_committed_entry()'s WAL_JOB_DISPATCH case (persist.c:3708-3757) then replays the dispatch fact via apply_job_update_transition(), which by explicit design and its own doc comment 'only ever touches state' -- job->job.allocs is never set anywhere in the replay path. Result: after restart, job.state is correctly RUN (both WAL_JOB_DISPATCH and the later WAL_JOB_START fact carry state=RUN and replay correctly) but job.allocs is permanently NULL for the remaining life of that job -- nothing ever re-derives it, since scheduler-dispatch-commit's design treats a job's dispatch as a one-time PEND->RUN event with no re-dispatch/re-reconciliation path for an already-RUN job (see bug-1631's confirmation that no live caller ever re-adds a RUN job back to the scheduler's dispatch pool). EXEC_HOST is rendered by the apiserver from this now-NULL allocation once the recovered, allocation-less job is pushed to it -- which is why restarting the apiserver alongside the controller (forcing a fresh push) reveals the bug, while leaving the apiserver running (as an earlier verification pass on this same issue did) masks it behind stale cached data.
Fix
FIXED (2026-08-25). No capnp schema change needed -- JobAlloc (xce_job_alloc_t) was already a full top-level capnp struct (the existing Job.allocs field) with generated encode/decode functions already present in the build. New glue only: src/controller/persist.c gains capnp_encode_joballoc()/decode_joballoc() (mirroring the existing capnp_encode_worker()/restore_job() patterns) and persist_submit_job_dispatch_fact(persist_t*, jobid, state, alloc) -- WAL_JOB_DISPATCH's payload is now a wal_jobstate_t header plus trailing capnp-encoded JobAlloc bytes, declared in src/include/persist.h. src/controller/job.c: dispatch_a_job_()'s success path now calls persist_submit_job_dispatch_fact(..., j->job.allocs) instead of persist_submit_job_fact(...), reading j->job.allocs (the freshly-constructed canonical struct) under the same write lock. Both apply seams updated: apply_committed_entry() (primary's own boot replay) gets one added else-if branch inside its existing WAL_JOB_START/WAL_JOB_FINISH stamp-chain (minimal diff, same shared plumbing) that decodes and adopts trailing allocation bytes when present. apply_wal_entry() (standby catch-up / replicated-apply -- also required, since a promoted standby would hit the exact same bug) has WAL_JOB_DISPATCH split into its own case, calling new restore_job_dispatch_state() (job.c, restore_job_state()'s sibling, takes ownership of an optional decoded allocation). A payload of exactly sizeof(wal_jobstate_t) (any WAL_JOB_DISPATCH entry written before this fix) decodes no allocation and behaves exactly as before -- full backward compatibility, verified by the 3 pre-existing test_wal_job_dispatch_fact.c tests passing completely unchanged.
Primary regression test: test_dispatch_commit.c's test_dc_success_appends_wal_job_dispatch_durably (already drove the REAL dispatch_a_job_() production path through a real WAL-write + persist_recover() restart cycle, previously only checking j2->job.state) extended to also assert j2->job.allocs matches what was dispatched (host, resource name, value) -- the user's exact acceptance criterion, 'after recover the job should get the same as it's state before shutdown.' RED->GREEN independently reproduced (not just narrated): git-stashed job.c/persist.c/persist.h with the test extension kept, rebuilt, confirmed FAIL at exactly this assertion; restored, rebuilt, confirmed PASS. Added 3 new tests to test_wal_job_dispatch_fact.c covering the richer payload directly on both apply seams plus a full restart-replay variant -- all pass. Full controller_unit_test suite independently re-run: 1136/1136 pass, zero regressions.
Live-verified at the WAL level via xce_test's waldump: a real dispatch_a_job_() call produced a WAL_JOB_DISPATCH entry with PLEN=176 (vs the old bare wal_jobstate_t shape's PLEN=16), confirming the richer payload is genuinely written on the production path.
IMPORTANT: full xcecli-visible E2E verification (submit -> restart -> check EXEC_HOST live) was blocked by discovering bug-1679 (see that entry) -- a separate, more severe, pre-existing bug where live job-status pushes to the apiserver never fire at all, restart or not. This fix's correctness rests on the unit-test evidence above (which inspects the controller's actual recovered in-memory state directly, with no dependency on the broken apiserver-notification path) and the direct WAL-payload inspection, not on an xcecli-visible live check.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1631, bug-1679
- Gitea: #43