Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
9.9 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1689 | fixed | 2026-09-24 | controller | unknown | 2026-08-25T18:10:00Z | 2026-09-19T09:00:00Z | 1 |
|
|
|
true |
bug-1689: xce-audit-invariants finding: WAL_JOB_FINISH silently drops job fields on a controller restart…
Description
xce-audit-invariants finding: WAL_JOB_FINISH silently drops job fields on a controller restart replay -- same bug class as bug-1678 (WAL_JOB_DISPATCH/job.allocs) and bug-1687 (WAL_JOB_START/job.pidinfo+start_time), but for the worker-confirmed-finish fact. apply_job_finish_transition() (job.c) sets job->job.exit_info->exitcode and job->job.finish_time/run_time, but the WAL_JOB_FINISH fact written by record_job_finish_fact() only carried wal_jobstate_t{jobid, state} -- exit_info was never in the log at all (and, worse, apply_job_finish_transition() itself is effectively dead code on the real production apply path -- both apply_committed_entry()'s and apply_wal_entry()'s WAL_JOB_FINISH handling route through apply_job_update_transition() (state-only) instead, per raft-apply-decoupling's deferred-apply design, so exit_info was NEVER populated through the real commit-apply pipeline, restart or not, prior to this fix), and finish_time was re-fabricated with time(NULL) at every apply_committed_entry() apply (and, on apply_wal_entry()'s standby catch-up seam, never touched at all -- an even larger gap than WAL_JOB_START's pre-1687 state) instead of carrying the true worker-reported receipt time.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
record_job_finish_fact() (job.c, on_job_finish()'s only caller) called persist_submit_job_fact(&c->persist, WAL_JOB_FINISH, job->job.jobid, state) -- a thin wrapper whose payload type wal_jobstate_t is exactly {jobid, state, _pad}, discarding finish->exit_code entirely even though it was available at the call site (the worker's real FINISH report). Both apply seams -- apply_committed_entry()'s shared WAL_JOB_START/FINISH/.../DISPATCH fall-through group (persist.c) for primary boot replay AND live commit-advance apply, and apply_wal_entry()'s identical shared group -> restore_job_state() (job.c, state-only) for standby catch-up -- never restored exit_info. apply_committed_entry() additionally fabricated finish_time = time(NULL) at apply time (rather than carrying the receipt-time value) and derived run_time from that fabricated finish_time; apply_wal_entry()'s shared group touched neither finish_time nor run_time at all for WAL_JOB_FINISH (a strictly worse gap than WAL_JOB_START's pre-bug-1687 apply_wal_entry behavior, which at least existed as a documented no-op). Confirmed via CodeGraph source read that apply_job_finish_transition() -- the richer core that DOES set exit_info/finish_time/run_time correctly -- is only ever invoked from record_job_finish_fact()'s duplicate-terminal-state peek (a no-op call whose own guard prevents any mutation), i.e. it is effectively dead code on the real production apply path today; the raft-apply-decoupling redesign defers all real mutation to apply_committed_entry()/apply_wal_entry(), which use the leaner apply_job_update_transition() (state-only) instead. job_exitinfo_t (xce/types.h) is a single int ({int exitcode;}) with NO existing Cap'n Proto mapping (confirmed -- no JobExitInfo struct in controller.capnp), unlike bug-1687's JobPIDInfo, so no capnp wrapper was invented for it.
Fix
FIXED (2026-08-25). Mirrors bug-1687's WAL_JOB_START fix pattern exactly, but uses a small NATIVE (non-capnp) trailing payload instead of Cap'n Proto, since job_exitinfo_t has no existing capnp mapping and no variable-length data (unlike JobPIDInfo). src/controller/persist.c gains decode_jobexitinfo() (a native memcpy decode, mirroring decode_pidinfo()/decode_joballoc()'s doc-comment style but without capnp) and persist_submit_job_finish_fact(persist_t*, jobid, state, finish_time, exit_code) -- WAL_JOB_FINISH's payload is now a wal_jobstate_t header + a fixed uint64_t finish_time (receipt-time clock, captured ONCE by record_job_finish_fact() rather than re-derived independently at each apply seam) + a fixed int32_t exit_code, declared in src/include/persist.h. src/controller/job.c's record_job_finish_fact() now calls persist_submit_job_finish_fact(..., time(NULL), finish->exit_code) instead of persist_submit_job_fact(...). Both apply seams updated: apply_wal_entry() (standby catch-up) splits WAL_JOB_FINISH out of its shared group into its own case (mirroring bug-1687's WAL_JOB_START split), decoding trailing finish_time/exit_code and calling new restore_job_finish_state() (job.c, restore_job_start_state()'s sibling -- finish_time==0 means 'not carried, leave finish_time/run_time untouched', matching this seam's pre-fix behavior exactly; exit_info==NULL means 'not carried, leave untouched'; job_exitinfo_t has no nested pointers so it is released with plain FREEUP, not a capnp free_*_ptr helper). apply_committed_entry() stays in its existing shared fall-through group but extends its wal_type==WAL_JOB_FINISH branch: a payload with trailing bytes (len >= sizeof(wal_jobstate_t)+sizeof(uint64_t)) decodes and adopts the wire finish_time and exit_code directly; a payload of exactly sizeof(wal_jobstate_t) (any WAL_JOB_FINISH entry written before this fix) falls through unchanged to the old fabricated-time(NULL)/exit_info-untouched behavior. run_time is NEVER carried on the wire in either case -- it is always re-derived as finish_time - start_time at apply time (matching apply_job_finish_transition()'s own formula), since start_time is already correctly recovered via bug-1687's own fix. Full backward compatibility verified by 2 dedicated old-format tests plus every pre-existing WAL_JOB_FINISH test in test_apply_committed_entry.c/test_apply_reapply_idempotency.c/test_apply_pump.c/test_commit_barrier_reply.c/test_commit_barrier_rollout.c (all still using the old bare-wal_jobstate_t shape) passing unchanged.
New regression tests: src/controller/testsuite/test_wal_job_finish_fact.c (6 tests, mirroring test_wal_job_start_fact.c's structure exactly, wired into src/controller/meson.build and testsuite/controller_test.c): (1) apply_committed_entry sets exit_info+finish_time (+re-derived run_time) from a new-format payload; (2) apply_committed_entry backward-compat with an old-format payload -- exit_info stays NULL, finish_time still fabricated via time(NULL); (3) apply_wal_entry sets exit_info+finish_time+run_time, fires no hook; (4) apply_wal_entry backward-compat -- exit_info/finish_time/run_time all left untouched (sentinel values survive), unlike apply_committed_entry's fabrication; (5) direct-payload restart-replay (WAL_JOB_SUBMIT + WAL_JOB_START + new-format WAL_JOB_FINISH via persist_wal_async + persist_recover()) recovers the exact wire finish_time, exit_code, and a correctly re-derived run_time; (6) test_wjf_real_on_job_finish_survives_restart -- drives the REAL production on_job_finish() path (record_job_finish_fact()'s only caller) with a real xce_jobfinish_t (nonzero exit_code=137), through a real WAL-write + persist_recover() restart cycle, asserting exit_info, finish_time (bounded to the real wall-clock window), and run_time all survive the restart intact -- the audit's exact confirmed root cause and the user's exact acceptance criterion.
RED->GREEN independently verified live (not just narrated): git-stashed job.c+persist.c only (git stash push -- src/controller/job.c src/controller/persist.c, keeping the new test file, persist.h, wal_types.h unchanged), rebuilt via the synchronous xce-build.sh script, ran ./controller_unit_test -t test_wal_job_finish_fact -- 6 Tests 4 Failures 0 Ignored (the 2 backward-compat tests correctly PASS unchanged; all 4 new-behavior tests correctly FAIL with exact expected mismatches, e.g. 'Expected 1700000050 Was 1787651049' for the fabricated-vs-wire finish_time on apply_committed_entry, 'Expected 1700000150 Was 0' for apply_wal_entry never touching finish_time pre-fix, and 'must set job.exit_info... not just state' for the real on_job_finish restart test). Restored via git stash pop, rebuilt, re-ran -- 6 Tests 0 Failures 0 Ignored, reproduced 3 consecutive times. Full controller_unit_test suite re-run after the fix: 1152 Tests, 1-2 Failures (flaky between runs) -- both failure signatures ('Expected 2 Was 3' / test_conf_trim_keeps_only_n_versions, and a garbage-uint64 mismatch / test_conf_get_saved_tag_no_config in test_config_repl.c) are the exact, previously-logged bug-1561 overlayfs-fdatasync sandbox artifact, confirmed identical across 2 consecutive re-runs of the unchanged binary (proving the flakiness is a sandbox I/O nondeterminism, not caused by this fix) -- zero new failures. Checked test_commit_barrier_reply.c's cbrp_count_jobstate_entries() (the exact helper bug-1687 had to fix for a payload-size-growth-induced hang) for the same risk: it already uses the generic hdr.payload_len >= sizeof(wal_jobstate_t) comparison bug-1687 fixed (not a WAL_JOB_START-specific special case), so WAL_JOB_FINISH's payload growth is already safely handled by that existing fix -- confirmed empirically by the full suite completing in ~38s with zero hangs, matching prior-session timing exactly.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1678, bug-1687, bug-1561