Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
7.3 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | gitea_issue | gitea_synced_status | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1843 | fixed | 2026-09-24 | scheduler | unknown | 2026-09-22T19:35:00Z | 2026-09-23T08:00:00Z | 1 |
|
|
|
true | 78 | fixed |
bug-1843: Same user report as bug-1842: replaying scheduler-replay-20260922-095207.replay via --sim…
Description
Same user report as bug-1842: replaying scheduler-replay-20260922-095207.replay via --sim --sim-verify also logs a small number of 'expected: -- actual: PEND jobid=... main_reason=0' mismatches -- live decisions with no recorded counterpart at all (beyond decoded_session->n_decisions), distinct from bug-1842's DISPATCH-vs-DISPATCH false positives (already fixed). 61 occurrences total across the full 643-session replay file (0.02% of the 3431 live PEND decisions produced).
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
CONFIRMED (2026-09-23), and the leading hypothesis from the prior session's investigation (event-batching-granularity in sim_loop.c's replay driver -- 'a job's first appearance lands one round earlier during replay than the session it was recorded in') is REFUTED by direct measurement, not just re-reasoned about: added temporary per-round diagnostic logging to sched_sim_driver_run() (sim_loop.c, reverted after use -- not part of the fix) and replayed the exact scheduler-replay-20260922-095207.replay capture this bug was originally filed against with --sim --sim-verify --sim-speed=0. For every one of the 61 mismatches, the LIVE (replay) side deterministically re-offers the full, correct set of already-known PEND candidates every round (e.g. round begin_round=...789016869984 produced 38 PEND decisions for jobs [22]..[59], with ZERO new events pushed into that round -- core state was provably identical to the immediately preceding round, which also produced 38). The RECORDED side, for that exact same round, has only 32 decisions (jobs [22]..[53]) -- missing exactly the newest-added cohort, jobs [54]..[59] -- while the immediately preceding and following recorded rounds both have all 38. This is the true shape of the bug: an isolated, self-healing gap in the ORIGINAL LIVE RECORDING, not a replay-side batching/discretization artifact (sim_loop.c's event-buffer-then-push-before-begin_session mechanism was verified, by direct code reading of both sim_loop.c's replay-side push loop and driver.c's live-side xce_sched_driver_translate_pending_events()/sched_record_write_events() call order, to be a byte-for-byte-faithful reproduction of the recorder's own file ordering -- there is no live-vs-replay discretization gap to close there). True root cause: sched_record_session_add_decision() (recording_session.c) grows its per-round ctx->decisions array via realloc(), doubling from an initial capacity of 8 (8 -> 16 -> 32 -> 64...). If that realloc() call fails, the existing code silently drops the CURRENT decision (free_record_decision(rd); return;) and leaves ctx->n/ctx->cap exactly where they were -- every SUBSEQUENT decision that round hits the same n==cap growth check and is ALSO silently dropped, so the round effectively truncates at whichever doubling boundary (8/16/32/64) the failing realloc() landed on (n=32 here -- exactly one of those boundaries, and exactly matching the empirically observed truncation point). Nothing marks the round as incomplete: sched_record_session_finalize() (recording_session.c) then serializes and writes ctx's now-short decisions array as an ordinary, apparently-complete ReplaySession -- indistinguishable on replay from a genuinely smaller live round. This is the ONE inconsistent failure-handling path in this file: every OTHER failure in the same recording subsystem (write_capnp_message() failures, XCE_DECISION_PREEMPT/NONE cases) is either propagated as a hard error (sched_record_abort_on_error()) or logged loudly by design -- only this one growth failure was silent-and-partial. The live SCHEDULING round itself was never affected either way (on_decision() still fires for every decision drained from core.decisions regardless of recording state) -- only the .replay FILE's own record of that one round silently lost its tail, which --sim-verify's replay comparison then had no way to distinguish from a genuinely smaller live round.
Fix
FIXED (2026-09-23, TDD). recording_session.c: added a bool truncated field to sched_record_session_ctx_s; sched_record_session_add_decision()'s realloc-failure branch now sets ctx->truncated = true (in addition to its existing ERROR log) before dropping the decision. sched_record_session_finalize() now checks ctx->truncated FIRST and, if set, frees everything and returns -ENOMEM WITHOUT encoding/writing anything -- routing through the exact same sched_record_abort_on_error() cleanup path (via sched_record_session_end(), driver.c) that every other recording failure in this subsystem already uses, instead of silently writing a truncated-looking-complete ReplaySession. sim_loop.c's own --sim-output path (sched_record_session_finalize() called directly) already treated any negative return as fatal, so it is covered for free with zero changes there. recording.h's doc comments for sched_record_session_add_decision()/sched_record_session_end()/sched_record_session_finalize() updated to describe the new contract. TDD: added test_decision_batch_growth_failure_drops_incomplete_session() (test_record_lifecycle.c) -- this codebase had no existing malloc/realloc fault-injection harness, so added -Wl,--wrap=realloc linker interposition scoped to ONLY this test executable's meson target (same established -Wl,--wrap= idiom already used elsewhere in this test suite for xcecli_*/sched_meet_host_selection, meson.build), with a __wrap_realloc() that passes through to __real_realloc() unless the one new test arms it. The test forces a REAL, deterministic realloc() failure on the 9th decision's array growth (8 -> 16) and asserts sched_record_session_end() returns an error and recording is cleanly disabled (matching test_write_failure_disables_recording_and_driver_keeps_running()'s existing assertion style). Verified genuine red->green: temporarily reverted just the production fix and reran -- scheduler record lifecycle test FAILED (exit 1) without the fix, confirming the new test actually exercises the bug; restored the fix and reran -- full 85/85 scheduler suite green. NOTE: this fix prevents FUTURE recordings from losing decisions this way -- it cannot retroactively repair the already-captured scheduler-replay-20260922-095207.replay file, which will continue to show the same 61 --sim-verify mismatches when replayed (now fully explained, not a live bug) unless/until it is re-captured.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1842, bug-1611, bug-1613, bug-1779, bug-1759
- Gitea: #78