Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
7.5 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | fix_commits | legacy | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1611 | fixed | 2026-09-24 | e2bb98c8, 984a6869 |
scheduler | unknown | 2026-08-21T15:10:00Z | 2026-09-19T14:00:00Z | 3 |
|
|
|
|
true |
bug-1611: Discovered analyzing a real 108-session replay (xce_scheduler -i -s…
Description
Discovered analyzing a real 108-session replay (xce_scheduler -i -s scheduler-replay-20260820-070103.replay, continue to EOF): sim_print_decision() (sim_loop.c) never printed a single DISPATCH or PREEMPT line across the entire replay -- only PEND -- even though 22 of the 30 jobs pushed at round 32 clearly transitioned to running (confirmed by ongoing JOB_UPDATE/JOB_RUSAGE ticks for jobids [0]-[21] every few rounds through round 107). An operator watching only decision output has zero confirmation those 22 jobs ever dispatched; they just silently stop appearing in PEND.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
Not yet root-caused with certainty. Plausible, benign explanation: in --sim/-s REPLAY mode, jobs already resolved (dispatched) in the ORIGINAL recorded run may be replayed purely as state deltas (JOB_UPDATE/JOB_RUSAGE) without the live picker/allocator re-deriving a fresh XCE_DECISION_DISPATCH -- only jobs still outstanding (PEND) get freshly re-decided every round, which is why PEND is the only decision op ever observed. Unconfirmed against sim_loop.c's actual replay-vs-live-decide branching; needs a source read of how sim_loop.c distinguishes replayed-decision playback from freshly-computed decisions before concluding this is 'working as designed' vs a real visibility gap.
Fix
Not yet fixed. If confirmed benign (dispatch legitimately has no fresh decision object in replay mode), the round-summary rollup being designed for bug-1609/1610 should not label its dispatch count misleadingly (a permanent 'X dispatch' column reading 0 for an entire replay looks like a bug even when it isn't) -- consider deriving a running-job delta from info-jobs-equivalent state instead of the decision tally for replay-mode display. If NOT benign (a real decision-visibility gap), this is a separate, more serious finding than the formatting issues in bug-1609/1610.
--- STATUS UPDATE 2026-08-24 (xce-audit-invariants scheduler audit) ---
Root cause confirmed and the literal mechanism is now built: sim_reconcile_unaccounted_jobs() (sim_loop.c:566-709, landed via scheduler-sim-decision-parity tasks 2.1-3, 2026-08-22) reconstructs and reports a decision for a RUN-state job whose DISPATCH decision was only in the recorded replay stream. BUT its trigger gate (sim_loop.c:592) requires JOB_IS_RUN(job->state) before searching, and the search only scans the CURRENT round's decoded_session (decoded fresh at sim_loop.c:871, freed at line 999 -- no cross-round persistence). So reconciliation only fires when a job's DISPATCH decision and its RUN confirmation land in the exact same replay round. This bug's own original scenario (dispatch at round 32, RUN confirmation through round 107) never matches -- the round-32 DISPATCH decision is gone by the time the job reaches RUN state. The scheduler-sim-decision-parity spec's own task 4.3 implementation notes explicitly acknowledge this residual does NOT fix bug-1611's dominant real-world manifestation. A follow-up spec already exists for this exact residual: .kiro/specs/scheduler-sim-eager-reconciliation/ (created 2026-08-22, phase=requirements-generated, ready_for_implementation=false -- design/tasks not yet generated). Its diagnosis: decouple allocation CAPTURE from the JOB_IS_RUN gate (capture as soon as a DISPATCH decision is seen for a job with job->accounts == NULL, regardless of state), leaving sched_account_jobs()'s existing JOB_IS_RUN + job->accounts != NULL gate to keep deferring actual charging until state transitions to RUN. Fix location: sim_reconcile_unaccounted_jobs() (sim_loop.c:566-709), trigger gate at line 592, plus its ownership-transfer logic (609-693) needs a double-capture guard since capture would no longer imply state==RUN. Do NOT fix inline as a drive-by patch per scheduler-driver-equivalence.md's testing discipline -- promote scheduler-sim-eager-reconciliation to design/tasks instead.
--- STATUS UPDATE 2026-08-27 (scheduler-sim-eager-reconciliation task 1.1) ---
Core production fix landed: sim_reconcile_unaccounted_jobs() trigger gate (sim_loop.c:592) had !JOB_IS_RUN(job->state) removed, leaving job->accounts == NULL as the sole capture-eligibility gate -- exactly the diagnosis from the 2026-08-24 status update. Match/capture/report mechanics (609-693) were confirmed already safe for early (pre-RUN) capture without changes (see scheduler-sim-eager-reconciliation/research.md's uncharge-safety research log -- sched_uncharge_account() has zero production callers, accounting is reset-and-resum not incremental, so no double-capture guard beyond the existing job->accounts != NULL check was needed). Old test test_reconcile_skips_non_run_state_job (which asserted the bug as correct behavior) replaced with test_reconcile_captures_non_run_state_job_when_dispatch_recorded (inverted expectation). RED/GREEN verified: new test failed under the old gate (job->allocs stayed NULL, confirmed via a real build+run before the fix), then passed after the one-line fix; full sched_sim_reconcile_unaccounted_test suite green (10/10, all prior regression tests unchanged). This is task 1.1 of the spec only -- task 1.2 (capture-safety unit coverage), task 2.1 (resInUse deferral integration test), and task 3.x (contention/--sim-verify/full-regression validation) are still pending before the spec itself is implementation-complete. CORRECTED STATUS UPDATE 2026-09-19 (superseding the immediately-preceding note in this same entry, which itself worked from stale premises): scheduler-sim-eager-reconciliation (spec completed 2026-09-11, BEFORE this session's audit ran) already decoupled sim_reconcile_unaccounted_jobs()'s trigger from the JOB_IS_RUN gate -- the function now fires on job->accounts == NULL alone, regardless of PEND/RUN state (sim_loop.c:560-572), so a job's DISPATCH decision is captured in the SAME round it is recorded, never deferred until a later, separately-recorded RUN-state confirmation event. This closes the exact real-world scenario that motivated this bug (dispatch round 32, confirmation round 107) for good -- the previous note's claimed "residual gap" (a job already-RUN before the replay window starts) is not a real gap either: such a job's allocation is populated directly from the snapshot at load time (xce_sched_core_load_snapshot(), confirmed field-complete for job->allocs/accounts by this same audit's dimension-4 pass), so job->accounts is already non-NULL immediately post-snapshot-load and such a job is never a reconciliation candidate in the first place. Combined with scheduler-sim-reconcile-terminal-guard (bug-1764, excludes terminal jobs from false rematch) and scheduler-sim-live-allocation (the live picker/allocator now handles most jobs directly, reconciliation is a narrow safety net), this entry is FULLY FIXED. STATUS: FIXED, not PARTIALLY FIXED.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1764, bug-1613, bug-1610, bug-1609