Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
6.3 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | fix_commits | legacy | gitea_issue | gitea_synced_status | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1846 | fixed | 2026-09-24 | 77f047f2, a9feded8, ac033f47 |
scheduler | unknown | 2026-09-23T10:05:00Z | 2026-09-23T10:05:00Z | 1 |
|
|
|
|
true | 84 | fixed |
bug-1846: Same replay session as bug-1844/1845 (scheduler-replay-20260923-085356.replay): after excluding…
Description
OPEN, NOT YET ROOT-CAUSED. Same replay session as bug-1844/1845 (scheduler-replay-20260923-085356.replay): after excluding instid (bug-1844) and reason-detail (bug-1845), --sim-verify still shows one job (jobid=0-8-1790154325-0[39] in the observed capture) genuinely DISPATCHING one round earlier in replay than in the original live recording -- not a false-positive field comparison, a real positional divergence: replay's round begin_round=...436582151823 dispatches a 13-job batch (jobs [27]..[39]) while the SAME round in the original recording only dispatched 12 (jobs [27]..[38]); job [39]'s recorded dispatch instead lands in the NEXT round, cascading into ~40+ downstream positional MISMATCH lines for that round pair alone.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
CONFIRMED (2026-09-23, via temporary per-decision diagnostics in the --sim-verify drain loop, since reverted) -- two distinct things. (1) ROOT-CAUSED + FIXED: match_job_host_limit_() (limit.c) runs on EVERY per-host allocation attempt (limit_prepare_allocation() via plugin_prepare_allocation()) and unconditionally xlist_store()d the matched vres into obj->hlimits[worker->idx]; obj->hlimits is only rebuilt by limit_job_election() on a configver bump. A job that kept pending on the same host accumulated one extra copy per attempt; sched_fill_limit_requests() staged every copy into ctx->alloc_req.host and seqjob's make_host_alloc() built a host alloc carrying all of them. In the captured replay the LIVE recording itself shows it (pc13v host allocs with mem+slots+70/79/94/112/136/175 identical limit=limits2/u=rshen/q=xce/p= entries, resetting only on a config push) -- i.e. a PRODUCTION bug, not sim-only. make_job_account() makes one res_account per entry and sched_update_res_accounts() charges each, so the host limit was charged (and later released) N-fold. Replay had the same accumulation with different counts, which is what --sim-verify flagged. (2) ROOT-CAUSED (2026-09-23): the 9 live rounds cut off after 11-12 decisions were each ended by a mid-round scheduler REJOIN. sched_publish_retry_loop() (scheduler.c) calls sched_check_pending_rejoin() after every successful publish -- i.e. inside the driver's decision-drain loop -- and after a client-library reconnect that issues xcecli_join(). The controller answers every join (on_scheduler_join(), controller/sched.c) with xcesched_push_configs() + xcesched_push_workers() + xcesched_push_jobs(). handler.c applied the config push synchronously on the handler thread (sched_init_config() -> sched_init_queues() -> builtin_fcfs on_queue_update()), which nulls priv->cursor when it reinserts the queue the cursor rests on; next_job() then returns NULL and the round ends. Evidence from the capture (temporary event-stream diagnostics, reverted): every truncated round is followed by a workersPush + full 50-job jobsPush (ntotal=50, pos=0, the join's resync -- e.g. jobs just dispatched reported back as PEND), every full-length dispatching round has none, and the config push itself never appears because it bypassed the pending-update queue the recorder writes from. Rejoins came roughly every ~100s (the configver bump also explains the periodic reset of part (1)'s duplicate counts). The underlying connection churn is a separate issue (see bug-1812/1813's pc13v reconnect/resync storm).
Fix
PARTIAL (2026-09-23, TDD). limit.c: new hlimits_contains_() pointer-identity check (vres is interned per name in privdata->instances) -- match_job_host_limit_() now stores a matched vres into obj->hlimits[worker] only once. New regression test test_limit_host_dedup.c (sched_limit_host_dedup_test, meson suite 'scheduler'): 5 plugin_prepare_allocation() attempts for one job on one worker must leave hlimits[worker]->count==1 and exactly one staged copy of the vres in alloc_req.host -- verified RED before the fix (Expected 1 Was 2 on attempt 2) and GREEN after. Replay re-run after the fix: replay-side host allocs are now always 3 entries; remaining DISPATCH alloc mismatches are the recording's own inflated entries (recorded with the buggy binary) and will disappear only with a fresh capture. PART (2) FIXED (2026-09-23, TDD, user-chosen design 'defer to round edge'): handler.c's configpushmsg case now calls new sched_stash_config() (parks the Conf4Sched in new xce_scheduler_t.pending_config under pending_config_lock, freeing any older unapplied push) instead of sched_init_config(); main.c's loop calls new sched_apply_pending_config() before each xce_sched_driver_run_normal_mode_round(), which runs the unchanged sched_init_config() strictly between sessions; sched_fini() frees a never-applied push. Same deferral direction as bug-1813's array-job fix. Tests: test_handlers.c's test_config_push_is_stashed_not_applied_on_handler_thread (RED: sched_init_config called 1x on the handler thread; GREEN after) and new test_config_push_deferred.c (sched_config_push_deferred_test: real scheduler_req_handler() + real builtin_fcfs mid-session -- RED: next_job() returned NULL right after the config push, job B never considered; GREEN: session walks B and C, configver unchanged until sched_apply_pending_config() applies it once). Known remaining gap (pre-existing, not addressed): a config push is still not written to a recording, so a replay spanning a config CHANGE won't see it.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1843, bug-1844, bug-1845, bug-1611, bug-1613
- Gitea: #84