Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
10 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1585 | fixed | 2026-09-24 | scheduler | unknown | 2026-08-19T02:47:00Z | 2026-09-19T09:00:00Z | 2 |
|
|
|
true |
bug-1585: User-reported, live xce_scheduler -i -s /tmp/x1.replay REPL session against a real .replay file…
Description
User-reported, live xce_scheduler -i -s /tmp/x1.replay REPL session against a real .replay file recorded from a live normal-mode deployment: info summary/info worker all show '0 jobs (0 pending, 0 running)' immediately after load, even though the scheduler log and xce_test confirm jobs were actually submitted/present during that recording. Startup also logs a WARN: xce_sched_core_load_snapshot failed to resolve job picker plugin (rc=-2); picker rebuild will be skipped for this load.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
TWO stacked, independently-confirmed causes, both live-reproduced this session against a real controller+scheduler+xcecd_virtual deployment (peers localhost:19991, worker-1):
(A) FULLY ROOT-CAUSED AND FIXED -- a genuine heap-use-after-free crash, NOT the picker-resolve WARN (confirmed a red herring, per this entry's own prior analysis of xce_sched_core_load_snapshot()) and not purely a test-methodology gap. core_events.c's XCE_CORE_EVENT_RESVALS_FULL (line ~750) and XCE_CORE_EVENT_RESVALS_DELTA (line ~786) handlers did zarray_put(&(zarray), idx, rv->val) -- a SHALLOW store of the wire-decoded xce_val_t* pointer (confirmed via zarray_put_ex(), src/lib/zarray.c: array->data[idx] = data[i], no copy). The zarray now owns that pointer and hands it to its killer (kill_sched_resval, schedobj.c) the next time that index is replaced. But driver.c's release_translated_payload() unconditionally frees the ENTIRE decoded ResValList/ResourceVal array (including that same rv->val) immediately after xce_core_push_events() returns, on EVERY drain, regardless of adoption. Result: the very next periodic resval report for the same worker+resource -- which happens every resource_sync interval (~1-15s) on literally any live deployment -- evicts the now-dangling first value via zarray_put(), crashing the scheduler with ASAN heap-use-after-free inside free_Value() (lib/controller.capnp.c:2898). Live-reproduced TWICE against a real deployment during this session's own repro attempts (first crash: PID 68752, silently lost because stderr went to /dev/null under start-scheduler.sh's nohup redirection; second crash, WITH stderr captured this time: PID 73919, full ASAN backtrace obtained -- xce_core_push_events -> zarray_put -> kill_sched_resval -> free_Value, use-after-free on a pointer release_translated_payload had freed one drain earlier). This exact hazard had ALREADY been noticed and explicitly documented as a 'PRE-EXISTING production ownership characteristic' in test_driver_translate.c's own test comment (task 5.1) but never fixed, because that test's kill_sched_resval is a no-op stub (its own comment wrongly generalizes 'fixtures are never torn down so these killers are never invoked' -- true for the OTHER kvmgr datakillers which fire only on core teardown, but NOT true for a zarray's killer, which fires on every REPLACEMENT of an occupied slot, not just teardown -- that blind spot is exactly why no existing test caught this). This crash means a live deployment's scheduler process is a near-certain crash within one or two resval-report cycles of any real session, corrupting/truncating whatever recording was in progress -- the practical dominant explanation for why a .replay file recorded from a real live session so often ends up missing jobs the operator knows were submitted.
(B) CONFIRMED BY DESIGN, NOT A DEFECT (bug-1569 precedent, re-verified live this session after A's fix) -- once the crash was fixed, a full clean recording session was captured live: submitted 3 jobs via xce_test while xceadmin scheduler record start was active, waited for real DISPATCH decisions, xceadmin scheduler record stop, then xceadmin scheduler dump. The dump showed the SnapshotEntry (initial) genuinely reflecting whatever core state existed at the exact instant record start was issued (6 pre-existing jobs in this session's case, since jobs from an earlier test batch were still live) -- ReplayEvents (jobsPush/jobUpdate/resvalsDelta) for state that changed AFTER that instant correctly appear later in the stream, and only become visible in the REPL after step/continue (querying info jobs/info summary immediately after [sim] ready., as the user's own transcript shows, only ever shows the starting snapshot's baseline -- exactly bug-1569's own prior finding for resvals, now confirmed to apply identically to job counts). Live-confirmed via step N: job count correctly grew from 6 to 9 once the REPL stepped past the recorded jobsPush/dispatch events, with correct per-job states (RUN/PEND) matching xce_test's live joblist at that exact point. The REAL underlying operator-facing gap this session's own analysis (and tasks.md's task-9.3 review note) already flagged: --record-snapshot-interval had NO CLI wiring anywhere in main.c, even though the struct field (xce_scheduler_t.record_snapshot_interval) and its consumption logic (recording_events.c's sched_record_write_events(), task 9.3) were already fully implemented and unit-tested -- so an operator had no way to ever get a periodically-refreshed starting snapshot; a .replay file with no periodic checkpointing legitimately has exactly ONE SnapshotEntry (written at record start time), and the REPL correctly starting from that baseline is expected behavior, not a bug.
Fix
(A) FIXED -- core_events.c's RESVALS_FULL/RESVALS_DELTA handlers now dup_val(rv->val) before zarray_put() (both call sites), matching the deep-copy-before-adopt convention this codebase already established for the identical hazard on the snapshot-load side (snapshot_load.c: zarray_put(&(core->sharedInst), r->idx, dup_val(rv->val)) / zarray_put(&(w->resvals), r->idx, dup_val(nv->val))). dup_val (defined in snapshot_wire.c) is now also extern-declared and consumed by core_events.c. TDD: new src/scheduler/testsuite/test_resval_ownership.c (2 tests, wired into meson.build as 'scheduler resval ownership test') white-box-includes core.c/core_clock.c/core_events.c/driver.c with a REAL (non-stubbed) kill_sched_resval (schedobj.c's implementation verbatim) plus a real dup_val reimplementation -- RED confirmed the exact live ASAN backtrace (heap-use-after-free in free_Value via zarray_put's eviction path) on the pre-fix code; GREEN after the fix, with an added assertion the evicted-and-replaced resval reads back correctly (no data corruption, not just 'didn't crash'). The other 5 pre-existing white-box test files that #include core_events.c (test_core_events.c, test_core_full_cycle.c, test_driver_decision_cb.c, test_driver_session_ordering.c, test_driver_translate.c) each needed a small real dup_val reimplementation added (link-satisfying, same 'reimplement the one small function' idiom as their existing sched_resolve_job_picker copies) since core_events.c now references it; test_driver_translate.c's own test_drain_translates_and_pushes_all_ten_op_types additionally needed its full_val/delta_val fixtures to set ->which = XCE_VAL_NUMERIC explicitly (a bare calloc() leaves which at 0 == XCE_VAL_STRING, which dup_val() now actually inspects -- exposed a latent test-fixture gap, not a production bug) and its stale 'PRE-EXISTING production ownership characteristic ... unsafe to dereference' comment was corrected to reflect the fix. Live-reproduced end-to-end post-fix: restarted the scheduler against the same live deployment, let 22+ periodic resval cycles for the same worker+resource pass with zero crash (previously guaranteed to crash on the 2nd cycle), then ran a full record-start/submit-jobs/record-stop/dump/REPL-step cycle cleanly. Full scheduler suite green (41/42, only bug-1560 unrelated). (B) FIXED -- --record-snapshot-interval N is now a real CLI flag on main.c's argp table (OPT_RECORD_SNAPSHOT_INTERVAL=262), mirroring --record-dir's exact pattern: parsed with the same strtoull()+validation shape --sim-speed already uses, assigned to sched.record_snapshot_interval right alongside sched.record_dir. No dedicated unit test was added (main.c defines its own main(), so it cannot be white-box #included into a test TU without a symbol collision, and this codebase's own precedent -- --record-dir itself -- has zero dedicated unit test either, relying on code review + the consumption side's existing coverage); verified instead via a genuine live E2E run: restarted the scheduler with --record-snapshot-interval 3, submitted 5 jobs across a record start/stop cycle, and confirmed via xceadmin scheduler dump that THREE periodic 'SnapshotEntry (checkpoint)' entries now appear mid-stream (previously only ever one, at record-start), each fully populated (14 jobs) -- then loaded the file via -i -s FILE and confirmed [sim] ready. now shows all 14 jobs immediately, with zero stepping required. Also added a one-line repl_print_banner() clarification (repl.c): '[sim] (jobs above reflect this snapshot only -- step/continue to replay later events)', printed between the loaded-snapshot line and the ready-prompt line -- verified this doesn't break test_repl_execution_control.c's own banner assertion (a strstr() substring/ordering check, unaffected by an inserted line).
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1569, bug-1560