Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
42 lines
10 KiB
Markdown
42 lines
10 KiB
Markdown
---
|
|
id: "bug-1585"
|
|
status: "fixed"
|
|
status_updated: "2026-09-24"
|
|
status_note: ""
|
|
component: "scheduler"
|
|
severity: "unknown"
|
|
timestamp: "2026-08-19T02:47:00Z"
|
|
last_seen: "2026-09-19T09:00:00Z"
|
|
occurrences: 2
|
|
files: ["src/scheduler/src/core_events.c (XCE_CORE_EVENT_RESVALS_FULL/RESVALS_DELTA handlers -- root cause)", "src/scheduler/src/main.c (--record-snapshot-interval CLI wiring)", "src/scheduler/src/repl.c (banner clarification)", "src/scheduler/testsuite/test_resval_ownership.c (new regression test)", "src/scheduler/testsuite/test_core_events.c/test_core_full_cycle.c/test_driver_decision_cb.c/test_driver_session_ordering.c/test_driver_translate.c (dup_val link stub + stale-comment/fixture fix)"]
|
|
tags: ["scheduler", "sim", "repl", "replay", "snapshot", "jobs", "picker", "heap-use-after-free", "resval", "zarray", "ownership-bug", "sigsegv", "cli-flag", "record-snapshot-interval", "fully-fixed", "live-reproduced", "tdd"]
|
|
related: ["bug-1569", "bug-1560"]
|
|
legacy: true
|
|
---
|
|
|
|
# bug-1585: User-reported, live `xce_scheduler -i -s /tmp/x1.replay` REPL session against a real .replay file…
|
|
|
|
## Description
|
|
User-reported, live `xce_scheduler -i -s /tmp/x1.replay` REPL session against a real .replay file recorded from a live normal-mode deployment: `info summary`/`info worker all` show '0 jobs (0 pending, 0 running)' immediately after load, even though the scheduler log and xce_test confirm jobs were actually submitted/present during that recording. Startup also logs a WARN: `xce_sched_core_load_snapshot failed to resolve job picker plugin (rc=-2); picker rebuild will be skipped for this load`.
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Root cause
|
|
TWO stacked, independently-confirmed causes, both live-reproduced this session against a real controller+scheduler+xcecd_virtual deployment (peers localhost:19991, worker-1):
|
|
|
|
(A) FULLY ROOT-CAUSED AND FIXED -- a genuine heap-use-after-free crash, NOT the picker-resolve WARN (confirmed a red herring, per this entry's own prior analysis of xce_sched_core_load_snapshot()) and not purely a test-methodology gap. core_events.c's XCE_CORE_EVENT_RESVALS_FULL (line ~750) and XCE_CORE_EVENT_RESVALS_DELTA (line ~786) handlers did `zarray_put(&(zarray), idx, rv->val)` -- a SHALLOW store of the wire-decoded xce_val_t* pointer (confirmed via zarray_put_ex(), src/lib/zarray.c: `array->data[idx] = data[i]`, no copy). The zarray now owns that pointer and hands it to its killer (kill_sched_resval, schedobj.c) the next time that index is replaced. But driver.c's release_translated_payload() unconditionally frees the ENTIRE decoded ResValList/ResourceVal array (including that same rv->val) immediately after xce_core_push_events() returns, on EVERY drain, regardless of adoption. Result: the very next periodic resval report for the same worker+resource -- which happens every resource_sync interval (~1-15s) on literally any live deployment -- evicts the now-dangling first value via zarray_put(), crashing the scheduler with ASAN heap-use-after-free inside free_Value() (lib/controller.capnp.c:2898). Live-reproduced TWICE against a real deployment during this session's own repro attempts (first crash: PID 68752, silently lost because stderr went to /dev/null under start-scheduler.sh's nohup redirection; second crash, WITH stderr captured this time: PID 73919, full ASAN backtrace obtained -- xce_core_push_events -> zarray_put -> kill_sched_resval -> free_Value, use-after-free on a pointer release_translated_payload had freed one drain earlier). This exact hazard had ALREADY been noticed and explicitly documented as a 'PRE-EXISTING production ownership characteristic' in test_driver_translate.c's own test comment (task 5.1) but never fixed, because that test's kill_sched_resval is a no-op stub (its own comment wrongly generalizes 'fixtures are never torn down so these killers are never invoked' -- true for the OTHER kvmgr datakillers which fire only on core teardown, but NOT true for a zarray's killer, which fires on every REPLACEMENT of an occupied slot, not just teardown -- that blind spot is exactly why no existing test caught this). This crash means a live deployment's scheduler process is a near-certain crash within one or two resval-report cycles of any real session, corrupting/truncating whatever recording was in progress -- the practical dominant explanation for why a `.replay` file recorded from a real live session so often ends up missing jobs the operator knows were submitted.
|
|
|
|
(B) CONFIRMED BY DESIGN, NOT A DEFECT (bug-1569 precedent, re-verified live this session after A's fix) -- once the crash was fixed, a full clean recording session was captured live: submitted 3 jobs via xce_test while `xceadmin scheduler record start` was active, waited for real DISPATCH decisions, `xceadmin scheduler record stop`, then `xceadmin scheduler dump`. The dump showed the SnapshotEntry (initial) genuinely reflecting whatever core state existed at the exact instant `record start` was issued (6 pre-existing jobs in this session's case, since jobs from an earlier test batch were still live) -- ReplayEvents (jobsPush/jobUpdate/resvalsDelta) for state that changed AFTER that instant correctly appear later in the stream, and only become visible in the REPL after `step`/`continue` (querying `info jobs`/`info summary` immediately after `[sim] ready.`, as the user's own transcript shows, only ever shows the starting snapshot's baseline -- exactly bug-1569's own prior finding for resvals, now confirmed to apply identically to job counts). Live-confirmed via `step N`: job count correctly grew from 6 to 9 once the REPL stepped past the recorded jobsPush/dispatch events, with correct per-job states (RUN/PEND) matching xce_test's live joblist at that exact point. The REAL underlying operator-facing gap this session's own analysis (and tasks.md's task-9.3 review note) already flagged: `--record-snapshot-interval` had NO CLI wiring anywhere in main.c, even though the struct field (xce_scheduler_t.record_snapshot_interval) and its consumption logic (recording_events.c's sched_record_write_events(), task 9.3) were already fully implemented and unit-tested -- so an operator had no way to ever get a periodically-refreshed starting snapshot; a `.replay` file with no periodic checkpointing legitimately has exactly ONE SnapshotEntry (written at `record start` time), and the REPL correctly starting from that baseline is expected behavior, not a bug.
|
|
|
|
## Fix
|
|
(A) FIXED -- core_events.c's RESVALS_FULL/RESVALS_DELTA handlers now `dup_val(rv->val)` before `zarray_put()` (both call sites), matching the deep-copy-before-adopt convention this codebase already established for the identical hazard on the snapshot-load side (snapshot_load.c: `zarray_put(&(core->sharedInst), r->idx, dup_val(rv->val))` / `zarray_put(&(w->resvals), r->idx, dup_val(nv->val))`). `dup_val` (defined in snapshot_wire.c) is now also `extern`-declared and consumed by core_events.c. TDD: new src/scheduler/testsuite/test_resval_ownership.c (2 tests, wired into meson.build as 'scheduler resval ownership test') white-box-includes core.c/core_clock.c/core_events.c/driver.c with a REAL (non-stubbed) kill_sched_resval (schedobj.c's implementation verbatim) plus a real dup_val reimplementation -- RED confirmed the exact live ASAN backtrace (heap-use-after-free in free_Value via zarray_put's eviction path) on the pre-fix code; GREEN after the fix, with an added assertion the evicted-and-replaced resval reads back correctly (no data corruption, not just 'didn't crash'). The other 5 pre-existing white-box test files that #include core_events.c (test_core_events.c, test_core_full_cycle.c, test_driver_decision_cb.c, test_driver_session_ordering.c, test_driver_translate.c) each needed a small real dup_val reimplementation added (link-satisfying, same 'reimplement the one small function' idiom as their existing sched_resolve_job_picker copies) since core_events.c now references it; test_driver_translate.c's own `test_drain_translates_and_pushes_all_ten_op_types` additionally needed its full_val/delta_val fixtures to set `->which = XCE_VAL_NUMERIC` explicitly (a bare calloc() leaves `which` at 0 == XCE_VAL_STRING, which dup_val() now actually inspects -- exposed a latent test-fixture gap, not a production bug) and its stale 'PRE-EXISTING production ownership characteristic ... unsafe to dereference' comment was corrected to reflect the fix. Live-reproduced end-to-end post-fix: restarted the scheduler against the same live deployment, let 22+ periodic resval cycles for the same worker+resource pass with zero crash (previously guaranteed to crash on the 2nd cycle), then ran a full record-start/submit-jobs/record-stop/dump/REPL-step cycle cleanly. Full scheduler suite green (41/42, only bug-1560 unrelated). (B) FIXED -- `--record-snapshot-interval N` is now a real CLI flag on main.c's argp table (OPT_RECORD_SNAPSHOT_INTERVAL=262), mirroring `--record-dir`'s exact pattern: parsed with the same strtoull()+validation shape --sim-speed already uses, assigned to `sched.record_snapshot_interval` right alongside `sched.record_dir`. No dedicated unit test was added (main.c defines its own `main()`, so it cannot be white-box #included into a test TU without a symbol collision, and this codebase's own precedent -- --record-dir itself -- has zero dedicated unit test either, relying on code review + the consumption side's existing coverage); verified instead via a genuine live E2E run: restarted the scheduler with `--record-snapshot-interval 3`, submitted 5 jobs across a record start/stop cycle, and confirmed via `xceadmin scheduler dump` that THREE periodic 'SnapshotEntry (checkpoint)' entries now appear mid-stream (previously only ever one, at record-start), each fully populated (14 jobs) -- then loaded the file via `-i -s FILE` and confirmed `[sim] ready.` now shows all 14 jobs immediately, with zero stepping required. Also added a one-line repl_print_banner() clarification (repl.c): '[sim] (jobs above reflect this snapshot only -- step/continue to replay later events)', printed between the loaded-snapshot line and the ready-prompt line -- verified this doesn't break test_repl_execution_control.c's own banner assertion (a strstr() substring/ordering check, unaffected by an inserted line).
|
|
|
|
## Verification
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## References
|
|
- Related: bug-1569, bug-1560
|