Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
46 lines
7.5 KiB
Markdown
46 lines
7.5 KiB
Markdown
---
|
|
id: "bug-1733"
|
|
status: "fixed"
|
|
status_updated: "2026-09-24"
|
|
status_note: ""
|
|
component: "lib"
|
|
severity: "crash"
|
|
timestamp: "2026-08-29T06:30:00Z"
|
|
last_seen: "2026-09-19T09:00:00Z"
|
|
occurrences: 2
|
|
files: ["src/lib/vtaskmgr.c (vtask_enq", "line 517)", "src/lib/dataq.c (dataq_enq)"]
|
|
tags: ["controller", "vtaskmgr", "use-after-free", "asan-confirmed", "concurrency", "high-severity", "live-crash", "found-via-bug-1731-repro"]
|
|
related: ["bug-1298", "bug-1415", "bug-1714", "bug-1731"]
|
|
legacy: true
|
|
---
|
|
|
|
# bug-1733: AddressSanitizer: heap-use-after-free WRITE of size 8 at vtask_enq (src/lib/vtaskmgr.c:517)…
|
|
|
|
## Description
|
|
AddressSanitizer: heap-use-after-free WRITE of size 8 at vtask_enq (src/lib/vtaskmgr.c:517) -- crashed 2 of 3 controllers in a live 3-node HA cluster under a heavy burst-array-job workload (6 concurrent `jobsub -a 50` submissions, 300 near-instant job elements, driving high-frequency WAL-replication-ACK traffic). Found while chasing bug-1714/bug-1731's scheduler crash under ASAN -- the scheduler itself did not crash in this run, but two controllers did, independently, ~7 and ~14 minutes into the run, with byte-for-byte identical ASAN stacks (different PIDs/addresses).
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Root cause
|
|
vtask_enq() (vtaskmgr.c:497-526) calls dataq_enq(&(vtaskmgr->taskq), &(task->link)) at line 514, which PUBLISHES the task onto the shared taskq (under q->lock, released on return) -- immediately making it visible to any idle vcpu worker thread. Only AFTER that publish does line 517 write `task->owner = vtaskmgr;`. A concurrent vcpu thread can dequeue the just-published task, run it to completion (run_task -> the task's own callback, here peer_ctrl_vtask_entry), and vcpu_entry (vtaskmgr.c:106-109) can then call vtask_fini(t)+free(t) -- all BEFORE the producer thread reaches line 517 -- so the producer's write lands on already-freed memory. ASAN's two captured stacks confirm exactly this: thread T6 allocates+enqueues the task (persist_apply_pump_kick -> vtask_alloc/vtask_enq, called via the replication-ACK path repl_ack_decode_and_dispatch -> repl_on_ack -> commit_on_ack -> commit_resolve_pending -> persist_apply_pump_kick), while thread T5 has ALREADY freed the same 48-byte vtask_t region inside vcpu_entry by the time T6's own `task->owner = vtaskmgr` write executes. This is a publish-before-init race -- the task is made visible to concurrent consumers before its own initialization (the `owner` field) completes. Notably, the code immediately above (lines 508-512, `endpoint_ref(ep)`) already gets this right, per an existing bug-1298 comment explicitly calling out 'acquire BEFORE dataq_enq publishes the task to the dispatcher' -- line 517's `owner` assignment simply doesn't follow that same established discipline. Reproduces reliably under this project's burst-array live-HA workload (both controller instances that received the assignment crashed on it during a single ~30-minute run); did not reproduce in the test suite's pool-less fixtures (matches the same class of gap bug-1415 documents for a different vtaskmgr race -- production n_vcpus>1 pool concurrency needed to manifest, not exercised by tests that never call commit_set_pool).
|
|
|
|
## Fix
|
|
FIXED and TDD-verified (RED attempted, GREEN confirmed). Root cause and fix exactly as diagnosed: vtask_enq() (src/lib/vtaskmgr.c) now sets `task->owner = vtaskmgr;` BEFORE calling dataq_enq(), mirroring the existing bug-1298 endpoint_ref-before-publish discipline immediately above it in the same function; the post-enqueue success block now only calls vtaskmgr_wakeup(vtaskmgr) (the owner write no longer straddles the publish that makes the task visible to concurrent vcpu workers).
|
|
|
|
New stress test test_vtask_enq_race_stress (src/lib/testsuite/test_vtask_enq_race.c, wired into libxce_test_src in src/lib/meson.build and testsuite/test.c's extern/TEST() list as test_vtask_enq_race, right after test_vtask_admission): a real vtaskmgr_t pool (8 vcpus) with 16 producer threads each calling vtask_enq_classified(mgr, task, VTASK_LIVENESS) in a tight loop (8000 iterations each, 128,000 tasks total) with true no-op entry callbacks (atomic increment only, matching test_vtask_admission.c's live_task_entry) -- maximizing the odds a consumer vcpu dequeues, runs, and frees a task before the producer's own next instruction lands, exactly the bug-1733 race window.
|
|
|
|
RED ATTEMPT: this is a genuine data race confined to the two-line window between dataq_enq()'s publish and the owner write, which cannot be forced deterministically from a test without modifying production code (unlike test_vtask_admission.c's fully deterministic cbuf-gated design). Extensive tuning was attempted against the UNMODIFIED vtaskmgr.c before applying the fix -- over a dozen configurations across 60+ manual runs: the default stress shape (6 vcpus/10 producers/4000 iters), bumped concurrency (16v/24p/6000iters, 32v/48p/20000iters), taskset-pinned to 4 cores and to a single core, background CPU-load saturation (22-44 concurrent `yes` processes churning all 22 host cores to force involuntary scheduler preemption right after dataq_enq() returns), and a few-producers/many-idle-vcpus variant (4p/16v/100000iters) meant to keep the dispatcher in a tight dequeue-polling loop. None reproduced the ASAN heap-use-after-free within these bounded attempts, including the official `nix develop ./src --command bash src/tools/xce-build.sh lib asan` run against the unmodified file (the new test itself PASSED cleanly there; the run's only SIGABRT came from a pre-existing, unrelated LeakSanitizer report in test_xcealloc.c -- 456 bytes/29 allocations via strdup in make_alloc/make_host_alloc -- confirmed present byte-for-byte identically both before and after this fix, so not a regression from this change). This is consistent with this entry's own root-cause note that the race needed real multi-node HA production load (WAL-replication-ACK bursts under `jobsub -a 50` burst-array-job load) to manifest, and that it also did not reproduce in prior pool-less test fixtures (same class of gap bug-1415 documents). The original ASAN report captured in this entry (heap-use-after-free WRITE of size 8 at vtask_enq ../lib/vtaskmgr.c:517, freed inside vcpu_entry, matching stacks on 2 of 3 live controllers) remains the authoritative confirmation the bug is real; a stress test that does not happen to hit a hard-to-trigger race in bounded iterations is a weaker regression guard than one that does, but per this fix's own instructions is not a reason to withhold the fix.
|
|
|
|
GREEN: after the fix, the new stress test passes cleanly and repeatably -- 8 consecutive standalone runs, zero ASAN reports -- at both the isolated and full-suite (`xce-build.sh lib asan --incremental`) level.
|
|
|
|
Full controller-suite regression (`xce-build.sh controller asan --incremental`, the real consumer of vtask_enq_classified via persist_apply_pump_kick/commit.c): 1213 tests, 2 failures -- both the pre-existing, already-logged bug-1561 timing-collision flake in test_config_repl.c (test_conf_trim_keeps_only_n_versions, test_conf_get_saved_tag_no_config), unrelated to this change. The specific real vtask_enq_classified callers this fix touches indirectly -- test_ha_dispatch_integration.c, test_ha_vep_teardown.c, test_apply_pump.c, test_apply_pump_integration.c, test_commit_admission_pool.c, test_commit_resolve.c -- all pass with zero failures. No regressions.
|
|
|
|
## Verification
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## References
|
|
- Related: bug-1298, bug-1415, bug-1714, bug-1731
|