---
id: "bug-1733"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "crash"
timestamp: "2026-08-29T06:30:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/lib/vtaskmgr.c (vtask_enq", "line 517)", "src/lib/dataq.c (dataq_enq)"]
tags: ["controller", "vtaskmgr", "use-after-free", "asan-confirmed", "concurrency", "high-severity", "live-crash", "found-via-bug-1731-repro"]
related: ["bug-1298", "bug-1415", "bug-1714", "bug-1731"]
legacy: true
---

# bug-1733: AddressSanitizer: heap-use-after-free WRITE of size 8 at vtask_enq (src/lib/vtaskmgr.c:517)…

## Description
AddressSanitizer: heap-use-after-free WRITE of size 8 at vtask_enq (src/lib/vtaskmgr.c:517) -- crashed 2 of 3 controllers in a live 3-node HA cluster under a heavy burst-array-job workload (6 concurrent `jobsub -a 50` submissions, 300 near-instant job elements, driving high-frequency WAL-replication-ACK traffic). Found while chasing bug-1714/bug-1731's scheduler crash under ASAN -- the scheduler itself did not crash in this run, but two controllers did, independently, ~7 and ~14 minutes into the run, with byte-for-byte identical ASAN stacks (different PIDs/addresses).

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
vtask_enq() (vtaskmgr.c:497-526) calls dataq_enq(&(vtaskmgr->taskq), &(task->link)) at line 514, which PUBLISHES the task onto the shared taskq (under q->lock, released on return) -- immediately making it visible to any idle vcpu worker thread. Only AFTER that publish does line 517 write `task->owner = vtaskmgr;`. A concurrent vcpu thread can dequeue the just-published task, run it to completion (run_task -> the task's own callback, here peer_ctrl_vtask_entry), and vcpu_entry (vtaskmgr.c:106-109) can then call vtask_fini(t)+free(t) -- all BEFORE the producer thread reaches line 517 -- so the producer's write lands on already-freed memory. ASAN's two captured stacks confirm exactly this: thread T6 allocates+enqueues the task (persist_apply_pump_kick -> vtask_alloc/vtask_enq, called via the replication-ACK path repl_ack_decode_and_dispatch -> repl_on_ack -> commit_on_ack -> commit_resolve_pending -> persist_apply_pump_kick), while thread T5 has ALREADY freed the same 48-byte vtask_t region inside vcpu_entry by the time T6's own `task->owner = vtaskmgr` write executes. This is a publish-before-init race -- the task is made visible to concurrent consumers before its own initialization (the `owner` field) completes. Notably, the code immediately above (lines 508-512, `endpoint_ref(ep)`) already gets this right, per an existing bug-1298 comment explicitly calling out 'acquire BEFORE dataq_enq publishes the task to the dispatcher' -- line 517's `owner` assignment simply doesn't follow that same established discipline. Reproduces reliably under this project's burst-array live-HA workload (both controller instances that received the assignment crashed on it during a single ~30-minute run); did not reproduce in the test suite's pool-less fixtures (matches the same class of gap bug-1415 documents for a different vtaskmgr race -- production n_vcpus>1 pool concurrency needed to manifest, not exercised by tests that never call commit_set_pool).

## Fix
FIXED and TDD-verified (RED attempted, GREEN confirmed). Root cause and fix exactly as diagnosed: vtask_enq() (src/lib/vtaskmgr.c) now sets `task->owner = vtaskmgr;` BEFORE calling dataq_enq(), mirroring the existing bug-1298 endpoint_ref-before-publish discipline immediately above it in the same function; the post-enqueue success block now only calls vtaskmgr_wakeup(vtaskmgr) (the owner write no longer straddles the publish that makes the task visible to concurrent vcpu workers).

New stress test test_vtask_enq_race_stress (src/lib/testsuite/test_vtask_enq_race.c, wired into libxce_test_src in src/lib/meson.build and testsuite/test.c's extern/TEST() list as test_vtask_enq_race, right after test_vtask_admission): a real vtaskmgr_t pool (8 vcpus) with 16 producer threads each calling vtask_enq_classified(mgr, task, VTASK_LIVENESS) in a tight loop (8000 iterations each, 128,000 tasks total) with true no-op entry callbacks (atomic increment only, matching test_vtask_admission.c's live_task_entry) -- maximizing the odds a consumer vcpu dequeues, runs, and frees a task before the producer's own next instruction lands, exactly the bug-1733 race window.

RED ATTEMPT: this is a genuine data race confined to the two-line window between dataq_enq()'s publish and the owner write, which cannot be forced deterministically from a test without modifying production code (unlike test_vtask_admission.c's fully deterministic cbuf-gated design). Extensive tuning was attempted against the UNMODIFIED vtaskmgr.c before applying the fix -- over a dozen configurations across 60+ manual runs: the default stress shape (6 vcpus/10 producers/4000 iters), bumped concurrency (16v/24p/6000iters, 32v/48p/20000iters), taskset-pinned to 4 cores and to a single core, background CPU-load saturation (22-44 concurrent `yes` processes churning all 22 host cores to force involuntary scheduler preemption right after dataq_enq() returns), and a few-producers/many-idle-vcpus variant (4p/16v/100000iters) meant to keep the dispatcher in a tight dequeue-polling loop. None reproduced the ASAN heap-use-after-free within these bounded attempts, including the official `nix develop ./src --command bash src/tools/xce-build.sh lib asan` run against the unmodified file (the new test itself PASSED cleanly there; the run's only SIGABRT came from a pre-existing, unrelated LeakSanitizer report in test_xcealloc.c -- 456 bytes/29 allocations via strdup in make_alloc/make_host_alloc -- confirmed present byte-for-byte identically both before and after this fix, so not a regression from this change). This is consistent with this entry's own root-cause note that the race needed real multi-node HA production load (WAL-replication-ACK bursts under `jobsub -a 50` burst-array-job load) to manifest, and that it also did not reproduce in prior pool-less test fixtures (same class of gap bug-1415 documents). The original ASAN report captured in this entry (heap-use-after-free WRITE of size 8 at vtask_enq ../lib/vtaskmgr.c:517, freed inside vcpu_entry, matching stacks on 2 of 3 live controllers) remains the authoritative confirmation the bug is real; a stress test that does not happen to hit a hard-to-trigger race in bounded iterations is a weaker regression guard than one that does, but per this fix's own instructions is not a reason to withhold the fix.

GREEN: after the fix, the new stress test passes cleanly and repeatably -- 8 consecutive standalone runs, zero ASAN reports -- at both the isolated and full-suite (`xce-build.sh lib asan --incremental`) level.

Full controller-suite regression (`xce-build.sh controller asan --incremental`, the real consumer of vtask_enq_classified via persist_apply_pump_kick/commit.c): 1213 tests, 2 failures -- both the pre-existing, already-logged bug-1561 timing-collision flake in test_config_repl.c (test_conf_trim_keeps_only_n_versions, test_conf_get_saved_tag_no_config), unrelated to this change. The specific real vtask_enq_classified callers this fix touches indirectly -- test_ha_dispatch_integration.c, test_ha_vep_teardown.c, test_apply_pump.c, test_apply_pump_integration.c, test_commit_admission_pool.c, test_commit_resolve.c -- all pass with zero failures. No regressions.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1298, bug-1415, bug-1714, bug-1731