myxce/.wolf/bugs/bug-1415.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	legacy
bug-1415	fixed	2026-09-24		controller	unknown	2026-07-04T19:30:00.000Z	2026-09-19T09:00:00Z	1	src/controller/commit.c (commit_resolve_pending commit_resolve/dispatch_continuation) src/controller/persist.c (persist_apply_pump_kick persist_apply_pump_run)	raft-apply-decoupling task-6.2 vtaskmgr race-condition orphaned-registration lost-reply commit_resolve_pending persist_apply_pump_kick self-healing	bug-1409	true

bug-1415: raft-apply-decoupling task 6.2 review finding (NOT reproduced under the test suite's pool-less…

Description

raft-apply-decoupling task 6.2 review finding (NOT reproduced under the test suite's pool-less fixtures, NOT fixed): a real vtaskmgr pool (n_vcpus > 1) can run the commit continuation's dispatch (job_commit_resolve_adapter -> jobsubmit_commit_continuation -> persist_register_pending_apply) and the apply pump's kick (persist_apply_pump_kick -> apply_pump_drain -> persist_apply_pump_run) on two DIFFERENT worker threads with no completion-order guarantee, even though commit_resolve_pending() enqueues them in that order. If the apply-pump task happens to finish first, persist_take_pending_apply() finds nothing registered yet for that lsn (0, correct no-op per its own contract), applies the entry, and advances last_applied_lsn past it -- then when the continuation task finally runs and calls persist_register_pending_apply() for that now-already-applied lsn, the registration is permanently orphaned: the walk only ever revisits [last_applied_lsn+1..committed_lsn], so nothing will ever call persist_take_pending_apply() for that lsn again. Net effect: the job is correctly constructed and visible in c->jobmgr, but the client that submitted it never receives a reply -- silently, forever.

Environment

Not recorded (migrated from buglog.json).

Reproduction

Not recorded (migrated from buglog.json).

Root cause

commit_resolve_pending() (commit.c ~793-835, landed by task 1.2/6.1) resolves every ready continuation via commit_resolve() -> dispatch_continuation(), which for a real pool (ce->pool != NULL) enqueues the continuation onto vtaskmgr via vtask_enq_classified(VTASK_LIVE_BLOCK) and returns WITHOUT waiting for it to run; commit_resolve_pending() then calls persist_apply_pump_kick(ce->p, ce->pool), which enqueues a SECOND, independent VTASK_LIVE_BLOCK task onto the SAME pool. vtaskmgr.c's dispatcher (vdispatcher_entry) hands queued tasks to whichever vcpu worker thread is next idle (pick_idle_vcpu/pick_eligible_task) -- with n_vcpus > 1 (the only realistic production configuration; every test fixture in this spec deliberately never calls commit_set_pool, so ce->pool stays NULL and both dispatches run synchronously inline instead, which is why no test in this suite reproduces the race), two tasks enqueued back-to-back can be picked up by two different idle vcpus at nearly the same time, and their COMPLETION order depends on each task's own runtime (a disk-reading WAL-scan-and-apply walk vs. a calloc+mutex-guarded list insert) -- not their enqueue order. This is a pre-existing structural gap from task 1.2 (async continuation dispatch) + task 6.1 (apply-pump-kick wired into the same choke point, same pool) that had no observable consequence until task 6.2 added a real consumer (persist_take_pending_apply) whose registration can now race the very entry it targets.

Fix

FIXED round 2 (task 6.2): persist_register_pending_apply() (persist.c) now self-heals option (b) from this entry's own proposed fixes -- before linking a new node onto pending_applies, it checks lsn <= persist_last_applied_lsn(p); if the apply pump's walk has already applied lsn (last_applied_lsn already at or past it), on_applied(ctx, lsn) fires immediately and synchronously on the CALLING thread instead of being linked for a walk pass that will never revisit that lsn. Race-safe in both interleavings: when the walk has not reached lsn yet, the check is false and registration proceeds exactly as before (unchanged behavior, verified against the full existing regression suite); when the walk already passed lsn, its own persist_take_pending_apply() call for that lsn already ran and found nothing (that is precisely how we know registration is late), so the immediate inline fire can never double-fire alongside it. New deterministic regression test test_apply_pump_late_pending_apply_registration_self_heals (test_apply_pump.c) wires a REAL multi-vcpu vtaskmgr_t (n_vcpus=4) via commit_set_pool -- the one test in this spec to do so -- and forces the exact bad interleaving via a barrier (the test's stand-in commit continuation polls persist_last_applied_lsn() until the apply pump has already applied the lsn, THEN registers), rather than relying on natural timing luck. Verified genuine RED (fails with 'Expected 1 Was 0' against the unfixed check, confirming the hang is real and reproducible) -> GREEN (fires exactly once, resource released exactly once, no orphaned registration) after the fix. Full regression sweep (test_apply_pump, test_commit_barrier_reply, test_apply_reapply_idempotency, test_commit_resolve, test_commit_admission_pool, test_demotion_convergence) shows zero new failures -- the only 2 failures present (test_jobsubmit_returns_before_commit_and_registers_pending_apply, test_jobsubmit_array_mixed_outcome_no_phantom_job_for_failed_elements, both in test_commit_barrier_reply.c) are pre-existing/expected per task 6.1's own Implementation Notes (their assertion text predates task 6's apply pump landing; task 7.3's job to update), confirmed identical before and after this fix by reverting to the pre-round-2 baseline and re-running.

Verification

Not recorded (migrated from buglog.json).

References

• Related: bug-1409