lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(raft-apply-decoupling): fire command continuation post-apply (task 6.2)

Browse source 
Wire persist_apply_pump_run() to call persist_take_pending_apply()
immediately after each entry's apply_committed_entry() call,
invoking the registered continuation strictly after the mutation.
Adds jobsubmit_reply_success_once() (job.c) so a successful job
submission finally sends a client reply once every array element
applies -- closes the standing gap from task 3.2 where committed
submissions never got a response.

Fixes bug-1415: commit_resolve_pending() dispatches the commit
continuation and the apply-pump kick as two independent
VTASK_LIVE_BLOCK vtasks with no completion-order guarantee. On any
real multi-vcpu pool (the production default, n_vcpus=8), if the
apply pump won the race and advanced last_applied_lsn past an LSN
before its continuation registered, the registration was permanently
orphaned -- the job would be constructed and visible, but the client
would hang forever. Untested by every existing fixture in this spec
(all use pool=NULL, which trivially serializes and can't reproduce
the race).

Fixed with a self-healing check in persist_register_pending_apply():
if the LSN is already <= last_applied_lsn, invoke the callback
immediately instead of registering for later pickup. Verified by the
spec's first genuine multi-vcpu regression test (real vtaskmgr_t,
n_vcpus=4), independently RED->GREEN reproduced by two review rounds.
This commit is contained in:
Rongsong Shen 2026-07-05 00:38:02 +00:00
parent 2fa955b92e
commit 88577f4658
6 changed files with 719 additions and 25 deletions
Download patch file Download diff file Expand all files Collapse all files

35
src/controller/persist.c
View file

@ -3139,6 +3139,24 @@ int persist_apply_pump_run(persist_t* p)
atomic_store_explicit(&p->last_applied_lsn, next, memory_order_relaxed);
applied++;
/* raft-apply-decoupling task 6.2 (Req 1.1, 2.1-2.3): consume this
* LSN's command hand-off, if any (task 3.2's
* persist_register_pending_apply(), e.g. handle_jobsubmit's
* commit-outcome continuation) -- strictly AFTER the mutation above
* and after last_applied_lsn already reflects it, never before or
* during. A fact entry never registers a hand-off, so
* persist_take_pending_apply() simply returns 0 for it and this is a
* correct no-op with no per-entry-type branching needed here.
* persist_take_pending_apply() unlinks the registration before
* returning it, so re-applying the same lsn (a future retry/replay
* landing on this exact walk position again) can never fire the
* continuation a second time. */
pending_apply_notify_fn on_applied = NULL;
void* apply_ctx = NULL;
if (persist_take_pending_apply(p, next, &on_applied, &apply_ctx))
on_applied(apply_ctx, next);
}
pthread_mutex_unlock(&p->apply_pump_lock);
@ -4585,6 +4603,23 @@ void persist_register_pending_apply(persist_t* p, uint64_t lsn,
if (p == NULL || on_applied == NULL)
return;
/* bug-1415 self-healing (round 2): commit.c's continuation dispatch and
* the apply pump's walk (persist_apply_pump_run) are two independent
* vtasks with no completion-order guarantee -- see this function's doc
* comment in persist.h for the full race. If the walk has already
* applied `lsn` (last_applied_lsn already at or past it) by the time we
* get here, linking this registration onto pending_applies would orphan
* it forever: the walk only ever visits each lsn once, and it already
* called persist_take_pending_apply() for this lsn and found nothing.
* Fire the hand-off immediately, synchronously, on THIS thread instead
* -- never leave a continuation unresolved (controllers never
* self-terminate; the same "fail loud and complete the work" posture
* commit.c's dispatch_continuation fallback uses). */
if (lsn <= persist_last_applied_lsn(p)) {
on_applied(ctx, lsn);
return;
}
node = calloc(1, sizeof(*node));
if (node == NULL) {
ERROR("persist_register_pending_apply: OOM registering the apply "