lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(raft-apply-decoupling): fire command continuation post-apply (task 6.2)

Browse source 
Wire persist_apply_pump_run() to call persist_take_pending_apply()
immediately after each entry's apply_committed_entry() call,
invoking the registered continuation strictly after the mutation.
Adds jobsubmit_reply_success_once() (job.c) so a successful job
submission finally sends a client reply once every array element
applies -- closes the standing gap from task 3.2 where committed
submissions never got a response.

Fixes bug-1415: commit_resolve_pending() dispatches the commit
continuation and the apply-pump kick as two independent
VTASK_LIVE_BLOCK vtasks with no completion-order guarantee. On any
real multi-vcpu pool (the production default, n_vcpus=8), if the
apply pump won the race and advanced last_applied_lsn past an LSN
before its continuation registered, the registration was permanently
orphaned -- the job would be constructed and visible, but the client
would hang forever. Untested by every existing fixture in this spec
(all use pool=NULL, which trivially serializes and can't reproduce
the race).

Fixed with a self-healing check in persist_register_pending_apply():
if the LSN is already <= last_applied_lsn, invoke the callback
immediately instead of registering for later pickup. Verified by the
spec's first genuine multi-vcpu regression test (real vtaskmgr_t,
n_vcpus=4), independently RED->GREEN reproduced by two review rounds.
This commit is contained in:
Rongsong Shen 2026-07-05 00:38:02 +00:00
parent 2fa955b92e
commit 88577f4658
6 changed files with 719 additions and 25 deletions
Download patch file Download diff file Expand all files Collapse all files

81
src/controller/job.c
View file

@ -623,6 +623,11 @@ typedef struct {
* once, in jobsubmit_async_ctx_release() */
uint64_t seq; /* xce_request_t.seq, copied for the reply */
uint32_t opcode; /* xce_request_t.opcode, copied for the reply */
uint64_t jobid; /* base jobid of the (possibly array) submission,
* echoed back on the success reply (task 6.2) --
* matches pre-3.1 semantics (git history commit
* 6f1fa97), which always reported the BASE id,
* never a per-element id */
pthread_mutex_t lock; /* guards pending/replied below */
int pending; /* entries not yet FULLY resolved (see above) */
int replied; /* 1 once a reply has been sent for this batch */
@ -663,24 +668,55 @@ static void jobsubmit_reply_failure_once(jobsubmit_async_ctx_t* actx, int rc)
xce_reply(actx->ep, actx->c->myhost, &reply);
}
/* raft-apply-decoupling task 6 hand-off target (task 6 itself is NOT built
* by this task): invoked by the future ordered apply pump immediately
* after it applies this entry via apply_committed_entry() — task 6.2's own
* contract, "fire a command's registered continuation immediately after
* its entry applies". Until task 6 exists, nothing ever calls this
* function; see jobsubmit_commit_continuation's COMMITTED branch below,
* which registers it as the hand-off target instead of applying or
* replying itself.
/* Sends exactly one success reply for the whole batch, carrying the base
* jobid — the sibling of jobsubmit_reply_failure_once, gated on the SAME
* actx->replied flag (Req 9.3) so a batch that already sent a failure reply
* for a sibling element never follows it with a contradictory success reply,
* and vice versa. Only ever reached from jobsubmit_apply_complete once every
* element's continuation has resolved (raft-apply-decoupling task 6.2). */
static void jobsubmit_reply_success_once(jobsubmit_async_ctx_t* actx)
{
xce_reply_t reply;
int send = 0;
pthread_mutex_lock(&actx->lock);
if (!actx->replied) {
actx->replied = 1;
send = 1;
}
pthread_mutex_unlock(&actx->lock);
if (!send)
return;
memset(&reply, 0, sizeof(reply));
reply.seq = actx->seq;
reply.opcode = actx->opcode;
reply.status = 0;
reply.data_which = RequestReply_data_jobid;
reply.data.jobid = actx->jobid;
xce_reply(actx->ep, actx->c->myhost, &reply);
}
/* raft-apply-decoupling task 6 hand-off target: invoked by the ordered apply
* pump (persist_apply_pump_run(), task 6.1/6.2) immediately after it applies
* this entry via apply_committed_entry() — task 6.2's own contract, "fire a
* command's registered continuation immediately after its entry applies".
* See jobsubmit_commit_continuation's COMMITTED branch below, which
* registers this as the hand-off target instead of applying or replying
* itself.
*
* NOTE for whoever builds task 6.2: apply_committed_entry()'s WAL_JOB_SUBMIT
* case (persist.c) already constructs the job (via restore_job()) AND fires
* the HOOK_DETAIL_JOB_NEW AFTER_EVENT notification, independently per entry
* — so by the time this function is invoked, the job is very likely already
* visible and already notified. This function most likely only needs to
* send the batch's success reply (echoing the jobid) once every entry has
* been applied, NOT reconstruct or re-notify anything — see the removed-code
* note above handle_jobsubmit's old (pre-3.2) construct/notify block for the
* full reasoning (git history, commit 6f1fa97, task 3.1). */
* apply_committed_entry()'s WAL_JOB_SUBMIT case (persist.c) already
* constructs the job (via restore_job()) AND fires the HOOK_DETAIL_JOB_NEW
* AFTER_EVENT notification, independently per entry, strictly BEFORE the
* apply pump reaches the persist_take_pending_apply()/invoke step that calls
* this function — so by the time this function runs, the job is already
* visible in c->jobmgr and already notified. This function therefore only
* sends the batch's success reply (echoing the base jobid) once every entry
* has been applied; it never reconstructs or re-notifies anything — see the
* removed-code note above handle_jobsubmit's old (pre-3.2) construct/notify
* block for the full reasoning (git history, commit 6f1fa97, task 3.1). */
static void jobsubmit_apply_complete(void* ctx, uint64_t lsn)
{
jobsubmit_async_ctx_t* actx = (jobsubmit_async_ctx_t*)ctx;
@ -693,14 +729,10 @@ static void jobsubmit_apply_complete(void* ctx, uint64_t lsn)
done = (actx->pending <= 0);
pthread_mutex_unlock(&actx->lock);
/* Task 6.2's success-reply belongs here (RequestReply_data_jobid,
* gated on actx->replied exactly like jobsubmit_reply_failure_once, so
* a partially-failed batch's earlier failure reply is never followed
* by a contradictory success reply) — not built by this task, which
* must not reply from the committed-outcome path at all (see the
* struct doc comment above). */
if (done)
if (done) {
jobsubmit_reply_success_once(actx);
jobsubmit_async_ctx_release(actx);
}
}
/* raft-apply-decoupling task 3.2: per-entry outcome of one array element's
@ -847,6 +879,7 @@ int handle_jobsubmit(cluster_t* c, endpoint_t* ep, xce_request_t* request)
actx->ep = ep;
actx->seq = request->seq;
actx->opcode = request->opcode;
actx->jobid = jobid;
actx->pending = plist.count;
actx->replied = 0;
pthread_mutex_init(&actx->lock, NULL);