lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(raft-apply-decoupling): convert job submit to continuation-based commit (task 3.2)

Browse source 
Replace handle_jobsubmit()'s blocking commit call (task 3.1) with a
real async registration against commit_submit_async(). On
FAILED/timeout, replies failure immediately with no job constructed.
On COMMITTED, hands off to a new pending-apply registration
(persist_register_pending_apply) for the ordered apply pump (task
6.1, not yet built) to consume -- does not apply or reply directly,
per design.

Deviates from design.md's stated integration site: appends via the
same ha_primary_log_append/persist_wal_async primitive the RETAIN-fact
path already uses and calls commit_submit_async directly on
p->repl.ce, bypassing repl_wait_majority_ack/repl_submit_majority_commit
in repl.c. Verified safe -- majority-ack detection is driven entirely
by (ce, lsn) events inside the commit engine, agnostic to which
wrapper registered the continuation -- but narrows the reusable
primitive design.md intended; documented in tasks.md.

Deletes (not preserves-dead) task 3.1's bug-1408 notification fix --
apply_committed_entry()'s WAL_JOB_SUBMIT case (task 2.2) already
fires the notification per-entry, immune to the batch-suppression
class bug-1408 fixed.

IMPORTANT: no live path applies a successfully-committed job
submission until task 6 lands (only restart replay does) -- every
successful submission commits durably but gets zero reply until
then. This commit range must not be soak-tested or deployed against
live traffic before task 6 completes.
This commit is contained in:
Rongsong Shen 2026-07-04 05:35:35 +00:00
parent 6f1fa97fab
commit df4d2fa0ed
5 changed files with 689 additions and 299 deletions
Download patch file Download diff file Expand all files Collapse all files

327
src/controller/job.c
View file

@ -595,7 +595,155 @@ int handle_jobdispatch(cluster_t* c, endpoint_t* ep, xce_request_t* request)
return rc;
}
static int xlist_deref_job_item_(job_t* j, void* arg) { return 0; }
/* raft-apply-decoupling task 3.2: shared state for one handle_jobsubmit()
* call's asynchronously-resolving per-entry commit continuations. Exactly
* one instance per client request — an array submission registers one
* continuation per array element, all sharing this struct, so the client
* still gets exactly one reply for the whole batch (matching pre-3.2
* per-request reply semantics), never one reply per array element.
*
* Lifetime: allocated in handle_jobsubmit() before it returns; released by
* whichever per-entry continuation (jobsubmit_commit_continuation) or
* apply-complete callback (jobsubmit_apply_complete) observes `pending`
* reach 0 — the last one out closes the door.
*
* IMPORTANT: a COMMITTED outcome does NOT decrement `pending` by itself
* (see jobsubmit_commit_continuation below) — task 3.2's own contract is
* that a committed entry is handed off to the not-yet-built ordered apply
* pump (task 6.1/6.2), and only jobsubmit_apply_complete (invoked BY that
* future pump) decrements `pending` for it. A batch with any committed
* entry is therefore only ever freed once task 6 exists and actually
* applies it. This mirrors the design's already-accepted "durably
* committed, not yet applied" posture for retained facts during quorum
* loss (Req 6.1) — it is not a leak. */
typedef struct {
cluster_t* c;
endpoint_t* ep; /* ref'd via endpoint_ref(); released exactly
* once, in jobsubmit_async_ctx_release() */
uint64_t seq; /* xce_request_t.seq, copied for the reply */
uint32_t opcode; /* xce_request_t.opcode, copied for the reply */
pthread_mutex_t lock; /* guards pending/replied below */
int pending; /* entries not yet FULLY resolved (see above) */
int replied; /* 1 once a reply has been sent for this batch */
} jobsubmit_async_ctx_t;
static void jobsubmit_async_ctx_release(jobsubmit_async_ctx_t* actx)
{
endpoint_unref(actx->ep);
pthread_mutex_destroy(&actx->lock);
free(actx);
}
/* Sends exactly one failure reply for the whole batch — the first failure
* observed wins; later failures/hand-offs never send a second, possibly
* contradictory reply for the same original request (Req 9.3, read at the
* per-REQUEST granularity a job-array submission actually has one of). */
static void jobsubmit_reply_failure_once(jobsubmit_async_ctx_t* actx, int rc)
{
xce_reply_t reply;
int send = 0;
pthread_mutex_lock(&actx->lock);
if (!actx->replied) {
actx->replied = 1;
send = 1;
}
pthread_mutex_unlock(&actx->lock);
if (!send)
return;
memset(&reply, 0, sizeof(reply));
reply.seq = actx->seq;
reply.opcode = actx->opcode;
reply.status = rc;
reply.data_which = RequestReply_data_nodata;
xce_reply(actx->ep, actx->c->myhost, &reply);
}
/* raft-apply-decoupling task 6 hand-off target (task 6 itself is NOT built
* by this task): invoked by the future ordered apply pump immediately
* after it applies this entry via apply_committed_entry() — task 6.2's own
* contract, "fire a command's registered continuation immediately after
* its entry applies". Until task 6 exists, nothing ever calls this
* function; see jobsubmit_commit_continuation's COMMITTED branch below,
* which registers it as the hand-off target instead of applying or
* replying itself.
*
* NOTE for whoever builds task 6.2: apply_committed_entry()'s WAL_JOB_SUBMIT
* case (persist.c) already constructs the job (via restore_job()) AND fires
* the HOOK_DETAIL_JOB_NEW AFTER_EVENT notification, independently per entry
* — so by the time this function is invoked, the job is very likely already
* visible and already notified. This function most likely only needs to
* send the batch's success reply (echoing the jobid) once every entry has
* been applied, NOT reconstruct or re-notify anything — see the removed-code
* note above handle_jobsubmit's old (pre-3.2) construct/notify block for the
* full reasoning (git history, commit 6f1fa97, task 3.1). */
static void jobsubmit_apply_complete(void* ctx, uint64_t lsn)
{
jobsubmit_async_ctx_t* actx = (jobsubmit_async_ctx_t*)ctx;
int done;
(void)lsn;
pthread_mutex_lock(&actx->lock);
actx->pending--;
done = (actx->pending <= 0);
pthread_mutex_unlock(&actx->lock);
/* Task 6.2's success-reply belongs here (RequestReply_data_jobid,
* gated on actx->replied exactly like jobsubmit_reply_failure_once, so
* a partially-failed batch's earlier failure reply is never followed
* by a contradictory success reply) — not built by this task, which
* must not reply from the committed-outcome path at all (see the
* struct doc comment above). */
if (done)
jobsubmit_async_ctx_release(actx);
}
/* raft-apply-decoupling task 3.2: per-entry outcome of one array element's
* commit_submit_async() registration (persist_submit_job_payloads_async's
* `fn` callback).
*
* On FAILED/timed-out (rc != 0): reply failure now — no apply is needed on
* this path (Req 2.3). This entry has nothing further to wait on, so it
* decrements `pending` itself.
*
* On COMMITTED (rc == 0): task 3.2 must NOT apply the job or reply here
* (the task's own explicit instruction) — hand the entry off to the
* not-yet-built ordered apply pump (task 6.1/6.2) via
* persist_register_pending_apply(), the producer-side hand-off this task
* builds. `pending` is intentionally NOT decremented here for a committed
* entry — see jobsubmit_async_ctx_t's doc comment. */
static void jobsubmit_commit_continuation(void* ctx, uint64_t lsn, int rc)
{
jobsubmit_async_ctx_t* actx = (jobsubmit_async_ctx_t*)ctx;
if (rc != 0) {
int done;
WARN("handle_jobsubmit: lsn=%lu commit failed/timed out (rc=%d) — "
"rejecting; no job ever constructed for it",
(unsigned long)lsn, rc);
jobsubmit_reply_failure_once(actx, rc);
pthread_mutex_lock(&actx->lock);
actx->pending--;
done = (actx->pending <= 0);
pthread_mutex_unlock(&actx->lock);
if (done)
jobsubmit_async_ctx_release(actx);
return;
}
/* COMMITTED: hand off to task 6's ordered apply pump instead of
* applying/replying directly from this continuation — see
* persist_register_pending_apply's doc comment (persist.h) for the
* full hand-off contract. */
persist_register_pending_apply(&actx->c->persist, lsn,
jobsubmit_apply_complete, actx);
}
int handle_jobsubmit(cluster_t* c, endpoint_t* ep, xce_request_t* request)
{
@ -603,10 +751,7 @@ int handle_jobsubmit(cluster_t* c, endpoint_t* ep, xce_request_t* request)
xce_job_submit_t* submit;
xce_reply_t reply;
xlist_t plist; /* xce_job_t* WAL payloads, pre-commit */
xlist_t jlist; /* job_t* for the post-commit notify hook */
uint64_t* lsns = NULL;
job_t* j = NULL;
int i;
int rc;
submit = request->body.jobsubmsg;
@ -660,138 +805,72 @@ int handle_jobsubmit(cluster_t* c, endpoint_t* ep, xce_request_t* request)
return rc;
}
/* Explicit raft step (raft-apply-decoupling item #1 / bug-1306; task
* 3.1): WAL-append + majority-commit barrier for every payload, reading
* the commit verdict DIRECTLY. This used to run inside the AFTER_EVENT
* hook (persist_job_hook_fun), but xce_run_hooks returns only the LAST
* hook's rc, so a later notify subscriber (sched/apiserver) returning 0
* CLOBBERED the -ETIMEDOUT — the client was acked for an entry that
* never reached majority, which a new-term leader then legally
* discarded (committed-job-loss). The client must not be acknowledged,
* and no job object may exist, until the submission is durable on a
* majority. */
int commit_rc = persist_submit_job_payloads_and_commit(&c->persist,
&plist, lsns);
if (commit_rc < 0) {
WARN("handle_jobsubmit: job not durable (rc=%d) — rejecting submission",
commit_rc);
/* Explicit raft step (raft-apply-decoupling item #1 / bug-1306, task
* 3.1; converted to the continuation mechanism by task 3.2): register
* every payload's commit outcome instead of blocking on the majority
* barrier. This handler returns as soon as every payload is durably
* appended locally and its outcome-resolution path is established —
* BEFORE any of them necessarily commit. From this point on, job
* construction and the client reply happen from
* jobsubmit_commit_continuation / jobsubmit_apply_complete (above),
* never again from this stack frame.
*
* REMOVED (raft-apply-decoupling task 3.2) relative to task 3.1's
* version of this function: the post-commit create_job()-loop +
* jlist-build + xce_run_hooks(AFTER_EVENT, HOOK_DETAIL_JOB_NEW) block
* (bug-1408's fix: unconditional per-successfully-constructed-element
* notification) is gone, not merely deferred. It is SUPERSEDED, not
* resurrected, by apply_committed_entry()'s existing WAL_JOB_SUBMIT
* case (persist.c, raft-apply-decoupling task 2.2, already built):
* that function already does restore_job() + fires HOOK_DETAIL_JOB_NEW
* per entry, independently per array element — which is structurally
* immune to the bug-1408 class (a sibling's apply failure can never
* suppress another sibling's notification, because each LSN applies
* through its own independent call). The full reasoning is preserved
* in this task's status report; the removed code itself remains in
* git history at commit 6f1fa97. Whoever builds task 6.1/6.2 should
* call apply_committed_entry() (not resurrect this loop) and should
* expect jobsubmit_apply_complete to need only the success reply. */
jobsubmit_async_ctx_t* actx = calloc(1, sizeof(*actx));
if (actx == NULL) {
free_job_submit_payloads(&plist);
free(lsns);
reply.status = commit_rc;
reply.status = -ENOMEM;
reply.data_which = RequestReply_data_nodata;
xce_reply(ep, c->myhost, &reply);
return commit_rc;
return -ENOMEM;
}
actx->c = c;
actx->ep = ep;
actx->seq = request->seq;
actx->opcode = request->opcode;
actx->pending = plist.count;
actx->replied = 0;
pthread_mutex_init(&actx->lock, NULL);
endpoint_ref(ep);
/* Committed on a majority — NOW construct and register each job object
* (moved here from before the commit barrier; raft-apply-decoupling
* task 3.1 / Req 2.1, 2.2, 2.3). A construction failure at this point
* cannot be rolled back (the entry is already durably committed); it is
* a genuine apply-time failure, not a retryable one (design.md Error
* Strategy) — logged loud, and the submission is NOT reported as a
* success. */
int construct_rc = 0;
for (i = 0; i < plist.count; i++) {
xce_job_t* jp = (xce_job_t*)xlist_get(&plist, i);
job_t* job = create_job(c, jp);
if (job == NULL) {
int saved_errno = errno;
ERROR("handle_jobsubmit: job %lu committed at lsn %lu but "
"could not be constructed (errno=%d)",
(unsigned long)jp->jobid, (unsigned long)lsns[i],
saved_errno);
if (construct_rc == 0)
construct_rc = -saved_errno;
continue;
}
job->submit_lsn = lsns[i];
}
(void)persist_submit_job_payloads_async(&c->persist, &plist, lsns,
jobsubmit_commit_continuation,
actx);
/* Every payload's WAL data is no longer needed once appended: the
* future apply pump (task 6) re-reads each committed entry's payload
* from the WAL itself — the same way restart replay already does —
* not from this in-memory copy. persist_submit_job_payloads_async has
* ALREADY invoked jobsubmit_commit_continuation, synchronously or
* asynchronously, exactly once for every payload by (or after) the
* time it returns; its own return value carries no information this
* function still needs to act on. */
free_job_submit_payloads(&plist);
free(lsns);
/* Build the job list for the AFTER_EVENT hook — the same lookup used
* before this task, just run after construction (which now itself runs
* after commit) instead of before either.
*
* IMPORTANT (raft-apply-decoupling task 3.1 remediation): this lookup
* and the hook it feeds run UNCONDITIONALLY here, even when
* construct_rc < 0 for one or more array elements. Every element whose
* create_job() succeeded above is already durably committed AND already
* inserted into c->jobmgr — key_find below naturally finds only those
* (a failed element was never inserted, so it is silently absent from
* jlist with no extra bookkeeping needed). Both the scheduler and the
* apiserver are purely event-driven off this hook with no periodic
* reconciliation, so returning early here (as a previous version of
* this function did whenever ANY sibling failed to construct) would
* strand the successfully constructed siblings — real, durable jobs —
* invisible until the next controller restart replays the WAL. A
* whole-batch construction failure must never suppress notification for
* the elements that individually succeeded. */
xlist_init(&jlist, submit->array_size);
for (i = 0; i < submit->array_size; i++) {
lock(c->jobmgr, RTTI_LOCK_READ);
rc = key_find(c->jobmgr, (void*)(jobid + i), (void**)&j);
if (j != NULL) {
reference(j);
}
unlock(c->jobmgr);
if ((rc <= 0) || (j == NULL)) {
char idstring[128];
fmt_jobid(idstring, sizeof(idstring) - 1, jobid);
ERROR("%s fail to lock job %s (%lu)", __func__, idstring, jobid);
} else {
if (xlist_store(&jlist, j) < 0) {
PERROR("xlist_store");
}
}
}
/* Committed + constructed — fire the AFTER_EVENT hook as NOTIFICATION
* only (scheduler push, apiserver SSE), observing already-committed,
* already-applied state. Its return no longer gates durability: a
* notify-hook failure cannot un-commit an already-durable job, so it is
* logged but never fails the client reply. Fired for every
* successfully constructed element regardless of construct_rc (see the
* comment above jlist's construction). */
int notify_rc = xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB,
HOOK_AFTER_EVENT,
HOOK_DETAIL_JOB_NEW, c, &jlist);
if (notify_rc != 0)
WARN("handle_jobsubmit: post-commit notification hook rc=%d "
"(job already durable; not failing the reply)", notify_rc);
xlist_iter(&jlist, (xlist_iter_fn_t)xlist_deref_job_item_, NULL);
xlist_fini(&jlist);
if (construct_rc < 0) {
/* At least one array element durably committed but could not be
* constructed (see the ERROR logged at the point of failure above).
* The client learns the submission did not fully succeed; the
* siblings that DID construct successfully were already notified
* above and must never have that notification gated on this. */
reply.status = construct_rc;
reply.data_which = RequestReply_data_nodata;
xce_reply(ep, c->myhost, &reply);
return construct_rc;
}
/* Reply success only AFTER WAL write + majority commit + construction. */
reply.status = 0;
reply.data_which = RequestReply_data_jobid;
reply.data.jobid = jobid;
xce_reply(ep, c->myhost, &reply);
/* The calling thread returns here, before any payload's commit outcome
* is necessarily known (raft-apply-decoupling task 3.2, Req 1.1, 1.4,
* 2.1, 2.3) — every reply from this point on is sent by
* jobsubmit_reply_failure_once or (once task 6 exists)
* jobsubmit_apply_complete, off this stack frame. */
return 0;
}