lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(raft-apply-decoupling): defer job construction past the commit barrier (task 3.1)

Browse source 
Reorder handle_jobsubmit() so job objects are constructed via
create_job() only after the WAL commit succeeds, not before. Adds
jobid_range_exists() for a fast-fail pre-check with no WAL work, and
persist_submit_job_payloads_and_commit() (persist.c) operating on raw
xce_job_t* payloads instead of pre-existing job_t*. Commit remains
synchronous -- converting to the continuation-based commit API is
task 3.2.

Fixes bug-1408 (found during review): the post-commit array-
construction loop suppressed notification for an entire array batch
if any one sibling element failed to construct, stranding other
durably-committed and successfully-constructed jobs invisible until
the next restart's replay. Notification now fires unconditionally
for every successfully-constructed element, independent of sibling
failures; the client reply still separately reports the batch's
construction status.
This commit is contained in:
Rongsong Shen 2026-07-04 04:44:43 +00:00
parent 8febd663d5
commit 6f1fa97fab
6 changed files with 871 additions and 399 deletions
Download patch file Download diff file Expand all files Collapse all files

108
src/controller/persist.c
View file

@ -3610,6 +3610,44 @@ static int wal_hook_append_classified(persist_t* p, uint32_t wal_type,
return rc; /* local append rc only — facts have no commit timeout */
}
/* Shared step for a single WAL_JOB_SUBMIT entry (raft-apply-decoupling task
* 3.1): encode `job`, append, and block on the majority-commit barrier.
* Returns 0 when the entry committed on a majority; a negative rc otherwise
* (-ETIMEDOUT: no majority within the deadline; -XCE_ERR_NOT_READY: admission
* cap; -ENOMEM: encode failure; or a local-write error). Whenever the LOCAL
* WAL write succeeded (rc == 0, -ETIMEDOUT, or -XCE_ERR_NOT_READY — the entry
* was appended locally and the LSN advanced even though replication is
* degraded), *out_lsn is set to the entry's assigned LSN so a caller can tag
* it for egress-watermark purposes even on a rejected submission. */
static int submit_job_wal_entry_and_commit(persist_t* p, const xce_job_t* job,
uint64_t* out_lsn)
{
uint8_t* buf = NULL;
size_t sz = 0;
int rc;
if (capnp_encode_job(job, &buf, &sz) != 0)
return -ENOMEM;
/* ha-convergence-recovery (3.3): snapshot the LSN this submit entry
* will receive BEFORE the append. persist_wal_async assigns
* hdr.lsn = current_lsn then increments, so the pre-append current_lsn
* is exactly the written entry's LSN. */
pthread_mutex_lock(&p->lock);
uint64_t submit_lsn = p->current_lsn;
pthread_mutex_unlock(&p->lock);
int sync_rc = 0;
rc = persist_wal_sync_append(p, WAL_JOB_SUBMIT, buf, (uint32_t)sz,
&sync_rc);
free(buf);
if (rc == 0 || rc == -ETIMEDOUT || rc == -XCE_ERR_NOT_READY)
*out_lsn = submit_lsn;
return rc;
}
/* Explicit controller-side raft step for a job submission (raft-apply-decoupling
* item #1 / bug-1306): WAL-append each job in `jlist` and block on the
* majority-commit barrier, returning the commit verdict DIRECTLY to the caller
@ -3633,32 +3671,36 @@ int persist_submit_wal_and_commit(persist_t* p, xlist_t* jlist)
for (int i = 0; i < jlist->count && rc == 0; i++) {
job_t* j = (job_t*)xlist_get(jlist, i);
uint8_t* buf = NULL;
size_t sz = 0;
if (capnp_encode_job(&j->job, &buf, &sz) != 0) {
rc = -ENOMEM;
break;
}
/* ha-convergence-recovery (3.3): snapshot the LSN this submit entry
* will receive BEFORE the append. persist_wal_async assigns
* hdr.lsn = current_lsn then increments, so the pre-append current_lsn
* is exactly the written entry's LSN. */
pthread_mutex_lock(&p->lock);
uint64_t submit_lsn = p->current_lsn;
pthread_mutex_unlock(&p->lock);
uint64_t lsn = 0;
int sync_rc = 0;
rc = persist_wal_sync_append(p, WAL_JOB_SUBMIT, buf, (uint32_t)sz,
&sync_rc);
free(buf);
/* Tag the job with its submit entry's LSN whenever the LOCAL WAL write
* succeeded. persist_wal_sync_append returns the persist_wal_async
* error only on local-write failure; on a replication-degraded outcome
* (0 / -ETIMEDOUT / -ENODEV / -XCE_ERR_NOT_READY) the entry was appended
* locally and the LSN advanced, so the egress tag is set. */
rc = submit_job_wal_entry_and_commit(p, &j->job, &lsn);
if (rc == 0 || rc == -ETIMEDOUT || rc == -XCE_ERR_NOT_READY)
j->submit_lsn = submit_lsn;
j->submit_lsn = lsn;
}
return rc;
}
/* Sibling of persist_submit_wal_and_commit for a job submission whose job
* object(s) do not exist yet (raft-apply-decoupling task 3.1 / Req 2.1, 2.3):
* `payloads` is an xlist_t of xce_job_t* that have never been registered in
* any job table. On full success (return 0), out_lsns[i] holds payloads[i]'s
* committed LSN — the caller constructs and tags the job object AFTER this
* call returns, so a failed/timed-out commit never leaves a job object
* behind (there is nothing to construct until this succeeds). Returns the
* first negative rc on failure. */
int persist_submit_job_payloads_and_commit(persist_t* p, xlist_t* payloads,
uint64_t* out_lsns)
{
int rc = 0;
if (p == NULL || payloads == NULL || out_lsns == NULL)
return -EINVAL;
for (int i = 0; i < payloads->count && rc == 0; i++) {
xce_job_t* j = (xce_job_t*)xlist_get(payloads, i);
rc = submit_job_wal_entry_and_commit(p, j, &out_lsns[i]);
}
return rc;
@ -3685,17 +3727,13 @@ int persist_job_hook_fun(int type, int situation, int detail,
mode = wal_classify(wal_type, detail);
if (mode == WAL_MODE_WAIT) {
/* WAL_JOB_SUBMIT durability (WAL append + majority-commit barrier) was
* MOVED OUT of this hook into the explicit raft step
* persist_submit_wal_and_commit(), called directly by handle_jobsubmit
* (raft-apply-decoupling item #1 / bug-1306). Rationale: xce_run_hooks
* returns only the LAST-registered hook's rc, so a later JOB AFTER_EVENT
* subscriber (sched/apiserver, returning 0) CLOBBERED this branch's
* -ETIMEDOUT — handle_jobsubmit then replied `jobsubmit ok` for an
* entry that never reached majority, and a new-term leader legally
* discarded it (committed-job-loss). The commit verdict must be read
* from the protocol step directly, never as a hook return. This branch
* is now a no-op; the AFTER_EVENT hook chain is notification-only. */
/* WAL_JOB_SUBMIT durability (WAL append + majority-commit barrier) is
* an explicit step invoked directly by handle_jobsubmit —
* persist_submit_job_payloads_and_commit(), called strictly before
* any job object is constructed (raft-apply-decoupling task 3.1 /
* bug-1306) — never from this hook. This hook fires only after that
* commit has already succeeded, purely as a notification, so there
* is nothing left for it to do here. */
return 0;
}