lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): extract job-submit commit barrier out of the hook fan-out — stop committed-job-loss (bug-1306)

Browse source 
raft-apply-decoupling item #1 (down payment). xce_run_hooks returns only the
LAST-registered hook's rc, so a later JOB AFTER_EVENT subscriber (sched/
apiserver) returning 0 clobbered persist_job_hook_fun's commit -ETIMEDOUT.
handle_jobsubmit then replied 'jobsubmit ok' for an entry that never reached
majority, and a new-term leader legally discarded it (committed-job-loss,
Commit Durability violation).

- persist.c/.h: new persist_submit_wal_and_commit(p, jlist) — the explicit raft
  step (WAL append + majority-commit barrier) returning the commit verdict
  directly. persist_job_hook_fun's WAL_MODE_WAIT branch is now a no-op.
- job.c handle_jobsubmit: call the explicit step, read commit_rc DIRECTLY,
  reject on <0 before firing the now notification-only AFTER hook
  (canonical commit -> notify -> reply).
- tests: +test_job_hook_new_is_notify_only regression pin; 4 hook tests
  relocated to drive the explicit step. persist_hooks 19/0, e2e_hooks 3/0,
  commit_barrier_reply 22/0, commit_durability 38/0.

Proven: 10-round chaotic soak committed_job_loss 5 -> 0 (same election code).
Residual soak FAILs (divergence catch-up lag + a round-4 election-escalation
tail) are NOT job loss — tracked as task 5.10.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
This commit is contained in:
Rongsong Shen 2026-07-01 00:42:30 +00:00
parent 6bb9d81225
commit 15be5d94e8
8 changed files with 1592 additions and 2495 deletions
Download patch file Download diff file Expand all files Collapse all files

110
src/controller/persist.c
View file

@ -2782,6 +2782,60 @@ static int wal_hook_append_classified(persist_t* p, uint32_t wal_type,
return rc; /* local append rc only — facts have no commit timeout */
}
/* Explicit controller-side raft step for a job submission (raft-apply-decoupling
* item #1 / bug-1306): WAL-append each job in `jlist` and block on the
* majority-commit barrier, returning the commit verdict DIRECTLY to the caller
* (handle_jobsubmit) instead of through the xce_run_hooks fan-out (which returns
* only the last hook's rc and so clobbered this -ETIMEDOUT with a later notify
* hook's 0 — the committed-job-loss root cause).
*
* Returns 0 when every job's entry committed on a majority; the FIRST negative
* rc otherwise (-ETIMEDOUT: no majority within the deadline; -XCE_ERR_NOT_READY:
* admission cap; -ENOMEM: encode failure; or a local-write error). The caller
* MUST reject the submission on a negative rc (never reply success) to preserve
* the Commit Durability invariant. On -ETIMEDOUT / -XCE_ERR_NOT_READY the entry
* WAS appended locally and the LSN advanced, so j->submit_lsn is still tagged —
* the entry remains a pending, uncommitted tail a new-term leader may discard. */
int persist_submit_wal_and_commit(persist_t* p, xlist_t* jlist)
{
int rc = 0;
if (p == NULL || jlist == NULL)
return -EINVAL;
for (int i = 0; i < jlist->count && rc == 0; i++) {
job_t* j = (job_t*)xlist_get(jlist, i);
uint8_t* buf = NULL;
size_t sz = 0;
if (capnp_encode_job(&j->job, &buf, &sz) != 0) {
rc = -ENOMEM;
break;
}
/* ha-convergence-recovery (3.3): snapshot the LSN this submit entry
* will receive BEFORE the append. persist_wal_async assigns
* hdr.lsn = current_lsn then increments, so the pre-append current_lsn
* is exactly the written entry's LSN. */
pthread_mutex_lock(&p->lock);
uint64_t submit_lsn = p->current_lsn;
pthread_mutex_unlock(&p->lock);
int sync_rc = 0;
rc = persist_wal_sync_append(p, WAL_JOB_SUBMIT, buf, (uint32_t)sz,
&sync_rc);
free(buf);
/* Tag the job with its submit entry's LSN whenever the LOCAL WAL write
* succeeded. persist_wal_sync_append returns the persist_wal_async
* error only on local-write failure; on a replication-degraded outcome
* (0 / -ETIMEDOUT / -ENODEV / -XCE_ERR_NOT_READY) the entry was appended
* locally and the LSN advanced, so the egress tag is set. */
if (rc == 0 || rc == -ETIMEDOUT || rc == -XCE_ERR_NOT_READY)
j->submit_lsn = submit_lsn;
}
return rc;
}
/* P0 payload: just the detail value (4 bytes).
* Full Cap'n Proto serialization is wired in Task 15. */
@ -2803,50 +2857,18 @@ int persist_job_hook_fun(int type, int situation, int detail,
mode = wal_classify(wal_type, detail);
if (mode == WAL_MODE_WAIT) {
/* WAL_JOB_SUBMIT — the only WAIT type in the job hook; reference
* implementation keeps the existing in-hook sync path.
* AFTER_EVENT hook args: cluster_t* c, xlist_t* jlist */
(void)va_arg(ap, cluster_t*);
xlist_t* jlist = va_arg(ap, xlist_t*);
int rc = 0;
for (int i = 0; i < jlist->count && rc == 0; i++) {
job_t* j = (job_t*)xlist_get(jlist, i);
uint8_t* buf = NULL;
size_t sz = 0;
if (capnp_encode_job(&j->job, &buf, &sz) != 0) {
rc = -ENOMEM;
break;
}
/* persist_wal_sync_append returns 0 on successful local WAL write
* even when replication is degraded (no standbys → -ENODEV, timeout
* → -ETIMEDOUT). Only abort the loop on a real disk-write failure
* (non-zero return). Replication status is informational here. */
/* ha-convergence-recovery (3.3): snapshot the LSN this submit
* entry will receive BEFORE the append. persist_wal_async assigns
* hdr.lsn = current_lsn then increments (persist.c:318), so the
* pre-append current_lsn is exactly the written entry's LSN — the
* same value persist_wal_sync_append derives as current_lsn - 1. */
pthread_mutex_lock(&p->lock);
uint64_t submit_lsn = p->current_lsn;
pthread_mutex_unlock(&p->lock);
int sync_rc = 0;
rc = persist_wal_sync_append(p, WAL_JOB_SUBMIT, buf, (uint32_t)sz,
&sync_rc);
free(buf);
/* Tag the job with its submit entry's LSN whenever the LOCAL WAL
* write succeeded. persist_wal_sync_append returns the
* persist_wal_async error only on local-write failure; on a
* replication-degraded outcome (0 / -ETIMEDOUT / -ENODEV /
* -XCE_ERR_NOT_READY) the entry was appended locally and the LSN
* advanced, so the tag must be set (the egress predicate is
* independent of replication status). */
if (rc == 0 || rc == -ETIMEDOUT || rc == -XCE_ERR_NOT_READY) {
j->submit_lsn = submit_lsn;
}
}
return rc;
/* WAL_JOB_SUBMIT durability (WAL append + majority-commit barrier) was
* MOVED OUT of this hook into the explicit raft step
* persist_submit_wal_and_commit(), called directly by handle_jobsubmit
* (raft-apply-decoupling item #1 / bug-1306). Rationale: xce_run_hooks
* returns only the LAST-registered hook's rc, so a later JOB AFTER_EVENT
* subscriber (sched/apiserver, returning 0) CLOBBERED this branch's
* -ETIMEDOUT — handle_jobsubmit then replied `jobsubmit ok` for an
* entry that never reached majority, and a new-term leader legally
* discarded it (committed-job-loss). The commit verdict must be read
* from the protocol step directly, never as a hook return. This branch
* is now a no-op; the AFTER_EVENT hook chain is notification-only. */
return 0;
}
/* All other job events: payload = wal_jobstate_t {jobid, state}.