lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): extract job-submit commit barrier out of the hook fan-out — stop committed-job-loss (bug-1306)

Browse source 
raft-apply-decoupling item #1 (down payment). xce_run_hooks returns only the
LAST-registered hook's rc, so a later JOB AFTER_EVENT subscriber (sched/
apiserver) returning 0 clobbered persist_job_hook_fun's commit -ETIMEDOUT.
handle_jobsubmit then replied 'jobsubmit ok' for an entry that never reached
majority, and a new-term leader legally discarded it (committed-job-loss,
Commit Durability violation).

- persist.c/.h: new persist_submit_wal_and_commit(p, jlist) — the explicit raft
  step (WAL append + majority-commit barrier) returning the commit verdict
  directly. persist_job_hook_fun's WAL_MODE_WAIT branch is now a no-op.
- job.c handle_jobsubmit: call the explicit step, read commit_rc DIRECTLY,
  reject on <0 before firing the now notification-only AFTER hook
  (canonical commit -> notify -> reply).
- tests: +test_job_hook_new_is_notify_only regression pin; 4 hook tests
  relocated to drive the explicit step. persist_hooks 19/0, e2e_hooks 3/0,
  commit_barrier_reply 22/0, commit_durability 38/0.

Proven: 10-round chaotic soak committed_job_loss 5 -> 0 (same election code).
Residual soak FAILs (divergence catch-up lag + a round-4 election-escalation
tail) are NOT job loss — tracked as task 5.10.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
This commit is contained in:
Rongsong Shen 2026-07-01 00:42:30 +00:00
parent 6bb9d81225
commit 15be5d94e8
8 changed files with 1592 additions and 2495 deletions
Download patch file Download diff file Expand all files Collapse all files

44
src/controller/job.c
View file

@ -590,27 +590,43 @@ int handle_jobsubmit(cluster_t* c, endpoint_t* ep, xce_request_t* request)
}
}
int hook_rc = xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB,
/* Explicit raft step (raft-apply-decoupling item #1 / bug-1306): WAL-append
* + majority-commit barrier, reading the commit verdict DIRECTLY. This used
* to run inside the AFTER_EVENT hook (persist_job_hook_fun), but xce_run_hooks
* returns only the LAST hook's rc, so a later notify subscriber (sched/
* apiserver) returning 0 CLOBBERED the -ETIMEDOUT — the client was acked for
* an entry that never reached majority, which a new-term leader then legally
* discarded (committed-job-loss). The client must not be acknowledged until
* the job is durable on a majority. */
int commit_rc = persist_submit_wal_and_commit(&c->persist, &jlist);
if (commit_rc < 0) {
WARN("handle_jobsubmit: job not durable (rc=%d) — rejecting submission",
commit_rc);
xlist_iter(&jlist, (xlist_iter_fn_t)xlist_deref_job_item_, NULL);
xlist_fini(&jlist);
reply.status = commit_rc;
reply.data_which = RequestReply_data_nodata;
xce_reply(ep, c->myhost, &reply);
return commit_rc;
}
/* Committed on a majority — fire the AFTER_EVENT hook as NOTIFICATION only
* (scheduler push, apiserver SSE), observing already-committed state. Its
* return no longer gates durability: a notify-hook failure cannot un-commit
* an already-durable job, so it is logged but never fails the client reply. */
int notify_rc = xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB,
HOOK_AFTER_EVENT,
HOOK_DETAIL_JOB_NEW, c, &jlist);
if (notify_rc != 0)
WARN("handle_jobsubmit: post-commit notification hook rc=%d "
"(job already durable; not failing the reply)", notify_rc);
xlist_iter(&jlist, (xlist_iter_fn_t)xlist_deref_job_item_, NULL);
xlist_fini(&jlist);
/* If persistence failed (replication timeout, I/O error), reject the
* submission so the client knows the job was not made durable. */
if (hook_rc != 0) {
WARN("handle_jobsubmit: persistence hook failed (rc=%d) — rejecting job",
hook_rc);
reply.status = hook_rc;
reply.data_which = RequestReply_data_nodata;
xce_reply(ep, c->myhost, &reply);
return hook_rc;
}
/* Reply only AFTER WAL write + majority replication succeeded. */
/* Reply success only AFTER WAL write + majority commit succeeded. */
reply.status = 0;
reply.data_which = RequestReply_data_jobid;
reply.data.jobid = jobid;