fix(controller): extract job-submit commit barrier out of the hook fan-out — stop committed-job-loss (bug-1306)

raft-apply-decoupling item #1 (down payment). xce_run_hooks returns only the
LAST-registered hook's rc, so a later JOB AFTER_EVENT subscriber (sched/
apiserver) returning 0 clobbered persist_job_hook_fun's commit -ETIMEDOUT.
handle_jobsubmit then replied 'jobsubmit ok' for an entry that never reached
majority, and a new-term leader legally discarded it (committed-job-loss,
Commit Durability violation).

- persist.c/.h: new persist_submit_wal_and_commit(p, jlist) — the explicit raft
  step (WAL append + majority-commit barrier) returning the commit verdict
  directly. persist_job_hook_fun's WAL_MODE_WAIT branch is now a no-op.
- job.c handle_jobsubmit: call the explicit step, read commit_rc DIRECTLY,
  reject on <0 before firing the now notification-only AFTER hook
  (canonical commit -> notify -> reply).
- tests: +test_job_hook_new_is_notify_only regression pin; 4 hook tests
  relocated to drive the explicit step. persist_hooks 19/0, e2e_hooks 3/0,
  commit_barrier_reply 22/0, commit_durability 38/0.

Proven: 10-round chaotic soak committed_job_loss 5 -> 0 (same election code).
Residual soak FAILs (divergence catch-up lag + a round-4 election-escalation
tail) are NOT job loss — tracked as task 5.10.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
This commit is contained in:
Rongsong Shen 2026-07-01 00:42:30 +00:00
parent 6bb9d81225
commit 15be5d94e8
8 changed files with 1592 additions and 2495 deletions

View file

@ -9,6 +9,7 @@
#include "config.h"
#include "wal_types.h" /* wal_entry_hdr_t, wal_entry_type — P2-4: shared with repl.h */
#include "repl.h" /* replicator_t — P2-4: embedded in persist_t */
#include "xlist.h" /* xlist_t — persist_submit_wal_and_commit job batch */
/* Forward declaration */
typedef struct cluster_s cluster_t;
@ -605,6 +606,14 @@ void persist_log_entry_metadata(const wal_entry_hdr_t* hdr);
* P0: writes a minimal 4-byte payload (detail value) to the WAL.
* Full Cap'n Proto serialization is wired in Task 15.
* ============================================================ */
/* Explicit controller-side raft step for job submission (raft-apply-decoupling
* item #1 / bug-1306): WAL-append every job in `jlist` and block on the
* majority-commit barrier, returning the commit verdict DIRECTLY (0 = all
* committed on a majority; first negative rc otherwise — -ETIMEDOUT / etc.).
* handle_jobsubmit calls this instead of reading the clobber-prone hook return;
* a negative rc MUST reject the submission (never reply success). */
int persist_submit_wal_and_commit(persist_t* p, xlist_t* jlist);
int persist_job_hook_fun(int type, int situation, int detail,
void* cls, va_list ap);
int persist_queue_hook_fun(int type, int situation, int detail,