myxce/.kiro/specs/raft-apply-decoupling/design.md

Design Document: raft-apply-decoupling

Overview

Purpose: This feature gives the XCE controller one canonical, ordered path from a client/worker request to committed, applied cluster state — append → majority-commit → apply → notify — replacing today's mix of an already-partially-fixed command path (bug-1306) and a still-fully-optimistic fact path (worker-reported job start/finish/fail/rusage, applied and reacted to on receipt, ahead of commit).

Users: Cluster operators (who need failover to never lose or fabricate job state) and maintainers of the controller's submit/commit/replay code (who need one apply seam instead of two divergent ones).

Impact: Changes the commit engine's COMMIT_MODE_WAIT path from a blocking wait to an event-driven resolve; introduces a single apply_committed_entry() seam invoked by both live commit-advance and restart replay; moves worker-fact state mutation and reactions from receipt-time to post-commit-time; adds a post-replay CLEAN/DIVERGENT verification.

Goals

• One explicit raft step (append + majority commit) per state-changing operation, invoked directly by its handler — not incidentally from a hook.
• One apply seam, used identically by live commit-advance and restart replay, applied strictly in commit order.
• Commands and facts converge on that one apply seam; only their ingress and commit-wait policy differ.
• A quorum-loss window degrades safely by construction (retained-but-unapplied), with no special-case "local apply" code path.
• A restart proves its own correctness (CLEAN/DIVERGENT) instead of only checking WAL structural integrity.
• No regression to any HA correctness or throughput property already proven by prior specs.

Non-Goals

• Rewriting the raft library's protocol core (raft_advance_commit, on_commit_advanced, etc.) — consumed as-is.
• Collapsing a job array into a single WAL record (job-array-single-wal-record).
• Moving snapshot send / config-push / peer-connect off the iomgr/delaymgr threads (raft-drive-async, already done separately).
• Building worker-rejoin-resync — this design depends on it as a future backstop, consistent with every sibling spec that already accepted this gap.
• Entity-bearing snapshot content or read-consistency/leader-lease work.

Boundary Commitments

This Spec Owns

• The explicit raft-commit step for command entries and its call sites (already partially landed for job submit via bug-1306; this spec completes the ordering and generalizes the mechanism).
• The commit engine's COMMIT_MODE_WAIT continuation/resolve mechanism (the keystone conversion away from blocking wait).
• The single apply_committed_entry() seam and its dispatch table, and the "apply pump" that drives it off commit-advance in strict LSN order.
• The fact-ingestion split (durable-append-only at receipt vs. mutation+reaction at apply).
• The post-replay CLEAN/DIVERGENT verification and its non-promotable fence.
• Conformance/regression assertions for apply-ordering, idempotency, and no-reaction-before-commit.

Out of Boundary

• The raft library's callback contracts (on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from, on_commit_advanced, raft_advance_commit) — consumed unchanged.
• Transport/timer-thread blocking (snapshot send, config-push wait, peer connect) — raft-drive-async's domain; this design only consumes its vtask_enq_classified / slow-lane primitives.
• Worker join/rejoin protocol changes — worker-join-refactor / worker-rejoin-resync.
• Entity-bearing snapshot format/content — ha-entity-snapshots / config-upgrade-snapshot. This design's replay-side changes must remain compatible with ha-entity-snapshots' restore-before-advance load_snapshot path, which already shares restore_job/restore_queue_state with WAL replay (see Architecture).
• Removing job_submit_committed() (egress.c:47) — becomes a harmless tautology once this design lands, but its removal is a documented follow-up, not in scope (research.md, Design Decisions).

Allowed Dependencies

• ha-commit-engine / ha-commit-barrier-rollout: majority-commit and idempotency primitives, consumed as-is.
• ha-term-safety (complete): on_commit_advanced / raft_advance_commit wiring on the ACK path (advisory raft_commit_index, Option A) — this design's apply pump is a new consumer of the existing durable-majority commit advance; it does not change how committed_lsn itself advances.
• raft-drive-async (complete): vtask_enq_classified + slow-lane executor + VTASK_LIVENESS/VTASK_LIVE_BLOCK/VTASK_REQ_BLOCK admission classes — reused for dispatching resolved-commit continuations off the ACK-processing thread.
• raft-sim-conformance: existing deterministic-simulator harness, extended (not replaced) with new assertions.

Revalidation Triggers

• Any change to the commit engine's per-LSN structure (commit_engine_t, ce->...) or its locking discipline (p->lock vs ce->lock ordering, persist.c:3347-3350).
• Any change to WAL_MODE_WAIT/WAL_MODE_RETAIN classification (wal_classify, persist.c:3218) — the apply pump's per-type behavior depends on this classification being exhaustive.
• Any new WAL entry type — must be added to the apply_committed_entry() dispatch table or it silently fails to apply.
• worker-rejoin-resync landing — must re-verify the fact-apply idempotency guards against its replay/resync ordering (shared mark_job_* guards).
• job-array-single-wal-record landing — builds its array-expand step directly on apply_committed_entry(); a signature change here is a breaking change there.

Architecture

Existing Architecture Analysis

Current state, verified against the live codebase (not the 9-day-old brief — see research.md):

Aspect	Current state	Gap vs. this design
Job-submit raft step	Already extracted from the hook fan-out into persist_submit_wal_and_commit(), called directly by handle_jobsubmit (job.c:626, bug-1306)	Requirement 1 done for submit. No other command type (config change) has this yet.
Job-submit apply ordering	create_job() (job.c:583) runs before the commit barrier; on commit failure the created job is not rolled back	Requirement 2 violated today — a failed/timed-out submit leaves a phantom job in c->jobmgr
Commit-wait drive	repl_wait_majority_ack → commit_submit(..., COMMIT_MODE_WAIT, ...) → blocking pthread_cond_timedwait (commit.c:409), pooled worker, 500ms default	Requirement 9's keystone: still fully blocking
Online vs. replay apply (submit)	Replay uses restore_job() (idempotent, LSN-tagged); online uses raw create_job() directly	Requirement 3: half-unified — replay already has the reusable primitive, online doesn't use it
Online vs. replay apply (facts)	Online uses guarded mark_job_start/finish/fail/update() (monotonic-rank checks); replay uses raw restore_job_state() (no guard)	Requirement 3/7: genuinely divergent — replay can regress state that online would reject
Fact mutation timing	mark_job_* mutates job->job.state synchronously at receipt, before the (already-async) WAL append	Requirement 5/6: fully unaddressed — no apply-after-commit exists for facts at all
Hook fan-out	AFTER_EVENT for job submit is already notification-only (dead WAIT branch, persist.c:3466); facts' AFTER_EVENT still triggers the WAL append itself	Requirement 4: partially true for submit, not for facts
Restart verification	CRC-chain + entry-CRC only (replay_wal, persist.c:2667-2723); no materialized-state-vs-log-tip check	Requirement 8: fully unaddressed
Async task submission	vtask_enq_classified + slow-lane executor already exist (raft-drive-async, complete)	Requirement 9: reusable as-is, no new primitive needed

Architecture Pattern & Boundary Map

graph TB
    Client[Client jobsubmit]
    Worker[Worker fact report]
    Handler[Command handler]
    FactRecv[Fact receipt handler]
    RaftStep[Explicit raft step]
    CommitEngine[Commit engine resolve]
    ApplyPump[Apply pump LSN ordered]
    ApplySeam[apply_committed_entry]
    Replay[Restart replay]
    Hooks[After hooks notify only]
    Sched[Scheduler dispatch]
    Reply[Client reply]

    Client --> Handler
    Handler --> RaftStep
    RaftStep --> CommitEngine
    Worker --> FactRecv
    FactRecv -->|durable append only| RaftStep
    CommitEngine -->|resolve on majority| ApplyPump
    Replay -->|same seam| ApplySeam
    ApplyPump --> ApplySeam
    ApplySeam --> Hooks
    Hooks --> Sched
    Hooks --> Reply
    CommitEngine -.->|resolve failure or timeout| Reply

Architecture Integration:

• Selected pattern: Commit-then-apply pipeline with a single apply seam, mirroring the pattern already used for the raft library's own log/commit split (raft-log-replication owns log, raft-commit-safety owns commit-index, controller owns apply) — this design is the controller finishing that same separation one layer up, for its own state machine.
• Domain/feature boundaries: the raft library and commit engine remain unaware of "jobs"; apply_committed_entry() is the only place WAL-entry semantics meet job/queue/worker state.
• Existing patterns preserved: the commit engine's per-LSN registry, wal_classify's WAIT/RETAIN table, the hook system's BEFORE/AFTER shape, restore_job/mark_job_*'s existing idempotency logic — all reused, not replaced.
• New components rationale: a continuation-registration API on the commit engine (replaces blocking wait); an LSN-ordered apply pump (didn't exist — nothing today reacts to RETAIN commit); a post-replay verification step.
• Steering compliance: preserves persist-before-ACK durability and the synchronous-commit-as-policy invariant (ha-async-io.md); the raft step continues to run off the iomgr thread; a divergent-verdict fence reuses the existing "fail loud, stay alive" mechanism (persist_recover_loud_fence_if_unrecoverable), never introducing a new terminal path.

Implementation Sequencing

This design has a hard build order, not just a hard conceptual one:

1. Commit Engine Resolve (the keystone) must land first — every other component either registers a continuation with it (Raft Step, Fact Ingestion's future apply trigger) or depends on the apply pump it drives.
2. Apply Seam (apply_committed_entry) depends on (1)'s apply pump existing to call it live, but its dispatch logic and the restore_job/guarded-mark_job_* extraction can be built and unit-tested in parallel with (1).
3. Raft Step (Command) ordering fix and Fact Ingestion's receipt/apply split both depend on (1) and (2) being in place; they touch disjoint call sites (handle_jobsubmit vs. on_job_start/finish/fail/update) and can proceed in parallel with each other once (1)-(2) land.
4. Restart Replay & Verification depends on (2) (apply_committed_entry must exist before replay can be pointed at it) but is otherwise independent of (3).
5. Conformance & Regression depends on all of the above being functionally complete — it verifies the assembled behavior, not any single component in isolation.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Controller core	C (gnu17), existing	All changes land in src/controller/{job.c,persist.c,commit.c,repl.c,ha.c}	No new language/runtime
Async dispatch	vtaskmgr.c vtask_enq_classified (raft-drive-async)	Dispatches resolved-commit continuations off the ACK-processing path	Reused, not new
Serialization	Cap'n Proto (existing controller.capnp)	WAL entry payload encode/decode, unchanged	No schema change
Test harness	raft_sim.c (raft-sim-conformance), xcelib_test/controller_unit_test, /xce-test-ha	Apply-ordering assertions, idempotency assertions, live soak	Extends existing harnesses

File Structure Plan

Modified Files

• src/controller/commit.c — Replace the COMMIT_MODE_WAIT blocking-wait branch with a continuation-registration + commit_resolve() mechanism; commit_advance_locked() walks newly-decided LSNs and calls commit_resolve() for each instead of cond_broadcast-ing a single waiter set. COMMIT_MODE_RETAIN registration is unchanged (already non-blocking). Also adds the timeout-arming mechanism for registered continuations: removing the blocking wait also removes the only thing that ever detected a commit timing out (the caller's own pthread_cond_timedwait deadline), so an explicit timeout must be armed at registration and swept/fired independently.
• src/controller/repl.c — NOT touched (implementation deviation, confirmed 2026-07-07 validate-impl audit). repl_wait_majority_ack()/repl_submit_majority_commit() were never built; task 3.2 instead calls commit_submit_async() directly on p->repl.ce, bypassing repl.c entirely (see tasks.md task 3.2's Implementation Note). This narrows the reusable primitive this design originally wanted — a third parallel "append+track a commit" implementation now exists alongside COMMIT_MODE_WAIT's and COMMIT_MODE_RETAIN's. Left as a known follow-up, not fixed retroactively (out of a validation pass's boundary).
• src/controller/persist.c — Add apply_committed_entry() as the single dispatch seam (extracted from today's apply_wal_entry() switch body, persist.c:2452-2562, plus the fact-transition guard fix). apply_wal_entry() (replay) and the new apply pump both call it. Add the post-replay CLEAN/DIVERGENT verification at the end of persist_recover()/replay_wal(). persist_job_hook_fun's dead WAL_MODE_WAIT branch is removed (no longer needed once repl_submit_majority_commit fully replaces the old call convention). wal_hook_append_classified() (fact receipt path) stops mutating job state — it becomes durable-append-only. Also adds a call from the existing WAL-truncation point into the commit engine's commit_resolve() so any pending registered continuation for a truncated LSN resolves as failed instead of hanging.
• src/controller/ha.c — Adds a call from the existing demotion/role-transition point into the commit engine's commit_resolve() so any pending registered continuation invalidated by losing leadership resolves as failed instead of hanging. This is in addition to ha.c's other listed change (wiring the apply pump, below).
• src/controller/job.c — handle_jobsubmit(): defer create_job()/job-object construction to fire from apply_committed_entry() after commit succeeds, not before. mark_job_start/finish/fail/update(): split into a guard+mutation core (callable from apply) and a receipt-time validation-only entry point; their AFTER_EVENT hook-firing relocates to fire from the apply step. restore_job_state() is retired in favor of routing replay through the same guarded core.
• Apply pump wiring — implementation deviation (confirmed 2026-07-07 validate-impl audit). Task 6.1 wired the pump's trigger into commit_resolve_pending() (commit.c), not ha.c's on_commit_advanced as originally planned above — every one of commit.c's 6 internal committed_lsn-advance sites kicks the pump directly, which covers on_commit_advanced's own call path plus every other advance site on_commit_advanced alone would have missed. The walk itself ([last_applied_lsn+1 .. committed_lsn] in order, invoking apply_committed_entry() for each, dispatched via vtask_enq_classified) matches this section as designed; only the trigger's wiring point differs.
• src/controller/egress.c — No functional change; job_submit_committed() (egress.c:47) is left in place as a now-tautological defensive predicate (see research.md).

New Test Files

• src/controller/testsuite/test_apply_seam.c — unit tests: apply-after-commit ordering, no-mutation-on-failed-commit, idempotent re-apply, strict LSN ordering.
• src/controller/testsuite/test_commit_resolve.c — unit tests for the continuation/resolve conversion: exactly-once resolution, resolve on timeout/demotion/truncation, admission behavior under concurrent in-flight commits.
• Extend src/lib/raft/raft_sim.c (or its conformance test driver) with apply-ordering and idempotent-re-apply assertions (Requirement 10.1-10.3).
• Extend the /xce-test-ha soak scenario set with a kill-mid-fact-stream case (Requirement 10.4).

System Flows

Command submission (target ordering)

sequenceDiagram
    participant C as Client
    participant H as handle_jobsubmit
    participant R as Raft step
    participant CE as Commit engine
    participant AP as Apply pump
    participant AS as apply_committed_entry
    participant N as After hooks notify

    C->>H: job submit request
    H->>H: BEFORE_EVENT validate
    H->>R: encode and append WAL_JOB_SUBMIT
    R->>CE: register continuation for LSN
    CE-->>CE: majority ack or timeout or demotion
    CE->>AP: resolve LSN outcome
    alt committed
        AP->>AS: apply_committed_entry creates job
        AS->>N: AFTER_EVENT notify scheduler apiserver
        AP->>H: reply success
    else failed or timed out
        AP->>H: reply failure, no apply
    end

Key decision not obvious from the diagram: the job object itself does not exist in c->jobmgr until the committed branch runs — there is no rollback needed because there is nothing to roll back (Requirement 2.3 by construction, not by compensating action).

Worker fact ingestion (target ordering)

sequenceDiagram
    participant W as Worker
    participant F as Fact receipt handler
    participant WAL as Durable append RETAIN
    participant CE as Commit engine
    participant AP as Apply pump
    participant AS as apply_committed_entry
    participant N as Reactions slot free dispatch notify

    W->>F: job start or finish report
    F->>F: BEFORE_EVENT validate only
    F->>WAL: append entry, return immediately
    WAL->>CE: register RETAIN LSN
    Note over CE: majority may be delayed by quorum loss; entry stays pending, never applied
    CE->>AP: resolve on eventual majority commit, in original order
    AP->>AS: apply_committed_entry mutates job state guarded
    AS->>N: fire reactions post commit

Key decision: quorum-loss safety (Requirement 6) requires no special-case code — the apply pump only ever walks committed LSNs, so an unreachable majority simply means the pump has nothing new to walk. Retained facts commit and apply in original order automatically once majority is restored, because the commit engine's committed_lsn itself only ever advances as a contiguous prefix.

Restart replay and verification

flowchart LR
    Start[persist_recover] --> CRC[CRC chain check]
    CRC -->|corrupt| Truncated[RECOVER_TRUNCATED]
    CRC -->|clean| Replay[replay_wal applies each entry via apply_committed_entry]
    Replay --> Verify[Compare materialized state vs committed log tip]
    Verify -->|match| Clean[CLEAN verdict]
    Verify -->|mismatch| Divergent[DIVERGENT verdict]
    Divergent --> Fence[Non promotable fence, stays alive, retries]

Requirements Traceability

Requirement	Summary	Components	Flows
1.1-1.4	Extract raft step from hook fan-out	Raft Step (job.c, persist.c)	Command submission
2.1-2.3	Apply-after-commit ordering for commands	Apply Pump, apply_committed_entry	Command submission
3.1-3.4	Single shared apply seam, online = replay	apply_committed_entry, Restart Replay	Command submission, Restart replay
4.1-4.3	Hooks demoted to notification-only	Notification (hook.c usage)	Command submission
5.1-5.3	Facts apply-after-commit	Fact Ingestion, apply_committed_entry	Fact ingestion
6.1-6.3	Quorum-loss retained-fact behavior	Apply Pump (commit-gated by construction)	Fact ingestion
7.1-7.3	Idempotent, ordered re-application	apply_committed_entry (guarded transitions)	Command submission, Fact ingestion, Restart replay
8.1-8.4	Restart apply-correctness verification	Restart Replay & Verification	Restart replay
9.1-9.4	Bounded resource behavior under concurrent commits	Commit Engine (continuation/resolve)	Command submission
10.1-10.4	Regression/conformance coverage	Conformance (raft_sim.c), soak suite	(cross-cutting)
11.1-11.4	Preserve existing behavior and performance	All components (verification gate)	(cross-cutting)

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
Commit Engine Resolve	commit.c/repl.c	Convert COMMIT_MODE_WAIT from blocking wait to continuation/resolve	1.3, 1.4, 9.1-9.4	vtaskmgr vtask_enq_classified (P0)	Service
Apply Seam	persist.c	Single dispatch point mutating state from a committed entry	2, 3, 5.2, 7	restore_job/mark_job_* guards (P0)	Service
Raft Step (Command)	job.c/persist.c	Explicit append+commit invoked by command handlers	1.1, 1.2, 2.1	Commit Engine Resolve (P0)	Service
Fact Ingestion	job.c/persist.c	Durable-append-only at receipt; defers mutation/reaction	5, 6	Apply Seam (P0)	Service
Restart Replay & Verification	persist.c	Replay via Apply Seam; post-replay CLEAN/DIVERGENT check	3.1-3.2, 8	Apply Seam (P0), existing loud-fence mechanism (P0)	Service
Notification (hooks)	hook.c usage in job.c/persist.c	AFTER_EVENT fires only post-apply, pure notify	4	Apply Seam (P0)	Event
Conformance & Regression	raft_sim.c, testsuite, /xce-test-ha	Assert ordering/idempotency invariants	10, 11	Apply Seam, Commit Engine Resolve (P1)	Batch

Commit Engine

Commit Engine Resolve

Field	Detail
Intent	Replace the blocking majority-commit wait with continuation registration and event-driven resolution
Requirements	1.3, 1.4, 9.1, 9.2, 9.3, 9.4

Responsibilities & Constraints

• Owns the per-LSN commit registry's WAIT-mode bookkeeping (already exists; behavior at resolution changes).
• Must resolve every registered LSN's outcome exactly once, across {majority-ack, timeout, demotion, truncation} (Req 9.4).
• Must not block the thread that calls commit_advance_locked() (the replication-ACK path) — dispatch continuations via vtask_enq_classified, never inline.
• Preserves the existing lock-ordering rule: never hold p->lock across ce->lock (persist.c:3347-3350) — the continuation dispatch happens after ce->lock is released.

Dependencies

• Outbound: vtaskmgr.c vtask_enq_classified — dispatch a resolved continuation (Criticality P0)
• Inbound: repl_submit_majority_commit() (repl.c, replaces repl_wait_majority_ack) — registers a continuation (Criticality P0)

Contracts: Service [x]

Service Interface

typedef enum { COMMIT_OUTCOME_COMMITTED, COMMIT_OUTCOME_FAILED } commit_outcome_t;

typedef void (*commit_continuation_fn)(void* ctx, uint64_t lsn,
                                        commit_outcome_t outcome, int rc);

/* Registers a continuation for lsn under COMMIT_MODE_WAIT. Returns 0 on
 * successful registration, negative on admission rejection (mirrors today's
 * -XCE_ERR_NOT_READY). Never blocks. */
int commit_submit_async(commit_engine_t* ce, uint64_t lsn,
                         commit_continuation_fn fn, void* ctx);

/* Called from commit_advance_locked() (majority path) and from the existing
 * timeout/demotion/truncation handling (failure path). Dispatches the
 * registered continuation, if any, via vtask_enq_classified. Idempotent:
 * a second resolve() for an already-resolved lsn is a no-op. */
void commit_resolve(commit_engine_t* ce, uint64_t lsn,
                     commit_outcome_t outcome, int rc);

• Preconditions: lsn was previously appended locally.
• Postconditions: exactly one of {COMMITTED, FAILED} is delivered to the registered continuation, exactly once.
• Invariants: resolution order across distinct LSNs is not required to be FIFO by this component alone — LSN application order is enforced by the Apply Seam's pump, not by resolve dispatch order.

Apply Seam

apply_committed_entry

Field	Detail
Intent	The single function that mutates in-memory state for one committed WAL entry, used by both live apply and replay
Requirements	2.2, 2.3, 3.1, 3.2, 3.3, 3.4, 5.2, 7.1, 7.2, 7.3

Responsibilities & Constraints

• Dispatches on WAL entry type: WAL_JOB_SUBMIT → job construction via the existing restore_job() primitive (already idempotent, already LSN-tagged); WAL_JOB_{START,FINISH,FAIL,UPDATE,...} → the guarded transition core factored out of mark_job_start/finish/fail/update(); WAL_QUEUE_CTRL → restore_queue_state(); WAL_WORKER_CTRL → worker_from_capnp().
• Fires the type's AFTER_EVENT hook after the mutation completes, never before.
• Must be safe to invoke more than once for the same entry (replay re-running an already-applied LSN, or a resync re-delivering an already-applied fact) — this safety comes from the guards already present in restore_job (-EEXIST-as-success) and the (relocated) mark_job_* monotonic-rank checks, not from new dedup logic.
• Must never be invoked out of LSN order by either caller (live pump or replay loop).

Dependencies

• Outbound: restore_job, restore_queue_state, worker_from_capnp, the guarded mark_job_* core (all job.c) (Criticality P0)
• Inbound: Apply Pump (live path), replay_wal (restart path) (Criticality P0)

Contracts: Service [x] / Batch [x]

Service Interface

/* Applies one committed WAL entry to in-memory state. Called with entries
 * strictly in ascending lsn order by both the live apply pump and replay.
 * Returns 0 on success (including a no-op idempotent re-apply), negative on
 * a genuine apply failure (which is a fenceable condition, not a retry). */
int apply_committed_entry(persist_t* p, uint32_t wal_type, uint64_t lsn,
                           const void* payload, uint32_t len);

• Preconditions: lsn is committed (majority-durable); every entry below lsn has already been applied.
• Postconditions: in-memory state reflects the entry; the type's AFTER_EVENT hook has fired exactly once for this application.
• Invariants: applying the same (wal_type, lsn, payload) twice produces the same observable state as applying it once.

Batch / Job Contract

• Trigger: commit-advance (live) walking [last_applied_lsn+1 .. committed_lsn], or replay iterating the recovered log.
• Input / validation: WAL entry header + payload, already CRC-verified by the point apply is reached.
• Output / destination: in-memory job/queue/worker state; AFTER_EVENT hook firing.
• Idempotency & recovery: guaranteed by the underlying restore_job/guarded-transition primitives; last_applied_lsn is the resumable cursor for restart.

Raft Step (Command)

Explicit command raft step

Field	Detail
Intent	Append + majority-commit a command entry, invoked directly by the command handler, never from a hook
Requirements	1.1, 1.2, 2.1

Responsibilities & Constraints

• For job submit: already implemented as persist_submit_wal_and_commit() (job.c:626) — this design changes its caller-side contract from "returns synchronously, caller replies inline" to "registers a continuation via repl_submit_majority_commit(), caller's reply logic becomes the continuation."
• The command handler (handle_jobsubmit) no longer calls create_job() before this step; object construction moves to the continuation's success branch, via apply_committed_entry().
• Must run off the iomgr thread (unchanged from today).

Dependencies

• Outbound: Commit Engine Resolve (commit_submit_async) (Criticality P0)
• Outbound: Apply Seam, on the continuation's success branch (Criticality P0)

Contracts: Service [x]

Implementation Notes

• Integration: the existing persist_submit_wal_and_commit() encode/append loop is retained; only its commit-wait call changes from blocking to continuation-registering.
• Validation: a config-change command handler follows the identical pattern once introduced (out of scope to build here, but the contract is designed to generalize — see Requirements Traceability 1.1 covering "job submit, configuration change").
• Risks: converting the one existing COMMIT_MODE_WAIT call site is narrow in surface area but touches a durability-critical path — full existing HA soak suite (Req 11.2) must pass before/after.

Fact Ingestion

Fact receipt split

Field	Detail
Intent	Split today's receipt-time mutate-then-append into durable-append-only at receipt, mutate+react at apply
Requirements	5.1, 5.2, 5.3, 6.1, 6.2, 6.3

Responsibilities & Constraints

• on_job_start/on_job_finish/on_job_fail/on_job_update (job.c) validate (existing BEFORE_EVENT, unchanged) and then only encode + durably append via wal_hook_append_classified (WAL_MODE_RETAIN) — no state mutation, no AFTER_EVENT firing at this point.
• The guarded mutation core (extracted from today's mark_job_*) and the AFTER_EVENT hook firing both relocate to run from apply_committed_entry().
• No local-apply allowlist under quorum loss (Req 6.2) — enforced structurally: the apply pump only ever processes committed LSNs, so there is no code path that can apply an uncommitted fact.

Dependencies

• Outbound: durable append (wal_hook_append_classified, existing, unchanged) (Criticality P0)
• Outbound: Apply Seam, for the deferred mutation (Criticality P0)

Contracts: Event [x] / Batch [x]

Event Contract

• Published events — implementation deviation (confirmed 2026-07-07 validate-impl audit). All nine job-state fact WAL types route through a single guard-only apply_job_update_transition() core in apply_committed_entry() and fire one generic HOOK_DETAIL_JOB_UPDATE, not per-type HOOK_DETAIL_JOB_START/_EXIT/_FAIL/etc. as stated below. Verified safe in practice: sched_job_update_fun/apiserver_job_update_fun (src/controller/sched.c:446-465) already exist and consume exactly the generic (cluster_t*, job_t*) form the apply path passes — no subscriber assumed per-type dispatch. Left as a documentation correction, not a functional gap.
• Published events: AFTER_EVENT for HOOK_DETAIL_JOB_START/_EXIT/etc., now fired from the apply step, not the receipt handler. (superseded by the note above)
• Subscribed events: none new.
• Ordering / delivery guarantees: strict commit order via the shared apply pump; at-least-once redelivery for facts lost before append remains worker-rejoin-resync's responsibility (out of boundary).

Implementation Notes

• Integration: mark_job_start/finish/fail/update's guard predicates (job_state_is_backward, job_state_is_terminal) become the guard logic inside apply_committed_entry's fact-transition case — the same predicates, relocated, not rewritten.
• Validation: audit every HOOK_DETAIL_JOB_* subscriber for an assumption that BEFORE_EVENT and the state mutation happen together at receipt; only the mutation moves, BEFORE_EVENT validation timing is unchanged.
• Risks: this is the widest behavioral change in the design — worker-visible latency for slot-free/dispatch grows by one commit round-trip (accepted cost, documented in facts-in-raft.md, Req 11.4 bounds it to "no per-fact synchronous barrier," not "no latency change").

Restart Replay & Verification

Post-replay apply-correctness verification

Field	Detail
Intent	Prove materialized state matches the committed log tip after replay, beyond CRC structural checks
Requirements	3.1, 3.2, 8.1, 8.2, 8.3, 8.4

Responsibilities & Constraints

• Runs after replay_wal()'s existing CRC-chain verification succeeds (RECOVER_CLEAN), applying every entry via apply_committed_entry() (replacing today's direct restore_job/restore_job_state calls in apply_wal_entry).
• Computes a lightweight expectation from the committed log tip (e.g., last-applied LSN reaches the recovered commit watermark, plus a count/hash of materialized entities) and compares it to the actual post-replay state.
• On mismatch: reuses persist_recover_loud_fence_if_unrecoverable()'s existing mechanism (persist.c:2990) and the recovery_unrecoverable sticky flag / ha->in_catch_up promotion gate — a DIVERGENT verdict is a new input to that existing fence function, not a new fence mechanism.

Dependencies

• Outbound: Apply Seam (Criticality P0)
• Outbound: existing loud-fence mechanism (persist_recover_loud_fence_if_unrecoverable, recovery_unrecoverable, ha->in_catch_up) (Criticality P0)

Contracts: Batch [x] / State [x]

State Management

• State model: last_applied_lsn (new, small addition to persist_t) tracks replay/apply progress; recovery_unrecoverable (existing) is the sticky fence flag.
• Persistence & consistency: last_applied_lsn is derived at replay time (not itself persisted) — restart always recomputes it from the WAL, so it cannot itself drift from disk state.
• Concurrency strategy: single-threaded during replay (unchanged); the live apply pump takes over last_applied_lsn advancement once replay completes.

Data Models

No new persistent schema. WAL_JOB_SUBMIT/WAL_JOB_START/WAL_JOB_FINISH/WAL_JOB_FAIL/WAL_JOB_UPDATE/WAL_QUEUE_CTRL/WAL_WORKER_CTRL entry formats, wal_classify's WAIT/RETAIN table, and job_t's state bits (JOB_STAT_PEND/RUN/DONE/EXIT) are all unchanged.

In-memory additions:

• persist_t.last_applied_lsn (_Atomic uint64_t) — the apply pump's resumable cursor, analogous to the existing committed_lsn / durable_lsn watermarks.
• commit_engine_t's per-LSN WAIT entry gains a commit_continuation_fn + void* ctx field (replacing the pthread_cond_t wait registration for that mode; COMMIT_MODE_RETAIN entries are unaffected).

Error Handling

Error Strategy

• Failed/timed-out command commit: commit_resolve(..., COMMIT_OUTCOME_FAILED, rc) fires the registered continuation with the original error-code vocabulary (-ETIMEDOUT, -XCE_ERR_NOT_READY, etc., unchanged from today); the continuation replies failure to the client and never calls apply_committed_entry() — no rollback needed because no mutation occurred (Req 2.3).
• Apply-time failure (e.g., a decode error on an already-committed, already-CRC-valid entry): a genuine programming-invariant violation, not an operational condition — treated per the controller's existing "never self-terminate" rule: log loud, do not apply, and let the DIVERGENT-verdict fence (restart path) or an equivalent live-path fence catch it; never assert()/abort on this path in production.
• DIVERGENT restart verdict: non-promotable fence via the existing recovery_unrecoverable mechanism — stays alive, keeps retrying catch-up, never exits (steering ha-async-io.md / controller CLAUDE.md process-lifecycle invariant).
• Quorum loss during a retained fact's commit: no error surfaces to the worker (facts have no requester awaiting confirmation, unchanged); the fact remains durably appended and unapplied until majority is restored (Req 6).

Monitoring

• Log a WARN when a retained fact's backlog age (durably appended, not yet committed) exceeds a threshold, consistent with facts-in-raft.md's D1 refinement (metric on retained-fact backlog age).
• Log the CLEAN/DIVERGENT verdict at INFO/ERROR respectively on every restart, alongside the existing CRC verdict.

Testing Strategy

• Unit Tests:
  • Apply-after-commit ordering: a failed commit produces zero observable state mutation (Req 2.3).
  • Idempotent re-apply: applying the same committed entry twice (simulated replay-of-already-applied) is a no-op (Req 7.1, 7.2).
  • Commit-resolve exactly-once: registering a continuation and forcing each of {majority-ack, timeout, demotion, truncation} resolves it exactly once (Req 9.4).
  • Fact monotonicity under replay: a duplicate/out-of-order FINISH-then-START replay sequence does not regress state (Req 7.2 — directly targets the restore_job_state gap found in research.md).
• Integration Tests:
  • End-to-end job submit: commit success → job visible and notified; commit failure → job never visible, client sees failure (Req 2, 11.3).
  • End-to-end fact flow: worker reports job start → no slot-free/dispatch/notify until the entry commits (Req 5.3).
  • Quorum-loss-then-restore: facts submitted during a majority-loss window apply in original order once quorum returns, with no local-apply escape hatch exercised (Req 6).
  • Restart CLEAN verdict on a normal shutdown/restart; forced DIVERGENT verdict (inject a materialization mismatch) confirms the fence engages and the process stays alive (Req 8).
• Regression / Conformance:
  • Extend raft_sim.c's conformance harness with "no reaction before commit" and "idempotent re-apply" assertions layered on its existing apply-after-on_commit_advanced loop (Req 10.1-10.3).
  • Extend /xce-test-ha with a kill-the-primary-mid-fact-stream scenario asserting no phantom slot-frees and no notification for an uncommitted fact (Req 10.4).
  • Full existing unit/integration/soak suite re-run as a regression gate (Req 11.1, 11.2), with the known, intentional exception of tests that asserted the old apply-before-commit ordering (updated, not skipped).

Performance & Scalability

• Target: fact-commit throughput under sustained high-volume worker churn must not regress relative to pre-feature behavior (Req 11.4) — the group-commit amortization for WAL_MODE_RETAIN entries is unchanged by this design; only the post-commit reaction timing shifts.
• The keystone conversion removes one pooled-worker-thread-per-in-flight-commit cost entirely (today's COMMIT_WAIT_RESERVED-gated blocking wait), which should improve concurrent-submit headroom, not merely preserve it — validate via Req 9's bounded-resource tests under a concurrent-submit burst.