myxce/.kiro/specs/raft-apply-decoupling/design.md

# Design Document: raft-apply-decoupling

## Overview

**Purpose**: This feature gives the XCE controller one canonical, ordered path from a client/worker request to committed, applied cluster state — append → majority-commit → apply → notify — replacing today's mix of an already-partially-fixed command path (bug-1306) and a still-fully-optimistic fact path (worker-reported job start/finish/fail/rusage, applied and reacted to on receipt, ahead of commit).

**Users**: Cluster operators (who need failover to never lose or fabricate job state) and maintainers of the controller's submit/commit/replay code (who need one apply seam instead of two divergent ones).

**Impact**: Changes the commit engine's `COMMIT_MODE_WAIT` path from a blocking wait to an event-driven resolve; introduces a single `apply_committed_entry()` seam invoked by both live commit-advance and restart replay; moves worker-fact state mutation and reactions from receipt-time to post-commit-time; adds a post-replay CLEAN/DIVERGENT verification.

### Goals
- One explicit raft step (append + majority commit) per state-changing operation, invoked directly by its handler — not incidentally from a hook.
- One apply seam, used identically by live commit-advance and restart replay, applied strictly in commit order.
- Commands and facts converge on that one apply seam; only their ingress and commit-wait policy differ.
- A quorum-loss window degrades safely by construction (retained-but-unapplied), with no special-case "local apply" code path.
- A restart proves its own correctness (CLEAN/DIVERGENT) instead of only checking WAL structural integrity.
- No regression to any HA correctness or throughput property already proven by prior specs.

### Non-Goals
- Rewriting the raft library's protocol core (`raft_advance_commit`, `on_commit_advanced`, etc.) — consumed as-is.
- Collapsing a job array into a single WAL record (`job-array-single-wal-record`).
- Moving snapshot send / config-push / peer-connect off the iomgr/delaymgr threads (`raft-drive-async`, already done separately).
- Building `worker-rejoin-resync` — this design depends on it as a future backstop, consistent with every sibling spec that already accepted this gap.
- Entity-bearing snapshot content or read-consistency/leader-lease work.

## Boundary Commitments

### This Spec Owns
- The explicit raft-commit step for command entries and its call sites (already partially landed for job submit via bug-1306; this spec completes the ordering and generalizes the mechanism).
- The commit engine's `COMMIT_MODE_WAIT` continuation/resolve mechanism (the keystone conversion away from blocking wait).
- The single `apply_committed_entry()` seam and its dispatch table, and the "apply pump" that drives it off commit-advance in strict LSN order.
- The fact-ingestion split (durable-append-only at receipt vs. mutation+reaction at apply).
- The post-replay CLEAN/DIVERGENT verification and its non-promotable fence.
- Conformance/regression assertions for apply-ordering, idempotency, and no-reaction-before-commit.

### Out of Boundary
- The raft library's callback contracts (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`, `raft_advance_commit`) — consumed unchanged.
- Transport/timer-thread blocking (snapshot send, config-push wait, peer connect) — `raft-drive-async`'s domain; this design only *consumes* its `vtask_enq_classified` / slow-lane primitives.
- Worker join/rejoin protocol changes — `worker-join-refactor` / `worker-rejoin-resync`.
- Entity-bearing snapshot format/content — `ha-entity-snapshots` / `config-upgrade-snapshot`. This design's replay-side changes must remain compatible with `ha-entity-snapshots`' restore-before-advance `load_snapshot` path, which already shares `restore_job`/`restore_queue_state` with WAL replay (see Architecture).
- Removing `job_submit_committed()` (egress.c:47) — becomes a harmless tautology once this design lands, but its removal is a documented follow-up, not in scope (research.md, Design Decisions).

### Allowed Dependencies
- `ha-commit-engine` / `ha-commit-barrier-rollout`: majority-commit and idempotency primitives, consumed as-is.
- `ha-term-safety` (complete): `on_commit_advanced` / `raft_advance_commit` wiring on the ACK path (advisory `raft_commit_index`, Option A) — this design's apply pump is a **new** consumer of the existing durable-majority commit advance; it does not change how `committed_lsn` itself advances.
- `raft-drive-async` (complete): `vtask_enq_classified` + slow-lane executor + `VTASK_LIVENESS`/`VTASK_LIVE_BLOCK`/`VTASK_REQ_BLOCK` admission classes — reused for dispatching resolved-commit continuations off the ACK-processing thread.
- `raft-sim-conformance`: existing deterministic-simulator harness, extended (not replaced) with new assertions.

### Revalidation Triggers
- Any change to the commit engine's per-LSN structure (`commit_engine_t`, `ce->...`) or its locking discipline (`p->lock` vs `ce->lock` ordering, persist.c:3347-3350).
- Any change to `WAL_MODE_WAIT`/`WAL_MODE_RETAIN` classification (`wal_classify`, persist.c:3218) — the apply pump's per-type behavior depends on this classification being exhaustive.
- Any new WAL entry type — must be added to the `apply_committed_entry()` dispatch table or it silently fails to apply.
- `worker-rejoin-resync` landing — must re-verify the fact-apply idempotency guards against its replay/resync ordering (shared `mark_job_*` guards).
- `job-array-single-wal-record` landing — builds its array-expand step directly on `apply_committed_entry()`; a signature change here is a breaking change there.

## Architecture

### Existing Architecture Analysis

Current state, verified against the live codebase (not the 9-day-old brief — see `research.md`):

| Aspect | Current state | Gap vs. this design |
|---|---|---|
| Job-submit raft step | Already extracted from the hook fan-out into `persist_submit_wal_and_commit()`, called directly by `handle_jobsubmit` (job.c:626, bug-1306) | Requirement 1 done for submit. No other command type (config change) has this yet. |
| Job-submit apply ordering | `create_job()` (job.c:583) runs **before** the commit barrier; on commit failure the created job is **not** rolled back | Requirement 2 violated today — a failed/timed-out submit leaves a phantom job in `c->jobmgr` |
| Commit-wait drive | `repl_wait_majority_ack` → `commit_submit(..., COMMIT_MODE_WAIT, ...)` → blocking `pthread_cond_timedwait` (commit.c:409), pooled worker, 500ms default | Requirement 9's keystone: still fully blocking |
| Online vs. replay apply (submit) | Replay uses `restore_job()` (idempotent, LSN-tagged); online uses raw `create_job()` directly | Requirement 3: half-unified — replay already has the reusable primitive, online doesn't use it |
| Online vs. replay apply (facts) | Online uses guarded `mark_job_start/finish/fail/update()` (monotonic-rank checks); replay uses raw `restore_job_state()` (**no** guard) | Requirement 3/7: genuinely divergent — replay can regress state that online would reject |
| Fact mutation timing | `mark_job_*` mutates `job->job.state` synchronously at receipt, **before** the (already-async) WAL append | Requirement 5/6: fully unaddressed — no apply-after-commit exists for facts at all |
| Hook fan-out | `AFTER_EVENT` for job submit is already notification-only (dead WAIT branch, persist.c:3466); facts' `AFTER_EVENT` still triggers the WAL append itself | Requirement 4: partially true for submit, not for facts |
| Restart verification | CRC-chain + entry-CRC only (`replay_wal`, persist.c:2667-2723); no materialized-state-vs-log-tip check | Requirement 8: fully unaddressed |
| Async task submission | `vtask_enq_classified` + slow-lane executor already exist (`raft-drive-async`, complete) | Requirement 9: reusable as-is, no new primitive needed |

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    Client[Client jobsubmit]
    Worker[Worker fact report]
    Handler[Command handler]
    FactRecv[Fact receipt handler]
    RaftStep[Explicit raft step]
    CommitEngine[Commit engine resolve]
    ApplyPump[Apply pump LSN ordered]
    ApplySeam[apply_committed_entry]
    Replay[Restart replay]
    Hooks[After hooks notify only]
    Sched[Scheduler dispatch]
    Reply[Client reply]

    Client --> Handler
    Handler --> RaftStep
    RaftStep --> CommitEngine
    Worker --> FactRecv
    FactRecv -->|durable append only| RaftStep
    CommitEngine -->|resolve on majority| ApplyPump
    Replay -->|same seam| ApplySeam
    ApplyPump --> ApplySeam
    ApplySeam --> Hooks
    Hooks --> Sched
    Hooks --> Reply
    CommitEngine -.->|resolve failure or timeout| Reply
```

**Architecture Integration**:
- **Selected pattern**: Commit-then-apply pipeline with a single apply seam, mirroring the pattern already used for the raft library's own log/commit split (`raft-log-replication` owns log, `raft-commit-safety` owns commit-index, controller owns apply) — this design is the controller finishing that same separation one layer up, for its own state machine.
- **Domain/feature boundaries**: the raft library and commit engine remain unaware of "jobs"; `apply_committed_entry()` is the only place WAL-entry semantics meet job/queue/worker state.
- **Existing patterns preserved**: the commit engine's per-LSN registry, `wal_classify`'s WAIT/RETAIN table, the hook system's BEFORE/AFTER shape, `restore_job`/`mark_job_*`'s existing idempotency logic — all reused, not replaced.
- **New components rationale**: a continuation-registration API on the commit engine (replaces blocking wait); an LSN-ordered apply pump (didn't exist — nothing today reacts to RETAIN commit); a post-replay verification step.
- **Steering compliance**: preserves persist-before-ACK durability and the synchronous-commit-as-policy invariant (`ha-async-io.md`); the raft step continues to run off the iomgr thread; a divergent-verdict fence reuses the existing "fail loud, stay alive" mechanism (`persist_recover_loud_fence_if_unrecoverable`), never introducing a new terminal path.

### Implementation Sequencing

This design has a hard build order, not just a hard conceptual one:

1. **Commit Engine Resolve** (the keystone) must land first — every other component either registers a continuation with it (Raft Step, Fact Ingestion's future apply trigger) or depends on the apply pump it drives.
2. **Apply Seam** (`apply_committed_entry`) depends on (1)'s apply pump existing to call it live, but its dispatch logic and the `restore_job`/guarded-`mark_job_*` extraction can be built and unit-tested in parallel with (1).
3. **Raft Step (Command)** ordering fix and **Fact Ingestion**'s receipt/apply split both depend on (1) and (2) being in place; they touch disjoint call sites (`handle_jobsubmit` vs. `on_job_start/finish/fail/update`) and can proceed in parallel with each other once (1)-(2) land.
4. **Restart Replay & Verification** depends on (2) (`apply_committed_entry` must exist before replay can be pointed at it) but is otherwise independent of (3).
5. **Conformance & Regression** depends on all of the above being functionally complete — it verifies the assembled behavior, not any single component in isolation.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|------------------|-------|
| Controller core | C (gnu17), existing | All changes land in `src/controller/{job.c,persist.c,commit.c,repl.c,ha.c}` | No new language/runtime |
| Async dispatch | `vtaskmgr.c` `vtask_enq_classified` (raft-drive-async) | Dispatches resolved-commit continuations off the ACK-processing path | Reused, not new |
| Serialization | Cap'n Proto (existing `controller.capnp`) | WAL entry payload encode/decode, unchanged | No schema change |
| Test harness | `raft_sim.c` (raft-sim-conformance), `xcelib_test`/`controller_unit_test`, `/xce-test-ha` | Apply-ordering assertions, idempotency assertions, live soak | Extends existing harnesses |

## File Structure Plan

### Modified Files
- `src/controller/commit.c` — Replace the `COMMIT_MODE_WAIT` blocking-wait branch with a continuation-registration + `commit_resolve()` mechanism; `commit_advance_locked()` walks newly-decided LSNs and calls `commit_resolve()` for each instead of `cond_broadcast`-ing a single waiter set. `COMMIT_MODE_RETAIN` registration is unchanged (already non-blocking). Also adds the timeout-arming mechanism for registered continuations: removing the blocking wait also removes the only thing that ever detected a commit timing out (the caller's own `pthread_cond_timedwait` deadline), so an explicit timeout must be armed at registration and swept/fired independently.
- `src/controller/repl.c` — **NOT touched (implementation deviation, confirmed 2026-07-07 validate-impl audit).** `repl_wait_majority_ack()`/`repl_submit_majority_commit()` were never built; task 3.2 instead calls `commit_submit_async()` directly on `p->repl.ce`, bypassing `repl.c` entirely (see tasks.md task 3.2's Implementation Note). This narrows the reusable primitive this design originally wanted — a third parallel "append+track a commit" implementation now exists alongside `COMMIT_MODE_WAIT`'s and `COMMIT_MODE_RETAIN`'s. Left as a known follow-up, not fixed retroactively (out of a validation pass's boundary).
- `src/controller/persist.c` — Add `apply_committed_entry()` as the single dispatch seam (extracted from today's `apply_wal_entry()` switch body, persist.c:2452-2562, plus the fact-transition guard fix). `apply_wal_entry()` (replay) and the new apply pump both call it. Add the post-replay CLEAN/DIVERGENT verification at the end of `persist_recover()`/`replay_wal()`. `persist_job_hook_fun`'s dead `WAL_MODE_WAIT` branch is removed (no longer needed once `repl_submit_majority_commit` fully replaces the old call convention). `wal_hook_append_classified()` (fact receipt path) stops mutating job state — it becomes durable-append-only. Also adds a call from the existing WAL-truncation point into the commit engine's `commit_resolve()` so any pending registered continuation for a truncated LSN resolves as failed instead of hanging.
- `src/controller/ha.c` — Adds a call from the existing demotion/role-transition point into the commit engine's `commit_resolve()` so any pending registered continuation invalidated by losing leadership resolves as failed instead of hanging. This is in addition to `ha.c`'s other listed change (wiring the apply pump, below).
- `src/controller/job.c` — `handle_jobsubmit()`: defer `create_job()`/job-object construction to fire from `apply_committed_entry()` after commit succeeds, not before. `mark_job_start/finish/fail/update()`: split into a guard+mutation core (callable from apply) and a receipt-time validation-only entry point; their `AFTER_EVENT` hook-firing relocates to fire from the apply step. `restore_job_state()` is retired in favor of routing replay through the same guarded core.
- **Apply pump wiring — implementation deviation (confirmed 2026-07-07 validate-impl audit).** Task 6.1 wired the pump's trigger into `commit_resolve_pending()` (`commit.c`), not `ha.c`'s `on_commit_advanced` as originally planned above — every one of `commit.c`'s 6 internal `committed_lsn`-advance sites kicks the pump directly, which covers `on_commit_advanced`'s own call path plus every other advance site `on_commit_advanced` alone would have missed. The walk itself (`[last_applied_lsn+1 .. committed_lsn]` in order, invoking `apply_committed_entry()` for each, dispatched via `vtask_enq_classified`) matches this section as designed; only the trigger's wiring point differs.
- `src/controller/egress.c` — No functional change; `job_submit_committed()` (egress.c:47) is left in place as a now-tautological defensive predicate (see research.md).

### New Test Files
- `src/controller/testsuite/test_apply_seam.c` — unit tests: apply-after-commit ordering, no-mutation-on-failed-commit, idempotent re-apply, strict LSN ordering.
- `src/controller/testsuite/test_commit_resolve.c` — unit tests for the continuation/resolve conversion: exactly-once resolution, resolve on timeout/demotion/truncation, admission behavior under concurrent in-flight commits.
- Extend `src/lib/raft/raft_sim.c` (or its conformance test driver) with apply-ordering and idempotent-re-apply assertions (Requirement 10.1-10.3).
- Extend the `/xce-test-ha` soak scenario set with a kill-mid-fact-stream case (Requirement 10.4).

## System Flows

### Command submission (target ordering)

```mermaid
sequenceDiagram
    participant C as Client
    participant H as handle_jobsubmit
    participant R as Raft step
    participant CE as Commit engine
    participant AP as Apply pump
    participant AS as apply_committed_entry
    participant N as After hooks notify

    C->>H: job submit request
    H->>H: BEFORE_EVENT validate
    H->>R: encode and append WAL_JOB_SUBMIT
    R->>CE: register continuation for LSN
    CE-->>CE: majority ack or timeout or demotion
    CE->>AP: resolve LSN outcome
    alt committed
        AP->>AS: apply_committed_entry creates job
        AS->>N: AFTER_EVENT notify scheduler apiserver
        AP->>H: reply success
    else failed or timed out
        AP->>H: reply failure, no apply
    end
```

Key decision not obvious from the diagram: the job object itself does not exist in `c->jobmgr` until the committed branch runs — there is no rollback needed because there is nothing to roll back (Requirement 2.3 by construction, not by compensating action).

### Worker fact ingestion (target ordering)

```mermaid
sequenceDiagram
    participant W as Worker
    participant F as Fact receipt handler
    participant WAL as Durable append RETAIN
    participant CE as Commit engine
    participant AP as Apply pump
    participant AS as apply_committed_entry
    participant N as Reactions slot free dispatch notify

    W->>F: job start or finish report
    F->>F: BEFORE_EVENT validate only
    F->>WAL: append entry, return immediately
    WAL->>CE: register RETAIN LSN
    Note over CE: majority may be delayed by quorum loss; entry stays pending, never applied
    CE->>AP: resolve on eventual majority commit, in original order
    AP->>AS: apply_committed_entry mutates job state guarded
    AS->>N: fire reactions post commit
```

Key decision: quorum-loss safety (Requirement 6) requires no special-case code — the apply pump only ever walks committed LSNs, so an unreachable majority simply means the pump has nothing new to walk. Retained facts commit and apply in original order automatically once majority is restored, because the commit engine's `committed_lsn` itself only ever advances as a contiguous prefix.

### Restart replay and verification

```mermaid
flowchart LR
    Start[persist_recover] --> CRC[CRC chain check]
    CRC -->|corrupt| Truncated[RECOVER_TRUNCATED]
    CRC -->|clean| Replay[replay_wal applies each entry via apply_committed_entry]
    Replay --> Verify[Compare materialized state vs committed log tip]
    Verify -->|match| Clean[CLEAN verdict]
    Verify -->|mismatch| Divergent[DIVERGENT verdict]
    Divergent --> Fence[Non promotable fence, stays alive, retries]
```

## Requirements Traceability

| Requirement | Summary | Components | Flows |
|---|---|---|---|
| 1.1-1.4 | Extract raft step from hook fan-out | Raft Step (job.c, persist.c) | Command submission |
| 2.1-2.3 | Apply-after-commit ordering for commands | Apply Pump, apply_committed_entry | Command submission |
| 3.1-3.4 | Single shared apply seam, online = replay | apply_committed_entry, Restart Replay | Command submission, Restart replay |
| 4.1-4.3 | Hooks demoted to notification-only | Notification (hook.c usage) | Command submission |
| 5.1-5.3 | Facts apply-after-commit | Fact Ingestion, apply_committed_entry | Fact ingestion |
| 6.1-6.3 | Quorum-loss retained-fact behavior | Apply Pump (commit-gated by construction) | Fact ingestion |
| 7.1-7.3 | Idempotent, ordered re-application | apply_committed_entry (guarded transitions) | Command submission, Fact ingestion, Restart replay |
| 8.1-8.4 | Restart apply-correctness verification | Restart Replay & Verification | Restart replay |
| 9.1-9.4 | Bounded resource behavior under concurrent commits | Commit Engine (continuation/resolve) | Command submission |
| 10.1-10.4 | Regression/conformance coverage | Conformance (raft_sim.c), soak suite | (cross-cutting) |
| 11.1-11.4 | Preserve existing behavior and performance | All components (verification gate) | (cross-cutting) |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|---|---|---|---|---|---|
| Commit Engine Resolve | commit.c/repl.c | Convert `COMMIT_MODE_WAIT` from blocking wait to continuation/resolve | 1.3, 1.4, 9.1-9.4 | vtaskmgr `vtask_enq_classified` (P0) | Service |
| Apply Seam | persist.c | Single dispatch point mutating state from a committed entry | 2, 3, 5.2, 7 | restore_job/mark_job_* guards (P0) | Service |
| Raft Step (Command) | job.c/persist.c | Explicit append+commit invoked by command handlers | 1.1, 1.2, 2.1 | Commit Engine Resolve (P0) | Service |
| Fact Ingestion | job.c/persist.c | Durable-append-only at receipt; defers mutation/reaction | 5, 6 | Apply Seam (P0) | Service |
| Restart Replay & Verification | persist.c | Replay via Apply Seam; post-replay CLEAN/DIVERGENT check | 3.1-3.2, 8 | Apply Seam (P0), existing loud-fence mechanism (P0) | Service |
| Notification (hooks) | hook.c usage in job.c/persist.c | AFTER_EVENT fires only post-apply, pure notify | 4 | Apply Seam (P0) | Event |
| Conformance & Regression | raft_sim.c, testsuite, `/xce-test-ha` | Assert ordering/idempotency invariants | 10, 11 | Apply Seam, Commit Engine Resolve (P1) | Batch |

### Commit Engine

#### Commit Engine Resolve

| Field | Detail |
|-------|--------|
| Intent | Replace the blocking majority-commit wait with continuation registration and event-driven resolution |
| Requirements | 1.3, 1.4, 9.1, 9.2, 9.3, 9.4 |

**Responsibilities & Constraints**
- Owns the per-LSN commit registry's WAIT-mode bookkeeping (already exists; behavior at resolution changes).
- Must resolve every registered LSN's outcome exactly once, across `{majority-ack, timeout, demotion, truncation}` (Req 9.4).
- Must not block the thread that calls `commit_advance_locked()` (the replication-ACK path) — dispatch continuations via `vtask_enq_classified`, never inline.
- Preserves the existing lock-ordering rule: never hold `p->lock` across `ce->lock` (persist.c:3347-3350) — the continuation dispatch happens after `ce->lock` is released.

**Dependencies**
- Outbound: `vtaskmgr.c` `vtask_enq_classified` — dispatch a resolved continuation (Criticality P0)
- Inbound: `repl_submit_majority_commit()` (repl.c, replaces `repl_wait_majority_ack`) — registers a continuation (Criticality P0)

**Contracts**: Service [x]

##### Service Interface
```c
typedef enum { COMMIT_OUTCOME_COMMITTED, COMMIT_OUTCOME_FAILED } commit_outcome_t;

typedef void (*commit_continuation_fn)(void* ctx, uint64_t lsn,
                                        commit_outcome_t outcome, int rc);

/* Registers a continuation for lsn under COMMIT_MODE_WAIT. Returns 0 on
 * successful registration, negative on admission rejection (mirrors today's
 * -XCE_ERR_NOT_READY). Never blocks. */
int commit_submit_async(commit_engine_t* ce, uint64_t lsn,
                         commit_continuation_fn fn, void* ctx);

/* Called from commit_advance_locked() (majority path) and from the existing
 * timeout/demotion/truncation handling (failure path). Dispatches the
 * registered continuation, if any, via vtask_enq_classified. Idempotent:
 * a second resolve() for an already-resolved lsn is a no-op. */
void commit_resolve(commit_engine_t* ce, uint64_t lsn,
                     commit_outcome_t outcome, int rc);
```
- Preconditions: `lsn` was previously appended locally.
- Postconditions: exactly one of `{COMMITTED, FAILED}` is delivered to the registered continuation, exactly once.
- Invariants: resolution order across distinct LSNs is not required to be FIFO by this component alone — LSN application order is enforced by the Apply Seam's pump, not by resolve dispatch order.

### Apply Seam

#### apply_committed_entry

| Field | Detail |
|-------|--------|
| Intent | The single function that mutates in-memory state for one committed WAL entry, used by both live apply and replay |
| Requirements | 2.2, 2.3, 3.1, 3.2, 3.3, 3.4, 5.2, 7.1, 7.2, 7.3 |

**Responsibilities & Constraints**
- Dispatches on WAL entry type: `WAL_JOB_SUBMIT` → job construction via the existing `restore_job()` primitive (already idempotent, already LSN-tagged); `WAL_JOB_{START,FINISH,FAIL,UPDATE,...}` → the guarded transition core factored out of `mark_job_start/finish/fail/update()`; `WAL_QUEUE_CTRL` → `restore_queue_state()`; `WAL_WORKER_CTRL` → `worker_from_capnp()`.
- Fires the type's `AFTER_EVENT` hook after the mutation completes, never before.
- Must be safe to invoke more than once for the same entry (replay re-running an already-applied LSN, or a resync re-delivering an already-applied fact) — this safety comes from the guards already present in `restore_job` (`-EEXIST`-as-success) and the (relocated) `mark_job_*` monotonic-rank checks, not from new dedup logic.
- Must never be invoked out of LSN order by either caller (live pump or replay loop).

**Dependencies**
- Outbound: `restore_job`, `restore_queue_state`, `worker_from_capnp`, the guarded `mark_job_*` core (all job.c) (Criticality P0)
- Inbound: Apply Pump (live path), `replay_wal` (restart path) (Criticality P0)

**Contracts**: Service [x] / Batch [x]

##### Service Interface
```c
/* Applies one committed WAL entry to in-memory state. Called with entries
 * strictly in ascending lsn order by both the live apply pump and replay.
 * Returns 0 on success (including a no-op idempotent re-apply), negative on
 * a genuine apply failure (which is a fenceable condition, not a retry). */
int apply_committed_entry(persist_t* p, uint32_t wal_type, uint64_t lsn,
                           const void* payload, uint32_t len);
```
- Preconditions: `lsn` is committed (majority-durable); every entry below `lsn` has already been applied.
- Postconditions: in-memory state reflects the entry; the type's `AFTER_EVENT` hook has fired exactly once for this application.
- Invariants: applying the same `(wal_type, lsn, payload)` twice produces the same observable state as applying it once.

##### Batch / Job Contract
- Trigger: commit-advance (live) walking `[last_applied_lsn+1 .. committed_lsn]`, or replay iterating the recovered log.
- Input / validation: WAL entry header + payload, already CRC-verified by the point apply is reached.
- Output / destination: in-memory job/queue/worker state; `AFTER_EVENT` hook firing.
- Idempotency & recovery: guaranteed by the underlying `restore_job`/guarded-transition primitives; `last_applied_lsn` is the resumable cursor for restart.

### Raft Step (Command)

#### Explicit command raft step

| Field | Detail |
|-------|--------|
| Intent | Append + majority-commit a command entry, invoked directly by the command handler, never from a hook |
| Requirements | 1.1, 1.2, 2.1 |

**Responsibilities & Constraints**
- For job submit: already implemented as `persist_submit_wal_and_commit()` (job.c:626) — this design changes its caller-side contract from "returns synchronously, caller replies inline" to "registers a continuation via `repl_submit_majority_commit()`, caller's reply logic becomes the continuation."
- The command handler (`handle_jobsubmit`) no longer calls `create_job()` before this step; object construction moves to the continuation's success branch, via `apply_committed_entry()`.
- Must run off the iomgr thread (unchanged from today).

**Dependencies**
- Outbound: Commit Engine Resolve (`commit_submit_async`) (Criticality P0)
- Outbound: Apply Seam, on the continuation's success branch (Criticality P0)

**Contracts**: Service [x]

**Implementation Notes**
- Integration: the existing `persist_submit_wal_and_commit()` encode/append loop is retained; only its commit-wait call changes from blocking to continuation-registering.
- Validation: a config-change command handler follows the identical pattern once introduced (out of scope to build here, but the contract is designed to generalize — see Requirements Traceability 1.1 covering "job submit, configuration change").
- Risks: converting the one existing `COMMIT_MODE_WAIT` call site is narrow in surface area but touches a durability-critical path — full existing HA soak suite (Req 11.2) must pass before/after.

### Fact Ingestion

#### Fact receipt split

| Field | Detail |
|-------|--------|
| Intent | Split today's receipt-time mutate-then-append into durable-append-only at receipt, mutate+react at apply |
| Requirements | 5.1, 5.2, 5.3, 6.1, 6.2, 6.3 |

**Responsibilities & Constraints**
- `on_job_start`/`on_job_finish`/`on_job_fail`/`on_job_update` (job.c) validate (existing `BEFORE_EVENT`, unchanged) and then only encode + durably append via `wal_hook_append_classified` (`WAL_MODE_RETAIN`) — no state mutation, no `AFTER_EVENT` firing at this point.
- The guarded mutation core (extracted from today's `mark_job_*`) and the `AFTER_EVENT` hook firing both relocate to run from `apply_committed_entry()`.
- No local-apply allowlist under quorum loss (Req 6.2) — enforced structurally: the apply pump only ever processes committed LSNs, so there is no code path that can apply an uncommitted fact.

**Dependencies**
- Outbound: durable append (`wal_hook_append_classified`, existing, unchanged) (Criticality P0)
- Outbound: Apply Seam, for the deferred mutation (Criticality P0)

**Contracts**: Event [x] / Batch [x]

##### Event Contract
- **Published events — implementation deviation (confirmed 2026-07-07 validate-impl audit).** All nine job-state fact WAL types route through a single guard-only `apply_job_update_transition()` core in `apply_committed_entry()` and fire one generic `HOOK_DETAIL_JOB_UPDATE`, not per-type `HOOK_DETAIL_JOB_START`/`_EXIT`/`_FAIL`/etc. as stated below. Verified safe in practice: `sched_job_update_fun`/`apiserver_job_update_fun` (`src/controller/sched.c:446-465`) already exist and consume exactly the generic `(cluster_t*, job_t*)` form the apply path passes — no subscriber assumed per-type dispatch. Left as a documentation correction, not a functional gap.
- ~~Published events: `AFTER_EVENT` for `HOOK_DETAIL_JOB_START`/`_EXIT`/etc., now fired from the apply step, not the receipt handler.~~ (superseded by the note above)
- Subscribed events: none new.
- Ordering / delivery guarantees: strict commit order via the shared apply pump; at-least-once redelivery for facts lost before append remains `worker-rejoin-resync`'s responsibility (out of boundary).

**Implementation Notes**
- Integration: `mark_job_start/finish/fail/update`'s guard predicates (`job_state_is_backward`, `job_state_is_terminal`) become the guard logic inside `apply_committed_entry`'s fact-transition case — the same predicates, relocated, not rewritten.
- Validation: audit every `HOOK_DETAIL_JOB_*` subscriber for an assumption that `BEFORE_EVENT` and the state mutation happen together at receipt; only the mutation moves, `BEFORE_EVENT` validation timing is unchanged.
- Risks: this is the widest behavioral change in the design — worker-visible latency for slot-free/dispatch grows by one commit round-trip (accepted cost, documented in `facts-in-raft.md`, Req 11.4 bounds it to "no per-fact synchronous barrier," not "no latency change").

### Restart Replay & Verification

#### Post-replay apply-correctness verification

| Field | Detail |
|-------|--------|
| Intent | Prove materialized state matches the committed log tip after replay, beyond CRC structural checks |
| Requirements | 3.1, 3.2, 8.1, 8.2, 8.3, 8.4 |

**Responsibilities & Constraints**
- Runs after `replay_wal()`'s existing CRC-chain verification succeeds (RECOVER_CLEAN), applying every entry via `apply_committed_entry()` (replacing today's direct `restore_job`/`restore_job_state` calls in `apply_wal_entry`).
- Computes a lightweight expectation from the committed log tip (e.g., last-applied LSN reaches the recovered commit watermark, plus a count/hash of materialized entities) and compares it to the actual post-replay state.
- On mismatch: reuses `persist_recover_loud_fence_if_unrecoverable()`'s existing mechanism (persist.c:2990) and the `recovery_unrecoverable` sticky flag / `ha->in_catch_up` promotion gate — a DIVERGENT verdict is a new *input* to that existing fence function, not a new fence mechanism.

**Dependencies**
- Outbound: Apply Seam (Criticality P0)
- Outbound: existing loud-fence mechanism (`persist_recover_loud_fence_if_unrecoverable`, `recovery_unrecoverable`, `ha->in_catch_up`) (Criticality P0)

**Contracts**: Batch [x] / State [x]

##### State Management
- State model: `last_applied_lsn` (new, small addition to `persist_t`) tracks replay/apply progress; `recovery_unrecoverable` (existing) is the sticky fence flag.
- Persistence & consistency: `last_applied_lsn` is derived at replay time (not itself persisted) — restart always recomputes it from the WAL, so it cannot itself drift from disk state.
- Concurrency strategy: single-threaded during replay (unchanged); the live apply pump takes over `last_applied_lsn` advancement once replay completes.

## Data Models

No new persistent schema. `WAL_JOB_SUBMIT`/`WAL_JOB_START`/`WAL_JOB_FINISH`/`WAL_JOB_FAIL`/`WAL_JOB_UPDATE`/`WAL_QUEUE_CTRL`/`WAL_WORKER_CTRL` entry formats, `wal_classify`'s WAIT/RETAIN table, and `job_t`'s state bits (`JOB_STAT_PEND/RUN/DONE/EXIT`) are all unchanged.

**In-memory additions**:
- `persist_t.last_applied_lsn` (`_Atomic uint64_t`) — the apply pump's resumable cursor, analogous to the existing `committed_lsn` / `durable_lsn` watermarks.
- `commit_engine_t`'s per-LSN WAIT entry gains a `commit_continuation_fn` + `void* ctx` field (replacing the `pthread_cond_t` wait registration for that mode; `COMMIT_MODE_RETAIN` entries are unaffected).

## Error Handling

### Error Strategy
- **Failed/timed-out command commit**: `commit_resolve(..., COMMIT_OUTCOME_FAILED, rc)` fires the registered continuation with the original error-code vocabulary (`-ETIMEDOUT`, `-XCE_ERR_NOT_READY`, etc., unchanged from today); the continuation replies failure to the client and never calls `apply_committed_entry()` — no rollback needed because no mutation occurred (Req 2.3).
- **Apply-time failure** (e.g., a decode error on an already-committed, already-CRC-valid entry): a genuine programming-invariant violation, not an operational condition — treated per the controller's existing "never self-terminate" rule: log loud, do not apply, and let the DIVERGENT-verdict fence (restart path) or an equivalent live-path fence catch it; never `assert()`/abort on this path in production.
- **DIVERGENT restart verdict**: non-promotable fence via the existing `recovery_unrecoverable` mechanism — stays alive, keeps retrying catch-up, never exits (steering `ha-async-io.md` / controller `CLAUDE.md` process-lifecycle invariant).
- **Quorum loss during a retained fact's commit**: no error surfaces to the worker (facts have no requester awaiting confirmation, unchanged); the fact remains durably appended and unapplied until majority is restored (Req 6).

### Monitoring
- Log a WARN when a retained fact's backlog age (durably appended, not yet committed) exceeds a threshold, consistent with `facts-in-raft.md`'s D1 refinement (metric on retained-fact backlog age).
- Log the CLEAN/DIVERGENT verdict at INFO/ERROR respectively on every restart, alongside the existing CRC verdict.

## Testing Strategy

- **Unit Tests**:
  - Apply-after-commit ordering: a failed commit produces zero observable state mutation (Req 2.3).
  - Idempotent re-apply: applying the same committed entry twice (simulated replay-of-already-applied) is a no-op (Req 7.1, 7.2).
  - Commit-resolve exactly-once: registering a continuation and forcing each of `{majority-ack, timeout, demotion, truncation}` resolves it exactly once (Req 9.4).
  - Fact monotonicity under replay: a duplicate/out-of-order FINISH-then-START replay sequence does not regress state (Req 7.2 — directly targets the `restore_job_state` gap found in `research.md`).
- **Integration Tests**:
  - End-to-end job submit: commit success → job visible and notified; commit failure → job never visible, client sees failure (Req 2, 11.3).
  - End-to-end fact flow: worker reports job start → no slot-free/dispatch/notify until the entry commits (Req 5.3).
  - Quorum-loss-then-restore: facts submitted during a majority-loss window apply in original order once quorum returns, with no local-apply escape hatch exercised (Req 6).
  - Restart CLEAN verdict on a normal shutdown/restart; forced DIVERGENT verdict (inject a materialization mismatch) confirms the fence engages and the process stays alive (Req 8).
- **Regression / Conformance**:
  - Extend `raft_sim.c`'s conformance harness with "no reaction before commit" and "idempotent re-apply" assertions layered on its existing apply-after-`on_commit_advanced` loop (Req 10.1-10.3).
  - Extend `/xce-test-ha` with a kill-the-primary-mid-fact-stream scenario asserting no phantom slot-frees and no notification for an uncommitted fact (Req 10.4).
  - Full existing unit/integration/soak suite re-run as a regression gate (Req 11.1, 11.2), with the known, intentional exception of tests that asserted the old apply-before-commit ordering (updated, not skipped).

### Performance & Scalability
- Target: fact-commit throughput under sustained high-volume worker churn must not regress relative to pre-feature behavior (Req 11.4) — the group-commit amortization for `WAL_MODE_RETAIN` entries is unchanged by this design; only the post-commit reaction timing shifts.
- The keystone conversion removes one pooled-worker-thread-per-in-flight-commit cost entirely (today's `COMMIT_WAIT_RESERVED`-gated blocking wait), which should *improve* concurrent-submit headroom, not merely preserve it — validate via Req 9's bounded-resource tests under a concurrent-submit burst.