myxce/.kiro/specs/raft-apply-decoupling/brief.md

Brief: raft-apply-decoupling

Problem

The controller's Raft consensus protocol is embedded inside the notification hook fan-out, and in-memory state is mutated before the entry is committed. This couples three concerns that canonical Raft keeps separate (log → commit → apply → notify):

• persist_job_hook_fun is registered as an AFTER_EVENT hook, and inside that callback it performs the WAL append (ha_primary_log_append → raft_log_append) and blocks on the synchronous majority-commit barrier (repl_wait_majority_ack). The raft protocol step is therefore literally a side effect of running a hook (persist.c:2532, persist.c:2198, ha.c:3080).
• handle_jobsubmit mutates the in-memory job table via create_job() before the WAL/commit step (job.c:558 runs before the AFTER hook at job.c:593). A commit that fails or times out does not cleanly revert that optimistic mutation, leaving a window where in-memory state diverges from the committed log.
• Because apply happens before commit, every downstream consumer (scheduler push, client reply) must defensively re-gate on a commit watermark (sched.c:38 job_submit_committed) to avoid acting on not-yet-committed state — scattering commit-awareness across the codebase instead of centralizing it at one apply point.

This is the structural root behind the bug-795/828/851/1060 divergence family: the controller has no single, ordered "apply committed entry to the state machine" seam.

Current State

• Hooks (src/lib/hook.c) are a typed pub/sub fan-out (xce_run_hooks) with HOOK_BEFORE_EVENT / HOOK_AFTER_EVENT situations. Subscribers: persist_job_hook_fun (durability — does WAL+commit), sched_job_hook_fun (scheduler push), apiserver SSE callbacks.
• Order today: BEFORE hook → create_job() (state mutation) → AFTER hook { WAL append + majority commit barrier } → client reply → scheduler push (commit-gated).
• The raft library (raft-log-replication, raft-commit-safety) already OWNS the replicated log and exposes pure callbacks (on_persist_entry, on_commit_advanced, etc.). The library half is the right protocol core; the controller half is what is mis-layered.
• Restart replay (persist_recover → replay_wal → apply_wal_entry, persist.c:1950/1735/1606) validates only the CRC chain. There is no post-replay check that materialized in-memory state equals the committed log.

Desired Outcome

The controller follows canonical Raft ordering with one explicit apply seam:

1. pre-hooks (validation / preparation only — no authoritative state change)
2. raft step — append entry → replicate → majority commit (an explicit protocol step, NOT a hook callback)
3. apply — a single apply_committed_entry() seam mutates the in-memory state machine after commit
4. after-hooks = pure notification delivery (scheduler push, client reply/SSE, apiserver) — they observe already-committed, already-applied state and no longer carry WAL/commit responsibility

A commit that never reaches majority therefore never applies, eliminating the optimistic-divergence window. On restart, the same apply_committed_entry() seam is reused by replay, and a verification step proves materialized state == committed log.

Approach

Apply-after-commit (canonical), chosen 2026-06-24. Invert the submit path so the raft commit gates the apply:

• KEYSTONE PRE-REQUIREMENT (must land first): convert the protocol drive from blocking to event-driven. repl_wait_majority_ack() today blocks on pthread_cond_timedwait (commit.c:359) — a parked thread per in-flight commit. Build its async counterpart: append → register pending(LSN) → resolve on majority-ACK/timeout/demotion → drive apply + fire reply continuation. B (apply self-deadlock), C (reply lifecycle), and vtask-pool exhaustion are all downstream of this one conversion. See facts-in-raft.md § Keystone. The rest of this approach assumes the drive no longer blocks.

• Extract WAL append + commit barrier out of persist_job_hook_fun into an explicit controller-side raft step invoked directly by command handlers (not via the hook system).

• Introduce a single apply_committed_entry() function that the commit path calls after on_commit_advanced, and that WAL replay reuses on restart — guaranteeing online-apply and replay-apply are the same code (no divergence-by-two-implementations).

• Demote the hook fan-out to notification-only: after-hooks fire from the apply step, observing committed state. Remove the now-redundant job_submit_committed watermark gating where the new ordering makes it unnecessary.

• Commands vs facts policy: make commands (WAL_MODE_WAIT: job submit, config) strictly apply-after-commit. Facts (WAL_MODE_RETAIN: worker-reported start/finish/fail/rusage) are applied on receipt today because the controller reacts immediately. Direction set 2026-06-25: route facts through commit-then-apply as well — candidate plan and 15-step list in facts-in-raft.md (this spec dir), formalized as the Facts-Apply invariant (criterion 2.9) in ha-consistency-invariants/requirements.md. After the change, commands and facts share the one apply_committed_entry() seam and differ only in ingress + commit-wait/failure policy (see the comparison table in facts-in-raft.md). The one remaining open decision is D1 (quorum-loss reaction policy: block-but-durable vs local-apply allowlist) — resolve before Phase C.

• Restart raft-log-applied validation (item #4, in-scope): add a post-replay verification that the materialized entity set matches the committed log tip (e.g., count/hash of applied entities vs WAL-derived expectation), surfacing a CLEAN/DIVERGENT verdict alongside the existing CRC verdict.

Scope

• In:
  • Decouple the raft protocol step from the hook fan-out (item #1).
  • Canonical ordering: pre-hooks → commit → apply → after-hooks-as-notification (item #2).
  • Single shared apply_committed_entry() seam used by both online commit and restart replay.
  • Post-restart validation that committed raft log was correctly applied to materialized state (item #4).
  • Requirement + regression test design for the above (item #5): unit tests for ordering/rollback, plus conformance assertions extended in raft-sim-conformance.
• Out:
  • Job-array single-record collapse — owned by the sibling job-array-single-wal-record spec.
  • Changes to the raft library protocol core (owned by raft-log-replication / raft-commit-safety).
  • Entity-bearing snapshots / config-upgrade snapshot (separate specs).
  • Read-consistency / leader-lease optimizations.

Boundary Candidates

• The explicit controller raft step (append + commit barrier) extracted from persist_job_hook_fun.
• The single apply_committed_entry() apply seam (shared online + replay).
• The hook system's demotion to notification-only (after-hook subscribers re-pointed at the apply step).
• The commands-vs-facts apply policy.
• The post-restart apply-correctness verification.

Out of Boundary

• The raft library callbacks/protocol (consumed, not modified here).
• Array-submission batching.
• Scheduler framework policy changes.

Upstream / Downstream

• Upstream: raft-log-replication + raft-commit-safety (library log/commit core providing on_persist_entry / on_commit_advanced); ha-term-safety (controller-side raft integration umbrella — this spec restructures the same submit→commit path ha-term-safety wires, so it must land in coordination with it); ha-commit-durability (durable-commit semantics, unchanged).
• Downstream: job-array-single-wal-record (builds the single atomic array record on top of the clean apply seam); worker-rejoin-resync and any future state-machine change benefit from the centralized apply point.

Existing Spec Touchpoints

• Extends / sequences with: ha-term-safety (controller raft integration umbrella — disjoint-but-adjacent edits to the submit/commit path; coordinate ordering to avoid conflicting refactors).
• Adjacent: raft-sim-conformance (item #5 regression assertions land here — apply-after-commit ordering + no-apply-on-failed-commit invariants); ha-commit-barrier-rollout (commit classification table — the WAIT vs RETAIN distinction this spec must respect); raft-log-replication / raft-commit-safety (library callbacks).

Constraints

• Must not block the iomgr thread (async-I/O invariant in ha-async-io.md): the explicit commit barrier must run off the iomgr thread, as the current persist_wal_sync_append already arranges.
• "Controllers never self-terminate" — a divergent restart verdict is a loud, non-promotable fence, never an exit/abort.
• Preserve persist-before-ACK durability and the synchronous commit barrier as policy (do not canonicalize them away).
• The shared apply seam must be deterministic so online-apply and replay-apply produce identical materialized state.