Rongsong Shen
5f307e297b
Adds the Raft Apply-Ordering & Hook Decoupling roadmap phase: canonical pre-hooks -> raft commit -> apply -> notify ordering, with the raft protocol step lifted out of the notification hook fan-out (root of the bug-795/828/ 851/1060 divergence family). Decision: apply-after-commit via a single shared apply_committed_entry() seam reused by online apply and restart replay. - raft-drive-async: full spec docs (brief/requirements/research/design) committed alongside its already-tracked tasks.md; async-drive prerequisite for the apply-decoupling work, sequenced to start first (threatens failover liveness today). - raft-apply-decoupling: new spec (brief + facts-in-raft.md) covering the apply seam and the facts (WAL_MODE_RETAIN) apply-timing question (open decision D1). - job-array-single-wal-record: promoted from backlog to a tracked spec, sequenced after raft-apply-decoupling. - ha-consistency-invariants: add criterion 2.9 (Facts-Apply invariant), documenting the current on-receipt fact-reaction deviation this family will reconcile. - raft-commit-safety / raft-log-replication: reconcile spec.json phase to "implemented" (both were mislabeled tasks-generated despite full 18/18 and 24/24 task completion + raft_sim GO). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
86 lines
8.8 KiB
Markdown
86 lines
8.8 KiB
Markdown
# Brief: raft-apply-decoupling
|
|
|
|
## Problem
|
|
|
|
The controller's Raft consensus protocol is **embedded inside the notification hook fan-out**, and in-memory state is mutated **before** the entry is committed. This couples three concerns that canonical Raft keeps separate (log → commit → apply → notify):
|
|
|
|
- `persist_job_hook_fun` is registered as an `AFTER_EVENT` hook, and *inside* that callback it performs the WAL append (`ha_primary_log_append` → `raft_log_append`) **and** blocks on the synchronous majority-commit barrier (`repl_wait_majority_ack`). The raft protocol step is therefore literally a side effect of running a hook (`persist.c:2532`, `persist.c:2198`, `ha.c:3080`).
|
|
- `handle_jobsubmit` mutates the in-memory job table via `create_job()` **before** the WAL/commit step (`job.c:558` runs before the AFTER hook at `job.c:593`). A commit that fails or times out does **not** cleanly revert that optimistic mutation, leaving a window where in-memory state diverges from the committed log.
|
|
- Because apply happens before commit, every downstream consumer (scheduler push, client reply) must defensively re-gate on a commit watermark (`sched.c:38` `job_submit_committed`) to avoid acting on not-yet-committed state — scattering commit-awareness across the codebase instead of centralizing it at one apply point.
|
|
|
|
This is the structural root behind the bug-795/828/851/1060 divergence family: the controller has no single, ordered "apply committed entry to the state machine" seam.
|
|
|
|
## Current State
|
|
|
|
- Hooks (`src/lib/hook.c`) are a typed pub/sub fan-out (`xce_run_hooks`) with `HOOK_BEFORE_EVENT` / `HOOK_AFTER_EVENT` situations. Subscribers: `persist_job_hook_fun` (durability — does WAL+commit), `sched_job_hook_fun` (scheduler push), apiserver SSE callbacks.
|
|
- Order today: `BEFORE hook → create_job() (state mutation) → AFTER hook { WAL append + majority commit barrier } → client reply → scheduler push (commit-gated)`.
|
|
- The raft library (raft-log-replication, raft-commit-safety) already OWNS the replicated log and exposes pure callbacks (`on_persist_entry`, `on_commit_advanced`, etc.). The library half is the right protocol core; the controller half is what is mis-layered.
|
|
- Restart replay (`persist_recover` → `replay_wal` → `apply_wal_entry`, `persist.c:1950/1735/1606`) validates only the CRC chain. There is **no** post-replay check that materialized in-memory state equals the committed log.
|
|
|
|
## Desired Outcome
|
|
|
|
The controller follows canonical Raft ordering with one explicit apply seam:
|
|
|
|
1. **pre-hooks** (validation / preparation only — no authoritative state change)
|
|
2. **raft step** — append entry → replicate → majority commit (an explicit protocol step, NOT a hook callback)
|
|
3. **apply** — a single `apply_committed_entry()` seam mutates the in-memory state machine *after* commit
|
|
4. **after-hooks** = pure notification delivery (scheduler push, client reply/SSE, apiserver) — they observe already-committed, already-applied state and no longer carry WAL/commit responsibility
|
|
|
|
A commit that never reaches majority therefore never applies, eliminating the optimistic-divergence window. On restart, the same `apply_committed_entry()` seam is reused by replay, and a verification step proves materialized state == committed log.
|
|
|
|
## Approach
|
|
|
|
Apply-after-commit (canonical), chosen 2026-06-24. Invert the submit path so the raft commit gates the apply:
|
|
|
|
- **KEYSTONE PRE-REQUIREMENT (must land first):** convert the protocol drive from blocking to event-driven. `repl_wait_majority_ack()` today blocks on `pthread_cond_timedwait` (commit.c:359) — a parked thread per in-flight commit. Build its async counterpart: append → register `pending(LSN)` → resolve on majority-ACK/timeout/demotion → drive apply + fire reply continuation. B (apply self-deadlock), C (reply lifecycle), and vtask-pool exhaustion are all downstream of this one conversion. See `facts-in-raft.md` § Keystone. The rest of this approach assumes the drive no longer blocks.
|
|
|
|
- Extract WAL append + commit barrier out of `persist_job_hook_fun` into an explicit controller-side raft step invoked directly by command handlers (not via the hook system).
|
|
- Introduce a single `apply_committed_entry()` function that the commit path calls **after** `on_commit_advanced`, and that WAL replay reuses on restart — guaranteeing online-apply and replay-apply are the same code (no divergence-by-two-implementations).
|
|
- Demote the hook fan-out to notification-only: after-hooks fire from the apply step, observing committed state. Remove the now-redundant `job_submit_committed` watermark gating where the new ordering makes it unnecessary.
|
|
- **Commands vs facts policy:** make commands (`WAL_MODE_WAIT`: job submit, config) strictly apply-after-commit. Facts (`WAL_MODE_RETAIN`: worker-reported start/finish/fail/rusage) are applied on receipt today because the controller reacts immediately. **Direction set 2026-06-25: route facts through commit-then-apply as well** — candidate plan and 15-step list in `facts-in-raft.md` (this spec dir), formalized as the **Facts-Apply invariant** (criterion 2.9) in `ha-consistency-invariants/requirements.md`. After the change, commands and facts share the one `apply_committed_entry()` seam and differ only in ingress + commit-wait/failure policy (see the comparison table in `facts-in-raft.md`). The one remaining open decision is **D1** (quorum-loss reaction policy: block-but-durable vs local-apply allowlist) — resolve before Phase C.
|
|
- **Restart raft-log-applied validation (item #4, in-scope):** add a post-replay verification that the materialized entity set matches the committed log tip (e.g., count/hash of applied entities vs WAL-derived expectation), surfacing a CLEAN/DIVERGENT verdict alongside the existing CRC verdict.
|
|
|
|
## Scope
|
|
|
|
- **In**:
|
|
- Decouple the raft protocol step from the hook fan-out (item #1).
|
|
- Canonical ordering: pre-hooks → commit → apply → after-hooks-as-notification (item #2).
|
|
- Single shared `apply_committed_entry()` seam used by both online commit and restart replay.
|
|
- Post-restart validation that committed raft log was correctly applied to materialized state (item #4).
|
|
- Requirement + regression test design for the above (item #5): unit tests for ordering/rollback, plus conformance assertions extended in `raft-sim-conformance`.
|
|
- **Out**:
|
|
- Job-array single-record collapse — owned by the sibling `job-array-single-wal-record` spec.
|
|
- Changes to the raft library protocol core (owned by raft-log-replication / raft-commit-safety).
|
|
- Entity-bearing snapshots / config-upgrade snapshot (separate specs).
|
|
- Read-consistency / leader-lease optimizations.
|
|
|
|
## Boundary Candidates
|
|
|
|
- The explicit controller raft step (append + commit barrier) extracted from `persist_job_hook_fun`.
|
|
- The single `apply_committed_entry()` apply seam (shared online + replay).
|
|
- The hook system's demotion to notification-only (after-hook subscribers re-pointed at the apply step).
|
|
- The commands-vs-facts apply policy.
|
|
- The post-restart apply-correctness verification.
|
|
|
|
## Out of Boundary
|
|
|
|
- The raft library callbacks/protocol (consumed, not modified here).
|
|
- Array-submission batching.
|
|
- Scheduler framework policy changes.
|
|
|
|
## Upstream / Downstream
|
|
|
|
- **Upstream**: raft-log-replication + raft-commit-safety (library log/commit core providing `on_persist_entry` / `on_commit_advanced`); ha-term-safety (controller-side raft integration umbrella — this spec restructures the same submit→commit path ha-term-safety wires, so it must land in coordination with it); ha-commit-durability (durable-commit semantics, unchanged).
|
|
- **Downstream**: `job-array-single-wal-record` (builds the single atomic array record on top of the clean apply seam); worker-rejoin-resync and any future state-machine change benefit from the centralized apply point.
|
|
|
|
## Existing Spec Touchpoints
|
|
|
|
- **Extends / sequences with**: `ha-term-safety` (controller raft integration umbrella — disjoint-but-adjacent edits to the submit/commit path; coordinate ordering to avoid conflicting refactors).
|
|
- **Adjacent**: `raft-sim-conformance` (item #5 regression assertions land here — apply-after-commit ordering + no-apply-on-failed-commit invariants); `ha-commit-barrier-rollout` (commit classification table — the WAIT vs RETAIN distinction this spec must respect); `raft-log-replication` / `raft-commit-safety` (library callbacks).
|
|
|
|
## Constraints
|
|
|
|
- Must not block the iomgr thread (async-I/O invariant in `ha-async-io.md`): the explicit commit barrier must run off the iomgr thread, as the current `persist_wal_sync_append` already arranges.
|
|
- "Controllers never self-terminate" — a divergent restart verdict is a loud, non-promotable fence, never an `exit`/`abort`.
|
|
- Preserve persist-before-ACK durability and the synchronous commit barrier as policy (do not canonicalize them away).
|
|
- The shared apply seam must be deterministic so online-apply and replay-apply produce identical materialized state.
|