myxce/.kiro/specs/raft-apply-decoupling/facts-in-raft.md

Design Note: Routing Facts Through Raft (apply-after-commit for RETAIN events)

Status: candidate plan (2026-06-25). Resolves the "commands vs facts apply policy" open question in brief.md. D1 resolved 2026-06-25: block-but-durable (retain-only) under quorum loss. Design-risk triage complete — see ## Design Risks & Tricky Cases.

Premise

Today XCE splits state changes into two paths:

• Commands (WAL_MODE_WAIT: job submit, config) — client-initiated; synchronous commit before reply.
• Facts (WAL_MODE_RETAIN: worker-reported job start/finish/fail/rusage) — worker-initiated; applied + reacted to on receipt, WAL-appended non-blocking, committed later via a covering standby ACK (wal_hook_append_classified, persist.c:2488).

Facts were made apply-on-receipt for liveness: a worker fact (job finished) must promptly free the slot and trigger the next dispatch, and the worker has already moved on (fire-and-forget, zero redelivery — bug-396).

"Raft handling facts" means: a fact becomes a normal committed log entry. Its state-machine mutation (job RUN/DONE/FAIL) and its reactions (slot free, dispatch trigger, client/apiserver notify) move to the post-commit apply_committed_entry() seam. Only the durable append stays on-receipt (so the fact is never lost). This makes online-apply ≡ replay-apply for facts too, and eliminates phantom reactions on uncommitted facts.

Do we still handle requests and facts differently? (yes — but only at the edges)

Once both requests (commands) and facts route through commit-then-apply, the apply step unifies — both go through the same apply_committed_entry() seam. What stays type-specific is the ingress and the commit-wait / failure policy, not the apply:

Aspect	Request / command (was WAL_MODE_WAIT)	Fact (WAL_MODE_RETAIN)
Origin / ingress	client; non-leader arrivals redirected via the libxce-failover taxonomy	worker; non-leader arrivals rejected/redirected, lost-window recovered by worker-rejoin-resync
Commit wait	synchronous — handler blocks on repl_wait_majority_ack; client reply gated on commit (criteria 2.1/2.2)	async — append, return immediately; commit via the group-commit batch; never a per-fact blocking barrier (high volume)
If commit fails / times out	report FAILED to the waiting requester (criterion 2.2)	nobody to tell — retain + retry while leader (criterion 2.8)
Quorum loss	block/fail (CP)	retained-and-retried; decision D1 decides any local-react escape hatch
State mutation + reaction (apply)	shared apply_committed_entry(), post-commit	shared apply_committed_entry(), post-commit

The key shift: the WAIT vs RETAIN classification's meaning narrows. Today it conflates two things — "apply-after-commit vs apply-on-receipt" and "synchronous-confirm vs async". After this change the apply timing is the same for both (after commit); the classification governs only "does a requester block and get a failure reply" vs "is it async-retained". So we still route them differently at the edges, but they converge at the one place correctness depends on: the apply seam.

Why this is correct, not just possible

• A fact is an external input to the replicated state machine. Raft already commits external inputs (that's what ClientRequest is). Nothing about a fact requires apply-on-receipt at the log level.
• Reactions are state-machine outputs. Canonical Raft produces outputs at apply time (post-commit), on the leader, idempotently. Today's on-receipt reaction violates this — it can fire on a leader that then loses the entry.
• The latency cost (one commit round-trip before slot-free/dispatch) is the same round-trip commands already pay, and the group-commit flusher (ha-commit-durability) already amortizes the high fact volume. The iomgr thread is never blocked: on_commit_advanced drives apply asynchronously.

Decisions to lock before building

• D1 — Quorum-loss fact policy — RESOLVED 2026-06-25: (a) block-but-durable, retain-only. When the primary cannot reach a majority, a fact is appended durably but NOT applied (no state mutation, no reaction) until its entry commits. No local-apply allowlist. Rationale: option (b) (local-apply allowlist) directly contradicts criterion 2.4 (provisional state must not drive irrevocable exports such as dispatch) and reopens the bug-795/828/851 divergence family. Cost of (a) is bounded and self-healing — only new dispatch stalls during the (already-degraded) quorum-loss window; running jobs continue (worker autonomy); retained facts commit+apply in order on quorum restore; primary-death-in-window recovered by worker-rejoin-resync. Consistent with CP + "controllers never self-terminate". Refinements: no carve-outs (allowlists rot); accept the documented stale-read deviation; emit a metric/WARN on retained-fact backlog age; facts inherit Requirement 7's quorum-loss posture.
  • Rejected: (b) local-apply allowlist — buys availability the operator should fix at the quorum level, at the cost of the core safety invariant.
• D2 — Reaction timing. Confirm ALL fact reactions (slot free, scheduler dispatch, client/apiserver notify) are safe to defer to post-commit. Expectation: yes — the worker is fire-and-forget; nothing external blocks on the controller's reaction.
• D3 — Idempotency key. Fact apply must be idempotent (replay, worker-rejoin-resync re-report, retry). Key on (jobid, monotonic job-state) so a re-applied or out-of-order fact is a no-op, never a regression of state. Reuse the mark_job_* idempotency guards (shared need with ha-consistency-invariants).

The list — follow in order

Phase A — Preconditions (shared with raft-apply-decoupling core)

1. Land the single apply_committed_entry() seam; route BOTH commands and facts through it.
2. Add/confirm mark_job_* idempotency guards (D3) so apply is replay/resync/retry safe.
3. Ensure the async group-commit flusher (ha-commit-durability fsync_worker) wakes a post-commit apply pump via on_commit_advanced(lsn) — no iomgr blocking.

Phase B — Reclassify facts onto the committed-apply path

4. Change fact ingestion (the WAL_MODE_RETAIN path in wal_hook_append_classified, persist.c:2488) so on receipt it ONLY: validate → ha_primary_log_append (append entry) → return. Remove the on-receipt state mutation and reactions.
5. Move the fact state mutation (restore_job_state-equivalent: RUN/DONE/FAIL/rusage) into apply_committed_entry(), keyed on committed LSN.
6. Move the reactions (slot free, scheduler dispatch trigger, client/apiserver notify) to fire from the apply step, after the mutation, leader-only.
7. Implement the apply pump: on_commit_advanced(lsn) applies all newly-committed entries (facts + commands) in LSN order exactly once.

Phase C — Failover / liveness correctness

8. Reject/redirect facts arriving at a non-leader (workers must target the leader; reuse the libxce-failover redirect taxonomy E_XCE_REDIRECT/E_XCE_NOMASTER/...).
9. Wire worker-rejoin-resync as the backstop for facts lost in the appended-but-not-committed or never-arrived window.
10. Implement D1 (RESOLVED: block-but-durable — append durably, no apply/react until commit, no allowlist); gate it, emit the retained-fact backlog metric, and test the quorum-loss-then-restore path.

Phase D — Restart / replay parity

11. Restart replay applies fact entries through the SAME apply_committed_entry() → online-apply ≡ replay-apply.
12. Extend the brief's post-restart apply-validation (item #4) to cover fact-derived final states, not just submits.

Phase E — Verification / regression (item #5)

13. raft-sim-conformance: assert no fact reaction (slot-free/dispatch/notify) occurs before that fact's entry commits; assert idempotent re-apply; assert State-Machine-Safety across interleaved fact+command logs.
14. Live soak (/xce-test-ha): kill the primary mid-fact-stream; assert no phantom slot-frees, no divergence, no final-state loss beyond what worker-rejoin-resync recovers.
15. Performance regression: high worker-churn fact stream must not regress — verify group-commit keeps fact commit amortized (NO per-fact synchronous barrier; facts never use the command repl_wait_majority_ack blocking path).

Keystone Pre-requirement: async, non-blocking commit drive (must land FIRST)

Everything in this spec is downstream of one conversion. The protocol drive is currently blocking: repl_wait_majority_ack() → pthread_cond_timedwait(&ce->cond, &ce->lock, &dl) (commit.c:359, bounded by DEFAULT_SYNC_TIMEOUT_MS=500ms, config.h:93) parks a thread until a majority of WalAcks arrive. This is blocking-offloaded (the wait was moved off the iomgr thread) but NOT event-driven — it honors the letter of "never block iomgr" while still parking one worker thread per in-flight commit.

Pre-requirement — convert the commit engine from wait to resolve (do NOT build a parallel registry; reuse the commit engine's existing per-LSN majority tracking — today it cond_broadcasts, replace that with resolve()):

• append → register pending(LSN) on the existing per-LSN commit structure, return immediately (no park).
• on each WalAck event: if majority reached → resolve(lsn, COMMITTED); on timeout/demotion/truncation → resolve(lsn, FAILED).
• resolve(lsn, outcome) → advance commit_index → drive apply (off the ACK thread) → fire the reply continuation (C) and any apply-time proposal callbacks (B).
• the cond_timedwait is deleted, not wrapped.

Why it is the keystone: B (apply→propose self-deadlock), C (reply continuation), and pool-exhaustion (parked workers starving the bounded vtask pool under a slow/degraded majority) are all downstream of the blocking drive. One conversion dissolves all three: with nothing parked, there is no thread to self-deadlock, replies become natural callbacks, and the pool can't be starved by commits. The apply-after-commit work sits on top of this — so the async drive is a hard prerequisite, sequenced before Phase A.

The blocking call is simple and obviously correct; the event-driven version is a state machine across async events (append-complete, each ACK, timeout, demotion, disconnect) carrying the same exactly-once + completeness burden as C, now on the protocol critical path. That cost is why "driving raft" — not the raft algorithm (already library-verified) — is the hardest part of this effort.

Two distinct prerequisites — the commit barrier is NOT the only blocking violation

A broad async-violation audit (2026-06-25) found the commit barrier is actually the least severe blocking class (it blocks a pooled vtask worker — bounded, degrades gracefully). Three CRITICAL violations block the iomgr thread itself — the single epoll loop driving ALL heartbeats/votes/ACKs/client I/O. Root cause: peer_ctrl_msg_handler is registered directly as ep->msg_handler (NOT handed to the vtask pool like normal cluster messages), so anything it does synchronously runs on the iomgr event loop. When iomgr stalls → heartbeats stop → peers declare the leader dead → election storm. This is a separate defect class from the apply refactor: it threatens protocol liveness today, independent of apply-after-commit, and is a likely root of the bug-851 chaotic-failover family.

CRITICAL (block iomgr / the heartbeat-driver thread):

1. Snapshot send on iomgr — ha_dispatch_repl_action_for_standby (ha.c:2702) → persist_checkpoint (fdatasync + dir scan + segment unlinks) + ha_send_snapshot_to_standby (fopen/fread chunk loop, config_repl.c:2446-2503), all on iomgr. A large snapshot freezes every heartbeat/vote/client request for its whole duration. Worse than the commit barrier.
2. Config push wait on iomgr — push_config_transport_only → wait_all_with_timeout busy-polls usleep(5000) + pthread_join (config_repl.c:1332,1340), called from peer_ctrl_msg_handler paths (ha.c:1172,1177,2672,2677,3973), up to CONF_PUSH_TIMEOUT_MS. Code comments (ha.c:75-77) acknowledge the safe-vs-unsafe caller split but unsafe paths exist.
3. Blocking DNS + connect on the delaymgr (heartbeat) thread — endpoint_connect → gethostbyname + blocking connect() (netio.c:289,302) via ha_connect_to_standby from primary_heartbeat_fn. One slow/unresolvable peer stalls ALL heartbeats/election-timeouts for seconds → false failover. (set_fd_nonblocking is applied AFTER connect, netio.c:716 — the connect is inherently blocking.)

MEDIUM (block a vtask pool worker — same class as the commit barrier):

• repl_wait_majority_ack for OPCode_upgradeConfigAck (config_repl.c:1459); 30-second usleep checkpoint-wait poll in do_upgrade_config_with_bytes (config_repl.c:1661); ha_persist_term fsync on a vote grant (ha.c:3407).

Correctly event-driven (confirmed): RequestVote fan-out, heartbeat fan-out, and WAL replication streaming all use non-blocking xce_request() enqueue; standby-side replicated WAL write + its fdatasync run on a vtask worker, not iomgr.

Verdict: the protocol drive is NOT fully event-driven. Two separable prerequisites result:

• P1 — commit-barrier async (this doc's keystone): unblocks the apply/reply/facts work (B, C, pool-exhaustion). Prerequisite for raft-apply-decoupling.

• P2 — drive hardening (move snapshot send, config-push wait, and peer DNS/connect OFF the iomgr/delaymgr threads): independently valuable, independently testable, threatens liveness today. Recommended home: its own spec raft-drive-async (or fold into election-iomgr-refactor, which already touches this thread model) — NOT buried inside raft-apply-decoupling.

  Offload mechanism — "trigger, don't execute" (delaymgr/iomgr only schedule; blocking work runs in a pool with admission headroom):

  • The timer/event thread enqueues a task and returns immediately; the blocking work runs on a worker. The danger is relocating the stall: dumping blocking work into the bounded vtask pool can starve request handling. XCE already solved this for the commit barrier — COMMIT_WAIT_RESERVED caps concurrent blocking waits at max(1, n_vcpus − RESERVED), reserving headroom (commit.c:195-203). Any offloaded blocking task MUST live under the same admission discipline.
  • API gap: the existing enqueue vtask_handle_msg (vtaskmgr.c:356) is endpoint+inbound-packet-bound (vtype=VTASK_IO, requires ep->fsm_in.pkt). A reconnect has no inbound packet → P2 needs a general (non-IO) task-submit path (vtask_enq with custom data), or generalize the API.
  • Lane by duration: short bounded tasks (reconnect, ≤200ms connect cap) → vtask pool under an admission cap (COMMIT_WAIT_RESERVED-style). Long tasks (snapshot send, config-push, seconds) → a dedicated slow-lane executor, NOT the request pool (a few concurrent multi-second tasks head-of-line-block request handling even with reservation).
  • Coordination per offloaded task: (1) idempotent in-flight dedup (pending[peer] atomic) so periodic timers don't pile up duplicates; (2) refcount the endpoint at enqueue, release at completion (arg outlives the tick); (3) marshal completion (epoll fd registration) back to the iomgr owning thread (bug-646); (4) resolve the outbound-vs-inbound connection race.
  • DNS: gethostbyname is LOW real-world risk (HPC peers are IP//etc/hosts) — defer unless measured. The connect fix is mostly reordering an existing pattern: non-blocking connect + bounded poll timeout already exists in xce_probe_config_tag (config_repl.c:1753-1761); the buggy path just sets non-blocking AFTER connect (netio.c:716).

  Heartbeat-problem completeness (does the offload fix it? — partially): Heartbeat liveness spans BOTH threads — delaymgr sends heartbeats + fires election timeouts; iomgr processes incoming ACKs/votes/replies (peer_ctrl_msg_handler). Connect-offload protects the delaymgr (sender) side and election-timeout firing — a real win — but is necessary-not-sufficient: the snapshot-send-on-iomgr blocker (#1) freezes the receiver loop, so the leader stops hearing its peers regardless of how cleanly it sends. Heartbeat is safe only when BOTH delaymgr AND iomgr never block → P2 must land all three CRITICALs, not just connect. Residual: delaymgr ticks at 200ms precision (delay.c) — a timer-granularity floor offload doesn't touch; set the failover-detection timeout with margin above 200ms (tuning, not blocking).

  Missing regression test: /xce-test-ha only kills the primary (process-dead → connect fails fast with RST; does NOT reproduce this). The real test is a partition variant — make one peer unreachable (blackhole packets so connect hangs and snapshot-send stalls) and assert (a) heartbeats to survivors keep flowing and (b) no spurious election. This specifically exercises the unreachable-not-dead case the blocking findings care about — the bug-851 chaotic-failover family.

Design Risks & Tricky Cases

Triaged 2026-06-25. After triage, only A and B are genuinely architectural; C was reclassified to request-handling correctness; D–G are correctness/cost refinements.

A — Reactions are not replayable → re-derive from committed state on recovery (architectural; RESOLVED by existing resync)

A committed log perfectly records what state should be, but is a bad script for what side effects to re-emit: replaying a job-finish must rebuild DONE state but must NOT re-notify the client or re-dispatch. So reactions originate from only two places: (1) live-leader apply of newly-committed entries, and (2) the full-state resync on consumer reconnect. Recovery never replays reactions — it rebuilds committed state, and the inevitable reconnect (failover / primary-restart / promotion all look identical to a consumer) re-derives the world. Three apply contexts: leader-live = state + react; standby = state only; recovery/replay = state only, then re-derive via resync.

This is solved by XCE's existing reconnect-and-resync (scheduler/apiserver full-state push on reconnect = the re-derivation step). Conditions for it to hold:

1. Resync must push committed state only (post-replay), never provisional (criterion 2.7).
2. Full reconcile, not delta (consumer discards old view) — guard against the "lightweight re-attach" optimization eroding this.
3. Resync re-reports flow through commit-then-apply too (a rejoining worker's re-reported fact is appended→committed→applied idempotently, never apply-on-receipt — same D1/2.9 gate).
4. Order resync after replay (re-reports above the recovered tip → append+commit; already-committed → idempotent no-op).

HARD prerequisite: the worker→controller half depends on worker-rejoin-resync, which is not yet implemented (workers send only hostattrs today, no job state — bug-396). Without it, a fact lost in a failover window has no redelivery path and A reopens for facts. → upgrade the worker-rejoin-resync dependency in brief.md/roadmap from "designated backstop" to HARD.

B — Apply → propose reentrancy (deadlock risk) (architectural; OPEN)

Apply runs inside the commit-advance callback. A reaction that produces new cluster state must enqueue a fresh proposal, never mutate inline and never block waiting for its commit (you'd deadlock inside the commit path). Rule: apply may enqueue proposals; apply may never block on consensus. The vtask-continuation discipline below (register-don't-block) is the same mechanism and covers B.

C — Reply ownership (RECLASSIFIED: request-handling correctness, NOT architecture; low risk)

"One request → exactly one reply" is a pre-existing request-handling invariant. Apply-decoupling only defers the reply to commit; it adds no new architecture. The genuinely architectural sub-concerns are already solved by the existing design:

• Send race / socket-epoll ownership (bug-646 class): solved — xce_reply is non-blocking (it enqueues to the endpoint's output queue; the io worker owns the wire). Reply is generated by the vtask handler-continuation (completion marshaled to the iomgr thread), enqueued, and delivered by the io worker.
• Endpoint lifetime (bug-1077 class): solved — the io socket is owned by the endpoint, which is refcounted; a continuation holds a ref, so a mid-flight disconnect discards the reply against a dead endpoint rather than a UAF.

Residual (request-handling correctness, single-threaded if produced on the iomgr thread):

1. Exactly-once deferred-reply resolution per request across {commit-success, commit-timeout, leadership-loss, uncommitted-tail truncation} — an atomic "claimed" flag on the obligation; centralize all exits through one resolve(lsn, outcome).
2. Bounded in-flight (the one system-level item, not per-request): the async model loses the blocking model's free thread-pool backpressure, so add admission control on the accept path.

D — Which facts need consensus? (refinement)

Job lifecycle (start/finish/fail) → RETAIN-through-commit (authoritative). rusage/telemetry → consider LOCAL (soft, re-derivable, highest-volume — don't commit-amplify it). Re-audit the wal_classify table.

E — Idempotency must be state-monotonic, not LSN-dedupe (refinement; mandatory)

Compare job-state rank (PEND<RUN<DONE) and no-op regressions/duplicates — not LSN dedupe. Resync amplifies this (it floods re-reported facts), so E is mandatory, not optional. (Implements D3.)

F — Strict LSN-ordered apply (refinement)

Serial, in-LSN-order apply pump, never reordered/parallelized — this is what makes "fact for a not-yet-created job" races vanish by construction. Lock it as a raft-sim-conformance assertion.

G — Notify reliability is unchanged (note, not a regression)

Moving after-hooks to notify-only does NOT make delivery reliable (still best-effort, bug-396). Resync backstops loss across a reconnect, not per-message. Don't let the cleaner architecture imply guaranteed delivery.