myxce/.kiro/specs/raft-apply-decoupling/requirements.md

Requirements Document

Introduction

The XCE controller's raft protocol step is currently a side effect of the notification hook fan-out rather than an explicit, ordered part of the request path. persist_job_hook_fun runs as an AFTER_EVENT hook and, inside that callback, performs the WAL append and blocks on the synchronous majority-commit barrier. Meanwhile handle_jobsubmit mutates the in-memory job table (create_job()) before that commit step runs, so a commit that fails or times out does not cleanly revert the optimistic mutation. Because apply happens before commit, downstream consumers (scheduler push, client reply) must defensively re-gate on a commit watermark to avoid acting on not-yet-committed state. This mis-layering is the structural root of the bug-795/828/851/1060 divergence family.

This feature restructures the controller onto canonical Raft ordering with one explicit apply seam: pre-hooks (validation only) → an explicit raft step (append → replicate → majority commit) → a single apply_committed_entry() seam that mutates in-memory state after commit → after-hooks that are pure notification delivery observing already-committed, already-applied state. The same apply_committed_entry() seam is reused by restart replay, and a post-replay verification proves materialized state matches the committed log. Per ha-consistency-invariants criterion 2.9 (the Facts-Apply invariant), this feature also reconciles worker-reported facts (WAL_MODE_RETAIN: job start/finish/fail/rusage) onto the same commit-then-apply seam — today they are applied and reacted to on receipt, ahead of commit.

Everything in this feature is downstream of one conversion: the commit-wait drive is currently blocking (repl_wait_majority_ack parks a pooled worker thread per in-flight commit via pthread_cond_timedwait). This feature includes converting that drive from blocking-wait to event-driven resolve as its keystone prerequisite, because the apply step, the deferred client reply, and the fact-reaction timing all sit on top of it. The separate transport/timer-thread blocking violations (snapshot send, config-push wait, peer connect on iomgr/delaymgr) are a distinct defect class owned by raft-drive-async, not this feature.

The authoritative subject for the acceptance criteria below is the XCE controller — specifically its command/fact submission, commit, and apply path.

Boundary Context

• In scope:
  • Extracting the raft protocol step (WAL append + majority-commit barrier) out of the AFTER_EVENT hook fan-out into an explicit step invoked directly by command handlers.
  • Canonical ordering for commands (job submit, configuration change): pre-hook validation → commit → apply → after-hook notification, with no in-memory state mutation before commit.
  • A single shared apply_committed_entry() seam used by both live commit-advance and restart replay, applying committed entries strictly in commit order.
  • Demoting the hook fan-out to notification-only: after-hooks observe already-committed, already-applied state and no longer carry WAL/commit responsibility, and downstream consumers no longer need to defensively re-check a commit watermark before acting on a notification.
  • Reconciling worker-reported facts (WAL_MODE_RETAIN) onto the same commit-then-apply seam (criterion 2.9): durable append stays on receipt, but the state mutation and reactions (worker slot release, scheduler dispatch trigger, client/apiserver notify) move to after commit.
  • The quorum-loss policy for retained facts: block-but-durable (append durably, do not apply or react until commit; no local-apply allowlist) — decision D1.
  • Idempotent, state-monotonic, strictly commit-ordered re-application of entries (replay, retry, worker resync).
  • Post-restart verification that materialized state matches the committed log tip, surfacing a CLEAN/DIVERGENT verdict.
  • The keystone conversion of the commit-wait drive from blocking (repl_wait_majority_ack / pthread_cond_timedwait) to an event-driven resolve, and the resulting bound on request-handling capacity under concurrent in-flight commits.
  • Regression and conformance coverage for the above ordering, apply-idempotency, and no-reaction-before-commit invariants.
  • No regression of existing HA behavior, correctness guarantees, or fact-commit throughput as a result of this reordering.
• Out of scope:
  • Collapsing a job-array submission into a single atomic WAL record — owned by job-array-single-wal-record, which builds on top of the apply seam this feature delivers.
  • Changes to the raft library's protocol core (raft_advance_commit, on_commit_advanced, on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from) — owned by raft-log-replication / raft-commit-safety and consumed as-is.
  • Entity-bearing snapshot content and the config-upgrade snapshot format — owned by ha-entity-snapshots / config-upgrade-snapshot.
  • Read-consistency and leader-lease optimizations.
  • Moving snapshot send, configuration-push waiting, or peer connect off the iomgr/delaymgr threads — owned by raft-drive-async (P2 transport/timer drive hardening); this feature owns only the commit-wait drive conversion (P1).
  • Changes to the worker join/rejoin protocol itself — owned by worker-join-refactor / worker-rejoin-resync.
  • Asynchronous DNS resolution for peer connections.
  • Extract-from-hook / apply-after-commit ordering for queue-control and worker-control commands: queue.c's AFTER_EVENT path and persist_queue_hook_fun/persist_worker_hook_fun still mutate in-memory state before firing the WAL append, the same double-append-class ordering this feature fixes for job submission and facts — un-fixed here (2026-07-09 validate-impl finding). Tracked as a follow-up, same shape as this feature's Requirement 1/2 but for the queue/worker command family.
• Adjacent expectations:
  • This feature relies on the majority-commit and idempotency primitives already delivered by ha-commit-engine and ha-commit-barrier-rollout; it does not re-implement them.
  • This feature touches the same submit/commit path that ha-term-safety is concurrently wiring into the raft library adapter; the two must be sequenced/coordinated so neither reintroduces a term-blind or apply-before-commit regression.
  • Full recovery of a fact that never reached the controller, or that was lost with the controller across a failover window, remains the responsibility of worker-rejoin-resync (roadmap backlog, not yet built) — this feature relies on it as the designated backstop but does not own or implement it. Until worker-rejoin-resync lands, such facts remain lost, consistent with the umbrella-locked decision already accepted by ha-consistency-invariants and ha-commit-barrier-rollout.
  • job-array-single-wal-record depends on the apply_committed_entry() seam this feature delivers and must land after it.
  • Ordering and apply-correctness assertions introduced here are consumed by raft-sim-conformance's regression harness.
  • This feature must preserve the existing "controllers never self-terminate" invariant and the existing persist-before-ACK durability and synchronous majority-commit-barrier policy; it changes ordering and apply timing, not the durability contract.

Requirements

Requirement 1: Extraction of the Raft Commit Step from the Hook Fan-Out

Objective: As a maintainer, I want the WAL append and majority-commit barrier for a command to run as an explicit step invoked directly by the command handler rather than as a side effect of an AFTER_EVENT hook callback, so that the raft protocol step is a first-class, ordered part of the request path instead of incidental hook behavior.

Acceptance Criteria

1. When a command handler (job submit, configuration change) processes a request, the controller shall perform the WAL append and majority-commit barrier as an explicit step invoked by the handler, not from within a BEFORE_EVENT or AFTER_EVENT hook callback.
2. The controller shall not register the majority-commit barrier as a hook subscriber for any command entry type.
3. The controller shall perform the explicit raft commit step without blocking the iomgr event-loop thread, preserving the existing async-I/O thread-role contract.
4. The controller shall preserve persist-before-ACK durability and the synchronous majority-commit barrier as policy; extracting the raft step from the hook fan-out shall not weaken or remove either.

Requirement 2: Canonical Command Ordering — Apply-After-Commit

Objective: As a cluster operator, I want a job-submit or configuration command's in-memory effect to occur only after its entry is durably committed to a majority, so that a commit that fails or times out never leaves the cluster's observable state showing an operation that did not durably happen.

Acceptance Criteria

1. When a command entry is accepted for processing, the controller shall not apply its state mutation until the entry has committed to a majority.
2. When a command entry commits to a majority, the controller shall apply its state mutation before producing any success reply or notification for that entry.
3. If a command entry's majority commit fails or times out, then the controller shall report failure to the requester and shall not apply the entry's state mutation, so the controller's observable state is unchanged by the failed command.

Requirement 3: Single Shared Apply Seam — Online and Replay Parity

Objective: As a maintainer, I want one shared apply_committed_entry() seam used both when a commit advances live and when the controller replays its log after restart, so that a node's materialized state is always the deterministic result of its committed log regardless of how it got there.

Acceptance Criteria

1. When the controller applies a committed entry during live operation, the controller shall invoke the same apply_committed_entry() seam that restart replay invokes for that entry type.
2. When restart replay processes a committed entry, the controller shall apply it through apply_committed_entry() rather than through a separate replay-only mutation path.
3. The controller shall apply committed entries strictly in commit order, never applying an entry out of order relative to an entry with a lower commit position.
4. The controller shall produce identical materialized state from a committed log regardless of whether that log is applied via live commit-advance or via restart replay.

Requirement 4: Hook Fan-Out Demotion to Notification-Only

Objective: As a maintainer, I want after-hooks to be pure notification delivery observing already-committed, already-applied state, so that scheduler push, client reply, and apiserver updates no longer need to carry or defensively re-check commit responsibility.

Acceptance Criteria

1. When an AFTER_EVENT hook fires for a command or fact entry, the controller shall treat that hook exclusively as a notification of already-committed, already-applied state (client reply, scheduler dispatch trigger, SSE/apiserver update).
2. The controller shall not perform WAL append, commit-barrier waiting, or state-machine mutation from within any AFTER_EVENT or BEFORE_EVENT hook callback.
3. The controller shall not require a downstream consumer (scheduler, apiserver) to re-check a commit watermark before acting on a notification delivered by an after-hook, since that notification only fires after the entry is committed and applied.

Requirement 5: Facts Apply-After-Commit

Objective: As a cluster operator, I want a worker-reported fact's state mutation and its consequential reactions (worker slot release, scheduler dispatch, notification) to occur only after the fact is committed to a majority, so that a leader that loses an uncommitted fact never leaves behind a phantom effect such as a freed slot, a triggered dispatch, or a sent notification.

Acceptance Criteria

1. When the controller receives a reported fact from a worker, the controller shall durably append the fact to its log without performing the fact's state mutation or reactions at that time.
2. When a reported fact's entry commits to a majority, the controller shall apply its state mutation and trigger its reactions through the same apply_committed_entry() seam used for commands.
3. The controller shall not release a worker slot, trigger scheduler dispatch, or send a client/apiserver notification for a fact whose entry has not committed to a majority.

Requirement 6: Quorum-Loss Behavior for Retained Facts

Objective: As a cluster operator, I want reported facts to be safely retained — neither dropped nor prematurely applied — when the controller cannot reach a majority, so that the cluster degrades safely and self-heals without operator intervention once quorum is restored.

Acceptance Criteria

1. While the controller cannot reach a majority of cluster members, the controller shall retain durably-appended, uncommitted facts without applying their state mutations or triggering their reactions.
2. The controller shall not apply a fact's state mutation or reactions through any path other than majority commit, regardless of quorum state; no local-only apply allowlist shall exist.
3. When a majority becomes reachable again after a quorum-loss period, the controller shall commit and apply the retained facts in their original log order.

Requirement 7: Idempotent, Ordered Re-Application

Objective: As a maintainer, I want re-applying an already-applied entry — via replay, retry, or worker resync — to be a safe no-op, so that duplicate or out-of-order delivery never regresses or corrupts job state.

Acceptance Criteria

1. When the controller applies an entry whose effect has already been applied, by prior commit, replay, or resync, the controller shall treat the re-application as a no-op with respect to observable job state.
2. If a duplicate or out-of-order fact is applied for a job, then the controller shall compare it against the job's current lifecycle state and shall not regress that job to an earlier lifecycle stage.
3. The controller shall apply entries in strict commit order and shall never apply an entry ahead of another entry that committed before it.

Requirement 8: Restart Apply-Correctness Verification

Objective: As a cluster operator, I want the controller to verify after restart that its materialized state matches its committed log, so that a hidden apply defect is surfaced immediately rather than silently causing future divergence.

Acceptance Criteria

1. When the controller completes WAL replay after a restart, the controller shall verify whether the materialized state matches the expectation derived from the committed log tip, in addition to the existing CRC chain verification.
2. When the post-replay verification matches, the controller shall report a CLEAN verdict.
3. If the post-replay verification does not match, then the controller shall report a DIVERGENT verdict as a loud, operator-visible condition.
4. If the controller reports a DIVERGENT verdict, then the controller shall enter a non-promotable fenced state and shall continue running and retrying catch-up rather than exiting or aborting, consistent with the "controllers never self-terminate" invariant.

Requirement 9: Bounded Resource Behavior Under Concurrent In-Flight Commits

Objective: As a cluster operator, I want the controller to remain responsive to new requests even while many operations are simultaneously awaiting commit, so that a burst of concurrent commands or facts cannot make the controller unresponsive.

Acceptance Criteria

1. While multiple entries are simultaneously awaiting majority commit, the controller shall continue to accept and begin processing new incoming requests.
2. The controller shall not exhaust its bounded request-handling capacity as a direct result of the number of entries currently awaiting commit.
3. When an entry's commit outcome (success, failure, or timeout) becomes known, the controller shall resolve exactly one reply or notification for that entry's originator.
4. If the controller loses leadership or a pending entry's log tail is truncated while the entry is awaiting commit, then the controller shall resolve that entry's outcome as a failure exactly once, rather than leaving it unresolved.

Requirement 10: Apply-Ordering Regression and Conformance Verification

Objective: As a maintainer, I want automated regression coverage asserting the apply-after-commit ordering, so that a future change cannot silently reintroduce apply-before-commit or lost-notification defects.

Acceptance Criteria

1. The project's conformance test suite shall assert that no entry's state mutation or reactions occur before that entry commits.
2. The project's conformance test suite shall assert that a failed or timed-out commit produces no residual state mutation.
3. The project's conformance test suite shall assert that re-applying an already-committed entry, whether by simulated replay or retry, is idempotent.
4. The project's regression suite shall include a live-cluster scenario that kills the primary mid-fact-stream and asserts no phantom slot-frees, no committed-state divergence, and no notification for a fact that did not commit.

Requirement 11: Preservation of Existing Behavior and Performance

Objective: As a cluster operator, I want this refactor to preserve every existing HA behavior, correctness guarantee, and performance characteristic that this feature does not specifically intend to change, so that fixing the apply-ordering defect does not introduce a new regression elsewhere.

Acceptance Criteria

1. The controller's existing automated unit and integration test suites shall continue to pass after this feature lands, except for tests that specifically assert the pre-existing apply-before-commit ordering, which shall be updated to assert the new apply-after-commit ordering instead.
2. The controller's existing HA failover regression and soak suites shall continue to report zero committed-job loss and zero committed-state divergence across a primary failover, matching the acceptance bar already established by prior HA specs.
3. When a client submits a job, cancels a job, or queries job/queue/worker status, or a worker reports a job's start, finish, failure, or resource usage, the controller shall produce the same externally observable end result as before this feature; this feature shall change only the internal timing of when that result becomes visible, not the result itself.
4. While a sustained high-volume worker-fact stream is in progress, the controller shall not regress fact-commit throughput relative to pre-feature behavior, and shall not introduce a per-fact synchronous commit barrier.