myxce/.kiro/specs/ha-commit-barrier-rollout/design.md

# Design: ha-commit-barrier-rollout

> Sub-spec 2 of 4 of the **ha-consistency-invariants** umbrella. This design slices the umbrella design's **Write-Path Classification**, **Config Barrier**, and **Idempotency Guards** components (`.kiro/specs/ha-consistency-invariants/design.md`), implementing umbrella Migration Strategy phase 2. The umbrella's WAL-type classification table and Reply-Ordering Contract are the contract; this design records their call-site verification (which the umbrella explicitly mandated) and the enforcement mechanism. The commit engine interface (`include/commit.h`, `commit_submit` WAIT/RETAIN) is consumed exactly as shipped by ha-commit-engine — this spec never modifies `commit.c` internals. Requirement IDs below are THIS spec's IDs (see `requirements.md` for the umbrella mapping).

## Overview

**Purpose**: This feature extends the majority-commit barrier from job submission to every cluster-state-changing operation: request-shaped operations reply to their requester only after commit (and fail on timeout), fact-shaped operations are appended and retained until committed with no timeout, configuration upgrades fail honestly to the admin tool, and job state transitions become idempotent so retained-fact retry and replay are safe.

**Users**: Cluster operators gain the guarantee that any confirmed control operation survives failover and that worker-reported reality is never silently dropped; downstream sub-specs (ha-single-leader-role, ha-convergence-recovery) gain a write path that is barrier-complete before the transitional leader state is removed.

**Impact**: Adds a classification dispatch at the persistence hook seam (persist.c) with a fail-fast guard for unclassified WAL types; restructures `do_jobcontrol_op` (job.c:2198-2224), `do_queuecontrol_op` (queue.c:128-154), `do_hostcontrol_op` (worker.c:796-822), and `handle_worker_join` (join.c:48-149) to commit-first-then-reply [line numbers re-verified against current source — task-evidence line numbers had drifted]; admits fact-bearing opcodes through the dispatch gate during quorum loss (`ha_dispatch_gate`, ha.c:4525, allowlist at ha.c:4644-4648); propagates the config majority-ACK result from `push_config_to_standbys` (config_repl.c:1609-1654) to the `upgradeConfig` requester (`do_upgrade_config_with_bytes`, config_repl.c:1696-1855 — Note: as of ha-term-safety, the requester's commit gate is `config_commit_change`'s WAL_CONFIG_MARKER barrier, which internally calls `push_config_to_standbys`; see Config Barrier section below); adds state-machine guards to `mark_job_*` (job.c: `mark_job_fail` 1830, `mark_job_update` 1929 — Note: `mark_job_start`/`mark_job_finish` were renamed and restructured by raft-apply-decoupling, see Idempotency Guards section below); and retires the legacy walAck counting fallback in `repl_wait_majority_ack` (repl.c:589-700) while hardening its configAck branch.

### Goals
- Commit-first-then-reply for every request-shaped control handler; `handle_jobsubmit` (job.c:1412 onward) is the reference implementation (Note: superseded by raft-apply-decoupling — `handle_jobsubmit` no longer "stays unchanged"; it was rearchitected to append via an async commit continuation (`jobsubmit_commit_continuation`/`jobsubmit_apply_complete`) rather than a synchronous append-then-reply, and job-array-single-wal-record added a sibling array-submit path. The commit-first-then-reply *outcome* for the requester is preserved, but the reference shape this design points to has changed; see those specs' design.md)
- Fact WAL types routed RETAIN (log-is-the-queue): appended + broadcast, pending purely by `lsn > committed_lsn`, never dropped while running+leading, accepted even during quorum loss
- An unclassified state-changing WAL type cannot silently bypass the barrier (debug assert; production fact-fallback)
- Config upgrade fails to the requester on missed majority and does not restart into the failed config
- `mark_job_*` idempotent and ordered (duplicate = no-op; terminal→non-terminal rejected)

### Non-Goals
- Commit engine internals: per-LSN evaluation, admission cap, `committed_lsn` advancement (ha-commit-engine — consumed as-is)
- Promotion no-op / own-term gate consumers / CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role; the gate's COMMITTING branch keeps rejecting facts until that spec deletes the state — known residual)
- Demotion truncate+restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum-transition logging (ha-convergence-recovery)
- Worker protocol changes; recovery of facts that never reached the controller or died with it (worker-rejoin-resync backlog — umbrella-locked: crash-loss of uncommitted facts accepted until then)
- WAL_NOOP introduction (ha-single-leader-role adds the type; the classification guard here forces it to be classified when it lands — confirmed landed: `wal_classify` (persist.c:5796) now classifies WAL_NOOP as RETAIN exactly as pre-declared below)

## Boundary Commitments

### This Spec Owns
- The WAL-type → commit-mode classification table, its dispatch at the persistence hook seam, and the unclassified-type guard
- The reply ordering of every request-shaped control handler on the primary (job control, queue control, worker control, worker join/update) and the receipt-ACK verdict for scheduler-origin handlers
- The commit-barrier helper (`persist_commit_barrier`) that handlers use to realize WAIT-mode completion
- Fact admission through the dispatch gate's no-write-quorum path (opcode allowlist only — not quorum accounting)
- Config-change barrier error propagation: `push_config_to_standbys` return contract, configAck majority arithmetic, and the `upgradeConfig` caller's failure path
- `mark_job_*` transition guards
- Retirement of the legacy (engine-less) walAck counting in `repl_wait_majority_ack`

### Out of Boundary
- `commit.c` / `include/commit.h` — consumed, never modified (interface changes there trigger revalidation here)
- `ha_has_write_quorum` and quorum accounting (ha-commit-engine); the gate's role-based branches other than the PRIMARY no-quorum opcode list
- `committed_lsn` advancement rules, admission cap, membership re-evaluation (commit engine)
- Egress filtering of provisional state to the scheduler (ha-convergence-recovery; dispatch of uncommitted submits is closed there)
- Standby-side apply, WAL wire format, catch-up streaming mechanics (ha-wal-replication)
- Worker/scheduler retry behavior on error replies (client-side, existing semantics)

### Allowed Dependencies
- `commit_submit(ce, lsn, COMMIT_MODE_WAIT|RETAIN, timeout_ms)` and the engine instance embedded in `cluster_t` (ha-commit-engine, P0) — the only commit-evaluation authority
- `persist_wal_async` / `persist_wal_sync_append` append+broadcast primitives and `persist_t.current_lsn` / `committed_lsn` atomics (persist.c, P0)
- `repl_on_config_ack` / `acked_config_lsn` per-standby config tracking (repl.h, P0) — read-only consumption for the config barrier
- Hook dispatch (`xce_run_hooks`, HOOK_AFTER_EVENT seam) — existing pattern, unchanged signatures
- Threading rules per `ha-async-io.md`: commit-barrier waits only on vtask threads; nothing here runs on iomgr
- Dependency direction: handlers (job.c/queue.c/worker.c/join.c) → persist.c barrier/classification → commit engine; persist.c never calls back into handlers

### Revalidation Triggers
- Any change to `commit.h` signatures or return codes (upstream) → re-verify barrier helper and handler error mapping
- ha-single-leader-role adding WAL_NOOP or removing CTRL_PRIMARY_COMMITTING → classification table gains the WAL_NOOP row (RETAIN, umbrella-fixed); gate COMMITTING branch disappears
- Any new state-changing opcode/WAL type → must be added to the classification table (the debug assert makes this unmissable) and its handler swept for reply ordering
- worker-rejoin-resync landing → re-verify `mark_job_*` guards against its replay ordering
- Changes to `acked_config_lsn` semantics or the upgradeConfigAck payload → re-verify configAck majority arithmetic

## Architecture

### Existing Architecture Analysis
- All state-changing writes flow through one seam: entity mutation fires `HOOK_AFTER_EVENT`, and `persist_job_hook_fun` / `persist_queue_hook_fun` / `persist_worker_hook_fun` (persist.c:6509/6589/6638 — re-verified, was 1093/1172/1204) map hook details to WAL types via `detail_to_wal_type` (persist.c:5769-5789, was 1068-1088) and append. WAL_JOB_SUBMIT is now committed via an explicit call from `handle_jobsubmit` (`persist_submit_job_payloads_and_commit`) strictly before any job object is constructed, never from this hook — the hook's WAL_MODE_WAIT branch is now a pure post-commit no-op (Note: superseded by raft-apply-decoupling task 3.1; the "in-hook sync barrier" description below and elsewhere in this doc no longer matches — see that spec's design.md). Everything else still routes through `wal_hook_append_classified` (append + RETAIN registration), functionally equivalent to the `persist_wal_async` fire-and-forget this doc describes
- Request handlers reply immediately after mutation: `do_jobcontrol_op` (job.c:2198), `do_queuecontrol_op` (queue.c:128), `do_hostcontrol_op` (worker.c:796), `handle_worker_join` (join.c:48) — all violate umbrella 2.1 today [historical framing — this spec fixes it; see Reply-Ordering Sweep]. `handle_jobsubmit` (job.c:1412) is the proven correct shape at the time this spec closed (Note: superseded by raft-apply-decoupling — see Goals section note above)
- Queue/worker/job-state hooks fire while the entity WRITE lock is held; job-submit's hook now fires post-commit outside any entity lock as a pure notification (raft-apply-decoupling task 3.1) — this asymmetry still drives WAIT placement (below), though the job-submit half of the asymmetry is now "no hook work at all" rather than "hook runs lock-free"
- Facts arrive via `do_notify` (cluster.c, no reply) and `do_update_op`/`do_set_preason`/`do_clr_preason` (receipt-style replies); the dispatch gate (`ha_dispatch_gate`, ha.c:4525) rejects `OPCode_notify` with `-XCE_ERR_NOT_READY` under quorum loss → permanent silent fact loss (bug-396 family, verified pre-existing; fixed by this spec, see Fact Gate Admission)
- Call-site verification of the umbrella classification table (mandated by the umbrella): WAL_JOB_STOP/RESUME are appended only from `on_job_stop`/`on_job_resume`, which are invoked exclusively from `do_notify` (worker-reported stop/resume facts). Client job control (`do_jobcontrol_op` → `job_ctrl`, job.c:957) either forwards to the worker (running job — no local entry) or mutates locally via `mark_job_update`/`mark_job_fail` (not-started job → WAL_JOB_UPDATE/WAL_JOB_FAIL). WAL_JOB_UPDATE is absent from the umbrella table and is dual-origin (request: job_ctrl; fact: on_job_postdone, rusage update). Resolution: STOP/RESUME/UPDATE/FAIL classify as fact/RETAIN at the seam, and the umbrella's intent for client job control — "error to client" — is realized by the handler-level commit barrier in `do_jobcontrol_op` (the Reply-Ordering Contract's handler sequence is the umbrella-authoritative mechanism). Recorded in research.md
- `push_config_to_standbys` (config_repl.c:1609-1654, was 866-930) calls `repl_wait_majority_ack(configAck)` but only logs the result and returns the push count; the caller ACKed the admin tool with status=0 and restarted regardless [historical framing — this spec fixes it]. The configAck counting (repl.c, was 418-438) has the same defects the engine fixed for walAck: `acked_config_lsn > 0` (any previous ack counts), `n_active == 0 → success`, denominator `n_active` instead of `n_peers` [historical — fixed, see Config Barrier]

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    subgraph requesters
        CLI[client and cli ops]
        SCHED[scheduler]
        WK[worker]
        ADM[admin tool]
    end
    subgraph controller_primary
        GATE[ha dispatch gate<br/>fact allowlist]
        H[request handlers<br/>job queue worker join]
        F[fact handlers<br/>notify update preason]
        HOOKS[persist hooks<br/>classification dispatch]
        BAR[persist_commit_barrier]
        CB[config barrier<br/>config_repl]
        CE[commit engine<br/>consumed]
        GUARD[mark_job guards]
    end
    CLI --> GATE
    SCHED --> GATE
    WK --> GATE
    ADM --> GATE
    GATE --> H
    GATE --> F
    H --> GUARD
    F --> GUARD
    GUARD --> HOOKS
    HOOKS -->|append plus RETAIN register| CE
    H -->|before success reply| BAR
    BAR -->|commit_submit WAIT| CE
    ADM -.upgradeConfig.-> CB
    CB -->|configAck majority| CE
```

**Architecture Integration** (sliced from umbrella):
- Selected pattern: classification at the single persistence seam + completion at the handler. The seam guarantees no entry type escapes classification (2.1, 2.2); the handler realizes the umbrella Reply-Ordering Contract sequence "validate → mutate (provisional) → WAL append → commit_submit(WAIT) → reply" (1.1)
- WAIT placement decision: WAL_JOB_SUBMIT keeps its in-hook sync barrier (hook runs lock-free, engine-wired upstream). All other request-shaped types complete via `persist_commit_barrier()` called by the handler after entity locks are released — blocking a commit wait under a queue/job/worker WRITE lock would stall readers for up to the commit timeout. Same completion semantics, umbrella-conformant placement (the contract names the handler sequence)
- Existing patterns preserved: hook-based persistence dispatch, provisional in-memory mutation order (umbrella-locked), `errnox` early-return error style, full fan-out broadcast
- Steering compliance: barrier waits only on vtask threads (handlers run on the vtask pool); iomgr untouched; TDD with testsuite/ tests first

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Backend / Services | C (gnu17), pthreads, stdatomic | classification, barrier helper, handler restructure | no new dependencies, no .capnp changes |
| Data / Storage | existing WAL (persist.c), wal_types.h enum | classification keyed on `wal_entry_type` + worker detail | no on-disk format change |
| Build / Test | meson, testsuite/ pattern | new `test_commit_barrier_rollout.c` | XCE_TESTING seams |

## File Structure Plan

### Directory Structure
```
src/controller/
├── persist.c                          # MODIFIED — classification dispatch, RETAIN registration,
│                                      #            persist_commit_barrier, unclassified guard
├── job.c                              # MODIFIED — do_jobcontrol_op reply ordering; mark_job_* guards
├── queue.c                            # MODIFIED — do_queuecontrol_op reply ordering
├── worker.c                           # MODIFIED — do_hostcontrol_op reply ordering
├── join.c                             # MODIFIED — handle_worker_join reply ordering
├── ha.c                               # MODIFIED — dispatch-gate fact allowlist (PRIMARY branch only)
├── config_repl.c                      # MODIFIED — push_config_to_standbys return contract; caller failure path
├── repl.c                             # MODIFIED — legacy walAck counting retired; configAck branch hardened
└── testsuite/
    └── test_commit_barrier_rollout.c  # NEW — classification, barrier ordering, fact retention,
                                       #       config barrier, idempotency guards
src/include/persist.h                  # MODIFIED — wal_commit_mode_t enum, wal_classify(),
                                       #            persist_commit_barrier() declarations
src/include/repl.h                     # MODIFIED — configAck evaluation reuses the entry_lsn
                                       #            parameter (added by ha-commit-engine) as the
                                       #            config push tag; no new parameter
```

### Modified Files (what and why)
[Line citations below re-verified against current source (was accurate as of this spec's 2026-06-13 close; substantial drift since from file growth — see per-line notes)]
- `src/include/persist.h` — declares `wal_commit_mode_t { WAL_MODE_WAIT, WAL_MODE_RETAIN, WAL_MODE_LOCAL }`, `wal_commit_mode_t wal_classify(uint32_t wal_type, int detail)`, `int persist_commit_barrier(persist_t *p, uint32_t timeout_ms)`
- `src/controller/persist.c` — `wal_classify` (persist.c:5796) table; all three hook functions consult it: WAIT → existing sync path (submit) or async append with handler-owed barrier; RETAIN → `persist_wal_async` + `commit_submit(RETAIN)` registration; LOCAL → unchanged (WAL_CHECKPOINT); unclassified → `assert(0)` under XCE_TESTING, ERROR + RETAIN fallback in production. `persist_commit_barrier` implemented here (persist.c:5875 — snapshot `current_lsn`, `commit_submit(WAIT)`). Note: the table has since grown two rows this design never described — `WAL_JOB_ARRAY_SUBMIT` (WAIT, added by job-array-single-wal-record) and `WAL_CONFIG_MARKER` (RETAIN, added by ha-term-safety task 4.2) — see both specs' design.md
- `src/controller/job.c` — `do_jobcontrol_op` (job.c:2198-2224, was 921-934): barrier between `job_ctrl` success and the reply. `mark_job_fail`/`mark_job_update` (job.c:1830/1929, was 696/726): transition guards before mutation. `mark_job_start`/`mark_job_finish` (was job.c:583/638) no longer exist under those names — renamed `record_job_start_fact`/`record_job_finish_fact` (job.c:1625/1734) with the guard+mutation split across receipt time and apply time (Note: superseded by raft-apply-decoupling, see Idempotency Guards below and that spec's design.md)
- `src/controller/queue.c` — `do_queuecontrol_op` (queue.c:128-154, was 119-132): barrier between `queue_ctrl` success and the reply
- `src/controller/worker.c` — `do_hostcontrol_op` (worker.c:796-822, was 689-702): barrier between `worker_ctrl` success and the reply
- `src/controller/join.c` — `handle_worker_join` (join.c:48-149, was 33-119): barrier between the AFTER_EVENT hook and the reply at join.c:128-135 (was 107-109); join failure (idx < 0) keeps replying immediately
- `src/controller/ha.c` — `ha_dispatch_gate` (ha.c:4525, was implicitly ~1527) PRIMARY no-quorum allowlist (ha.c:4644-4648, was 1537-1556): adds `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` — confirmed present in current source. The gate has since grown two additional fence layers checked before this allowlist (`config_serve_fenced`, `upgrade_materialization_fenced` — ha-term-safety / config-upgrade-snapshot work); those fences do NOT include the fact opcodes in their own allowlists, so a config-fenced primary rejects facts too — a narrower, config-specific case this design did not anticipate and does not contradict (quorum-loss admission, this spec's concern, is unaffected)
- `src/controller/config_repl.c` — `push_config_to_standbys` (config_repl.c:1609-1654, was 866-930) returns the barrier result (0 / `-ETIMEDOUT`), push count via out-param; passes the in-flight tag for per-push evaluation; caller `do_upgrade_config_with_bytes` (config_repl.c:1696-1855, was 1057): on failure send error reply, skip restart. Note: superseded by ha-term-safety task 4.2 — the direct caller of `push_config_to_standbys` is now `config_commit_change` (config_repl.c:535), which `do_upgrade_config_with_bytes` calls instead; the requester's success/failure now gates on a WAL_CONFIG_MARKER consensus record reaching the committed prefix, with `push_config_to_standbys`'s majority-ACK wait as the out-of-band byte-delivery quorum underneath it. See Config Barrier section and ha-term-safety's design.md
- `src/controller/repl.c` — walAck branch of `repl_wait_majority_ack` (repl.c:589-700, was 386-462): legacy counting removed (engine is mandatory on this path; `ce == NULL` becomes a programming error); configAck branch: per-push comparison (`acked_config_lsn >= push_tag`), majority over `n_peers` counting self, `n_peers > 1` with zero standby acks never succeeds — confirmed unchanged in current source
- `src/controller/meson.build` — register `testsuite/test_commit_barrier_rollout.c` (confirmed present)

## System Flows

### Request-shaped control operation (job/queue/worker control, worker join)

```mermaid
sequenceDiagram
    participant R as requester
    participant V as vtask thread handler
    participant E as entity lock scope
    participant P as persist hooks
    participant CE as commit engine
    R->>V: control request
    V->>V: dispatch gate incl write quorum
    V->>E: lock, validate, mutate provisional
    E->>P: HOOK_AFTER_EVENT append entries
    P->>CE: persist_wal_async + register
    E->>V: unlock, rc
    alt rc != 0
        V->>R: error reply
    else rc == 0
        V->>CE: persist_commit_barrier WAIT
        alt committed
            V->>R: success reply
        else timeout or not ready
            V->>R: error reply
        end
    end
```

Flow decisions: the barrier waits on a snapshot of `current_lsn` taken after the mutation returns — single-writer appends and contiguous commit advancement guarantee the snapshot covers every entry the operation produced (1.1, 1.4); waiting may also cover unrelated concurrent entries (bounded by the commit timeout, harmless — they must commit anyway); `-ETIMEDOUT`/`-XCE_ERR_NOT_READY` map to the error reply via the existing `errnox` style (1.2, 1.3); no in-memory rollback (provisional model, umbrella-locked — outcome converges per umbrella 2.5, owned downstream).

### Fact-shaped delivery (worker notify, rusage update, preason, worker dead)
Gate admits the opcode even without write quorum (3.2) → handler applies the transition through the `mark_job_*` guards (5.1-5.3) → hook classifies the WAL type RETAIN → `persist_wal_async` append + broadcast + `commit_submit(RETAIN)` (returns immediately, no timeout) → handler returns (notify: no reply; update/preason: receipt-style reply unchanged, verdict per 2.3/2.4). The fact is pending purely by `lsn > committed_lsn`; catch-up streaming re-delivers the uncommitted suffix and `committed_lsn` advances when ACKs resume (3.1, 3.3, 3.4). No new queue structure exists (log-is-the-queue, umbrella-locked).

### Configuration upgrade

> **Note: the diagram and "request restart" step below are superseded by ha-term-safety (task 4.2), see that spec's design.md.** Current source inserts `config_commit_change` (config_repl.c:535, appending a WAL_CONFIG_MARKER) between `do_upgrade_config_with_bytes` and `push_config_to_standbys`, and no restart is requested on success — materialization now happens via `apply_committed_entry`'s WAL_CONFIG_MARKER case on every member's own apply of the committed marker. The majority-ACK shape below (push → parallel threads → per-standby ack → majority-of-n_peers) is otherwise still accurate as the mechanism `push_config_to_standbys` uses underneath the marker.

```mermaid
sequenceDiagram
    participant A as admin tool
    participant U as do_upgrade_config_with_bytes
    participant PS as push_config_to_standbys
    participant S as standbys
    A->>U: OP_upgradeConfig
    U->>U: validate, save to disk, bump generation
    U->>PS: push bytes, in-flight tag
    PS->>S: parallel push threads
    S-->>PS: OP_upgradeConfigAck tag
    PS->>PS: majority of n_peers acked tag, self counted
    alt majority within timeout
        PS-->>U: 0, acked count
        U->>U: committed_config_tag advanced
        U->>A: success reply
        U->>U: request restart
    else timeout
        PS-->>U: -ETIMEDOUT
        U->>A: error reply upgrade failed
        U->>U: no restart, tag not committed
    end
```

Flow decisions: `committed_config_tag` advances only inside the majority branch (4.3 — already correct, preserved, now framed by ha-term-safety as a non-authoritative cache of the marker-derived committed config); the failure path uses the existing `send_upgrade_error` shape so the admin tool sees a definite failure (4.2); restart was originally gated on success (4.4 — closed the roadmap residual "don't restart before the commit barrier passes") but ha-term-safety later removed the restart step entirely in favor of marker-driven materialization on every member — see the supersession note above; config remains a parallel, simpler track sharing only the majority arithmetic rule with the engine (umbrella-locked: one in-flight config change at a time).

## Requirements Traceability

| Requirement | Summary | Components | Flows |
|-------------|---------|------------|-------|
| 1.1 | success reply only after majority commit | Reply-Ordering Sweep, Commit Barrier Helper | request control |
| 1.2 | commit timeout → error reply | Commit Barrier Helper, Reply-Ordering Sweep | request control |
| 1.3 | not-ready rejection → error reply | Commit Barrier Helper (engine code passthrough) | request control |
| 1.4 | no success-interpretable reply pre-commit | Reply-Ordering Sweep (integration-test assertion) | request control |
| 2.1 | every WAL type classified WAIT/RETAIN | Write-Path Classification | both write flows |
| 2.2 | unclassified type fails fast / fact-fallback | Write-Path Classification (guard) | — |
| 2.3 | dispatch ACK is receipt-only | Reply-Ordering Sweep (handle_jobdispatch verdict) | — |
| 2.4 | exhaustive reply-site verification | Reply-Ordering Sweep (audit) | — |
| 3.1 | facts appended, retried, no timeout | Write-Path Classification (RETAIN routing) | fact delivery |
| 3.2 | facts accepted during quorum loss | Fact Gate Admission | fact delivery |
| 3.3 | uncommitted facts never discarded | Write-Path Classification (RETAIN; log-is-the-queue) | fact delivery |
| 3.4 | facts commit after quorum restore, no redelivery | Write-Path Classification + existing catch-up (verified by test) | fact delivery |
| 4.1 | config confirmed only after majority | Config Barrier | config upgrade |
| 4.2 | config timeout → failed to requester | Config Barrier | config upgrade |
| 4.3 | unacked config never committed | Config Barrier (tag advance in success branch only) | config upgrade |
| 4.4 | failed upgrade does not restart | Config Barrier (caller) | config upgrade |
| 5.1 | duplicate transition is logged no-op | Idempotency Guards | fact delivery |
| 5.2 | terminal→non-terminal rejected with WARN | Idempotency Guards | fact delivery |
| 5.3 | replay-once equivalence | Idempotency Guards | fact delivery |

## Components and Interfaces

| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|-------|--------|--------------|------------------|-----------|
| Write-Path Classification | controller/persist.c | route every WAL type to its commit mode at the hook seam | 2.1, 2.2, 3.1, 3.3, 3.4 | commit engine (P0), hooks (P0) | Service |
| Commit Barrier Helper | controller/persist.c | handler-callable WAIT completion on the appended watermark | 1.1, 1.2, 1.3 | commit engine (P0) | Service |
| Reply-Ordering Sweep | controller/job.c, queue.c, worker.c, join.c | commit-first-then-reply in every request handler; receipt-ACK verdicts | 1.1, 1.2, 1.3, 1.4, 2.3, 2.4 | Commit Barrier Helper (P0) | Service |
| Fact Gate Admission | controller/ha.c | admit fact-bearing opcodes without write quorum | 3.2 | dispatch gate (P0) | Service |
| Config Barrier | controller/config_repl.c, repl.c | per-push config majority with error propagation | 4.1, 4.2, 4.3, 4.4 | repl_on_config_ack tracking (P0) | Service |
| Idempotency Guards | controller/job.c | safe duplicate/replay of job transitions | 5.1, 5.2, 5.3 | — | Service |

### Write-Path Classification (`persist.c`, `src/include/persist.h`)

| Field | Detail |
|-------|--------|
| Intent | Single seam where every state-changing WAL append is classified and registered with the engine |
| Requirements | 2.1, 2.2, 3.1, 3.3, 3.4 |

**Classification table** (umbrella table verified against call sites — verification deviations recorded in research.md §2; this verified table is the implementation contract):

| WAL type (+ detail) | Verified origin | Shape | Mode | Failure behavior |
|---|---|---|---|---|
| WAL_JOB_SUBMIT | handle_jobsubmit (client) | request | WAIT (existing in-hook sync path) | error to client (reference impl, unchanged) |
| WAL_QUEUE_CTRL | do_queuecontrol_op (client) | request | WAIT (handler barrier) | error to client |
| WAL_WORKER_CTRL, detail JOIN/UPDATE | handle_worker_join, do_hostcontrol_op | request | WAIT (handler barrier) | error reply to worker/client |
| WAL_WORKER_CTRL, detail DEAD | controller-detected (worker.c:565/578) | fact | RETAIN | none — retained until committed |
| WAL_JOB_START / FINISH / FAIL / STOP / RESUME / UPDATE / RUSAGE / CKPT / PREASON | worker notify, scheduler preason, postdone, rusage; FAIL/UPDATE also from job_ctrl (request origin → covered by do_jobcontrol_op's handler barrier) | fact | RETAIN | none — retained until committed |
| WAL_NOOP | promotion (ha-single-leader-role) | internal | RETAIN | landed and confirmed classified exactly as pre-declared (persist.c:5835-5839) — no longer future |
| WAL_JOB_ARRAY_SUBMIT | handle_jobsubmit_array (client, job-array-single-wal-record) | request | WAIT (handler barrier, single barrier for the whole array) | error to client — row added by job-array-single-wal-record, not in this design's original table (Note: superseded/extended, see that spec's design.md) |
| WAL_CONFIG_MARKER | config_commit_change (ha-term-safety task 4.2) | request-adjacent consensus record | RETAIN | commits through the same barrier as job-state facts; the upgrade requester's own barrier is owned by `do_upgrade_config_with_bytes`, not this row — row added by ha-term-safety, not in this design's original table (Note: superseded/extended, see that spec's design.md and the Config Barrier section below) |
| WAL_CHECKPOINT | local checkpoint marker | local-only | LOCAL | unchanged (never replicated, by design) |

**Responsibilities & Constraints**
- `wal_classify(wal_type, detail)` is a pure lookup; `detail` disambiguates only WAL_WORKER_CTRL (JOIN/UPDATE vs DEAD)
- Hook dispatch: RETAIN → `persist_wal_async` (append + broadcast, the surviving RETAIN fast path) followed by `commit_submit(ce, appended_lsn, COMMIT_MODE_RETAIN, 0)` registration (returns immediately; observability counter). WAIT → WAL_JOB_SUBMIT keeps `persist_wal_sync_append`; other WAIT types append via `persist_wal_async` and completion is owed by the handler barrier (placement decision, Architecture)
- Unclassified guard: under `XCE_TESTING`, `assert(0)` with the offending type logged; in production, `ERROR` log + RETAIN treatment (never dropped, never falsely confirmed) (2.2)
- Fact appends never return commit errors to their callers — `persist_job_hook_fun`'s non-submit return value stays the local-append rc only (3.1)

**Dependencies**
- Inbound: hook system (HOOK_AFTER_EVENT) — all state-changing writes (P0)
- Outbound: `commit_submit` (P0); `persist_wal_async`/`persist_wal_sync_append` (P0)

**Contracts**: Service [x]

##### Service Interface
```c
/* src/include/persist.h */
typedef enum {
    WAL_MODE_WAIT,    /* request-shaped: barrier before requester reply */
    WAL_MODE_RETAIN,  /* fact-shaped: retained until committed, no timeout */
    WAL_MODE_LOCAL,   /* not replicated (WAL_CHECKPOINT) */
} wal_commit_mode_t;

/* Pure classification lookup. detail disambiguates WAL_WORKER_CTRL only.
 * Unknown type: XCE_TESTING assert; production returns WAL_MODE_RETAIN
 * after ERROR log. */
wal_commit_mode_t wal_classify(uint32_t wal_type, int detail);
```
- Preconditions: called at the hook seam with the type produced by `detail_to_wal_type`
- Postconditions: total over all `wal_entry_type` values; deterministic
- Invariants: no WAL append path bypasses classification

### Commit Barrier Helper (`persist.c`, declared in `src/include/persist.h`)

| Field | Detail |
|-------|--------|
| Intent | Give request handlers a single call realizing commit_submit(WAIT) on everything their mutation appended |
| Requirements | 1.1, 1.2, 1.3 |

**Responsibilities & Constraints**
- Snapshot `lsn = p->current_lsn` (under `p->lock`, matching `persist_wal_sync_append`'s read at persist.c:1019-1021); if `committed_lsn >= lsn` already, return 0 without blocking
- Delegate to `commit_submit(ce, lsn, COMMIT_MODE_WAIT, timeout_ms ? timeout_ms : p->config.sync_timeout_ms)`; return its code unmodified (0 / `-ETIMEDOUT` / `-XCE_ERR_NOT_READY`)
- Single-writer + contiguous advancement ⇒ the snapshot covers every entry the calling operation produced; over-waiting on unrelated concurrent entries is accepted (bounded by the timeout; they must commit anyway). `n_peers <= 1` returns 0 immediately (engine self-quorum, standalone behavior preserved)
- Must be called only from vtask threads after entity locks are released; counts against the engine's admission cap (deliberate CP back-pressure)
- Migration completion: with all request paths on the engine (directly or via this helper), the walAck legacy counting in `repl_wait_majority_ack` is removed (upstream marked it "retired by ha-commit-barrier-rollout"); the configAck branch remains the config track's mechanism

**Contracts**: Service [x]

##### Service Interface
```c
/* src/include/persist.h */
/* Block until committed_lsn covers everything appended so far.
 * timeout_ms = 0 → p->config.sync_timeout_ms.
 * Returns 0 (committed), -ETIMEDOUT (not committed in time — entry
 * remains in log; outcome converges per umbrella 2.5, owned downstream),
 * -XCE_ERR_NOT_READY (admission cap / own-term gate, retriable). */
int persist_commit_barrier(persist_t *p, uint32_t timeout_ms);
```
- Preconditions: commit engine wired (`cluster_init` phase, upstream); vtask thread; no entity locks held
- Postconditions: returns 0 ⇔ `committed_lsn >=` the snapshot watermark
- Invariants: never sends replies, never mutates state

### Reply-Ordering Sweep (`job.c`, `queue.c`, `worker.c`, `join.c`)

| Field | Detail |
|-------|--------|
| Intent | Enforce the umbrella Reply-Ordering Contract at every control-path reply site |
| Requirements | 1.1, 1.2, 1.3, 1.4, 2.3, 2.4 |

**Responsibilities & Constraints**
- Restructure pattern (uniform): `rc = <ctrl fn>(...); if (rc == 0) rc = persist_commit_barrier(&c->persist, 0); reply.status = rc; xce_reply(...)`. Confirmed still in place at (line numbers re-verified against current source, drifted from file growth since this spec's 2026-06-13 close — see File Structure Plan §Modified Files for the specific old→new lines):
  - `do_jobcontrol_op` (job.c:2198-2224) — covers the not-started branch's WAL_JOB_UPDATE/WAL_JOB_FAIL entries; the running-job branch (forward to worker, no local entry) waits on an already-covered watermark and returns immediately
  - `do_queuecontrol_op` (queue.c:128-154)
  - `do_hostcontrol_op` (worker.c:796-822)
  - `handle_worker_join` (join.c:48-149) — barrier inserted between the AFTER_EVENT hook (join.c:118-120) and the success reply (join.c:137-139); error replies (validate_host, hostconfig, idx < 0) stay immediate; `on_worker_join` dataex push stays after the reply
- Receipt-ACK verdicts (verified, kept, classification-guard-enforced): `handle_jobdispatch` (job.c:1119, was 459-491) — early reply is ACK-of-receipt; `dispatch_a_job_` produces no WAL entries (allocation results return via `notify_dispatch`, and resulting job-state changes arrive later as worker facts) → per the umbrella contract the early reply stands (2.3). `do_set_preason`/`do_clr_preason` (job.c:2330/2358, was 1040/1068) — scheduler-origin facts (WAL_JOB_PREASON, RETAIN); early reply is receipt-only. `do_update_op` paths (opupd.c:48/64/88) — worker-origin facts (rusage → WAL_JOB_RUSAGE/WAL_JOB_UPDATE RETAIN; resvals produce no WAL); replies are receipt-only
- Exhaustive audit (2.4): every `xce_reply` call site under src/controller/ (excluding testsuite) is enumerated by grep and assigned one of {post-commit confirmation, receipt-only ACK, read-only/replication/error reply}; the audit table was committed with the task evidence. The design-time line-number snapshot originally recorded here (job.c/queue.c/worker.c/join.c/opupd.c/cluster.c/config_repl.c/ha.c/opget.c/reply.c/task.c line lists) has drifted wholesale with subsequent file growth across a dozen-plus later specs and is no longer independently meaningful as a citation list; the audit's *coverage claim* (every reply site classified, zero unclassified) is not re-verifiable line-by-line within this pass's scope — re-run the grep sweep against current source before relying on the exhaustiveness claim
- A missed handler silently violates umbrella 2.1 — the integration test asserting no-success-reply-before-commit is the behavioral backstop (1.4)

**Contracts**: Service [x] — external reply semantics: success now implies majority durability; new error codes `-ETIMEDOUT`/`-XCE_ERR_NOT_READY` surface to requesters through existing reply envelopes (Go tier passes them through, per umbrella out-of-boundary)

**Implementation Notes**
- Integration: handlers run on vtask threads (cluster.c dispatch switch at cluster.c:325-363) — barrier blocking is steering-compliant
- Validation: integration test drives each restructured opcode against a 3-node-simulated registry with ACKs withheld → error reply, `committed_lsn` unmoved; with ACKs → success reply strictly after commit
- Risks: scheduler/worker callers seeing new timeout errors where they previously saw instant success — deliberate CP behavior (umbrella); retry semantics unchanged on their side

### Fact Gate Admission (`ha.c`)
Summary-only: add `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` to the CTRL_PRIMARY no-write-quorum allowlist in `ha_dispatch_gate` (ha.c:1537-1556) so reported facts reach the RETAIN path during quorum loss (3.2 — closes the bug-396 silent-loss hole; umbrella error strategy: "fact-shaped ops are still appended and retained"). Request-shaped opcodes (jobsub, jobcontrol, queuecontrol, hostcontrol, jobdispatch, join) remain rejected without quorum. The CTRL_PRIMARY_COMMITTING branch is intentionally untouched (deleted wholesale by ha-single-leader-role — known residual until then). Quorum accounting itself is out of boundary (upstream).

### Config Barrier (`config_repl.c`, `repl.c` configAck branch)

| Field | Detail |
|-------|--------|
| Intent | Make the config majority-ACK result authoritative for the upgrade outcome |
| Requirements | 4.1, 4.2, 4.3, 4.4 |

**Responsibilities & Constraints**

> **Note: partially superseded by ha-term-safety (task 4.2), see that spec's design.md.** As of this spec's 2026-06-13 close, `do_upgrade_config_with_bytes` called `push_config_to_standbys` directly and its return code alone gated the requester's success/failure reply (described below, still architecturally accurate for `push_config_to_standbys` itself). ha-term-safety inserted a layer: `do_upgrade_config_with_bytes` now calls `config_commit_change` (config_repl.c:535), which appends a `WAL_CONFIG_MARKER` as the authoritative (term,lsn) consensus record and *internally* calls `push_config_to_standbys` as the out-of-band byte-delivery quorum underneath that marker. The requester's success/failure now gates on the marker reaching the committed prefix, not directly on `push_config_to_standbys`'s return. The majority-ACK arithmetic itself (below) is unchanged and still lives in `push_config_to_standbys`/`repl_wait_majority_ack`.

- `push_config_to_standbys` (config_repl.c:1609-1654, was 866-930) signature: `int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen, const uint8_t sha256[32], uint64_t new_tag, int *acked_out)` — confirmed unchanged — returns the barrier result (0 / `-ETIMEDOUT`), push count via `acked_out`. The majority wait passes `new_tag` down for per-push evaluation; `committed_config_tag` advance stays inside the success branch only (4.3 preserved, now additionally described by ha-term-safety as a "non-authoritative cache" of the marker-derived committed config)
- configAck branch of `repl_wait_majority_ack` (repl.c:616-699, was 418-438) hardened to mirror the engine's rules on the config track (umbrella: "same majority arithmetic, parallel simpler track") — confirmed unchanged: count standbys with `acked_config_lsn >= push_tag` (not `> 0` — a previous upgrade's ack no longer counts), majority = `1 (self) + count > n_peers / 2` over configured members (not `n_active`), `n_peers <= 1` → immediate success, `n_peers > 1` with zero covering acks → `-ETIMEDOUT` at deadline (the `n_active == 0 → success` short-circuit is removed from this branch). The configAck branch reuses the `entry_lsn` parameter already added to `repl_wait_majority_ack` by ha-commit-engine as the config push tag — no second parameter is added (`repl.h` comment updated accordingly); the function is now configAck-only (walAck callers are on the engine)
- Caller `do_upgrade_config_with_bytes` (config_repl.c:1696-1855, was 1056-1094): on non-zero return from `config_commit_change` (was directly from `push_config_to_standbys`, see supersession note above) — `send_upgrade_error(ep, req, c, ...)` to the admin tool (4.2), skip the success ACK, skip restart. Disk save and generation bump have already happened (pre-existing sequence); the reply makes the failure explicit and the cluster-wide tag stays uncommitted — convergence/rollback of a half-pushed config remains the existing OP_replConfig anti-entropy behavior (out of boundary). Materialization on success now happens via `apply_committed_entry`'s WAL_CONFIG_MARKER case on every member's own apply of the marker — no restart is requested or required (a further ha-term-safety change beyond this spec's original "restart is gated on success" framing; see the Flow decisions paragraph below, now stale on this point)
- One in-flight config change at a time (umbrella-locked) — no per-entry registry needed

**Contracts**: Service [x] — `upgradeConfig` reply contract: success ⇔ majority-acked config; failure carries an error string and implies no restart

**Implementation Notes**
- Integration: `CONF_PUSH_TIMEOUT_MS` remains the config commit timeout
- Validation: integration test withholds configAcks → admin reply is an error, `committed_config_tag` unmoved, no restart request; with acks → success and tag advanced
- Risks: admin tooling that treated any reply as success must observe `status` — existing CLI already surfaces reply status (admin-config-tool backlog item consumes this contract)

### Idempotency Guards (`job.c`)

| Field | Detail |
|-------|--------|
| Intent | Make job state transitions safe under duplicate, out-of-order, and replayed application |
| Requirements | 5.1, 5.2, 5.3 |

**Responsibilities & Constraints**

> **Note: partially superseded by raft-apply-decoupling, see that spec's design.md.** `mark_job_fail`/`mark_job_update` (job.c:1830/1929) still exist essentially as described below — guard-then-mutate synchronously under the caller-held WRITE lock, used by the client-request path (`job_ctrl`). `mark_job_start`/`mark_job_finish` no longer exist under those names: they were renamed `record_job_start_fact`/`record_job_finish_fact` (job.c:1625/1734, worker-fact receipt path only) and split into two phases — a receipt-time "peek" of the identical guard (log-only, for the rejected-transition WARN; no mutation) plus a durable-append-only WAL write with no immediate mutation and no AFTER_EVENT hook, followed by the *actual* guarded mutation + hook firing later, deferred to `apply_committed_entry()` (via the extracted `apply_job_start_transition`/`apply_job_finish_transition` cores) once the entry reaches the committed prefix and the apply pump dispatches it. The guard *rules* (duplicate → no-op, backward → rejected+WARN) and their *outcome* are unchanged from what this design specifies below; the *timing* — guard-before-append vs. guard-at-apply-time — is not.

- Guard placement (as designed; see supersession note above for the current split for START/FINISH): at the top of `mark_job_start` (job.c:583), `mark_job_finish` (job.c:638), `mark_job_fail` (job.c:696), `mark_job_update` (job.c:726), under the caller-held job WRITE lock, before the BEFORE_EVENT hook fires (a guarded-out transition runs no hooks → no WAL entry, no double-fire)
- Rules (terminal = any of DONE | EXIT, with or without PERR/PDONE modifiers):
  - duplicate: requested resulting state == current state → INFO log, return 0, no mutation, no hooks (5.1)
  - backward: current state terminal AND requested state non-terminal (START on finished job, UPDATE clearing terminal bits) → WARN log, return `-EALREADY`-style rejection, state preserved (5.2)
  - forward/normal: proceed unchanged; `mark_job_fail`'s existing post-exec special case (job.c:702-716) already tolerates fail-after-finish and is kept as the terminal-state-compatible path
- Replay equivalence (5.3): guards make `restore_job_state`-driven re-application and RETAIN retry converge to single-application state; callers (`on_job_*`, `job_ctrl`) treat the duplicate code as success-no-op
- Shared prerequisite of worker-rejoin-resync (umbrella adjacent expectation) — guards live here, resync consumes them later

**Contracts**: Service [x] — `mark_job_*` return contract gains the logged no-op/rejection outcomes; all existing callers checked during the sweep

## Data Models

### Domain Model (changes only)
- **wal_commit_mode_t** — new enum (persist.h); pure classification metadata, never persisted
- **WAL entries, job_t, queue_t, worker_t** — no format or field changes in this spec (`submit_lsn` tagging is ha-convergence-recovery's egress filter, not here)
- Invariants: every appended state-changing entry is classified; a success reply to a request-shaped operation implies `committed_lsn >=` all entries that operation appended; `committed_config_tag` advances only on per-push config majority; job state never transitions terminal → non-terminal

## Error Handling

### Error Strategy
- Request-shaped commit failure: `-ETIMEDOUT` / `-XCE_ERR_NOT_READY` propagate to the requester reply via the existing `errnox` early-return pattern; no in-memory rollback (provisional model, umbrella-locked; convergence owned downstream)
- Fact-shaped paths never surface commit errors to reporters: local append failure (disk) keeps its existing ERROR logging; commit progress is the engine's concern (RETAIN)
- Unclassified WAL type: XCE_TESTING assert (fail fast in CI); production ERROR + RETAIN fallback (2.2 — fail-safe toward retention, never silent confirmation)
- Config failure: explicit error reply via `send_upgrade_error`; no restart; WARN with tag and acked count
- Guard rejections: INFO (duplicate) / WARN (backward) with jobid and both states — log-once per application, no log storms

### Monitoring
- Retained-uncommitted depth (committed-vs-appended lag) — engine counter, consumed in logs here for fact-backlog visibility
- Per-handler barrier failures WARN with opcode and rc; config barrier WARN already present (config_repl.c:924-926), now coupled to the failed reply

## Testing Strategy

### Unit Tests (`testsuite/test_commit_barrier_rollout.c`, XCE_TESTING)
- Classification totality: every `wal_entry_type` value maps to a mode; WAL_WORKER_CTRL detail split (JOIN/UPDATE → WAIT, DEAD → RETAIN); unknown type asserts under XCE_TESTING (2.1, 2.2)
- Barrier helper: returns 0 immediately when `committed_lsn` covers the watermark; `-ETIMEDOUT` with ACKs withheld and `committed_lsn` unmoved; 0 after covering ACK; `n_peers <= 1` immediate success (1.1, 1.2, 1.3)
- Idempotency guards: duplicate FINISH → state unchanged, rc = no-op, no AFTER hook fired (no WAL entry); START on DONE job → rejected, WARN, state preserved; double application of a transition equals single application; fail-after-finish keeps existing PERR behavior (5.1, 5.2, 5.3)
- configAck arithmetic: stale `acked_config_lsn` from a previous tag does not count; zero covering acks with n_peers=3 → timeout; majority with self-count → 0; n_peers=1 → immediate (4.1, 4.3)

### Integration Tests (testsuite/)
- No-success-reply-before-commit: for each restructured opcode (jobcontrol, queuecontrol, hostcontrol, worker join), with ACKs withheld the reply is an error and arrives only at/after the timeout; with ACKs the success reply is observable only after `committed_lsn` covers the operation's entries (1.1, 1.2, 1.4)
- Fact retention across quorum loss: simulate quorum loss → notify-driven transitions still append (RETAIN, no error), gate admits the fact opcodes; restore ACKs → entries commit without re-delivery (3.1-3.4)
- Dispatch receipt-ACK: handle_jobdispatch still replies immediately; no WAL entry is produced by dispatch processing itself (classification assert clean) (2.3)
- Config barrier: upgradeConfig with withheld configAcks → error reply to requester, `committed_config_tag` unmoved, no restart requested; with acks → success, tag advanced (4.1-4.4)
- Reply-site audit (2.4): scripted grep over all `xce_reply` call sites in control paths, each assigned a category; zero unclassified sites

### Regression
- Full controller testsuite + ASAN (`/xce-build controller`) green; existing job-submit barrier tests (upstream) unaffected; `test_repl_majority.c` configAck cases updated to the hardened per-push arithmetic

## Performance & Scalability
- Request-control latency gains one majority RTT (the same cost job submit already pays); fact paths gain no latency (RETAIN returns immediately)
- Concurrent blocked handlers are bounded by the engine's `n_vcpus`-relative admission cap (upstream); excess load degrades to fast NOT_READY rejections — deliberate CP
- Barrier over-wait on concurrent unrelated entries is bounded by `sync_timeout_ms` and only occurs while uncommitted traffic is pending

## Migration Strategy
This spec is phase 2 of the umbrella's hard-ordered landing ("barrier before state removal"):
1. Classification + RETAIN registration + barrier helper (no behavior change for replies yet)
2. Handler restructure sweep + fact gate admission (reply semantics switch on)
3. Config barrier error propagation; legacy walAck counting retirement; idempotency guards
Each step leaves the system no less consistent than before it; rollback trigger per step: controller testsuite regression or job-control E2E failure. ha-single-leader-role must not land until the sweep (step 2) is complete and audited.