myxce/.kiro/specs/ha-commit-barrier-rollout/design.md

# Design: ha-commit-barrier-rollout

> Sub-spec 2 of 4 of the **ha-consistency-invariants** umbrella. This design slices the umbrella design's **Write-Path Classification**, **Config Barrier**, and **Idempotency Guards** components (`.kiro/specs/ha-consistency-invariants/design.md`), implementing umbrella Migration Strategy phase 2. The umbrella's WAL-type classification table and Reply-Ordering Contract are the contract; this design records their call-site verification (which the umbrella explicitly mandated) and the enforcement mechanism. The commit engine interface (`include/commit.h`, `commit_submit` WAIT/RETAIN) is consumed exactly as shipped by ha-commit-engine — this spec never modifies `commit.c` internals. Requirement IDs below are THIS spec's IDs (see `requirements.md` for the umbrella mapping).

## Overview

**Purpose**: This feature extends the majority-commit barrier from job submission to every cluster-state-changing operation: request-shaped operations reply to their requester only after commit (and fail on timeout), fact-shaped operations are appended and retained until committed with no timeout, configuration upgrades fail honestly to the admin tool, and job state transitions become idempotent so retained-fact retry and replay are safe.

**Users**: Cluster operators gain the guarantee that any confirmed control operation survives failover and that worker-reported reality is never silently dropped; downstream sub-specs (ha-single-leader-role, ha-convergence-recovery) gain a write path that is barrier-complete before the transitional leader state is removed.

**Impact**: Adds a classification dispatch at the persistence hook seam (persist.c) with a fail-fast guard for unclassified WAL types; restructures `do_jobcontrol_op` (job.c:921-934), `do_queuecontrol_op` (queue.c:119-132), `do_hostcontrol_op` (worker.c:689-702), and `handle_worker_join` (join.c:33-119) to commit-first-then-reply; admits fact-bearing opcodes through the dispatch gate during quorum loss (ha.c:1527); propagates the config majority-ACK result from `push_config_to_standbys` (config_repl.c:866-930) to the `upgradeConfig` requester (config_repl.c:1057-1094); adds state-machine guards to `mark_job_*` (job.c:583/638/696/726); and retires the legacy walAck counting fallback in `repl_wait_majority_ack` (repl.c:386-462) while hardening its configAck branch.

### Goals
- Commit-first-then-reply for every request-shaped control handler; `handle_jobsubmit` (job.c:495-581) is the reference implementation and stays unchanged
- Fact WAL types routed RETAIN (log-is-the-queue): appended + broadcast, pending purely by `lsn > committed_lsn`, never dropped while running+leading, accepted even during quorum loss
- An unclassified state-changing WAL type cannot silently bypass the barrier (debug assert; production fact-fallback)
- Config upgrade fails to the requester on missed majority and does not restart into the failed config
- `mark_job_*` idempotent and ordered (duplicate = no-op; terminal→non-terminal rejected)

### Non-Goals
- Commit engine internals: per-LSN evaluation, admission cap, `committed_lsn` advancement (ha-commit-engine — consumed as-is)
- Promotion no-op / own-term gate consumers / CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role; the gate's COMMITTING branch keeps rejecting facts until that spec deletes the state — known residual)
- Demotion truncate+restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum-transition logging (ha-convergence-recovery)
- Worker protocol changes; recovery of facts that never reached the controller or died with it (worker-rejoin-resync backlog — umbrella-locked: crash-loss of uncommitted facts accepted until then)
- WAL_NOOP introduction (ha-single-leader-role adds the type; the classification guard here forces it to be classified when it lands)

## Boundary Commitments

### This Spec Owns
- The WAL-type → commit-mode classification table, its dispatch at the persistence hook seam, and the unclassified-type guard
- The reply ordering of every request-shaped control handler on the primary (job control, queue control, worker control, worker join/update) and the receipt-ACK verdict for scheduler-origin handlers
- The commit-barrier helper (`persist_commit_barrier`) that handlers use to realize WAIT-mode completion
- Fact admission through the dispatch gate's no-write-quorum path (opcode allowlist only — not quorum accounting)
- Config-change barrier error propagation: `push_config_to_standbys` return contract, configAck majority arithmetic, and the `upgradeConfig` caller's failure path
- `mark_job_*` transition guards
- Retirement of the legacy (engine-less) walAck counting in `repl_wait_majority_ack`

### Out of Boundary
- `commit.c` / `include/commit.h` — consumed, never modified (interface changes there trigger revalidation here)
- `ha_has_write_quorum` and quorum accounting (ha-commit-engine); the gate's role-based branches other than the PRIMARY no-quorum opcode list
- `committed_lsn` advancement rules, admission cap, membership re-evaluation (commit engine)
- Egress filtering of provisional state to the scheduler (ha-convergence-recovery; dispatch of uncommitted submits is closed there)
- Standby-side apply, WAL wire format, catch-up streaming mechanics (ha-wal-replication)
- Worker/scheduler retry behavior on error replies (client-side, existing semantics)

### Allowed Dependencies
- `commit_submit(ce, lsn, COMMIT_MODE_WAIT|RETAIN, timeout_ms)` and the engine instance embedded in `cluster_t` (ha-commit-engine, P0) — the only commit-evaluation authority
- `persist_wal_async` / `persist_wal_sync_append` append+broadcast primitives and `persist_t.current_lsn` / `committed_lsn` atomics (persist.c, P0)
- `repl_on_config_ack` / `acked_config_lsn` per-standby config tracking (repl.h, P0) — read-only consumption for the config barrier
- Hook dispatch (`xce_run_hooks`, HOOK_AFTER_EVENT seam) — existing pattern, unchanged signatures
- Threading rules per `ha-async-io.md`: commit-barrier waits only on vtask threads; nothing here runs on iomgr
- Dependency direction: handlers (job.c/queue.c/worker.c/join.c) → persist.c barrier/classification → commit engine; persist.c never calls back into handlers

### Revalidation Triggers
- Any change to `commit.h` signatures or return codes (upstream) → re-verify barrier helper and handler error mapping
- ha-single-leader-role adding WAL_NOOP or removing CTRL_PRIMARY_COMMITTING → classification table gains the WAL_NOOP row (RETAIN, umbrella-fixed); gate COMMITTING branch disappears
- Any new state-changing opcode/WAL type → must be added to the classification table (the debug assert makes this unmissable) and its handler swept for reply ordering
- worker-rejoin-resync landing → re-verify `mark_job_*` guards against its replay ordering
- Changes to `acked_config_lsn` semantics or the upgradeConfigAck payload → re-verify configAck majority arithmetic

## Architecture

### Existing Architecture Analysis
- All state-changing writes flow through one seam: entity mutation fires `HOOK_AFTER_EVENT`, and `persist_job_hook_fun` / `persist_queue_hook_fun` / `persist_worker_hook_fun` (persist.c:1093/1172/1204) map hook details to WAL types via `detail_to_wal_type` (persist.c:1068-1088) and append. Only WAL_JOB_SUBMIT uses the sync barrier (persist.c:1107-1129); everything else is `persist_wal_async` fire-and-forget
- Request handlers reply immediately after mutation: `do_jobcontrol_op` (job.c:932), `do_queuecontrol_op` (queue.c:130), `do_hostcontrol_op` (worker.c:700), `handle_worker_join` (join.c:109) — all violate umbrella 2.1 today. `handle_jobsubmit` (job.c:563-578) is the proven correct shape
- Queue/worker/job-state hooks fire while the entity WRITE lock is held (queue.c:76-112, job.c:629-631/687-689); job-submit's hook runs after the job locks are released (job.c:553) — this asymmetry drives WAIT placement (below)
- Facts arrive via `do_notify` (cluster.c:118-157, no reply) and `do_update_op`/`do_set_preason`/`do_clr_preason` (receipt-style replies); the dispatch gate (ha.c:1527) rejects `OPCode_notify` with `-XCE_ERR_NOT_READY` under quorum loss → permanent silent fact loss (bug-396 family, verified pre-existing)
- Call-site verification of the umbrella classification table (mandated by the umbrella): WAL_JOB_STOP/RESUME are appended only from `on_job_stop`/`on_job_resume`, which are invoked exclusively from `do_notify` (worker-reported stop/resume facts). Client job control (`do_jobcontrol_op` → `job_ctrl`, job.c:306-438) either forwards to the worker (running job — no local entry) or mutates locally via `mark_job_update`/`mark_job_fail` (not-started job → WAL_JOB_UPDATE/WAL_JOB_FAIL). WAL_JOB_UPDATE is absent from the umbrella table and is dual-origin (request: job_ctrl; fact: on_job_postdone, rusage update). Resolution: STOP/RESUME/UPDATE/FAIL classify as fact/RETAIN at the seam, and the umbrella's intent for client job control — "error to client" — is realized by the handler-level commit barrier in `do_jobcontrol_op` (the Reply-Ordering Contract's handler sequence is the umbrella-authoritative mechanism). Recorded in research.md
- `push_config_to_standbys` (config_repl.c:866-930) calls `repl_wait_majority_ack(configAck)` at 916-917 but only logs the result and returns the push count; the caller (config_repl.c:1057) ACKs the admin tool with status=0 and restarts regardless. The configAck counting (repl.c:418-438) has the same defects the engine fixed for walAck: `acked_config_lsn > 0` (any previous ack counts), `n_active == 0 → success`, denominator `n_active` instead of `n_peers`

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    subgraph requesters
        CLI[client and cli ops]
        SCHED[scheduler]
        WK[worker]
        ADM[admin tool]
    end
    subgraph controller_primary
        GATE[ha dispatch gate<br/>fact allowlist]
        H[request handlers<br/>job queue worker join]
        F[fact handlers<br/>notify update preason]
        HOOKS[persist hooks<br/>classification dispatch]
        BAR[persist_commit_barrier]
        CB[config barrier<br/>config_repl]
        CE[commit engine<br/>consumed]
        GUARD[mark_job guards]
    end
    CLI --> GATE
    SCHED --> GATE
    WK --> GATE
    ADM --> GATE
    GATE --> H
    GATE --> F
    H --> GUARD
    F --> GUARD
    GUARD --> HOOKS
    HOOKS -->|append plus RETAIN register| CE
    H -->|before success reply| BAR
    BAR -->|commit_submit WAIT| CE
    ADM -.upgradeConfig.-> CB
    CB -->|configAck majority| CE
```

**Architecture Integration** (sliced from umbrella):
- Selected pattern: classification at the single persistence seam + completion at the handler. The seam guarantees no entry type escapes classification (2.1, 2.2); the handler realizes the umbrella Reply-Ordering Contract sequence "validate → mutate (provisional) → WAL append → commit_submit(WAIT) → reply" (1.1)
- WAIT placement decision: WAL_JOB_SUBMIT keeps its in-hook sync barrier (hook runs lock-free, engine-wired upstream). All other request-shaped types complete via `persist_commit_barrier()` called by the handler after entity locks are released — blocking a commit wait under a queue/job/worker WRITE lock would stall readers for up to the commit timeout. Same completion semantics, umbrella-conformant placement (the contract names the handler sequence)
- Existing patterns preserved: hook-based persistence dispatch, provisional in-memory mutation order (umbrella-locked), `errnox` early-return error style, full fan-out broadcast
- Steering compliance: barrier waits only on vtask threads (handlers run on the vtask pool); iomgr untouched; TDD with testsuite/ tests first

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Backend / Services | C (gnu17), pthreads, stdatomic | classification, barrier helper, handler restructure | no new dependencies, no .capnp changes |
| Data / Storage | existing WAL (persist.c), wal_types.h enum | classification keyed on `wal_entry_type` + worker detail | no on-disk format change |
| Build / Test | meson, testsuite/ pattern | new `test_commit_barrier_rollout.c` | XCE_TESTING seams |

## File Structure Plan

### Directory Structure
```
src/controller/
├── persist.c                          # MODIFIED — classification dispatch, RETAIN registration,
│                                      #            persist_commit_barrier, unclassified guard
├── job.c                              # MODIFIED — do_jobcontrol_op reply ordering; mark_job_* guards
├── queue.c                            # MODIFIED — do_queuecontrol_op reply ordering
├── worker.c                           # MODIFIED — do_hostcontrol_op reply ordering
├── join.c                             # MODIFIED — handle_worker_join reply ordering
├── ha.c                               # MODIFIED — dispatch-gate fact allowlist (PRIMARY branch only)
├── config_repl.c                      # MODIFIED — push_config_to_standbys return contract; caller failure path
├── repl.c                             # MODIFIED — legacy walAck counting retired; configAck branch hardened
└── testsuite/
    └── test_commit_barrier_rollout.c  # NEW — classification, barrier ordering, fact retention,
                                       #       config barrier, idempotency guards
src/include/persist.h                  # MODIFIED — wal_commit_mode_t enum, wal_classify(),
                                       #            persist_commit_barrier() declarations
src/include/repl.h                     # MODIFIED — configAck evaluation reuses the entry_lsn
                                       #            parameter (added by ha-commit-engine) as the
                                       #            config push tag; no new parameter
```

### Modified Files (what and why)
- `src/include/persist.h` — declares `wal_commit_mode_t { WAL_MODE_WAIT, WAL_MODE_RETAIN, WAL_MODE_LOCAL }`, `wal_commit_mode_t wal_classify(uint32_t wal_type, int detail)`, `int persist_commit_barrier(persist_t *p, uint32_t timeout_ms)`
- `src/controller/persist.c` — `wal_classify` table; all three hook functions consult it: WAIT → existing sync path (submit) or async append with handler-owed barrier; RETAIN → `persist_wal_async` + `commit_submit(RETAIN)` registration; LOCAL → unchanged (WAL_CHECKPOINT); unclassified → `assert(0)` under XCE_TESTING, ERROR + RETAIN fallback in production. `persist_commit_barrier` implemented here (snapshot `current_lsn`, `commit_submit(WAIT)`)
- `src/controller/job.c` — `do_jobcontrol_op` (job.c:921-934): barrier between `job_ctrl` success and the reply. `mark_job_start/finish/fail/update` (job.c:583/638/696/726): transition guards before mutation
- `src/controller/queue.c` — `do_queuecontrol_op` (queue.c:119-132): barrier between `queue_ctrl` success and the reply
- `src/controller/worker.c` — `do_hostcontrol_op` (worker.c:689-702): barrier between `worker_ctrl` success and the reply
- `src/controller/join.c` — `handle_worker_join` (join.c:33-119): barrier between the AFTER_EVENT hook and the reply at join.c:107-109; join failure (idx < 0) keeps replying immediately
- `src/controller/ha.c` — `ha_dispatch_gate` PRIMARY no-quorum allowlist (ha.c:1537-1556): add `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason`
- `src/controller/config_repl.c` — `push_config_to_standbys` returns the barrier result (0 / `-ETIMEDOUT`), push count via out-param; passes the in-flight tag for per-push evaluation; caller `do_upgrade_config_with_bytes` (config_repl.c:1057): on failure send error reply, skip restart
- `src/controller/repl.c` — walAck branch of `repl_wait_majority_ack`: legacy counting removed (engine is mandatory on this path; `ce == NULL` becomes a programming error); configAck branch: per-push comparison (`acked_config_lsn >= push_tag`), majority over `n_peers` counting self, `n_peers > 1` with zero standby acks never succeeds
- `src/controller/meson.build` — register `testsuite/test_commit_barrier_rollout.c`

## System Flows

### Request-shaped control operation (job/queue/worker control, worker join)

```mermaid
sequenceDiagram
    participant R as requester
    participant V as vtask thread handler
    participant E as entity lock scope
    participant P as persist hooks
    participant CE as commit engine
    R->>V: control request
    V->>V: dispatch gate incl write quorum
    V->>E: lock, validate, mutate provisional
    E->>P: HOOK_AFTER_EVENT append entries
    P->>CE: persist_wal_async + register
    E->>V: unlock, rc
    alt rc != 0
        V->>R: error reply
    else rc == 0
        V->>CE: persist_commit_barrier WAIT
        alt committed
            V->>R: success reply
        else timeout or not ready
            V->>R: error reply
        end
    end
```

Flow decisions: the barrier waits on a snapshot of `current_lsn` taken after the mutation returns — single-writer appends and contiguous commit advancement guarantee the snapshot covers every entry the operation produced (1.1, 1.4); waiting may also cover unrelated concurrent entries (bounded by the commit timeout, harmless — they must commit anyway); `-ETIMEDOUT`/`-XCE_ERR_NOT_READY` map to the error reply via the existing `errnox` style (1.2, 1.3); no in-memory rollback (provisional model, umbrella-locked — outcome converges per umbrella 2.5, owned downstream).

### Fact-shaped delivery (worker notify, rusage update, preason, worker dead)
Gate admits the opcode even without write quorum (3.2) → handler applies the transition through the `mark_job_*` guards (5.1-5.3) → hook classifies the WAL type RETAIN → `persist_wal_async` append + broadcast + `commit_submit(RETAIN)` (returns immediately, no timeout) → handler returns (notify: no reply; update/preason: receipt-style reply unchanged, verdict per 2.3/2.4). The fact is pending purely by `lsn > committed_lsn`; catch-up streaming re-delivers the uncommitted suffix and `committed_lsn` advances when ACKs resume (3.1, 3.3, 3.4). No new queue structure exists (log-is-the-queue, umbrella-locked).

### Configuration upgrade

```mermaid
sequenceDiagram
    participant A as admin tool
    participant U as do_upgrade_config_with_bytes
    participant PS as push_config_to_standbys
    participant S as standbys
    A->>U: OP_upgradeConfig
    U->>U: validate, save to disk, bump generation
    U->>PS: push bytes, in-flight tag
    PS->>S: parallel push threads
    S-->>PS: OP_upgradeConfigAck tag
    PS->>PS: majority of n_peers acked tag, self counted
    alt majority within timeout
        PS-->>U: 0, acked count
        U->>U: committed_config_tag advanced
        U->>A: success reply
        U->>U: request restart
    else timeout
        PS-->>U: -ETIMEDOUT
        U->>A: error reply upgrade failed
        U->>U: no restart, tag not committed
    end
```

Flow decisions: `committed_config_tag` advances only inside the majority branch (4.3 — already correct, preserved); the failure path uses the existing `send_upgrade_error` shape so the admin tool sees a definite failure (4.2); restart is gated on success (4.4 — closes the roadmap residual "don't restart before the commit barrier passes"); config remains a parallel, simpler track sharing only the majority arithmetic rule with the engine (umbrella-locked: one in-flight config change at a time).

## Requirements Traceability

| Requirement | Summary | Components | Flows |
|-------------|---------|------------|-------|
| 1.1 | success reply only after majority commit | Reply-Ordering Sweep, Commit Barrier Helper | request control |
| 1.2 | commit timeout → error reply | Commit Barrier Helper, Reply-Ordering Sweep | request control |
| 1.3 | not-ready rejection → error reply | Commit Barrier Helper (engine code passthrough) | request control |
| 1.4 | no success-interpretable reply pre-commit | Reply-Ordering Sweep (integration-test assertion) | request control |
| 2.1 | every WAL type classified WAIT/RETAIN | Write-Path Classification | both write flows |
| 2.2 | unclassified type fails fast / fact-fallback | Write-Path Classification (guard) | — |
| 2.3 | dispatch ACK is receipt-only | Reply-Ordering Sweep (handle_jobdispatch verdict) | — |
| 2.4 | exhaustive reply-site verification | Reply-Ordering Sweep (audit) | — |
| 3.1 | facts appended, retried, no timeout | Write-Path Classification (RETAIN routing) | fact delivery |
| 3.2 | facts accepted during quorum loss | Fact Gate Admission | fact delivery |
| 3.3 | uncommitted facts never discarded | Write-Path Classification (RETAIN; log-is-the-queue) | fact delivery |
| 3.4 | facts commit after quorum restore, no redelivery | Write-Path Classification + existing catch-up (verified by test) | fact delivery |
| 4.1 | config confirmed only after majority | Config Barrier | config upgrade |
| 4.2 | config timeout → failed to requester | Config Barrier | config upgrade |
| 4.3 | unacked config never committed | Config Barrier (tag advance in success branch only) | config upgrade |
| 4.4 | failed upgrade does not restart | Config Barrier (caller) | config upgrade |
| 5.1 | duplicate transition is logged no-op | Idempotency Guards | fact delivery |
| 5.2 | terminal→non-terminal rejected with WARN | Idempotency Guards | fact delivery |
| 5.3 | replay-once equivalence | Idempotency Guards | fact delivery |

## Components and Interfaces

| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|-------|--------|--------------|------------------|-----------|
| Write-Path Classification | controller/persist.c | route every WAL type to its commit mode at the hook seam | 2.1, 2.2, 3.1, 3.3, 3.4 | commit engine (P0), hooks (P0) | Service |
| Commit Barrier Helper | controller/persist.c | handler-callable WAIT completion on the appended watermark | 1.1, 1.2, 1.3 | commit engine (P0) | Service |
| Reply-Ordering Sweep | controller/job.c, queue.c, worker.c, join.c | commit-first-then-reply in every request handler; receipt-ACK verdicts | 1.1, 1.2, 1.3, 1.4, 2.3, 2.4 | Commit Barrier Helper (P0) | Service |
| Fact Gate Admission | controller/ha.c | admit fact-bearing opcodes without write quorum | 3.2 | dispatch gate (P0) | Service |
| Config Barrier | controller/config_repl.c, repl.c | per-push config majority with error propagation | 4.1, 4.2, 4.3, 4.4 | repl_on_config_ack tracking (P0) | Service |
| Idempotency Guards | controller/job.c | safe duplicate/replay of job transitions | 5.1, 5.2, 5.3 | — | Service |

### Write-Path Classification (`persist.c`, `src/include/persist.h`)

| Field | Detail |
|-------|--------|
| Intent | Single seam where every state-changing WAL append is classified and registered with the engine |
| Requirements | 2.1, 2.2, 3.1, 3.3, 3.4 |

**Classification table** (umbrella table verified against call sites — verification deviations recorded in research.md §2; this verified table is the implementation contract):

| WAL type (+ detail) | Verified origin | Shape | Mode | Failure behavior |
|---|---|---|---|---|
| WAL_JOB_SUBMIT | handle_jobsubmit (client) | request | WAIT (existing in-hook sync path) | error to client (reference impl, unchanged) |
| WAL_QUEUE_CTRL | do_queuecontrol_op (client) | request | WAIT (handler barrier) | error to client |
| WAL_WORKER_CTRL, detail JOIN/UPDATE | handle_worker_join, do_hostcontrol_op | request | WAIT (handler barrier) | error reply to worker/client |
| WAL_WORKER_CTRL, detail DEAD | controller-detected (worker.c:565/578) | fact | RETAIN | none — retained until committed |
| WAL_JOB_START / FINISH / FAIL / STOP / RESUME / UPDATE / RUSAGE / CKPT / PREASON | worker notify, scheduler preason, postdone, rusage; FAIL/UPDATE also from job_ctrl (request origin → covered by do_jobcontrol_op's handler barrier) | fact | RETAIN | none — retained until committed |
| WAL_NOOP (future) | promotion (ha-single-leader-role) | internal | RETAIN | added downstream; guard forces classification |
| WAL_CHECKPOINT | local checkpoint marker | local-only | LOCAL | unchanged (never replicated, by design) |

**Responsibilities & Constraints**
- `wal_classify(wal_type, detail)` is a pure lookup; `detail` disambiguates only WAL_WORKER_CTRL (JOIN/UPDATE vs DEAD)
- Hook dispatch: RETAIN → `persist_wal_async` (append + broadcast, the surviving RETAIN fast path) followed by `commit_submit(ce, appended_lsn, COMMIT_MODE_RETAIN, 0)` registration (returns immediately; observability counter). WAIT → WAL_JOB_SUBMIT keeps `persist_wal_sync_append`; other WAIT types append via `persist_wal_async` and completion is owed by the handler barrier (placement decision, Architecture)
- Unclassified guard: under `XCE_TESTING`, `assert(0)` with the offending type logged; in production, `ERROR` log + RETAIN treatment (never dropped, never falsely confirmed) (2.2)
- Fact appends never return commit errors to their callers — `persist_job_hook_fun`'s non-submit return value stays the local-append rc only (3.1)

**Dependencies**
- Inbound: hook system (HOOK_AFTER_EVENT) — all state-changing writes (P0)
- Outbound: `commit_submit` (P0); `persist_wal_async`/`persist_wal_sync_append` (P0)

**Contracts**: Service [x]

##### Service Interface
```c
/* src/include/persist.h */
typedef enum {
    WAL_MODE_WAIT,    /* request-shaped: barrier before requester reply */
    WAL_MODE_RETAIN,  /* fact-shaped: retained until committed, no timeout */
    WAL_MODE_LOCAL,   /* not replicated (WAL_CHECKPOINT) */
} wal_commit_mode_t;

/* Pure classification lookup. detail disambiguates WAL_WORKER_CTRL only.
 * Unknown type: XCE_TESTING assert; production returns WAL_MODE_RETAIN
 * after ERROR log. */
wal_commit_mode_t wal_classify(uint32_t wal_type, int detail);
```
- Preconditions: called at the hook seam with the type produced by `detail_to_wal_type`
- Postconditions: total over all `wal_entry_type` values; deterministic
- Invariants: no WAL append path bypasses classification

### Commit Barrier Helper (`persist.c`, declared in `src/include/persist.h`)

| Field | Detail |
|-------|--------|
| Intent | Give request handlers a single call realizing commit_submit(WAIT) on everything their mutation appended |
| Requirements | 1.1, 1.2, 1.3 |

**Responsibilities & Constraints**
- Snapshot `lsn = p->current_lsn` (under `p->lock`, matching `persist_wal_sync_append`'s read at persist.c:1019-1021); if `committed_lsn >= lsn` already, return 0 without blocking
- Delegate to `commit_submit(ce, lsn, COMMIT_MODE_WAIT, timeout_ms ? timeout_ms : p->config.sync_timeout_ms)`; return its code unmodified (0 / `-ETIMEDOUT` / `-XCE_ERR_NOT_READY`)
- Single-writer + contiguous advancement ⇒ the snapshot covers every entry the calling operation produced; over-waiting on unrelated concurrent entries is accepted (bounded by the timeout; they must commit anyway). `n_peers <= 1` returns 0 immediately (engine self-quorum, standalone behavior preserved)
- Must be called only from vtask threads after entity locks are released; counts against the engine's admission cap (deliberate CP back-pressure)
- Migration completion: with all request paths on the engine (directly or via this helper), the walAck legacy counting in `repl_wait_majority_ack` is removed (upstream marked it "retired by ha-commit-barrier-rollout"); the configAck branch remains the config track's mechanism

**Contracts**: Service [x]

##### Service Interface
```c
/* src/include/persist.h */
/* Block until committed_lsn covers everything appended so far.
 * timeout_ms = 0 → p->config.sync_timeout_ms.
 * Returns 0 (committed), -ETIMEDOUT (not committed in time — entry
 * remains in log; outcome converges per umbrella 2.5, owned downstream),
 * -XCE_ERR_NOT_READY (admission cap / own-term gate, retriable). */
int persist_commit_barrier(persist_t *p, uint32_t timeout_ms);
```
- Preconditions: commit engine wired (`cluster_init` phase, upstream); vtask thread; no entity locks held
- Postconditions: returns 0 ⇔ `committed_lsn >=` the snapshot watermark
- Invariants: never sends replies, never mutates state

### Reply-Ordering Sweep (`job.c`, `queue.c`, `worker.c`, `join.c`)

| Field | Detail |
|-------|--------|
| Intent | Enforce the umbrella Reply-Ordering Contract at every control-path reply site |
| Requirements | 1.1, 1.2, 1.3, 1.4, 2.3, 2.4 |

**Responsibilities & Constraints**
- Restructure pattern (uniform): `rc = <ctrl fn>(...); if (rc == 0) rc = persist_commit_barrier(&c->persist, 0); reply.status = rc; xce_reply(...)`. Applies to:
  - `do_jobcontrol_op` (job.c:921-934) — covers the not-started branch's WAL_JOB_UPDATE/WAL_JOB_FAIL entries; the running-job branch (forward to worker, no local entry) waits on an already-covered watermark and returns immediately
  - `do_queuecontrol_op` (queue.c:119-132)
  - `do_hostcontrol_op` (worker.c:689-702)
  - `handle_worker_join` (join.c:33-119) — barrier inserted between the AFTER_EVENT hook (join.c:104-105) and the success reply (join.c:107-109); error replies (validate_host, hostconfig, idx < 0) stay immediate; `on_worker_join` dataex push stays after the reply
- Receipt-ACK verdicts (verified, kept, classification-guard-enforced): `handle_jobdispatch` (job.c:459-491) — early reply at job.c:466-471 is ACK-of-receipt; `dispatch_a_job_` produces no WAL entries (allocation results return via `notify_dispatch`, and resulting job-state changes arrive later as worker facts) → per the umbrella contract the early reply stands (2.3). `do_set_preason`/`do_clr_preason` (job.c:1040/1068) — scheduler-origin facts (WAL_JOB_PREASON, RETAIN); early reply is receipt-only. `do_update_op` paths (opupd.c) — worker-origin facts (rusage → WAL_JOB_RUSAGE/WAL_JOB_UPDATE RETAIN; resvals produce no WAL); replies are receipt-only
- Exhaustive audit (2.4): every `xce_reply` call site under src/controller/ (excluding testsuite) is enumerated by grep and assigned one of {post-commit confirmation, receipt-only ACK, read-only/replication/error reply}; the audit table is committed with the task evidence. Known control-path population at design time: job.c:471,524,569,578,932,1050,1078; queue.c:130; worker.c:700; join.c:28,54,66,109,139,152,198,225,248,257,283; opupd.c:48,64,88; cluster.c (gate/shutdown/hello/notify-path replies); config_repl.c:619-1129 (config/replication track); ha.c:1035; opget.c:222 (read-only); reply.c:61; task.c:97 — line numbers re-verified by grep during implementation
- A missed handler silently violates umbrella 2.1 — the integration test asserting no-success-reply-before-commit is the behavioral backstop (1.4)

**Contracts**: Service [x] — external reply semantics: success now implies majority durability; new error codes `-ETIMEDOUT`/`-XCE_ERR_NOT_READY` surface to requesters through existing reply envelopes (Go tier passes them through, per umbrella out-of-boundary)

**Implementation Notes**
- Integration: handlers run on vtask threads (cluster.c dispatch switch at cluster.c:325-363) — barrier blocking is steering-compliant
- Validation: integration test drives each restructured opcode against a 3-node-simulated registry with ACKs withheld → error reply, `committed_lsn` unmoved; with ACKs → success reply strictly after commit
- Risks: scheduler/worker callers seeing new timeout errors where they previously saw instant success — deliberate CP behavior (umbrella); retry semantics unchanged on their side

### Fact Gate Admission (`ha.c`)
Summary-only: add `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` to the CTRL_PRIMARY no-write-quorum allowlist in `ha_dispatch_gate` (ha.c:1537-1556) so reported facts reach the RETAIN path during quorum loss (3.2 — closes the bug-396 silent-loss hole; umbrella error strategy: "fact-shaped ops are still appended and retained"). Request-shaped opcodes (jobsub, jobcontrol, queuecontrol, hostcontrol, jobdispatch, join) remain rejected without quorum. The CTRL_PRIMARY_COMMITTING branch is intentionally untouched (deleted wholesale by ha-single-leader-role — known residual until then). Quorum accounting itself is out of boundary (upstream).

### Config Barrier (`config_repl.c`, `repl.c` configAck branch)

| Field | Detail |
|-------|--------|
| Intent | Make the config majority-ACK result authoritative for the upgrade outcome |
| Requirements | 4.1, 4.2, 4.3, 4.4 |

**Responsibilities & Constraints**
- `push_config_to_standbys` (config_repl.c:866-930) signature: `int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen, const uint8_t sha256[32], uint64_t new_tag, int *acked_out)` — returns the barrier result (0 / `-ETIMEDOUT`), push count via `acked_out`. The majority wait (config_repl.c:916-917) passes `new_tag` down for per-push evaluation; `committed_config_tag` advance stays inside the success branch only (config_repl.c:918-922 — 4.3 preserved)
- configAck branch of `repl_wait_majority_ack` (repl.c:418-438) hardened to mirror the engine's rules on the config track (umbrella: "same majority arithmetic, parallel simpler track"): count standbys with `acked_config_lsn >= push_tag` (not `> 0` — a previous upgrade's ack no longer counts), majority = `1 (self) + count > n_peers / 2` over configured members (not `n_active`), `n_peers <= 1` → immediate success, `n_peers > 1` with zero covering acks → `-ETIMEDOUT` at deadline (the `n_active == 0 → success` short-circuit is removed from this branch). The configAck branch reuses the `entry_lsn` parameter already added to `repl_wait_majority_ack` by ha-commit-engine as the config push tag — no second parameter is added (`repl.h` comment updated accordingly); the function is now configAck-only (walAck callers are on the engine)
- Caller `do_upgrade_config_with_bytes` (config_repl.c:1056-1094): on non-zero return — `send_upgrade_error(ep, req, c, ...)` to the admin tool (4.2), skip the success ACK, skip `restart_requested` (4.4). Disk save and generation bump have already happened (pre-existing sequence); the reply makes the failure explicit and the cluster-wide tag stays uncommitted — convergence/rollback of a half-pushed config remains the existing OP_replConfig anti-entropy behavior (out of boundary)
- One in-flight config change at a time (umbrella-locked) — no per-entry registry needed

**Contracts**: Service [x] — `upgradeConfig` reply contract: success ⇔ majority-acked config; failure carries an error string and implies no restart

**Implementation Notes**
- Integration: `CONF_PUSH_TIMEOUT_MS` remains the config commit timeout
- Validation: integration test withholds configAcks → admin reply is an error, `committed_config_tag` unmoved, no restart request; with acks → success and tag advanced
- Risks: admin tooling that treated any reply as success must observe `status` — existing CLI already surfaces reply status (admin-config-tool backlog item consumes this contract)

### Idempotency Guards (`job.c`)

| Field | Detail |
|-------|--------|
| Intent | Make job state transitions safe under duplicate, out-of-order, and replayed application |
| Requirements | 5.1, 5.2, 5.3 |

**Responsibilities & Constraints**
- Guard placement: at the top of `mark_job_start` (job.c:583), `mark_job_finish` (job.c:638), `mark_job_fail` (job.c:696), `mark_job_update` (job.c:726), under the caller-held job WRITE lock, before the BEFORE_EVENT hook fires (a guarded-out transition runs no hooks → no WAL entry, no double-fire)
- Rules (terminal = any of DONE | EXIT, with or without PERR/PDONE modifiers):
  - duplicate: requested resulting state == current state → INFO log, return 0, no mutation, no hooks (5.1)
  - backward: current state terminal AND requested state non-terminal (START on finished job, UPDATE clearing terminal bits) → WARN log, return `-EALREADY`-style rejection, state preserved (5.2)
  - forward/normal: proceed unchanged; `mark_job_fail`'s existing post-exec special case (job.c:702-716) already tolerates fail-after-finish and is kept as the terminal-state-compatible path
- Replay equivalence (5.3): guards make `restore_job_state`-driven re-application and RETAIN retry converge to single-application state; callers (`on_job_*`, `job_ctrl`) treat the duplicate code as success-no-op
- Shared prerequisite of worker-rejoin-resync (umbrella adjacent expectation) — guards live here, resync consumes them later

**Contracts**: Service [x] — `mark_job_*` return contract gains the logged no-op/rejection outcomes; all existing callers checked during the sweep

## Data Models

### Domain Model (changes only)
- **wal_commit_mode_t** — new enum (persist.h); pure classification metadata, never persisted
- **WAL entries, job_t, queue_t, worker_t** — no format or field changes in this spec (`submit_lsn` tagging is ha-convergence-recovery's egress filter, not here)
- Invariants: every appended state-changing entry is classified; a success reply to a request-shaped operation implies `committed_lsn >=` all entries that operation appended; `committed_config_tag` advances only on per-push config majority; job state never transitions terminal → non-terminal

## Error Handling

### Error Strategy
- Request-shaped commit failure: `-ETIMEDOUT` / `-XCE_ERR_NOT_READY` propagate to the requester reply via the existing `errnox` early-return pattern; no in-memory rollback (provisional model, umbrella-locked; convergence owned downstream)
- Fact-shaped paths never surface commit errors to reporters: local append failure (disk) keeps its existing ERROR logging; commit progress is the engine's concern (RETAIN)
- Unclassified WAL type: XCE_TESTING assert (fail fast in CI); production ERROR + RETAIN fallback (2.2 — fail-safe toward retention, never silent confirmation)
- Config failure: explicit error reply via `send_upgrade_error`; no restart; WARN with tag and acked count
- Guard rejections: INFO (duplicate) / WARN (backward) with jobid and both states — log-once per application, no log storms

### Monitoring
- Retained-uncommitted depth (committed-vs-appended lag) — engine counter, consumed in logs here for fact-backlog visibility
- Per-handler barrier failures WARN with opcode and rc; config barrier WARN already present (config_repl.c:924-926), now coupled to the failed reply

## Testing Strategy

### Unit Tests (`testsuite/test_commit_barrier_rollout.c`, XCE_TESTING)
- Classification totality: every `wal_entry_type` value maps to a mode; WAL_WORKER_CTRL detail split (JOIN/UPDATE → WAIT, DEAD → RETAIN); unknown type asserts under XCE_TESTING (2.1, 2.2)
- Barrier helper: returns 0 immediately when `committed_lsn` covers the watermark; `-ETIMEDOUT` with ACKs withheld and `committed_lsn` unmoved; 0 after covering ACK; `n_peers <= 1` immediate success (1.1, 1.2, 1.3)
- Idempotency guards: duplicate FINISH → state unchanged, rc = no-op, no AFTER hook fired (no WAL entry); START on DONE job → rejected, WARN, state preserved; double application of a transition equals single application; fail-after-finish keeps existing PERR behavior (5.1, 5.2, 5.3)
- configAck arithmetic: stale `acked_config_lsn` from a previous tag does not count; zero covering acks with n_peers=3 → timeout; majority with self-count → 0; n_peers=1 → immediate (4.1, 4.3)

### Integration Tests (testsuite/)
- No-success-reply-before-commit: for each restructured opcode (jobcontrol, queuecontrol, hostcontrol, worker join), with ACKs withheld the reply is an error and arrives only at/after the timeout; with ACKs the success reply is observable only after `committed_lsn` covers the operation's entries (1.1, 1.2, 1.4)
- Fact retention across quorum loss: simulate quorum loss → notify-driven transitions still append (RETAIN, no error), gate admits the fact opcodes; restore ACKs → entries commit without re-delivery (3.1-3.4)
- Dispatch receipt-ACK: handle_jobdispatch still replies immediately; no WAL entry is produced by dispatch processing itself (classification assert clean) (2.3)
- Config barrier: upgradeConfig with withheld configAcks → error reply to requester, `committed_config_tag` unmoved, no restart requested; with acks → success, tag advanced (4.1-4.4)
- Reply-site audit (2.4): scripted grep over all `xce_reply` call sites in control paths, each assigned a category; zero unclassified sites

### Regression
- Full controller testsuite + ASAN (`/xce-build controller`) green; existing job-submit barrier tests (upstream) unaffected; `test_repl_majority.c` configAck cases updated to the hardened per-push arithmetic

## Performance & Scalability
- Request-control latency gains one majority RTT (the same cost job submit already pays); fact paths gain no latency (RETAIN returns immediately)
- Concurrent blocked handlers are bounded by the engine's `n_vcpus`-relative admission cap (upstream); excess load degrades to fast NOT_READY rejections — deliberate CP
- Barrier over-wait on concurrent unrelated entries is bounded by `sync_timeout_ms` and only occurs while uncommitted traffic is pending

## Migration Strategy
This spec is phase 2 of the umbrella's hard-ordered landing ("barrier before state removal"):
1. Classification + RETAIN registration + barrier helper (no behavior change for replies yet)
2. Handler restructure sweep + fact gate admission (reply semantics switch on)
3. Config barrier error propagation; legacy walAck counting retirement; idempotency guards
Each step leaves the system no less consistent than before it; rollback trigger per step: controller testsuite regression or job-control E2E failure. ha-single-leader-role must not land until the sweep (step 2) is complete and audited.