myxce/.kiro/specs/ha-commit-barrier-rollout/design.md

Design: ha-commit-barrier-rollout

Sub-spec 2 of 4 of the ha-consistency-invariants umbrella. This design slices the umbrella design's Write-Path Classification, Config Barrier, and Idempotency Guards components (.kiro/specs/ha-consistency-invariants/design.md), implementing umbrella Migration Strategy phase 2. The umbrella's WAL-type classification table and Reply-Ordering Contract are the contract; this design records their call-site verification (which the umbrella explicitly mandated) and the enforcement mechanism. The commit engine interface (include/commit.h, commit_submit WAIT/RETAIN) is consumed exactly as shipped by ha-commit-engine — this spec never modifies commit.c internals. Requirement IDs below are THIS spec's IDs (see requirements.md for the umbrella mapping).

Overview

Purpose: This feature extends the majority-commit barrier from job submission to every cluster-state-changing operation: request-shaped operations reply to their requester only after commit (and fail on timeout), fact-shaped operations are appended and retained until committed with no timeout, configuration upgrades fail honestly to the admin tool, and job state transitions become idempotent so retained-fact retry and replay are safe.

Users: Cluster operators gain the guarantee that any confirmed control operation survives failover and that worker-reported reality is never silently dropped; downstream sub-specs (ha-single-leader-role, ha-convergence-recovery) gain a write path that is barrier-complete before the transitional leader state is removed.

Impact: Adds a classification dispatch at the persistence hook seam (persist.c) with a fail-fast guard for unclassified WAL types; restructures do_jobcontrol_op (job.c:921-934), do_queuecontrol_op (queue.c:119-132), do_hostcontrol_op (worker.c:689-702), and handle_worker_join (join.c:33-119) to commit-first-then-reply; admits fact-bearing opcodes through the dispatch gate during quorum loss (ha.c:1527); propagates the config majority-ACK result from push_config_to_standbys (config_repl.c:866-930) to the upgradeConfig requester (config_repl.c:1057-1094); adds state-machine guards to mark_job_* (job.c:583/638/696/726); and retires the legacy walAck counting fallback in repl_wait_majority_ack (repl.c:386-462) while hardening its configAck branch.

Goals

• Commit-first-then-reply for every request-shaped control handler; handle_jobsubmit (job.c:495-581) is the reference implementation and stays unchanged
• Fact WAL types routed RETAIN (log-is-the-queue): appended + broadcast, pending purely by lsn > committed_lsn, never dropped while running+leading, accepted even during quorum loss
• An unclassified state-changing WAL type cannot silently bypass the barrier (debug assert; production fact-fallback)
• Config upgrade fails to the requester on missed majority and does not restart into the failed config
• mark_job_* idempotent and ordered (duplicate = no-op; terminal→non-terminal rejected)

Non-Goals

• Commit engine internals: per-LSN evaluation, admission cap, committed_lsn advancement (ha-commit-engine — consumed as-is)
• Promotion no-op / own-term gate consumers / CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role; the gate's COMMITTING branch keeps rejecting facts until that spec deletes the state — known residual)
• Demotion truncate+restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum-transition logging (ha-convergence-recovery)
• Worker protocol changes; recovery of facts that never reached the controller or died with it (worker-rejoin-resync backlog — umbrella-locked: crash-loss of uncommitted facts accepted until then)
• WAL_NOOP introduction (ha-single-leader-role adds the type; the classification guard here forces it to be classified when it lands)

Boundary Commitments

This Spec Owns

• The WAL-type → commit-mode classification table, its dispatch at the persistence hook seam, and the unclassified-type guard
• The reply ordering of every request-shaped control handler on the primary (job control, queue control, worker control, worker join/update) and the receipt-ACK verdict for scheduler-origin handlers
• The commit-barrier helper (persist_commit_barrier) that handlers use to realize WAIT-mode completion
• Fact admission through the dispatch gate's no-write-quorum path (opcode allowlist only — not quorum accounting)
• Config-change barrier error propagation: push_config_to_standbys return contract, configAck majority arithmetic, and the upgradeConfig caller's failure path
• mark_job_* transition guards
• Retirement of the legacy (engine-less) walAck counting in repl_wait_majority_ack

Out of Boundary

• commit.c / include/commit.h — consumed, never modified (interface changes there trigger revalidation here)
• ha_has_write_quorum and quorum accounting (ha-commit-engine); the gate's role-based branches other than the PRIMARY no-quorum opcode list
• committed_lsn advancement rules, admission cap, membership re-evaluation (commit engine)
• Egress filtering of provisional state to the scheduler (ha-convergence-recovery; dispatch of uncommitted submits is closed there)
• Standby-side apply, WAL wire format, catch-up streaming mechanics (ha-wal-replication)
• Worker/scheduler retry behavior on error replies (client-side, existing semantics)

Allowed Dependencies

• commit_submit(ce, lsn, COMMIT_MODE_WAIT|RETAIN, timeout_ms) and the engine instance embedded in cluster_t (ha-commit-engine, P0) — the only commit-evaluation authority
• persist_wal_async / persist_wal_sync_append append+broadcast primitives and persist_t.current_lsn / committed_lsn atomics (persist.c, P0)
• repl_on_config_ack / acked_config_lsn per-standby config tracking (repl.h, P0) — read-only consumption for the config barrier
• Hook dispatch (xce_run_hooks, HOOK_AFTER_EVENT seam) — existing pattern, unchanged signatures
• Threading rules per ha-async-io.md: commit-barrier waits only on vtask threads; nothing here runs on iomgr
• Dependency direction: handlers (job.c/queue.c/worker.c/join.c) → persist.c barrier/classification → commit engine; persist.c never calls back into handlers

Revalidation Triggers

• Any change to commit.h signatures or return codes (upstream) → re-verify barrier helper and handler error mapping
• ha-single-leader-role adding WAL_NOOP or removing CTRL_PRIMARY_COMMITTING → classification table gains the WAL_NOOP row (RETAIN, umbrella-fixed); gate COMMITTING branch disappears
• Any new state-changing opcode/WAL type → must be added to the classification table (the debug assert makes this unmissable) and its handler swept for reply ordering
• worker-rejoin-resync landing → re-verify mark_job_* guards against its replay ordering
• Changes to acked_config_lsn semantics or the upgradeConfigAck payload → re-verify configAck majority arithmetic

Architecture

Existing Architecture Analysis

• All state-changing writes flow through one seam: entity mutation fires HOOK_AFTER_EVENT, and persist_job_hook_fun / persist_queue_hook_fun / persist_worker_hook_fun (persist.c:1093/1172/1204) map hook details to WAL types via detail_to_wal_type (persist.c:1068-1088) and append. Only WAL_JOB_SUBMIT uses the sync barrier (persist.c:1107-1129); everything else is persist_wal_async fire-and-forget
• Request handlers reply immediately after mutation: do_jobcontrol_op (job.c:932), do_queuecontrol_op (queue.c:130), do_hostcontrol_op (worker.c:700), handle_worker_join (join.c:109) — all violate umbrella 2.1 today. handle_jobsubmit (job.c:563-578) is the proven correct shape
• Queue/worker/job-state hooks fire while the entity WRITE lock is held (queue.c:76-112, job.c:629-631/687-689); job-submit's hook runs after the job locks are released (job.c:553) — this asymmetry drives WAIT placement (below)
• Facts arrive via do_notify (cluster.c:118-157, no reply) and do_update_op/do_set_preason/do_clr_preason (receipt-style replies); the dispatch gate (ha.c:1527) rejects OPCode_notify with -XCE_ERR_NOT_READY under quorum loss → permanent silent fact loss (bug-396 family, verified pre-existing)
• Call-site verification of the umbrella classification table (mandated by the umbrella): WAL_JOB_STOP/RESUME are appended only from on_job_stop/on_job_resume, which are invoked exclusively from do_notify (worker-reported stop/resume facts). Client job control (do_jobcontrol_op → job_ctrl, job.c:306-438) either forwards to the worker (running job — no local entry) or mutates locally via mark_job_update/mark_job_fail (not-started job → WAL_JOB_UPDATE/WAL_JOB_FAIL). WAL_JOB_UPDATE is absent from the umbrella table and is dual-origin (request: job_ctrl; fact: on_job_postdone, rusage update). Resolution: STOP/RESUME/UPDATE/FAIL classify as fact/RETAIN at the seam, and the umbrella's intent for client job control — "error to client" — is realized by the handler-level commit barrier in do_jobcontrol_op (the Reply-Ordering Contract's handler sequence is the umbrella-authoritative mechanism). Recorded in research.md
• push_config_to_standbys (config_repl.c:866-930) calls repl_wait_majority_ack(configAck) at 916-917 but only logs the result and returns the push count; the caller (config_repl.c:1057) ACKs the admin tool with status=0 and restarts regardless. The configAck counting (repl.c:418-438) has the same defects the engine fixed for walAck: acked_config_lsn > 0 (any previous ack counts), n_active == 0 → success, denominator n_active instead of n_peers

Architecture Pattern & Boundary Map

graph TB
    subgraph requesters
        CLI[client and cli ops]
        SCHED[scheduler]
        WK[worker]
        ADM[admin tool]
    end
    subgraph controller_primary
        GATE[ha dispatch gate<br/>fact allowlist]
        H[request handlers<br/>job queue worker join]
        F[fact handlers<br/>notify update preason]
        HOOKS[persist hooks<br/>classification dispatch]
        BAR[persist_commit_barrier]
        CB[config barrier<br/>config_repl]
        CE[commit engine<br/>consumed]
        GUARD[mark_job guards]
    end
    CLI --> GATE
    SCHED --> GATE
    WK --> GATE
    ADM --> GATE
    GATE --> H
    GATE --> F
    H --> GUARD
    F --> GUARD
    GUARD --> HOOKS
    HOOKS -->|append plus RETAIN register| CE
    H -->|before success reply| BAR
    BAR -->|commit_submit WAIT| CE
    ADM -.upgradeConfig.-> CB
    CB -->|configAck majority| CE

Architecture Integration (sliced from umbrella):

• Selected pattern: classification at the single persistence seam + completion at the handler. The seam guarantees no entry type escapes classification (2.1, 2.2); the handler realizes the umbrella Reply-Ordering Contract sequence "validate → mutate (provisional) → WAL append → commit_submit(WAIT) → reply" (1.1)
• WAIT placement decision: WAL_JOB_SUBMIT keeps its in-hook sync barrier (hook runs lock-free, engine-wired upstream). All other request-shaped types complete via persist_commit_barrier() called by the handler after entity locks are released — blocking a commit wait under a queue/job/worker WRITE lock would stall readers for up to the commit timeout. Same completion semantics, umbrella-conformant placement (the contract names the handler sequence)
• Existing patterns preserved: hook-based persistence dispatch, provisional in-memory mutation order (umbrella-locked), errnox early-return error style, full fan-out broadcast
• Steering compliance: barrier waits only on vtask threads (handlers run on the vtask pool); iomgr untouched; TDD with testsuite/ tests first

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Backend / Services	C (gnu17), pthreads, stdatomic	classification, barrier helper, handler restructure	no new dependencies, no .capnp changes
Data / Storage	existing WAL (persist.c), wal_types.h enum	classification keyed on wal_entry_type + worker detail	no on-disk format change
Build / Test	meson, testsuite/ pattern	new test_commit_barrier_rollout.c	XCE_TESTING seams

File Structure Plan

Directory Structure

src/controller/
├── persist.c                          # MODIFIED — classification dispatch, RETAIN registration,
│                                      #            persist_commit_barrier, unclassified guard
├── job.c                              # MODIFIED — do_jobcontrol_op reply ordering; mark_job_* guards
├── queue.c                            # MODIFIED — do_queuecontrol_op reply ordering
├── worker.c                           # MODIFIED — do_hostcontrol_op reply ordering
├── join.c                             # MODIFIED — handle_worker_join reply ordering
├── ha.c                               # MODIFIED — dispatch-gate fact allowlist (PRIMARY branch only)
├── config_repl.c                      # MODIFIED — push_config_to_standbys return contract; caller failure path
├── repl.c                             # MODIFIED — legacy walAck counting retired; configAck branch hardened
└── testsuite/
    └── test_commit_barrier_rollout.c  # NEW — classification, barrier ordering, fact retention,
                                       #       config barrier, idempotency guards
src/include/persist.h                  # MODIFIED — wal_commit_mode_t enum, wal_classify(),
                                       #            persist_commit_barrier() declarations
src/include/repl.h                     # MODIFIED — configAck evaluation reuses the entry_lsn
                                       #            parameter (added by ha-commit-engine) as the
                                       #            config push tag; no new parameter

Modified Files (what and why)

• src/include/persist.h — declares wal_commit_mode_t { WAL_MODE_WAIT, WAL_MODE_RETAIN, WAL_MODE_LOCAL }, wal_commit_mode_t wal_classify(uint32_t wal_type, int detail), int persist_commit_barrier(persist_t *p, uint32_t timeout_ms)
• src/controller/persist.c — wal_classify table; all three hook functions consult it: WAIT → existing sync path (submit) or async append with handler-owed barrier; RETAIN → persist_wal_async + commit_submit(RETAIN) registration; LOCAL → unchanged (WAL_CHECKPOINT); unclassified → assert(0) under XCE_TESTING, ERROR + RETAIN fallback in production. persist_commit_barrier implemented here (snapshot current_lsn, commit_submit(WAIT))
• src/controller/job.c — do_jobcontrol_op (job.c:921-934): barrier between job_ctrl success and the reply. mark_job_start/finish/fail/update (job.c:583/638/696/726): transition guards before mutation
• src/controller/queue.c — do_queuecontrol_op (queue.c:119-132): barrier between queue_ctrl success and the reply
• src/controller/worker.c — do_hostcontrol_op (worker.c:689-702): barrier between worker_ctrl success and the reply
• src/controller/join.c — handle_worker_join (join.c:33-119): barrier between the AFTER_EVENT hook and the reply at join.c:107-109; join failure (idx < 0) keeps replying immediately
• src/controller/ha.c — ha_dispatch_gate PRIMARY no-quorum allowlist (ha.c:1537-1556): add OPCode_notify, OPCode_update, OPCode_setpreason, OPCode_clrpreason
• src/controller/config_repl.c — push_config_to_standbys returns the barrier result (0 / -ETIMEDOUT), push count via out-param; passes the in-flight tag for per-push evaluation; caller do_upgrade_config_with_bytes (config_repl.c:1057): on failure send error reply, skip restart
• src/controller/repl.c — walAck branch of repl_wait_majority_ack: legacy counting removed (engine is mandatory on this path; ce == NULL becomes a programming error); configAck branch: per-push comparison (acked_config_lsn >= push_tag), majority over n_peers counting self, n_peers > 1 with zero standby acks never succeeds
• src/controller/meson.build — register testsuite/test_commit_barrier_rollout.c

System Flows

Request-shaped control operation (job/queue/worker control, worker join)

sequenceDiagram
    participant R as requester
    participant V as vtask thread handler
    participant E as entity lock scope
    participant P as persist hooks
    participant CE as commit engine
    R->>V: control request
    V->>V: dispatch gate incl write quorum
    V->>E: lock, validate, mutate provisional
    E->>P: HOOK_AFTER_EVENT append entries
    P->>CE: persist_wal_async + register
    E->>V: unlock, rc
    alt rc != 0
        V->>R: error reply
    else rc == 0
        V->>CE: persist_commit_barrier WAIT
        alt committed
            V->>R: success reply
        else timeout or not ready
            V->>R: error reply
        end
    end

Flow decisions: the barrier waits on a snapshot of current_lsn taken after the mutation returns — single-writer appends and contiguous commit advancement guarantee the snapshot covers every entry the operation produced (1.1, 1.4); waiting may also cover unrelated concurrent entries (bounded by the commit timeout, harmless — they must commit anyway); -ETIMEDOUT/-XCE_ERR_NOT_READY map to the error reply via the existing errnox style (1.2, 1.3); no in-memory rollback (provisional model, umbrella-locked — outcome converges per umbrella 2.5, owned downstream).

Fact-shaped delivery (worker notify, rusage update, preason, worker dead)

Gate admits the opcode even without write quorum (3.2) → handler applies the transition through the mark_job_* guards (5.1-5.3) → hook classifies the WAL type RETAIN → persist_wal_async append + broadcast + commit_submit(RETAIN) (returns immediately, no timeout) → handler returns (notify: no reply; update/preason: receipt-style reply unchanged, verdict per 2.3/2.4). The fact is pending purely by lsn > committed_lsn; catch-up streaming re-delivers the uncommitted suffix and committed_lsn advances when ACKs resume (3.1, 3.3, 3.4). No new queue structure exists (log-is-the-queue, umbrella-locked).

Configuration upgrade

sequenceDiagram
    participant A as admin tool
    participant U as do_upgrade_config_with_bytes
    participant PS as push_config_to_standbys
    participant S as standbys
    A->>U: OP_upgradeConfig
    U->>U: validate, save to disk, bump generation
    U->>PS: push bytes, in-flight tag
    PS->>S: parallel push threads
    S-->>PS: OP_upgradeConfigAck tag
    PS->>PS: majority of n_peers acked tag, self counted
    alt majority within timeout
        PS-->>U: 0, acked count
        U->>U: committed_config_tag advanced
        U->>A: success reply
        U->>U: request restart
    else timeout
        PS-->>U: -ETIMEDOUT
        U->>A: error reply upgrade failed
        U->>U: no restart, tag not committed
    end

Flow decisions: committed_config_tag advances only inside the majority branch (4.3 — already correct, preserved); the failure path uses the existing send_upgrade_error shape so the admin tool sees a definite failure (4.2); restart is gated on success (4.4 — closes the roadmap residual "don't restart before the commit barrier passes"); config remains a parallel, simpler track sharing only the majority arithmetic rule with the engine (umbrella-locked: one in-flight config change at a time).

Requirements Traceability

Requirement	Summary	Components	Flows
1.1	success reply only after majority commit	Reply-Ordering Sweep, Commit Barrier Helper	request control
1.2	commit timeout → error reply	Commit Barrier Helper, Reply-Ordering Sweep	request control
1.3	not-ready rejection → error reply	Commit Barrier Helper (engine code passthrough)	request control
1.4	no success-interpretable reply pre-commit	Reply-Ordering Sweep (integration-test assertion)	request control
2.1	every WAL type classified WAIT/RETAIN	Write-Path Classification	both write flows
2.2	unclassified type fails fast / fact-fallback	Write-Path Classification (guard)	—
2.3	dispatch ACK is receipt-only	Reply-Ordering Sweep (handle_jobdispatch verdict)	—
2.4	exhaustive reply-site verification	Reply-Ordering Sweep (audit)	—
3.1	facts appended, retried, no timeout	Write-Path Classification (RETAIN routing)	fact delivery
3.2	facts accepted during quorum loss	Fact Gate Admission	fact delivery
3.3	uncommitted facts never discarded	Write-Path Classification (RETAIN; log-is-the-queue)	fact delivery
3.4	facts commit after quorum restore, no redelivery	Write-Path Classification + existing catch-up (verified by test)	fact delivery
4.1	config confirmed only after majority	Config Barrier	config upgrade
4.2	config timeout → failed to requester	Config Barrier	config upgrade
4.3	unacked config never committed	Config Barrier (tag advance in success branch only)	config upgrade
4.4	failed upgrade does not restart	Config Barrier (caller)	config upgrade
5.1	duplicate transition is logged no-op	Idempotency Guards	fact delivery
5.2	terminal→non-terminal rejected with WARN	Idempotency Guards	fact delivery
5.3	replay-once equivalence	Idempotency Guards	fact delivery

Components and Interfaces

Component	Layer	Intent	Req Coverage	Key Dependencies	Contracts
Write-Path Classification	controller/persist.c	route every WAL type to its commit mode at the hook seam	2.1, 2.2, 3.1, 3.3, 3.4	commit engine (P0), hooks (P0)	Service
Commit Barrier Helper	controller/persist.c	handler-callable WAIT completion on the appended watermark	1.1, 1.2, 1.3	commit engine (P0)	Service
Reply-Ordering Sweep	controller/job.c, queue.c, worker.c, join.c	commit-first-then-reply in every request handler; receipt-ACK verdicts	1.1, 1.2, 1.3, 1.4, 2.3, 2.4	Commit Barrier Helper (P0)	Service
Fact Gate Admission	controller/ha.c	admit fact-bearing opcodes without write quorum	3.2	dispatch gate (P0)	Service
Config Barrier	controller/config_repl.c, repl.c	per-push config majority with error propagation	4.1, 4.2, 4.3, 4.4	repl_on_config_ack tracking (P0)	Service
Idempotency Guards	controller/job.c	safe duplicate/replay of job transitions	5.1, 5.2, 5.3	—	Service

Write-Path Classification (persist.c, src/include/persist.h)

Field	Detail
Intent	Single seam where every state-changing WAL append is classified and registered with the engine
Requirements	2.1, 2.2, 3.1, 3.3, 3.4

Classification table (umbrella table verified against call sites — verification deviations recorded in research.md §2; this verified table is the implementation contract):

WAL type (+ detail)	Verified origin	Shape	Mode	Failure behavior
WAL_JOB_SUBMIT	handle_jobsubmit (client)	request	WAIT (existing in-hook sync path)	error to client (reference impl, unchanged)
WAL_QUEUE_CTRL	do_queuecontrol_op (client)	request	WAIT (handler barrier)	error to client
WAL_WORKER_CTRL, detail JOIN/UPDATE	handle_worker_join, do_hostcontrol_op	request	WAIT (handler barrier)	error reply to worker/client
WAL_WORKER_CTRL, detail DEAD	controller-detected (worker.c:565/578)	fact	RETAIN	none — retained until committed
WAL_JOB_START / FINISH / FAIL / STOP / RESUME / UPDATE / RUSAGE / CKPT / PREASON	worker notify, scheduler preason, postdone, rusage; FAIL/UPDATE also from job_ctrl (request origin → covered by do_jobcontrol_op's handler barrier)	fact	RETAIN	none — retained until committed
WAL_NOOP (future)	promotion (ha-single-leader-role)	internal	RETAIN	added downstream; guard forces classification
WAL_CHECKPOINT	local checkpoint marker	local-only	LOCAL	unchanged (never replicated, by design)

Responsibilities & Constraints

• wal_classify(wal_type, detail) is a pure lookup; detail disambiguates only WAL_WORKER_CTRL (JOIN/UPDATE vs DEAD)
• Hook dispatch: RETAIN → persist_wal_async (append + broadcast, the surviving RETAIN fast path) followed by commit_submit(ce, appended_lsn, COMMIT_MODE_RETAIN, 0) registration (returns immediately; observability counter). WAIT → WAL_JOB_SUBMIT keeps persist_wal_sync_append; other WAIT types append via persist_wal_async and completion is owed by the handler barrier (placement decision, Architecture)
• Unclassified guard: under XCE_TESTING, assert(0) with the offending type logged; in production, ERROR log + RETAIN treatment (never dropped, never falsely confirmed) (2.2)
• Fact appends never return commit errors to their callers — persist_job_hook_fun's non-submit return value stays the local-append rc only (3.1)

Dependencies

• Inbound: hook system (HOOK_AFTER_EVENT) — all state-changing writes (P0)
• Outbound: commit_submit (P0); persist_wal_async/persist_wal_sync_append (P0)

Contracts: Service [x]

Service Interface

/* src/include/persist.h */
typedef enum {
    WAL_MODE_WAIT,    /* request-shaped: barrier before requester reply */
    WAL_MODE_RETAIN,  /* fact-shaped: retained until committed, no timeout */
    WAL_MODE_LOCAL,   /* not replicated (WAL_CHECKPOINT) */
} wal_commit_mode_t;

/* Pure classification lookup. detail disambiguates WAL_WORKER_CTRL only.
 * Unknown type: XCE_TESTING assert; production returns WAL_MODE_RETAIN
 * after ERROR log. */
wal_commit_mode_t wal_classify(uint32_t wal_type, int detail);

• Preconditions: called at the hook seam with the type produced by detail_to_wal_type
• Postconditions: total over all wal_entry_type values; deterministic
• Invariants: no WAL append path bypasses classification

Commit Barrier Helper (persist.c, declared in src/include/persist.h)

Field	Detail
Intent	Give request handlers a single call realizing commit_submit(WAIT) on everything their mutation appended
Requirements	1.1, 1.2, 1.3

Responsibilities & Constraints

• Snapshot lsn = p->current_lsn (under p->lock, matching persist_wal_sync_append's read at persist.c:1019-1021); if committed_lsn >= lsn already, return 0 without blocking
• Delegate to commit_submit(ce, lsn, COMMIT_MODE_WAIT, timeout_ms ? timeout_ms : p->config.sync_timeout_ms); return its code unmodified (0 / -ETIMEDOUT / -XCE_ERR_NOT_READY)
• Single-writer + contiguous advancement ⇒ the snapshot covers every entry the calling operation produced; over-waiting on unrelated concurrent entries is accepted (bounded by the timeout; they must commit anyway). n_peers <= 1 returns 0 immediately (engine self-quorum, standalone behavior preserved)
• Must be called only from vtask threads after entity locks are released; counts against the engine's admission cap (deliberate CP back-pressure)
• Migration completion: with all request paths on the engine (directly or via this helper), the walAck legacy counting in repl_wait_majority_ack is removed (upstream marked it "retired by ha-commit-barrier-rollout"); the configAck branch remains the config track's mechanism

Contracts: Service [x]

Service Interface

/* src/include/persist.h */
/* Block until committed_lsn covers everything appended so far.
 * timeout_ms = 0 → p->config.sync_timeout_ms.
 * Returns 0 (committed), -ETIMEDOUT (not committed in time — entry
 * remains in log; outcome converges per umbrella 2.5, owned downstream),
 * -XCE_ERR_NOT_READY (admission cap / own-term gate, retriable). */
int persist_commit_barrier(persist_t *p, uint32_t timeout_ms);

• Preconditions: commit engine wired (cluster_init phase, upstream); vtask thread; no entity locks held
• Postconditions: returns 0 ⇔ committed_lsn >= the snapshot watermark
• Invariants: never sends replies, never mutates state

Reply-Ordering Sweep (job.c, queue.c, worker.c, join.c)

Field	Detail
Intent	Enforce the umbrella Reply-Ordering Contract at every control-path reply site
Requirements	1.1, 1.2, 1.3, 1.4, 2.3, 2.4

Responsibilities & Constraints

• Restructure pattern (uniform): rc = <ctrl fn>(...); if (rc == 0) rc = persist_commit_barrier(&c->persist, 0); reply.status = rc; xce_reply(...). Applies to:
  • do_jobcontrol_op (job.c:921-934) — covers the not-started branch's WAL_JOB_UPDATE/WAL_JOB_FAIL entries; the running-job branch (forward to worker, no local entry) waits on an already-covered watermark and returns immediately
  • do_queuecontrol_op (queue.c:119-132)
  • do_hostcontrol_op (worker.c:689-702)
  • handle_worker_join (join.c:33-119) — barrier inserted between the AFTER_EVENT hook (join.c:104-105) and the success reply (join.c:107-109); error replies (validate_host, hostconfig, idx < 0) stay immediate; on_worker_join dataex push stays after the reply
• Receipt-ACK verdicts (verified, kept, classification-guard-enforced): handle_jobdispatch (job.c:459-491) — early reply at job.c:466-471 is ACK-of-receipt; dispatch_a_job_ produces no WAL entries (allocation results return via notify_dispatch, and resulting job-state changes arrive later as worker facts) → per the umbrella contract the early reply stands (2.3). do_set_preason/do_clr_preason (job.c:1040/1068) — scheduler-origin facts (WAL_JOB_PREASON, RETAIN); early reply is receipt-only. do_update_op paths (opupd.c) — worker-origin facts (rusage → WAL_JOB_RUSAGE/WAL_JOB_UPDATE RETAIN; resvals produce no WAL); replies are receipt-only
• Exhaustive audit (2.4): every xce_reply call site under src/controller/ (excluding testsuite) is enumerated by grep and assigned one of {post-commit confirmation, receipt-only ACK, read-only/replication/error reply}; the audit table is committed with the task evidence. Known control-path population at design time: job.c:471,524,569,578,932,1050,1078; queue.c:130; worker.c:700; join.c:28,54,66,109,139,152,198,225,248,257,283; opupd.c:48,64,88; cluster.c (gate/shutdown/hello/notify-path replies); config_repl.c:619-1129 (config/replication track); ha.c:1035; opget.c:222 (read-only); reply.c:61; task.c:97 — line numbers re-verified by grep during implementation
• A missed handler silently violates umbrella 2.1 — the integration test asserting no-success-reply-before-commit is the behavioral backstop (1.4)

Contracts: Service [x] — external reply semantics: success now implies majority durability; new error codes -ETIMEDOUT/-XCE_ERR_NOT_READY surface to requesters through existing reply envelopes (Go tier passes them through, per umbrella out-of-boundary)

Implementation Notes

• Integration: handlers run on vtask threads (cluster.c dispatch switch at cluster.c:325-363) — barrier blocking is steering-compliant
• Validation: integration test drives each restructured opcode against a 3-node-simulated registry with ACKs withheld → error reply, committed_lsn unmoved; with ACKs → success reply strictly after commit
• Risks: scheduler/worker callers seeing new timeout errors where they previously saw instant success — deliberate CP behavior (umbrella); retry semantics unchanged on their side

Fact Gate Admission (ha.c)

Summary-only: add OPCode_notify, OPCode_update, OPCode_setpreason, OPCode_clrpreason to the CTRL_PRIMARY no-write-quorum allowlist in ha_dispatch_gate (ha.c:1537-1556) so reported facts reach the RETAIN path during quorum loss (3.2 — closes the bug-396 silent-loss hole; umbrella error strategy: "fact-shaped ops are still appended and retained"). Request-shaped opcodes (jobsub, jobcontrol, queuecontrol, hostcontrol, jobdispatch, join) remain rejected without quorum. The CTRL_PRIMARY_COMMITTING branch is intentionally untouched (deleted wholesale by ha-single-leader-role — known residual until then). Quorum accounting itself is out of boundary (upstream).

Config Barrier (config_repl.c, repl.c configAck branch)

Field	Detail
Intent	Make the config majority-ACK result authoritative for the upgrade outcome
Requirements	4.1, 4.2, 4.3, 4.4

Responsibilities & Constraints

• push_config_to_standbys (config_repl.c:866-930) signature: int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen, const uint8_t sha256[32], uint64_t new_tag, int *acked_out) — returns the barrier result (0 / -ETIMEDOUT), push count via acked_out. The majority wait (config_repl.c:916-917) passes new_tag down for per-push evaluation; committed_config_tag advance stays inside the success branch only (config_repl.c:918-922 — 4.3 preserved)
• configAck branch of repl_wait_majority_ack (repl.c:418-438) hardened to mirror the engine's rules on the config track (umbrella: "same majority arithmetic, parallel simpler track"): count standbys with acked_config_lsn >= push_tag (not > 0 — a previous upgrade's ack no longer counts), majority = 1 (self) + count > n_peers / 2 over configured members (not n_active), n_peers <= 1 → immediate success, n_peers > 1 with zero covering acks → -ETIMEDOUT at deadline (the n_active == 0 → success short-circuit is removed from this branch). The configAck branch reuses the entry_lsn parameter already added to repl_wait_majority_ack by ha-commit-engine as the config push tag — no second parameter is added (repl.h comment updated accordingly); the function is now configAck-only (walAck callers are on the engine)
• Caller do_upgrade_config_with_bytes (config_repl.c:1056-1094): on non-zero return — send_upgrade_error(ep, req, c, ...) to the admin tool (4.2), skip the success ACK, skip restart_requested (4.4). Disk save and generation bump have already happened (pre-existing sequence); the reply makes the failure explicit and the cluster-wide tag stays uncommitted — convergence/rollback of a half-pushed config remains the existing OP_replConfig anti-entropy behavior (out of boundary)
• One in-flight config change at a time (umbrella-locked) — no per-entry registry needed

Contracts: Service [x] — upgradeConfig reply contract: success ⇔ majority-acked config; failure carries an error string and implies no restart

Implementation Notes

• Integration: CONF_PUSH_TIMEOUT_MS remains the config commit timeout
• Validation: integration test withholds configAcks → admin reply is an error, committed_config_tag unmoved, no restart request; with acks → success and tag advanced
• Risks: admin tooling that treated any reply as success must observe status — existing CLI already surfaces reply status (admin-config-tool backlog item consumes this contract)

Idempotency Guards (job.c)

Field	Detail
Intent	Make job state transitions safe under duplicate, out-of-order, and replayed application
Requirements	5.1, 5.2, 5.3

Responsibilities & Constraints

• Guard placement: at the top of mark_job_start (job.c:583), mark_job_finish (job.c:638), mark_job_fail (job.c:696), mark_job_update (job.c:726), under the caller-held job WRITE lock, before the BEFORE_EVENT hook fires (a guarded-out transition runs no hooks → no WAL entry, no double-fire)
• Rules (terminal = any of DONE | EXIT, with or without PERR/PDONE modifiers):
  • duplicate: requested resulting state == current state → INFO log, return 0, no mutation, no hooks (5.1)
  • backward: current state terminal AND requested state non-terminal (START on finished job, UPDATE clearing terminal bits) → WARN log, return -EALREADY-style rejection, state preserved (5.2)
  • forward/normal: proceed unchanged; mark_job_fail's existing post-exec special case (job.c:702-716) already tolerates fail-after-finish and is kept as the terminal-state-compatible path
• Replay equivalence (5.3): guards make restore_job_state-driven re-application and RETAIN retry converge to single-application state; callers (on_job_*, job_ctrl) treat the duplicate code as success-no-op
• Shared prerequisite of worker-rejoin-resync (umbrella adjacent expectation) — guards live here, resync consumes them later

Contracts: Service [x] — mark_job_* return contract gains the logged no-op/rejection outcomes; all existing callers checked during the sweep

Data Models

Domain Model (changes only)

• wal_commit_mode_t — new enum (persist.h); pure classification metadata, never persisted
• WAL entries, job_t, queue_t, worker_t — no format or field changes in this spec (submit_lsn tagging is ha-convergence-recovery's egress filter, not here)
• Invariants: every appended state-changing entry is classified; a success reply to a request-shaped operation implies committed_lsn >= all entries that operation appended; committed_config_tag advances only on per-push config majority; job state never transitions terminal → non-terminal

Error Handling

Error Strategy

• Request-shaped commit failure: -ETIMEDOUT / -XCE_ERR_NOT_READY propagate to the requester reply via the existing errnox early-return pattern; no in-memory rollback (provisional model, umbrella-locked; convergence owned downstream)
• Fact-shaped paths never surface commit errors to reporters: local append failure (disk) keeps its existing ERROR logging; commit progress is the engine's concern (RETAIN)
• Unclassified WAL type: XCE_TESTING assert (fail fast in CI); production ERROR + RETAIN fallback (2.2 — fail-safe toward retention, never silent confirmation)
• Config failure: explicit error reply via send_upgrade_error; no restart; WARN with tag and acked count
• Guard rejections: INFO (duplicate) / WARN (backward) with jobid and both states — log-once per application, no log storms

Monitoring

• Retained-uncommitted depth (committed-vs-appended lag) — engine counter, consumed in logs here for fact-backlog visibility
• Per-handler barrier failures WARN with opcode and rc; config barrier WARN already present (config_repl.c:924-926), now coupled to the failed reply

Testing Strategy

Unit Tests (testsuite/test_commit_barrier_rollout.c, XCE_TESTING)

• Classification totality: every wal_entry_type value maps to a mode; WAL_WORKER_CTRL detail split (JOIN/UPDATE → WAIT, DEAD → RETAIN); unknown type asserts under XCE_TESTING (2.1, 2.2)
• Barrier helper: returns 0 immediately when committed_lsn covers the watermark; -ETIMEDOUT with ACKs withheld and committed_lsn unmoved; 0 after covering ACK; n_peers <= 1 immediate success (1.1, 1.2, 1.3)
• Idempotency guards: duplicate FINISH → state unchanged, rc = no-op, no AFTER hook fired (no WAL entry); START on DONE job → rejected, WARN, state preserved; double application of a transition equals single application; fail-after-finish keeps existing PERR behavior (5.1, 5.2, 5.3)
• configAck arithmetic: stale acked_config_lsn from a previous tag does not count; zero covering acks with n_peers=3 → timeout; majority with self-count → 0; n_peers=1 → immediate (4.1, 4.3)

Integration Tests (testsuite/)

• No-success-reply-before-commit: for each restructured opcode (jobcontrol, queuecontrol, hostcontrol, worker join), with ACKs withheld the reply is an error and arrives only at/after the timeout; with ACKs the success reply is observable only after committed_lsn covers the operation's entries (1.1, 1.2, 1.4)
• Fact retention across quorum loss: simulate quorum loss → notify-driven transitions still append (RETAIN, no error), gate admits the fact opcodes; restore ACKs → entries commit without re-delivery (3.1-3.4)
• Dispatch receipt-ACK: handle_jobdispatch still replies immediately; no WAL entry is produced by dispatch processing itself (classification assert clean) (2.3)
• Config barrier: upgradeConfig with withheld configAcks → error reply to requester, committed_config_tag unmoved, no restart requested; with acks → success, tag advanced (4.1-4.4)
• Reply-site audit (2.4): scripted grep over all xce_reply call sites in control paths, each assigned a category; zero unclassified sites

Regression

• Full controller testsuite + ASAN (/xce-build controller) green; existing job-submit barrier tests (upstream) unaffected; test_repl_majority.c configAck cases updated to the hardened per-push arithmetic

Performance & Scalability

• Request-control latency gains one majority RTT (the same cost job submit already pays); fact paths gain no latency (RETAIN returns immediately)
• Concurrent blocked handlers are bounded by the engine's n_vcpus-relative admission cap (upstream); excess load degrades to fast NOT_READY rejections — deliberate CP
• Barrier over-wait on concurrent unrelated entries is bounded by sync_timeout_ms and only occurs while uncommitted traffic is pending

Migration Strategy

This spec is phase 2 of the umbrella's hard-ordered landing ("barrier before state removal"):

1. Classification + RETAIN registration + barrier helper (no behavior change for replies yet)
2. Handler restructure sweep + fact gate admission (reply semantics switch on)
3. Config barrier error propagation; legacy walAck counting retirement; idempotency guards Each step leaves the system no less consistent than before it; rollback trigger per step: controller testsuite regression or job-control E2E failure. ha-single-leader-role must not land until the sweep (step 2) is complete and audited.