lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

chore: baseline snapshot — prior-session work + ha-consistency-invariants spec family

Browse source 
Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x),
heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup.
New: ha-consistency-invariants umbrella (requirements/design/research) and
its four implementation sub-specs (specs generated via batch, cross-spec
review applied), steering roadmap + product.md Raft+CP decision.
This commit is contained in:
Rongsong Shen 2026-06-11 13:21:50 +00:00
parent 084244941b
commit 06135954ff
88 changed files with 25611 additions and 5285 deletions
Download patch file Download diff file Expand all files Collapse all files

24
.kiro/specs/ha-commit-barrier-rollout/brief.md Normal file
View file

@ -0,0 +1,24 @@
# Brief: ha-commit-barrier-rollout
> Sub-spec 2 of 4 implementing the **ha-consistency-invariants** umbrella contract.
> Authoritative requirements/design/research: `.kiro/specs/ha-consistency-invariants/` (design.md §Write-Path Classification + §Config Barrier + §Idempotency Guards). This brief scopes the slice; do not re-derive decisions.
## Problem
Only WAL_JOB_SUBMIT goes through a commit barrier. Job state changes, queue control, and worker control apply locally with fire-and-forget replication; job/queue control replies before any durability; config push ignores its majority-ACK result; worker-reported facts are silently lost on rejection (bug-396); `mark_job_*` have no idempotency guards.
## Desired Outcome
Every state-changing WAL type routed per the design's classification table: request-shaped → WAIT mode with **commit-first-then-reply** (validate → mutate provisional → append → commit_submit → reply); fact-shaped → RETAIN mode (log-is-the-queue, appended + retried until committed, no timeout, never dropped while running+leading). Config upgrade fails to the requester on missed majority. `mark_job_*` transitions are idempotent and ordered.
## Scope
- **In**: classification table + dispatch in `persist.c` (debug assert on unclassified types); reply-ordering sweep of all control handlers in `job.c` (incl. `do_jobcontrol_op`), queue and worker join/update paths; `handle_jobdispatch` classification (per design Reply-Ordering Contract); RETAIN routing for fact WAL types; `push_config_to_standbys` error propagation + caller failure; `mark_job_*` state-machine guards; integration tests (no success reply observable before commit; fact retention across quorum loss).
- **Out**: the commit engine itself (ha-commit-engine); promotion/COMMITTING (ha-single-leader-role); egress filter and demotion (ha-convergence-recovery); worker protocol changes (worker-rejoin-resync backlog).
## Requirements (umbrella IDs)
2.1 (all operation types), 2.2, 2.8, 5.1, 5.2, 5.3.
## Upstream / Downstream
- **Upstream**: ha-commit-engine (WAIT/RETAIN API).
- **Downstream**: ha-single-leader-role (must land after this — barrier before state removal); worker-rejoin-resync (shares idempotency guards).
## Constraints
Commit-first-then-reply is non-negotiable for request-shaped ops; a missed handler silently violates umbrella Req 2.1 — the sweep must be exhaustive (grep all xce_reply sites in control paths). Crash-loss of uncommitted facts accepted until worker-rejoin-resync lands.

446
.kiro/specs/ha-commit-barrier-rollout/design.md Normal file
View file

@ -0,0 +1,446 @@
# Design: ha-commit-barrier-rollout
> Sub-spec 2 of 4 of the **ha-consistency-invariants** umbrella. This design slices the umbrella design's **Write-Path Classification**, **Config Barrier**, and **Idempotency Guards** components (`.kiro/specs/ha-consistency-invariants/design.md`), implementing umbrella Migration Strategy phase 2. The umbrella's WAL-type classification table and Reply-Ordering Contract are the contract; this design records their call-site verification (which the umbrella explicitly mandated) and the enforcement mechanism. The commit engine interface (`include/commit.h`, `commit_submit` WAIT/RETAIN) is consumed exactly as shipped by ha-commit-engine — this spec never modifies `commit.c` internals. Requirement IDs below are THIS spec's IDs (see `requirements.md` for the umbrella mapping).
## Overview
**Purpose**: This feature extends the majority-commit barrier from job submission to every cluster-state-changing operation: request-shaped operations reply to their requester only after commit (and fail on timeout), fact-shaped operations are appended and retained until committed with no timeout, configuration upgrades fail honestly to the admin tool, and job state transitions become idempotent so retained-fact retry and replay are safe.
**Users**: Cluster operators gain the guarantee that any confirmed control operation survives failover and that worker-reported reality is never silently dropped; downstream sub-specs (ha-single-leader-role, ha-convergence-recovery) gain a write path that is barrier-complete before the transitional leader state is removed.
**Impact**: Adds a classification dispatch at the persistence hook seam (persist.c) with a fail-fast guard for unclassified WAL types; restructures `do_jobcontrol_op` (job.c:921-934), `do_queuecontrol_op` (queue.c:119-132), `do_hostcontrol_op` (worker.c:689-702), and `handle_worker_join` (join.c:33-119) to commit-first-then-reply; admits fact-bearing opcodes through the dispatch gate during quorum loss (ha.c:1527); propagates the config majority-ACK result from `push_config_to_standbys` (config_repl.c:866-930) to the `upgradeConfig` requester (config_repl.c:1057-1094); adds state-machine guards to `mark_job_*` (job.c:583/638/696/726); and retires the legacy walAck counting fallback in `repl_wait_majority_ack` (repl.c:386-462) while hardening its configAck branch.
### Goals
- Commit-first-then-reply for every request-shaped control handler; `handle_jobsubmit` (job.c:495-581) is the reference implementation and stays unchanged
- Fact WAL types routed RETAIN (log-is-the-queue): appended + broadcast, pending purely by `lsn > committed_lsn`, never dropped while running+leading, accepted even during quorum loss
- An unclassified state-changing WAL type cannot silently bypass the barrier (debug assert; production fact-fallback)
- Config upgrade fails to the requester on missed majority and does not restart into the failed config
- `mark_job_*` idempotent and ordered (duplicate = no-op; terminal→non-terminal rejected)
### Non-Goals
- Commit engine internals: per-LSN evaluation, admission cap, `committed_lsn` advancement (ha-commit-engine — consumed as-is)
- Promotion no-op / own-term gate consumers / CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role; the gate's COMMITTING branch keeps rejecting facts until that spec deletes the state — known residual)
- Demotion truncate+restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum-transition logging (ha-convergence-recovery)
- Worker protocol changes; recovery of facts that never reached the controller or died with it (worker-rejoin-resync backlog — umbrella-locked: crash-loss of uncommitted facts accepted until then)
- WAL_NOOP introduction (ha-single-leader-role adds the type; the classification guard here forces it to be classified when it lands)
## Boundary Commitments
### This Spec Owns
- The WAL-type → commit-mode classification table, its dispatch at the persistence hook seam, and the unclassified-type guard
- The reply ordering of every request-shaped control handler on the primary (job control, queue control, worker control, worker join/update) and the receipt-ACK verdict for scheduler-origin handlers
- The commit-barrier helper (`persist_commit_barrier`) that handlers use to realize WAIT-mode completion
- Fact admission through the dispatch gate's no-write-quorum path (opcode allowlist only — not quorum accounting)
- Config-change barrier error propagation: `push_config_to_standbys` return contract, configAck majority arithmetic, and the `upgradeConfig` caller's failure path
- `mark_job_*` transition guards
- Retirement of the legacy (engine-less) walAck counting in `repl_wait_majority_ack`
### Out of Boundary
- `commit.c` / `include/commit.h` — consumed, never modified (interface changes there trigger revalidation here)
- `ha_has_write_quorum` and quorum accounting (ha-commit-engine); the gate's role-based branches other than the PRIMARY no-quorum opcode list
- `committed_lsn` advancement rules, admission cap, membership re-evaluation (commit engine)
- Egress filtering of provisional state to the scheduler (ha-convergence-recovery; dispatch of uncommitted submits is closed there)
- Standby-side apply, WAL wire format, catch-up streaming mechanics (ha-wal-replication)
- Worker/scheduler retry behavior on error replies (client-side, existing semantics)
### Allowed Dependencies
- `commit_submit(ce, lsn, COMMIT_MODE_WAIT|RETAIN, timeout_ms)` and the engine instance embedded in `cluster_t` (ha-commit-engine, P0) — the only commit-evaluation authority
- `persist_wal_async` / `persist_wal_sync_append` append+broadcast primitives and `persist_t.current_lsn` / `committed_lsn` atomics (persist.c, P0)
- `repl_on_config_ack` / `acked_config_lsn` per-standby config tracking (repl.h, P0) — read-only consumption for the config barrier
- Hook dispatch (`xce_run_hooks`, HOOK_AFTER_EVENT seam) — existing pattern, unchanged signatures
- Threading rules per `ha-async-io.md`: commit-barrier waits only on vtask threads; nothing here runs on iomgr
- Dependency direction: handlers (job.c/queue.c/worker.c/join.c) → persist.c barrier/classification → commit engine; persist.c never calls back into handlers
### Revalidation Triggers
- Any change to `commit.h` signatures or return codes (upstream) → re-verify barrier helper and handler error mapping
- ha-single-leader-role adding WAL_NOOP or removing CTRL_PRIMARY_COMMITTING → classification table gains the WAL_NOOP row (RETAIN, umbrella-fixed); gate COMMITTING branch disappears
- Any new state-changing opcode/WAL type → must be added to the classification table (the debug assert makes this unmissable) and its handler swept for reply ordering
- worker-rejoin-resync landing → re-verify `mark_job_*` guards against its replay ordering
- Changes to `acked_config_lsn` semantics or the upgradeConfigAck payload → re-verify configAck majority arithmetic
## Architecture
### Existing Architecture Analysis
- All state-changing writes flow through one seam: entity mutation fires `HOOK_AFTER_EVENT`, and `persist_job_hook_fun` / `persist_queue_hook_fun` / `persist_worker_hook_fun` (persist.c:1093/1172/1204) map hook details to WAL types via `detail_to_wal_type` (persist.c:1068-1088) and append. Only WAL_JOB_SUBMIT uses the sync barrier (persist.c:1107-1129); everything else is `persist_wal_async` fire-and-forget
- Request handlers reply immediately after mutation: `do_jobcontrol_op` (job.c:932), `do_queuecontrol_op` (queue.c:130), `do_hostcontrol_op` (worker.c:700), `handle_worker_join` (join.c:109) — all violate umbrella 2.1 today. `handle_jobsubmit` (job.c:563-578) is the proven correct shape
- Queue/worker/job-state hooks fire while the entity WRITE lock is held (queue.c:76-112, job.c:629-631/687-689); job-submit's hook runs after the job locks are released (job.c:553) — this asymmetry drives WAIT placement (below)
- Facts arrive via `do_notify` (cluster.c:118-157, no reply) and `do_update_op`/`do_set_preason`/`do_clr_preason` (receipt-style replies); the dispatch gate (ha.c:1527) rejects `OPCode_notify` with `-XCE_ERR_NOT_READY` under quorum loss → permanent silent fact loss (bug-396 family, verified pre-existing)
- Call-site verification of the umbrella classification table (mandated by the umbrella): WAL_JOB_STOP/RESUME are appended only from `on_job_stop`/`on_job_resume`, which are invoked exclusively from `do_notify` (worker-reported stop/resume facts). Client job control (`do_jobcontrol_op` → `job_ctrl`, job.c:306-438) either forwards to the worker (running job — no local entry) or mutates locally via `mark_job_update`/`mark_job_fail` (not-started job → WAL_JOB_UPDATE/WAL_JOB_FAIL). WAL_JOB_UPDATE is absent from the umbrella table and is dual-origin (request: job_ctrl; fact: on_job_postdone, rusage update). Resolution: STOP/RESUME/UPDATE/FAIL classify as fact/RETAIN at the seam, and the umbrella's intent for client job control — "error to client" — is realized by the handler-level commit barrier in `do_jobcontrol_op` (the Reply-Ordering Contract's handler sequence is the umbrella-authoritative mechanism). Recorded in research.md
- `push_config_to_standbys` (config_repl.c:866-930) calls `repl_wait_majority_ack(configAck)` at 916-917 but only logs the result and returns the push count; the caller (config_repl.c:1057) ACKs the admin tool with status=0 and restarts regardless. The configAck counting (repl.c:418-438) has the same defects the engine fixed for walAck: `acked_config_lsn > 0` (any previous ack counts), `n_active == 0 → success`, denominator `n_active` instead of `n_peers`
### Architecture Pattern & Boundary Map
```mermaid
graph TB
subgraph requesters
CLI[client and cli ops]
SCHED[scheduler]
WK[worker]
ADM[admin tool]
end
subgraph controller_primary
GATE[ha dispatch gate<br/>fact allowlist]
H[request handlers<br/>job queue worker join]
F[fact handlers<br/>notify update preason]
HOOKS[persist hooks<br/>classification dispatch]
BAR[persist_commit_barrier]
CB[config barrier<br/>config_repl]
CE[commit engine<br/>consumed]
GUARD[mark_job guards]
end
CLI --> GATE
SCHED --> GATE
WK --> GATE
ADM --> GATE
GATE --> H
GATE --> F
H --> GUARD
F --> GUARD
GUARD --> HOOKS
HOOKS -->|append plus RETAIN register| CE
H -->|before success reply| BAR
BAR -->|commit_submit WAIT| CE
ADM -.upgradeConfig.-> CB
CB -->|configAck majority| CE
```
**Architecture Integration** (sliced from umbrella):
- Selected pattern: classification at the single persistence seam + completion at the handler. The seam guarantees no entry type escapes classification (2.1, 2.2); the handler realizes the umbrella Reply-Ordering Contract sequence "validate → mutate (provisional) → WAL append → commit_submit(WAIT) → reply" (1.1)
- WAIT placement decision: WAL_JOB_SUBMIT keeps its in-hook sync barrier (hook runs lock-free, engine-wired upstream). All other request-shaped types complete via `persist_commit_barrier()` called by the handler after entity locks are released — blocking a commit wait under a queue/job/worker WRITE lock would stall readers for up to the commit timeout. Same completion semantics, umbrella-conformant placement (the contract names the handler sequence)
- Existing patterns preserved: hook-based persistence dispatch, provisional in-memory mutation order (umbrella-locked), `errnox` early-return error style, full fan-out broadcast
- Steering compliance: barrier waits only on vtask threads (handlers run on the vtask pool); iomgr untouched; TDD with testsuite/ tests first
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Backend / Services | C (gnu17), pthreads, stdatomic | classification, barrier helper, handler restructure | no new dependencies, no .capnp changes |
| Data / Storage | existing WAL (persist.c), wal_types.h enum | classification keyed on `wal_entry_type` + worker detail | no on-disk format change |
| Build / Test | meson, testsuite/ pattern | new `test_commit_barrier_rollout.c` | XCE_TESTING seams |
## File Structure Plan
### Directory Structure
```
src/controller/
├── persist.c # MODIFIED — classification dispatch, RETAIN registration,
│ # persist_commit_barrier, unclassified guard
├── job.c # MODIFIED — do_jobcontrol_op reply ordering; mark_job_* guards
├── queue.c # MODIFIED — do_queuecontrol_op reply ordering
├── worker.c # MODIFIED — do_hostcontrol_op reply ordering
├── join.c # MODIFIED — handle_worker_join reply ordering
├── ha.c # MODIFIED — dispatch-gate fact allowlist (PRIMARY branch only)
├── config_repl.c # MODIFIED — push_config_to_standbys return contract; caller failure path
├── repl.c # MODIFIED — legacy walAck counting retired; configAck branch hardened
└── testsuite/
└── test_commit_barrier_rollout.c # NEW — classification, barrier ordering, fact retention,
# config barrier, idempotency guards
src/include/persist.h # MODIFIED — wal_commit_mode_t enum, wal_classify(),
# persist_commit_barrier() declarations
src/include/repl.h # MODIFIED — configAck evaluation reuses the entry_lsn
# parameter (added by ha-commit-engine) as the
# config push tag; no new parameter
```
### Modified Files (what and why)
- `src/include/persist.h` — declares `wal_commit_mode_t { WAL_MODE_WAIT, WAL_MODE_RETAIN, WAL_MODE_LOCAL }`, `wal_commit_mode_t wal_classify(uint32_t wal_type, int detail)`, `int persist_commit_barrier(persist_t *p, uint32_t timeout_ms)`
- `src/controller/persist.c` — `wal_classify` table; all three hook functions consult it: WAIT → existing sync path (submit) or async append with handler-owed barrier; RETAIN → `persist_wal_async` + `commit_submit(RETAIN)` registration; LOCAL → unchanged (WAL_CHECKPOINT); unclassified → `assert(0)` under XCE_TESTING, ERROR + RETAIN fallback in production. `persist_commit_barrier` implemented here (snapshot `current_lsn`, `commit_submit(WAIT)`)
- `src/controller/job.c` — `do_jobcontrol_op` (job.c:921-934): barrier between `job_ctrl` success and the reply. `mark_job_start/finish/fail/update` (job.c:583/638/696/726): transition guards before mutation
- `src/controller/queue.c` — `do_queuecontrol_op` (queue.c:119-132): barrier between `queue_ctrl` success and the reply
- `src/controller/worker.c` — `do_hostcontrol_op` (worker.c:689-702): barrier between `worker_ctrl` success and the reply
- `src/controller/join.c` — `handle_worker_join` (join.c:33-119): barrier between the AFTER_EVENT hook and the reply at join.c:107-109; join failure (idx < 0) keeps replying immediately
- `src/controller/ha.c` — `ha_dispatch_gate` PRIMARY no-quorum allowlist (ha.c:1537-1556): add `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason`
- `src/controller/config_repl.c` — `push_config_to_standbys` returns the barrier result (0 / `-ETIMEDOUT`), push count via out-param; passes the in-flight tag for per-push evaluation; caller `do_upgrade_config_with_bytes` (config_repl.c:1057): on failure send error reply, skip restart
- `src/controller/repl.c` — walAck branch of `repl_wait_majority_ack`: legacy counting removed (engine is mandatory on this path; `ce == NULL` becomes a programming error); configAck branch: per-push comparison (`acked_config_lsn >= push_tag`), majority over `n_peers` counting self, `n_peers > 1` with zero standby acks never succeeds
- `src/controller/meson.build` — register `testsuite/test_commit_barrier_rollout.c`
## System Flows
### Request-shaped control operation (job/queue/worker control, worker join)
```mermaid
sequenceDiagram
participant R as requester
participant V as vtask thread handler
participant E as entity lock scope
participant P as persist hooks
participant CE as commit engine
R->>V: control request
V->>V: dispatch gate incl write quorum
V->>E: lock, validate, mutate provisional
E->>P: HOOK_AFTER_EVENT append entries
P->>CE: persist_wal_async + register
E->>V: unlock, rc
alt rc != 0
V->>R: error reply
else rc == 0
V->>CE: persist_commit_barrier WAIT
alt committed
V->>R: success reply
else timeout or not ready
V->>R: error reply
end
end
```
Flow decisions: the barrier waits on a snapshot of `current_lsn` taken after the mutation returns — single-writer appends and contiguous commit advancement guarantee the snapshot covers every entry the operation produced (1.1, 1.4); waiting may also cover unrelated concurrent entries (bounded by the commit timeout, harmless — they must commit anyway); `-ETIMEDOUT`/`-XCE_ERR_NOT_READY` map to the error reply via the existing `errnox` style (1.2, 1.3); no in-memory rollback (provisional model, umbrella-locked — outcome converges per umbrella 2.5, owned downstream).
### Fact-shaped delivery (worker notify, rusage update, preason, worker dead)
Gate admits the opcode even without write quorum (3.2) → handler applies the transition through the `mark_job_*` guards (5.1-5.3) → hook classifies the WAL type RETAIN → `persist_wal_async` append + broadcast + `commit_submit(RETAIN)` (returns immediately, no timeout) → handler returns (notify: no reply; update/preason: receipt-style reply unchanged, verdict per 2.3/2.4). The fact is pending purely by `lsn > committed_lsn`; catch-up streaming re-delivers the uncommitted suffix and `committed_lsn` advances when ACKs resume (3.1, 3.3, 3.4). No new queue structure exists (log-is-the-queue, umbrella-locked).
### Configuration upgrade
```mermaid
sequenceDiagram
participant A as admin tool
participant U as do_upgrade_config_with_bytes
participant PS as push_config_to_standbys
participant S as standbys
A->>U: OP_upgradeConfig
U->>U: validate, save to disk, bump generation
U->>PS: push bytes, in-flight tag
PS->>S: parallel push threads
S-->>PS: OP_upgradeConfigAck tag
PS->>PS: majority of n_peers acked tag, self counted
alt majority within timeout
PS-->>U: 0, acked count
U->>U: committed_config_tag advanced
U->>A: success reply
U->>U: request restart
else timeout
PS-->>U: -ETIMEDOUT
U->>A: error reply upgrade failed
U->>U: no restart, tag not committed
end
```
Flow decisions: `committed_config_tag` advances only inside the majority branch (4.3 — already correct, preserved); the failure path uses the existing `send_upgrade_error` shape so the admin tool sees a definite failure (4.2); restart is gated on success (4.4 — closes the roadmap residual "don't restart before the commit barrier passes"); config remains a parallel, simpler track sharing only the majority arithmetic rule with the engine (umbrella-locked: one in-flight config change at a time).
## Requirements Traceability
| Requirement | Summary | Components | Flows |
|-------------|---------|------------|-------|
| 1.1 | success reply only after majority commit | Reply-Ordering Sweep, Commit Barrier Helper | request control |
| 1.2 | commit timeout → error reply | Commit Barrier Helper, Reply-Ordering Sweep | request control |
| 1.3 | not-ready rejection → error reply | Commit Barrier Helper (engine code passthrough) | request control |
| 1.4 | no success-interpretable reply pre-commit | Reply-Ordering Sweep (integration-test assertion) | request control |
| 2.1 | every WAL type classified WAIT/RETAIN | Write-Path Classification | both write flows |
| 2.2 | unclassified type fails fast / fact-fallback | Write-Path Classification (guard) | — |
| 2.3 | dispatch ACK is receipt-only | Reply-Ordering Sweep (handle_jobdispatch verdict) | — |
| 2.4 | exhaustive reply-site verification | Reply-Ordering Sweep (audit) | — |
| 3.1 | facts appended, retried, no timeout | Write-Path Classification (RETAIN routing) | fact delivery |
| 3.2 | facts accepted during quorum loss | Fact Gate Admission | fact delivery |
| 3.3 | uncommitted facts never discarded | Write-Path Classification (RETAIN; log-is-the-queue) | fact delivery |
| 3.4 | facts commit after quorum restore, no redelivery | Write-Path Classification + existing catch-up (verified by test) | fact delivery |
| 4.1 | config confirmed only after majority | Config Barrier | config upgrade |
| 4.2 | config timeout → failed to requester | Config Barrier | config upgrade |
| 4.3 | unacked config never committed | Config Barrier (tag advance in success branch only) | config upgrade |
| 4.4 | failed upgrade does not restart | Config Barrier (caller) | config upgrade |
| 5.1 | duplicate transition is logged no-op | Idempotency Guards | fact delivery |
| 5.2 | terminal→non-terminal rejected with WARN | Idempotency Guards | fact delivery |
| 5.3 | replay-once equivalence | Idempotency Guards | fact delivery |
## Components and Interfaces
| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|-------|--------|--------------|------------------|-----------|
| Write-Path Classification | controller/persist.c | route every WAL type to its commit mode at the hook seam | 2.1, 2.2, 3.1, 3.3, 3.4 | commit engine (P0), hooks (P0) | Service |
| Commit Barrier Helper | controller/persist.c | handler-callable WAIT completion on the appended watermark | 1.1, 1.2, 1.3 | commit engine (P0) | Service |
| Reply-Ordering Sweep | controller/job.c, queue.c, worker.c, join.c | commit-first-then-reply in every request handler; receipt-ACK verdicts | 1.1, 1.2, 1.3, 1.4, 2.3, 2.4 | Commit Barrier Helper (P0) | Service |
| Fact Gate Admission | controller/ha.c | admit fact-bearing opcodes without write quorum | 3.2 | dispatch gate (P0) | Service |
| Config Barrier | controller/config_repl.c, repl.c | per-push config majority with error propagation | 4.1, 4.2, 4.3, 4.4 | repl_on_config_ack tracking (P0) | Service |
| Idempotency Guards | controller/job.c | safe duplicate/replay of job transitions | 5.1, 5.2, 5.3 | — | Service |
### Write-Path Classification (`persist.c`, `src/include/persist.h`)
| Field | Detail |
|-------|--------|
| Intent | Single seam where every state-changing WAL append is classified and registered with the engine |
| Requirements | 2.1, 2.2, 3.1, 3.3, 3.4 |
**Classification table** (umbrella table verified against call sites — verification deviations recorded in research.md §2; this verified table is the implementation contract):
| WAL type (+ detail) | Verified origin | Shape | Mode | Failure behavior |
|---|---|---|---|---|
| WAL_JOB_SUBMIT | handle_jobsubmit (client) | request | WAIT (existing in-hook sync path) | error to client (reference impl, unchanged) |
| WAL_QUEUE_CTRL | do_queuecontrol_op (client) | request | WAIT (handler barrier) | error to client |
| WAL_WORKER_CTRL, detail JOIN/UPDATE | handle_worker_join, do_hostcontrol_op | request | WAIT (handler barrier) | error reply to worker/client |
| WAL_WORKER_CTRL, detail DEAD | controller-detected (worker.c:565/578) | fact | RETAIN | none — retained until committed |
| WAL_JOB_START / FINISH / FAIL / STOP / RESUME / UPDATE / RUSAGE / CKPT / PREASON | worker notify, scheduler preason, postdone, rusage; FAIL/UPDATE also from job_ctrl (request origin → covered by do_jobcontrol_op's handler barrier) | fact | RETAIN | none — retained until committed |
| WAL_NOOP (future) | promotion (ha-single-leader-role) | internal | RETAIN | added downstream; guard forces classification |
| WAL_CHECKPOINT | local checkpoint marker | local-only | LOCAL | unchanged (never replicated, by design) |
**Responsibilities & Constraints**
- `wal_classify(wal_type, detail)` is a pure lookup; `detail` disambiguates only WAL_WORKER_CTRL (JOIN/UPDATE vs DEAD)
- Hook dispatch: RETAIN → `persist_wal_async` (append + broadcast, the surviving RETAIN fast path) followed by `commit_submit(ce, appended_lsn, COMMIT_MODE_RETAIN, 0)` registration (returns immediately; observability counter). WAIT → WAL_JOB_SUBMIT keeps `persist_wal_sync_append`; other WAIT types append via `persist_wal_async` and completion is owed by the handler barrier (placement decision, Architecture)
- Unclassified guard: under `XCE_TESTING`, `assert(0)` with the offending type logged; in production, `ERROR` log + RETAIN treatment (never dropped, never falsely confirmed) (2.2)
- Fact appends never return commit errors to their callers — `persist_job_hook_fun`'s non-submit return value stays the local-append rc only (3.1)
**Dependencies**
- Inbound: hook system (HOOK_AFTER_EVENT) — all state-changing writes (P0)
- Outbound: `commit_submit` (P0); `persist_wal_async`/`persist_wal_sync_append` (P0)
**Contracts**: Service [x]
##### Service Interface
```c
/* src/include/persist.h */
typedef enum {
WAL_MODE_WAIT, /* request-shaped: barrier before requester reply */
WAL_MODE_RETAIN, /* fact-shaped: retained until committed, no timeout */
WAL_MODE_LOCAL, /* not replicated (WAL_CHECKPOINT) */
} wal_commit_mode_t;
/* Pure classification lookup. detail disambiguates WAL_WORKER_CTRL only.
* Unknown type: XCE_TESTING assert; production returns WAL_MODE_RETAIN
* after ERROR log. */
wal_commit_mode_t wal_classify(uint32_t wal_type, int detail);
```
- Preconditions: called at the hook seam with the type produced by `detail_to_wal_type`
- Postconditions: total over all `wal_entry_type` values; deterministic
- Invariants: no WAL append path bypasses classification
### Commit Barrier Helper (`persist.c`, declared in `src/include/persist.h`)
| Field | Detail |
|-------|--------|
| Intent | Give request handlers a single call realizing commit_submit(WAIT) on everything their mutation appended |
| Requirements | 1.1, 1.2, 1.3 |
**Responsibilities & Constraints**
- Snapshot `lsn = p->current_lsn` (under `p->lock`, matching `persist_wal_sync_append`'s read at persist.c:1019-1021); if `committed_lsn >= lsn` already, return 0 without blocking
- Delegate to `commit_submit(ce, lsn, COMMIT_MODE_WAIT, timeout_ms ? timeout_ms : p->config.sync_timeout_ms)`; return its code unmodified (0 / `-ETIMEDOUT` / `-XCE_ERR_NOT_READY`)
- Single-writer + contiguous advancement ⇒ the snapshot covers every entry the calling operation produced; over-waiting on unrelated concurrent entries is accepted (bounded by the timeout; they must commit anyway). `n_peers <= 1` returns 0 immediately (engine self-quorum, standalone behavior preserved)
- Must be called only from vtask threads after entity locks are released; counts against the engine's admission cap (deliberate CP back-pressure)
- Migration completion: with all request paths on the engine (directly or via this helper), the walAck legacy counting in `repl_wait_majority_ack` is removed (upstream marked it "retired by ha-commit-barrier-rollout"); the configAck branch remains the config track's mechanism
**Contracts**: Service [x]
##### Service Interface
```c
/* src/include/persist.h */
/* Block until committed_lsn covers everything appended so far.
* timeout_ms = 0 → p->config.sync_timeout_ms.
* Returns 0 (committed), -ETIMEDOUT (not committed in time — entry
* remains in log; outcome converges per umbrella 2.5, owned downstream),
* -XCE_ERR_NOT_READY (admission cap / own-term gate, retriable). */
int persist_commit_barrier(persist_t *p, uint32_t timeout_ms);
```
- Preconditions: commit engine wired (`cluster_init` phase, upstream); vtask thread; no entity locks held
- Postconditions: returns 0 ⇔ `committed_lsn >=` the snapshot watermark
- Invariants: never sends replies, never mutates state
### Reply-Ordering Sweep (`job.c`, `queue.c`, `worker.c`, `join.c`)
| Field | Detail |
|-------|--------|
| Intent | Enforce the umbrella Reply-Ordering Contract at every control-path reply site |
| Requirements | 1.1, 1.2, 1.3, 1.4, 2.3, 2.4 |
**Responsibilities & Constraints**
- Restructure pattern (uniform): `rc = <ctrl fn>(...); if (rc == 0) rc = persist_commit_barrier(&c->persist, 0); reply.status = rc; xce_reply(...)`. Applies to:
- `do_jobcontrol_op` (job.c:921-934) — covers the not-started branch's WAL_JOB_UPDATE/WAL_JOB_FAIL entries; the running-job branch (forward to worker, no local entry) waits on an already-covered watermark and returns immediately
- `do_queuecontrol_op` (queue.c:119-132)
- `do_hostcontrol_op` (worker.c:689-702)
- `handle_worker_join` (join.c:33-119) — barrier inserted between the AFTER_EVENT hook (join.c:104-105) and the success reply (join.c:107-109); error replies (validate_host, hostconfig, idx < 0) stay immediate; `on_worker_join` dataex push stays after the reply
- Receipt-ACK verdicts (verified, kept, classification-guard-enforced): `handle_jobdispatch` (job.c:459-491) — early reply at job.c:466-471 is ACK-of-receipt; `dispatch_a_job_` produces no WAL entries (allocation results return via `notify_dispatch`, and resulting job-state changes arrive later as worker facts) → per the umbrella contract the early reply stands (2.3). `do_set_preason`/`do_clr_preason` (job.c:1040/1068) — scheduler-origin facts (WAL_JOB_PREASON, RETAIN); early reply is receipt-only. `do_update_op` paths (opupd.c) — worker-origin facts (rusage → WAL_JOB_RUSAGE/WAL_JOB_UPDATE RETAIN; resvals produce no WAL); replies are receipt-only
- Exhaustive audit (2.4): every `xce_reply` call site under src/controller/ (excluding testsuite) is enumerated by grep and assigned one of {post-commit confirmation, receipt-only ACK, read-only/replication/error reply}; the audit table is committed with the task evidence. Known control-path population at design time: job.c:471,524,569,578,932,1050,1078; queue.c:130; worker.c:700; join.c:28,54,66,109,139,152,198,225,248,257,283; opupd.c:48,64,88; cluster.c (gate/shutdown/hello/notify-path replies); config_repl.c:619-1129 (config/replication track); ha.c:1035; opget.c:222 (read-only); reply.c:61; task.c:97 — line numbers re-verified by grep during implementation
- A missed handler silently violates umbrella 2.1 — the integration test asserting no-success-reply-before-commit is the behavioral backstop (1.4)
**Contracts**: Service [x] — external reply semantics: success now implies majority durability; new error codes `-ETIMEDOUT`/`-XCE_ERR_NOT_READY` surface to requesters through existing reply envelopes (Go tier passes them through, per umbrella out-of-boundary)
**Implementation Notes**
- Integration: handlers run on vtask threads (cluster.c dispatch switch at cluster.c:325-363) — barrier blocking is steering-compliant
- Validation: integration test drives each restructured opcode against a 3-node-simulated registry with ACKs withheld → error reply, `committed_lsn` unmoved; with ACKs → success reply strictly after commit
- Risks: scheduler/worker callers seeing new timeout errors where they previously saw instant success — deliberate CP behavior (umbrella); retry semantics unchanged on their side
### Fact Gate Admission (`ha.c`)
Summary-only: add `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` to the CTRL_PRIMARY no-write-quorum allowlist in `ha_dispatch_gate` (ha.c:1537-1556) so reported facts reach the RETAIN path during quorum loss (3.2 — closes the bug-396 silent-loss hole; umbrella error strategy: "fact-shaped ops are still appended and retained"). Request-shaped opcodes (jobsub, jobcontrol, queuecontrol, hostcontrol, jobdispatch, join) remain rejected without quorum. The CTRL_PRIMARY_COMMITTING branch is intentionally untouched (deleted wholesale by ha-single-leader-role — known residual until then). Quorum accounting itself is out of boundary (upstream).
### Config Barrier (`config_repl.c`, `repl.c` configAck branch)
| Field | Detail |
|-------|--------|
| Intent | Make the config majority-ACK result authoritative for the upgrade outcome |
| Requirements | 4.1, 4.2, 4.3, 4.4 |
**Responsibilities & Constraints**
- `push_config_to_standbys` (config_repl.c:866-930) signature: `int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen, const uint8_t sha256[32], uint64_t new_tag, int *acked_out)` — returns the barrier result (0 / `-ETIMEDOUT`), push count via `acked_out`. The majority wait (config_repl.c:916-917) passes `new_tag` down for per-push evaluation; `committed_config_tag` advance stays inside the success branch only (config_repl.c:918-922 — 4.3 preserved)
- configAck branch of `repl_wait_majority_ack` (repl.c:418-438) hardened to mirror the engine's rules on the config track (umbrella: "same majority arithmetic, parallel simpler track"): count standbys with `acked_config_lsn >= push_tag` (not `> 0` — a previous upgrade's ack no longer counts), majority = `1 (self) + count > n_peers / 2` over configured members (not `n_active`), `n_peers <= 1` → immediate success, `n_peers > 1` with zero covering acks → `-ETIMEDOUT` at deadline (the `n_active == 0 → success` short-circuit is removed from this branch). The configAck branch reuses the `entry_lsn` parameter already added to `repl_wait_majority_ack` by ha-commit-engine as the config push tag — no second parameter is added (`repl.h` comment updated accordingly); the function is now configAck-only (walAck callers are on the engine)
- Caller `do_upgrade_config_with_bytes` (config_repl.c:1056-1094): on non-zero return — `send_upgrade_error(ep, req, c, ...)` to the admin tool (4.2), skip the success ACK, skip `restart_requested` (4.4). Disk save and generation bump have already happened (pre-existing sequence); the reply makes the failure explicit and the cluster-wide tag stays uncommitted — convergence/rollback of a half-pushed config remains the existing OP_replConfig anti-entropy behavior (out of boundary)
- One in-flight config change at a time (umbrella-locked) — no per-entry registry needed
**Contracts**: Service [x] — `upgradeConfig` reply contract: success ⇔ majority-acked config; failure carries an error string and implies no restart
**Implementation Notes**
- Integration: `CONF_PUSH_TIMEOUT_MS` remains the config commit timeout
- Validation: integration test withholds configAcks → admin reply is an error, `committed_config_tag` unmoved, no restart request; with acks → success and tag advanced
- Risks: admin tooling that treated any reply as success must observe `status` — existing CLI already surfaces reply status (admin-config-tool backlog item consumes this contract)
### Idempotency Guards (`job.c`)
| Field | Detail |
|-------|--------|
| Intent | Make job state transitions safe under duplicate, out-of-order, and replayed application |
| Requirements | 5.1, 5.2, 5.3 |
**Responsibilities & Constraints**
- Guard placement: at the top of `mark_job_start` (job.c:583), `mark_job_finish` (job.c:638), `mark_job_fail` (job.c:696), `mark_job_update` (job.c:726), under the caller-held job WRITE lock, before the BEFORE_EVENT hook fires (a guarded-out transition runs no hooks → no WAL entry, no double-fire)
- Rules (terminal = any of DONE | EXIT, with or without PERR/PDONE modifiers):
- duplicate: requested resulting state == current state → INFO log, return 0, no mutation, no hooks (5.1)
- backward: current state terminal AND requested state non-terminal (START on finished job, UPDATE clearing terminal bits) → WARN log, return `-EALREADY`-style rejection, state preserved (5.2)
- forward/normal: proceed unchanged; `mark_job_fail`'s existing post-exec special case (job.c:702-716) already tolerates fail-after-finish and is kept as the terminal-state-compatible path
- Replay equivalence (5.3): guards make `restore_job_state`-driven re-application and RETAIN retry converge to single-application state; callers (`on_job_*`, `job_ctrl`) treat the duplicate code as success-no-op
- Shared prerequisite of worker-rejoin-resync (umbrella adjacent expectation) — guards live here, resync consumes them later
**Contracts**: Service [x] — `mark_job_*` return contract gains the logged no-op/rejection outcomes; all existing callers checked during the sweep
## Data Models
### Domain Model (changes only)
- **wal_commit_mode_t** — new enum (persist.h); pure classification metadata, never persisted
- **WAL entries, job_t, queue_t, worker_t** — no format or field changes in this spec (`submit_lsn` tagging is ha-convergence-recovery's egress filter, not here)
- Invariants: every appended state-changing entry is classified; a success reply to a request-shaped operation implies `committed_lsn >=` all entries that operation appended; `committed_config_tag` advances only on per-push config majority; job state never transitions terminal → non-terminal
## Error Handling
### Error Strategy
- Request-shaped commit failure: `-ETIMEDOUT` / `-XCE_ERR_NOT_READY` propagate to the requester reply via the existing `errnox` early-return pattern; no in-memory rollback (provisional model, umbrella-locked; convergence owned downstream)
- Fact-shaped paths never surface commit errors to reporters: local append failure (disk) keeps its existing ERROR logging; commit progress is the engine's concern (RETAIN)
- Unclassified WAL type: XCE_TESTING assert (fail fast in CI); production ERROR + RETAIN fallback (2.2 — fail-safe toward retention, never silent confirmation)
- Config failure: explicit error reply via `send_upgrade_error`; no restart; WARN with tag and acked count
- Guard rejections: INFO (duplicate) / WARN (backward) with jobid and both states — log-once per application, no log storms
### Monitoring
- Retained-uncommitted depth (committed-vs-appended lag) — engine counter, consumed in logs here for fact-backlog visibility
- Per-handler barrier failures WARN with opcode and rc; config barrier WARN already present (config_repl.c:924-926), now coupled to the failed reply
## Testing Strategy
### Unit Tests (`testsuite/test_commit_barrier_rollout.c`, XCE_TESTING)
- Classification totality: every `wal_entry_type` value maps to a mode; WAL_WORKER_CTRL detail split (JOIN/UPDATE → WAIT, DEAD → RETAIN); unknown type asserts under XCE_TESTING (2.1, 2.2)
- Barrier helper: returns 0 immediately when `committed_lsn` covers the watermark; `-ETIMEDOUT` with ACKs withheld and `committed_lsn` unmoved; 0 after covering ACK; `n_peers <= 1` immediate success (1.1, 1.2, 1.3)
- Idempotency guards: duplicate FINISH → state unchanged, rc = no-op, no AFTER hook fired (no WAL entry); START on DONE job → rejected, WARN, state preserved; double application of a transition equals single application; fail-after-finish keeps existing PERR behavior (5.1, 5.2, 5.3)
- configAck arithmetic: stale `acked_config_lsn` from a previous tag does not count; zero covering acks with n_peers=3 → timeout; majority with self-count → 0; n_peers=1 → immediate (4.1, 4.3)
### Integration Tests (testsuite/)
- No-success-reply-before-commit: for each restructured opcode (jobcontrol, queuecontrol, hostcontrol, worker join), with ACKs withheld the reply is an error and arrives only at/after the timeout; with ACKs the success reply is observable only after `committed_lsn` covers the operation's entries (1.1, 1.2, 1.4)
- Fact retention across quorum loss: simulate quorum loss → notify-driven transitions still append (RETAIN, no error), gate admits the fact opcodes; restore ACKs → entries commit without re-delivery (3.1-3.4)
- Dispatch receipt-ACK: handle_jobdispatch still replies immediately; no WAL entry is produced by dispatch processing itself (classification assert clean) (2.3)
- Config barrier: upgradeConfig with withheld configAcks → error reply to requester, `committed_config_tag` unmoved, no restart requested; with acks → success, tag advanced (4.1-4.4)
- Reply-site audit (2.4): scripted grep over all `xce_reply` call sites in control paths, each assigned a category; zero unclassified sites
### Regression
- Full controller testsuite + ASAN (`/xce-build controller`) green; existing job-submit barrier tests (upstream) unaffected; `test_repl_majority.c` configAck cases updated to the hardened per-push arithmetic
## Performance & Scalability
- Request-control latency gains one majority RTT (the same cost job submit already pays); fact paths gain no latency (RETAIN returns immediately)
- Concurrent blocked handlers are bounded by the engine's `n_vcpus`-relative admission cap (upstream); excess load degrades to fast NOT_READY rejections — deliberate CP
- Barrier over-wait on concurrent unrelated entries is bounded by `sync_timeout_ms` and only occurs while uncommitted traffic is pending
## Migration Strategy
This spec is phase 2 of the umbrella's hard-ordered landing ("barrier before state removal"):
1. Classification + RETAIN registration + barrier helper (no behavior change for replies yet)
2. Handler restructure sweep + fact gate admission (reply semantics switch on)
3. Config barrier error propagation; legacy walAck counting retirement; idempotency guards
Each step leaves the system no less consistent than before it; rollback trigger per step: controller testsuite regression or job-control E2E failure. ha-single-leader-role must not land until the sweep (step 2) is complete and audited.

92
.kiro/specs/ha-commit-barrier-rollout/requirements.md Normal file
View file

@ -0,0 +1,92 @@
# Requirements Document
> Sub-spec 2 of 4 implementing the **ha-consistency-invariants** umbrella contract.
> The umbrella documents (`.kiro/specs/ha-consistency-invariants/requirements.md`, `design.md`, `research.md`) are the authoritative contract. This document restates the slice owned by this spec — umbrella criteria 2.1 (all operation types), 2.2, 2.8, 5.1, 5.2, 5.3 — as this spec's own numbered requirements. Each criterion notes its umbrella ID; the traceability table at the end gives the full mapping. Locked umbrella decisions (log-is-the-queue for facts, commit-first-then-reply for all request-shaped operations, crash-loss of uncommitted facts accepted until worker-rejoin-resync) are restated, never re-derived.
## Project Description (Input)
Only WAL_JOB_SUBMIT goes through a commit barrier today. Job state changes, queue control, and worker control apply locally with fire-and-forget replication; job/queue/worker control handlers reply to their requesters before any durability is established; config push ignores its majority-ACK result and confirms the upgrade to the admin tool regardless; worker-reported facts are silently lost when the dispatch gate rejects them (bug-396 family); and `mark_job_*` transitions have no idempotency guards, so duplicate or replayed application corrupts state. Cluster operators need every state-changing operation routed per the umbrella design's write-path classification table: request-shaped operations confirm to their requester only after majority commit (validate → mutate provisional → append → commit barrier → reply) and fail to the requester on commit timeout; fact-shaped operations are appended and retained until committed with no timeout — never dropped while the primary runs and leads; configuration upgrades fail to the requester on missed majority; and fact transitions are idempotent and ordered so retry/replay is safe. This spec consumes the commit engine delivered by ha-commit-engine (commit_submit WAIT/RETAIN) and never modifies the engine's internals.
## Introduction
This spec delivers the barrier-rollout slice of the umbrella contract. After this spec, every cluster-state-changing operation on the primary participates in the majority-commit contract: a requester (client, scheduler, worker, or admin tool) that receives a success reply is guaranteed the operation is durable on a majority of cluster members; a requester whose operation cannot reach majority within the commit timeout receives an error; and reported facts with no requester awaiting confirmation are retained in the log and retried until committed — including across quorum-loss periods. The write path enforces this through an exhaustive classification of WAL entry types: a state-changing entry type that is not classified cannot silently bypass the barrier. Configuration changes follow the same contract on their own track. Job state transitions become idempotent and ordered so that retained-fact retry, replay, and the future worker-rejoin-resync can re-apply them safely.
## Boundary Context
- **In scope**: The write-path classification table and its dispatch at the persistence seam (with a fail-fast guard for unclassified types); commit-first-then-reply restructuring of every request-shaped control handler (job control, queue control, worker control, worker join/update) and verification of every control-path reply site; classification verdict for the scheduler dispatch acknowledgment; RETAIN routing for fact-shaped WAL types, including acceptance of reported facts during quorum loss; configuration-change commit-barrier error propagation to the upgrade requester; idempotency and ordering guards for job state transitions; integration tests proving no success reply is observable before commit and that facts survive quorum-loss periods.
- **Out of scope**: The commit engine itself — per-LSN evaluation, admission cap, `committed_lsn` advancement (ha-commit-engine, consumed as-is); promotion no-op, own-term gate consumers, and CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role — must land after this spec); demotion truncate+restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum-transition logging (ha-convergence-recovery); worker protocol changes of any kind — facts that never reached the controller, or that died with it across failover, are recovered only by worker-rejoin-resync (roadmap backlog, designated backstop); election and vote correctness (raft-library); replication wire format and standby apply mechanics (ha-wal-replication).
- **Adjacent expectations**: Upstream, this spec consumes the `commit_submit` WAIT/RETAIN service interface and the quorum-accounting fixes delivered by ha-commit-engine; changes to that interface require revalidation here. Downstream, ha-single-leader-role depends on this spec having placed the barrier on all write paths before the transitional leader state is removed (hard ordering), and worker-rejoin-resync shares the idempotency guards introduced here. Crash-loss of facts that were appended but not yet committed when the primary process dies is accepted until worker-rejoin-resync lands (umbrella-locked decision).
## Requirements
### Requirement 1: Commit-First-Then-Reply for Request-Shaped Operations
**Objective:** As a cluster operator, I want every operation with a requester awaiting confirmation to be confirmed only after its log entries are durable on a majority of cluster members, so that no acknowledged change can be lost by a single node failure or failover.
#### Acceptance Criteria
1. When the primary receives a request-shaped cluster-state-changing operation (job control, queue control, worker control, worker join, or worker update), the primary shall send a success reply to the requester only after every log entry produced by the operation has been acknowledged by a majority of cluster members, counting itself. *(umbrella 2.1)*
2. If majority acknowledgment for a request-shaped operation is not obtained within the configured commit timeout, the primary shall reply to that requester with an error reporting the operation as failed. *(umbrella 2.2)*
3. If the commit barrier rejects a request-shaped operation without waiting (cluster not ready to commit), the primary shall reply to the requester with the corresponding error, never with success. *(umbrella 2.2)*
4. While the log entries produced by a request-shaped operation are not yet committed, the primary shall not emit any reply that the requester can interpret as confirmation of that operation. *(umbrella 2.1)*
### Requirement 2: Complete Write-Path Classification
**Objective:** As a cluster operator, I want every state-changing entry type explicitly classified as request-shaped or fact-shaped at the write path, so that no operation type can silently bypass the commit contract.
#### Acceptance Criteria
1. The controller shall route every state-changing log entry type through the commit engine under an explicit classification: request-shaped types complete through the commit barrier before requester confirmation, and fact-shaped types are retained until committed. *(umbrella 2.1, 2.8)*
2. If a state-changing log entry type with no classification reaches the write path, the controller shall fail fast at that point in testing builds; in production builds it shall log an error and treat the entry as fact-shaped, never dropping it and never confirming it early. *(umbrella 2.1)*
3. When the scheduler delivers a job dispatch message, the primary's immediate acknowledgment shall constitute only receipt of the message; per-job dispatch outcomes shall continue to be delivered through the existing per-job notifications, and the receipt acknowledgment shall not confirm any state-changing log entry. *(umbrella 2.1)*
4. The controller shall contain no control-path reply site that confirms a request-shaped state-changing operation before its entries are committed; every control-path reply site shall be either a post-commit confirmation or a receipt-only acknowledgment for fact-shaped deliveries. *(umbrella 2.1)*
### Requirement 3: Fact Retention — Log Is the Queue
**Objective:** As a cluster operator, I want reported facts that have no requester awaiting confirmation to be retained and retried until committed, so that a quorum-loss window never silently erases what actually happened on the cluster.
#### Acceptance Criteria
1. When the primary receives a reported fact with no requester awaiting confirmation (a worker-reported job state transition, worker-reported resource usage, scheduler-reported pending reason, or controller-detected worker death), the primary shall append the fact to its log and continue attempting commitment until the fact's entry is committed, with no commit timeout. *(umbrella 2.8)*
2. While the primary cannot reach a majority of cluster members, when a reported fact arrives, the primary shall accept and append the fact rather than rejecting it. *(umbrella 2.8)*
3. While the primary remains running and holds leadership, the primary shall not discard an uncommitted reported fact. *(umbrella 2.8)*
4. When majority connectivity is restored, the primary shall bring previously appended uncommitted facts to committed state without requiring redelivery from the original reporter. *(umbrella 2.8)*
### Requirement 4: Configuration Change Commit Barrier
**Objective:** As a cluster operator, I want configuration changes to follow the same majority-commit contract as other state changes, so that a failover never reverts a configuration upgrade that was reported as successful.
#### Acceptance Criteria
1. When the primary distributes a configuration change, the primary shall confirm the change to its requester only after a majority of cluster members have acknowledged it. *(umbrella 5.1)*
2. If majority acknowledgment for a configuration change is not obtained within the configured commit timeout, the primary shall report the configuration change as failed to its requester. *(umbrella 5.2)*
3. If a configuration change has not achieved majority acknowledgment, the primary shall not mark that configuration as committed. *(umbrella 5.3)*
4. If a configuration change was reported as failed to its requester, the primary shall not proceed with the restart that activates the new configuration. *(umbrella 5.2)*
### Requirement 5: Idempotent and Ordered Fact Application
**Objective:** As a cluster operator, I want job state transitions to be safe under duplicate and replayed application, so that retained-fact retry, log replay, and future worker rejoin re-synchronization cannot corrupt job state.
#### Acceptance Criteria
1. When a job state transition that is already reflected in the job's current state is applied again, the controller shall leave the job's state unchanged and record the duplicate application in its log. *(umbrella 2.8)*
2. If a job state transition would move a job from a terminal state back to a non-terminal state, the controller shall reject the transition, preserve the terminal state, and log a warning. *(umbrella 2.8)*
3. When a fact's log entry is applied more than once through retry or replay, the resulting job state shall equal the state produced by exactly one application. *(umbrella 2.8)*
## Umbrella Traceability
| This spec | Umbrella ID | Notes |
|-----------|-------------|-------|
| 1.1, 1.4 | 2.1 | Commit-first-then-reply generalized from job submit to all request-shaped operation types |
| 1.2, 1.3 | 2.2 | Requester-awaiting operations fail on commit timeout / not-ready, never silently degrade |
| 2.1 | 2.1, 2.8 | Classification table is the enforcement mechanism for "all operation types" |
| 2.2 | 2.1 | Fail-fast guard: a new/unclassified WAL type cannot silently bypass the barrier (umbrella revalidation trigger) |
| 2.3 | 2.1 | handle_jobdispatch classification per the umbrella Reply-Ordering Contract (receipt-ACK verdict, task-verified) |
| 2.4 | 2.1 | Exhaustive reply-site sweep; a missed handler silently violates umbrella 2.1 |
| 3.1, 3.3 | 2.8 | Log-is-the-queue: RETAIN mode, no timeout, retry until committed (umbrella-locked decision) |
| 3.2 | 2.8 | Facts accepted across quorum-loss periods (closes the bug-396 silent-loss hole) |
| 3.4 | 2.8 | Commitment resumes from the retained log suffix, not from reporter redelivery |
| 4.1 | 5.1 | Config change confirmed only after majority ACK |
| 4.2, 4.4 | 5.2 | Config change fails to requester on timeout; restart sequencing gated on the barrier (roadmap residual) |
| 4.3 | 5.3 | committed_config_tag advances only on majority ACK |
| 5.1, 5.2, 5.3 | 2.8 | Idempotency guards make RETAIN retry/replay safe; shared prerequisite of worker-rejoin-resync |

108
.kiro/specs/ha-commit-barrier-rollout/research.md Normal file
View file

@ -0,0 +1,108 @@
# Research & Design Decisions: ha-commit-barrier-rollout
## Summary
- **Feature**: `ha-commit-barrier-rollout` (sub-spec 2 of 4, ha-consistency-invariants umbrella)
- **Discovery Scope**: Extension (light discovery — umbrella research.md is the primary discovery artifact; this pass verified its anchors against the current tree, 2026-06-11)
- **Key Findings**:
- Every reply-before-commit call site named by the umbrella still exists; line anchors re-verified (table below)
- Call-site verification of the umbrella classification table found two deviations the umbrella anticipated ("verified against call sites during implementation; table is the contract"): WAL_JOB_STOP/RESUME are fact-origin only, and WAL_JOB_UPDATE (absent from the umbrella table) is dual-origin
- The configAck branch of `repl_wait_majority_ack` has the same three counting defects the commit engine fixed for walAck (`> 0` counting, `n_active == 0 → success`, `n_active` denominator)
- The dispatch gate rejects `OPCode_notify` (and update/preason) under quorum loss — the bug-396 silent fact-loss hole is gate-level, not just replication-level
## Verified Code Anchors (2026-06-11, persistent_ha working tree)
| Concern | Anchor (verified by grep/read) | Status vs umbrella audit |
|---|---|---|
| do_jobcontrol_op replies after mutation, async WAL only | job.c:921-934 (reply at 932) | unchanged |
| handle_jobdispatch early reply | job.c:459-491 (reply at 466-471) | umbrella cited 459-471 — still accurate |
| handle_jobsubmit reference implementation | job.c:495-581 (hook rc checked 563-571, reply 573-578) | umbrella cited 563-578 — still accurate |
| do_queuecontrol_op | queue.c:119-132 (reply at 130); queue_ctrl fires AFTER hook under queue WRITE lock (queue.c:76-112, hook at 107) | new precision: lock-scope drives WAIT placement |
| do_hostcontrol_op | worker.c:689-702 (reply at 700) | new anchor |
| handle_worker_join | join.c:33-119 (AFTER hook 104-105, success reply 107-109) | new anchor |
| do_set_preason / do_clr_preason early replies | job.c:1040-1066 (reply at 1050) / job.c:1068+ (reply at 1078) | scheduler-origin facts — receipt-ACK verdict |
| do_update_op family | opupd.c:48/64/88 reply sites | worker-origin facts — receipt-ACK verdict |
| do_notify (no reply, fire-and-forget) | cluster.c:118-157 | unchanged |
| Hook → WAL type map | persist.c:1068-1088 (`detail_to_wal_type`) | classification dispatch insertion point |
| Job hook: submit sync path / fact async path | persist.c:1093-1169 (sync 1107-1129, async 1169) | unchanged |
| Queue / worker hooks (async) | persist.c:1172-1202 (append 1199) / persist.c:1204-1244 (append 1241) | unchanged |
| persist_wal_sync_append + timeout | persist.c:1003-1049; timeout `p->config.sync_timeout_ms` (DEFAULT_SYNC_TIMEOUT_MS=50, config.h:93) | committed_lsn store at 1035-1036 is removed by ha-commit-engine (upstream) |
| Dispatch gate PRIMARY no-quorum allowlist | ha.c:1527 (`ha_dispatch_gate`), allowlist 1537-1556; notify falls to `default → -XCE_ERR_NOT_READY` | fact admission insertion point |
| push_config_to_standbys ACK result ignored | config_repl.c:866-930 (wait at 916-917, rc logged-only 918-927, returns acked at 929) | umbrella cited 916-928 — still accurate |
| upgradeConfig caller ignores result | config_repl.c:1056-1094 (push at 1057, unconditional success ACK 1062-1073, restart 1091-1093) | umbrella cited 1057 — still accurate |
| configAck counting defects | repl.c:386-462 (`acked_config_lsn > 0` at 422-424, `n_active==0 → success` at 432-435, denominator `n_active` at 437) | mirrors the walAck defects fixed upstream |
| repl_on_config_ack stores reported tag/lsn | repl.c:351-371; `standby_conn_t.acked_config_lsn` (repl.h:25) | enables per-push `>= push_tag` comparison |
| mark_job_start/finish/fail/update unguarded | job.c:583/638/696/726 | unchanged |
| job_ctrl branches | job.c:306-438 — running job forwards to worker (357, no local entry); not-started job mutates via mark_job_update (391/401) / mark_job_fail (415) | classification evidence (deviation §2) |
| dispatch_a_job_ produces no WAL | job.c:180-304 — sends xce_startjob_req, mutates j->job.allocs, fires no hooks | handle_jobdispatch receipt-ACK evidence |
| WAL type enum | src/include/wal_types.h:44-58 (no WAL_NOOP yet) | WAL_NOOP arrives with ha-single-leader-role |
| Control-path xce_reply population | job.c:471,524,569,578,932,1050,1078; queue.c:130; worker.c:700; join.c:28,54,66,109,139,152,198,225,248,257,283; opupd.c:48,64,88; cluster.c:255-490; config_repl.c:619-1129; ha.c:1035; opget.c:222; reply.c:61; task.c:97 | sweep-task input; re-grep before editing |
| Opcode → handler dispatch | cluster.c:325-363 (jobsub/jobdispatch/notify/update/jobcontrol/queuecontrol/hostcontrol/setpreason/clrpreason/join) | gate allowlist + sweep scoping |
Upstream interface consumed (ha-commit-engine design, generated 2026-06-11): `commit_submit(ce, lsn, COMMIT_MODE_WAIT|RETAIN, timeout_ms)` → 0 / -ETIMEDOUT / -XCE_ERR_NOT_READY; RETAIN returns immediately; n_peers<=1 self-quorum; admission cap n_vcpus-relative; engine embedded in `cluster_t`, reachable from `persist_t` via `p->repl.ce`. `commit.c`/`commit.h` do not exist in the tree yet — ha-commit-engine must be implemented first (hard upstream dependency; tasks assume its interface as specified).
## Research Log
### §1 Reply-ordering reality and the WAIT placement question
- **Context**: The umbrella Reply-Ordering Contract fixes the handler sequence (validate → mutate → append → commit_submit(WAIT) → reply). Where should the WAIT physically run — inside the persistence hook (like job submit) or in the handler?
- **Findings**: queue/worker/job-state hooks fire while the entity WRITE lock is held (queue.c:107 inside lock taken at 76; mark_job_* called under job WRITE lock, e.g. job.c:629-631). Job submit is the exception: its AFTER hook runs against a job list after per-job locks are released (job.c:553), which is why its in-hook sync wait is safe.
- **Implications**: In-hook WAIT for queue/worker/job-control would block entity readers for up to the commit timeout. Handler-level barrier (after locks released, before reply) keeps the same completion semantics and matches the umbrella's handler-sequence wording. Decision below.
### §2 Classification table call-site verification (umbrella-mandated)
- **Context**: Umbrella table marks WAL_JOB_STOP/RESUME as "request (client job control) → WAIT" and omits WAL_JOB_UPDATE.
- **Findings**: WAL_JOB_STOP/RESUME entries are appended only by `on_job_stop`/`on_job_resume`, reachable solely from `do_notify` (worker-reported stoppedJob/resumedJob facts — cluster.c:131-136). Client job control (`do_jobcontrol_op` → `job_ctrl`) never produces them: for a running job it forwards the signal to the worker (job.c:357) and the resulting transition comes back later as a worker fact; for a not-yet-started job it mutates locally through `mark_job_update` (WAL_JOB_UPDATE) or `mark_job_fail` (WAL_JOB_FAIL). WAL_JOB_UPDATE/WAL_JOB_FAIL are therefore dual-origin (request via job_ctrl, fact via postdone/notify/rusage).
- **Implications**: Classify STOP/RESUME/UPDATE/FAIL as fact/RETAIN at the seam; realize the umbrella's request-shaped guarantee ("error to client") for client job control at the handler barrier in `do_jobcontrol_op`. This satisfies the umbrella's intent through its own prescribed handler sequence — the barrier is a property of the handler, not of the entry type. Not a contradiction: the umbrella explicitly delegated call-site verification, and the failure-column behavior is preserved.
### §3 handle_jobdispatch classification (umbrella left it open)
- **Findings**: `dispatch_a_job_` (job.c:180-304) sends `xce_startjob_req` to the worker and updates `j->job.allocs` in memory; it fires no hooks and appends no WAL entries. Per-job outcomes return to the scheduler via `notify_dispatch` (job.c:440-457, called at 479-486). The job's externally meaningful state change (RUN) arrives later as a worker fact (WAL_JOB_START).
- **Implications**: Per the umbrella's own criterion ("if its outcomes are fact-shaped … the early reply is an ACK-of-receipt and may stand"), the verdict is **receipt-ACK stands** (this spec's Req 2.3). The orphan-dispatch-of-uncommitted-submit race is closed by the egress watermark filter (ha-convergence-recovery), not here. The classification guard plus a no-WAL-from-dispatch test pin the verdict.
### §4 Fact loss is gate-level too (bug-396 family)
- **Findings**: `ha_dispatch_gate`'s CTRL_PRIMARY branch rejects `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` with `-XCE_ERR_NOT_READY` whenever write quorum is absent (ha.c:1537-1556 default arm). The worker never reads notify replies (umbrella research §A), so each rejection is a permanently lost fact.
- **Implications**: Umbrella 2.8 ("retain … including across quorum-loss periods") requires admitting these opcodes without write quorum; durability is then owned by RETAIN. Allowlist change is confined to the PRIMARY branch; the COMMITTING branch is left for ha-single-leader-role's wholesale deletion.
### §5 Config track hardening scope
- **Findings**: configAck counting (repl.c:418-438) shares all three walAck defects; `repl_on_config_ack` already stores the standby's acknowledged tag (repl.c:351-371), so per-push comparison (`>= push_tag`) needs only a parameter, not new state. `committed_config_tag` advance is already success-gated (config_repl.c:918-922).
- **Implications**: Hardening lives in the configAck branch + `push_config_to_standbys` return contract + caller failure path. No engine involvement (umbrella-locked: config is a parallel, simpler track sharing the majority arithmetic rule — one in-flight change at a time).
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Verdict |
|--------|-------------|-----------|---------------------|---------|
| In-hook WAIT for all request types | extend the job-submit sync-append pattern to queue/worker/job-control hooks | one mechanism, no handler edits | blocks commit waits under entity WRITE locks (readers stall up to timeout); job-control hooks fire per-transition, not per-operation | rejected |
| Handler-level barrier (chosen) | hooks append (RETAIN-registered); request handlers call `persist_commit_barrier()` after locks, before reply | matches umbrella handler sequence verbatim; no lock-hold waits; uniform mechanical sweep pattern; covers dual-origin types from request contexts | barrier watermark may over-wait on unrelated concurrent entries (bounded by timeout) | selected |
| Per-entry LSN plumbing (hooks return entry LSNs to handlers) | exact-entry waits | no over-wait | invasive signature changes through the hook system for negligible benefit (over-wait is bounded and rare) | rejected (YAGNI) |
## Design Decisions
### Decision: Handler-level commit barrier on the appended watermark
- **Context**: §1 lock-scope finding vs the umbrella WAIT mode for queue/worker/job-control.
- **Selected Approach**: `persist_commit_barrier(p, timeout_ms)` = snapshot `current_lsn`, `commit_submit(WAIT)` on it; called by each request handler between mutation success and reply.
- **Rationale**: Single-writer appends + contiguous commit advancement (engine postcondition) make the watermark cover everything the operation appended; the umbrella's Reply-Ordering Contract literally specifies the handler sequence.
- **Trade-offs**: Possible over-wait on concurrent unrelated uncommitted entries (bounded by `sync_timeout_ms`; those entries must commit anyway). A no-op control operation may wait behind unrelated traffic — accepted (request-shaped ops are gate-rejected during quorum loss, so the wait resolves quickly in healthy clusters).
- **Follow-up**: Integration test pins reply-after-commit ordering per opcode.
### Decision: Verified classification deviations (STOP/RESUME fact-only; UPDATE dual-origin)
- See Research Log §2. Umbrella conformance preserved through the handler barrier; the verified table in design.md is the implementation contract. Inline umbrella tags kept (2.1/2.8).
### Decision: Production fallback for unclassified WAL types = RETAIN
- **Context**: Req 2.2 requires fail-fast in testing; production must not crash the primary on a programmer omission.
- **Selected Approach**: XCE_TESTING assert; production ERROR log + RETAIN treatment.
- **Rationale**: Fail-safe direction is retention (never dropped, never falsely confirmed) — a misrouted request-shaped type degrades to a missing barrier caught by the sweep audit/tests, not to data loss.
### Decision: Failed config upgrade does not restart
- **Context**: Roadmap residual ("don't restart the primary before the commit barrier passes"); umbrella 5.2.
- **Selected Approach**: `do_upgrade_config_with_bytes` gates the success ACK and `restart_requested` on the barrier result; disk save/generation bump remain pre-barrier (existing sequence), reconciled by the existing OP_replConfig anti-entropy (out of boundary).
- **Trade-offs**: A failed upgrade leaves the new config on the primary's disk but uncommitted cluster-wide; explicit failure to the operator makes the state visible. Full save-after-commit resequencing deferred (would touch checkpoint/restart machinery beyond this slice).
## Risks & Mitigations
- Missed reply site silently violates umbrella 2.1 — exhaustive grep audit task over all `xce_reply` sites + behavioral integration test (no success reply before commit)
- Upstream ha-commit-engine not yet implemented — tasks consume only its published interface; if signatures drift, revalidation trigger fires (boundary section)
- New timeout errors surfacing to scheduler/worker/CLI callers — deliberate CP semantics (umbrella); error codes flow through existing reply envelopes; callers' retry behavior unchanged
- Guard rejections changing `mark_job_*` return codes — sweep checks all callers treat duplicate-as-success-no-op
## References
- `.kiro/specs/ha-consistency-invariants/{requirements,design,research}.md` — authoritative umbrella contract (esp. design §Write-Path Classification, §Config Barrier, §Idempotency Guards; research locked decisions 1, 2, 4)
- `.kiro/specs/ha-commit-engine/design.md` — consumed engine interface (commit.h verbatim), migration note "retired by ha-commit-barrier-rollout"
- `.kiro/steering/roadmap.md` — dependency ordering; superseded item "Quorum-ACK gate before config restart" residual
- `.kiro/steering/ha-async-io.md`, `testing.md`, `error-handling.md` — threading, TDD, errnox conventions
- Raft paper §5.3 (commit rule) — via umbrella; this spec adds no new consensus logic

23
.kiro/specs/ha-commit-barrier-rollout/spec.json Normal file
View file

@ -0,0 +1,23 @@
{
"feature_name": "ha-commit-barrier-rollout",
"created_at": "2026-06-11T09:20:51Z",
"updated_at": "2026-06-11T09:45:00Z",
"language": "en",
"phase": "tasks-generated",
"umbrella": "ha-consistency-invariants",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true
}

106
.kiro/specs/ha-commit-barrier-rollout/tasks.md Normal file
View file

@ -0,0 +1,106 @@
# Implementation Plan: ha-commit-barrier-rollout
> TDD ordering per project steering (`testing.md`): every implementation sub-task is preceded by a testsuite sub-task that lands failing (RED) tests first. New tests live in `testsuite/test_commit_barrier_rollout.c` plus targeted updates to `test_repl_majority.c` — this shared-file reality, plus job.c being touched by three groups, is why no `(P)` markers are applied: the plan is sequential.
> **Hard upstream prerequisite**: ha-commit-engine must be implemented and green first — every task below consumes `commit_submit(WAIT/RETAIN)` from `include/commit.h` and never modifies `commit.c` internals.
> Verify exact line numbers with grep before editing (research.md §Verified Code Anchors has the 2026-06-11 verified table; files drift).
- [ ] 1. Write-path classification: table, dispatch, RETAIN routing
- [ ] 1.1 Write RED tests for classification totality and RETAIN registration
- Create `testsuite/test_commit_barrier_rollout.c` (XCE_TESTING pattern, AAA structure) and register it in `controller_unit_test_src` (meson.build); declare `wal_commit_mode_t` + `wal_classify()` in src/include/persist.h so tests compile
- Tests: every value of `wal_entry_type` (wal_types.h:44-58) maps to a mode — WAL_JOB_SUBMIT/WAL_QUEUE_CTRL → WAIT, WAL_WORKER_CTRL with detail JOIN/UPDATE → WAIT and detail DEAD → RETAIN, all WAL_JOB_* state types → RETAIN, WAL_CHECKPOINT → LOCAL; an unknown type triggers the XCE_TESTING fail-fast path; a fact-type append through `persist_job_hook_fun` registers the entry with the engine in RETAIN mode (observable via the engine's retained/registered accounting against a stub engine) and returns without blocking
- Observable completion: build succeeds with the new header declarations; all new test cases FAIL (red) against the unmodified persist.c
- _Requirements: 2.1, 2.2, 3.1_
- [ ] 1.2 Implement wal_classify and the hook-seam dispatch to green
- Implement `wal_classify(wal_type, detail)` in persist.c per the design's verified classification table; consult it in `persist_job_hook_fun` / `persist_queue_hook_fun` / `persist_worker_hook_fun` (persist.c:1093/1172/1204 anchors): RETAIN types → `persist_wal_async` + `commit_submit(ce, appended_lsn, COMMIT_MODE_RETAIN, 0)` registration; WAIT types → WAL_JOB_SUBMIT keeps the existing sync path, queue/worker appends stay async (handler barrier owes completion, task 3); LOCAL → unchanged
- Unclassified guard: `assert` under XCE_TESTING with the offending type logged; production ERROR + RETAIN fallback
- Fact appends never propagate commit-progress errors to hook callers (local append rc only)
- Observable completion: all task 1.1 tests PASS; grep shows no bare `persist_wal_async` call in the three hook functions that bypasses classification
- _Requirements: 2.1, 2.2, 3.1_
- [ ] 2. Commit barrier helper and legacy adapter retirement
- [ ] 2.1 Write RED tests for persist_commit_barrier semantics
- Declare `persist_commit_barrier(persist_t *p, uint32_t timeout_ms)` in src/include/persist.h
- Tests (stub engine + replicator, n_peers=3 unless noted): returns 0 immediately when `committed_lsn` already covers `current_lsn`; returns `-ETIMEDOUT` when ACKs are withheld past the timeout and `committed_lsn` stays unmoved; returns 0 once a covering ACK advances the commit index; `timeout_ms = 0` falls back to `p->config.sync_timeout_ms`; n_peers=1 returns 0 immediately (standalone preserved); `-XCE_ERR_NOT_READY` from the engine passes through unmodified
- Observable completion: new cases FAIL (red) — the helper does not exist yet
- _Requirements: 1.1, 1.2, 1.3_
- [ ] 2.2 Implement the barrier helper and retire the legacy walAck counting
- Implement `persist_commit_barrier` in persist.c: snapshot `current_lsn` under `p->lock` (mirror persist.c:1019-1021), short-circuit when covered, else `commit_submit(ce, lsn, COMMIT_MODE_WAIT, timeout)`; return engine code unmodified
- repl.c `repl_wait_majority_ack` (repl.c:386-462 anchor): remove the legacy walAck counting branch (upstream marked it "retired by ha-commit-barrier-rollout") — the walAck path delegates to the engine unconditionally; the function's remaining body is the configAck track (hardened in task 5)
- Observable completion: all task 2.1 tests PASS; grep shows no remaining `acked_lsn > 0` counting and no `n_active == 0 → success` short-circuit reachable from any walAck call path
- _Requirements: 1.1, 1.2, 1.3_
- [ ] 3. Reply-ordering restructure of request-shaped handlers
- [ ] 3.1 Write RED integration tests for commit-first-then-reply per opcode
- Tests drive the four handlers directly (vtask-context test harness, stub standby registry): for each of `do_jobcontrol_op` (not-started-job branch, producing WAL_JOB_UPDATE/WAL_JOB_FAIL), `do_queuecontrol_op`, `do_hostcontrol_op`, `handle_worker_join` — with ACKs withheld the requester reply carries an error (`-ETIMEDOUT`) and no success reply is ever observable while the operation's entries exceed `committed_lsn`; with ACKs supplied the success reply is observable only after `committed_lsn` covers the entries; local validation failures still reply immediately with their existing error codes
- Observable completion: new cases FAIL (red) against the current reply-before-commit handlers (anchors: job.c:932, queue.c:130, worker.c:700, join.c:107-109 — re-verify with grep)
- _Requirements: 1.1, 1.2, 1.3, 1.4_
- [ ] 3.2 Restructure the four handlers to green
- Apply the uniform pattern: `rc = <ctrl fn>; if (rc == 0) rc = persist_commit_barrier(&c->persist, 0); reply.status = rc; xce_reply(...)` — in `do_jobcontrol_op` (job.c:921-934), `do_queuecontrol_op` (queue.c:119-132), `do_hostcontrol_op` (worker.c:689-702); in `handle_worker_join` (join.c:33-119) insert the barrier between the AFTER_EVENT hook and the success reply, keep error replies immediate, keep `on_worker_join` after the reply
- Barrier runs after entity locks are released in every handler (the ctrl fns unlock before returning)
- Observable completion: all task 3.1 tests PASS; `handle_jobsubmit` untouched (reference implementation)
- _Requirements: 1.1, 1.2, 1.3, 1.4_
- [ ] 3.3 Pin the receipt-ACK verdicts with tests
- Tests: `handle_jobdispatch` (job.c:459-491) replies immediately AND dispatch processing appends no WAL entries (classification guard stays silent through a dispatch cycle; `notify_dispatch` outcomes unaffected); `do_set_preason`/`do_clr_preason` and the `do_update_op` rusage path remain receipt-style replies whose subsequent fact appends are RETAIN-registered
- Document the verdict in the test comments with the umbrella Reply-Ordering Contract reference
- Observable completion: verdict tests PASS without modifying those handlers; any future WAL append from dispatch processing trips the classification guard in CI
- _Requirements: 2.3_
- [ ] 4. Fact admission during quorum loss and retention proof
- [ ] 4.1 Write RED tests for the dispatch-gate fact allowlist
- Tests against `ha_dispatch_gate` (ha.c:1527 anchor) with CTRL_PRIMARY and write quorum absent: `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` return 0; `OPCode_jobsub`, `OPCode_jobcontrol`, `OPCode_queuecontrol`, `OPCode_hostcontrol`, `OPCode_jobdispatch`, `OPCode_join` still return `-XCE_ERR_NOT_READY`; with quorum present everything returns 0 as today
- Observable completion: the four fact-opcode cases FAIL (red) against the current allowlist
- _Requirements: 3.2_
- [ ] 4.2 Implement the allowlist to green
- Add the four fact-bearing opcodes to the CTRL_PRIMARY no-write-quorum switch (ha.c:1537-1556 anchor); CTRL_PRIMARY_COMMITTING branch untouched (deleted wholesale by ha-single-leader-role)
- Observable completion: all task 4.1 tests PASS; existing gate tests for request opcodes unchanged
- _Requirements: 3.2_
- [ ] 4.3 Write the fact-retention-across-quorum-loss integration test
- Scenario (stub registry, n_peers=3): disconnect standbys (quorum lost) → drive worker-fact paths (job start/finish via `do_notify`-level calls) → appends succeed with no error, entries pending (`lsn > committed_lsn`), nothing dropped; reconnect + supply covering ACKs → `committed_lsn` advances over the retained suffix with no re-delivery of the facts; assert no timeout was ever applied to the fact entries
- Observable completion: test PASSES against tasks 1.2 + 4.2 output (if it fails, fix the RETAIN/gate path, not the test); retained-depth observability counter visible in test logs
- _Requirements: 3.1, 3.3, 3.4_
- [ ] 5. Configuration change commit barrier
- [ ] 5.1 Write RED tests for configAck arithmetic and the upgrade failure path
- Unit tests (repl.c configAck branch via `repl_wait_majority_ack`): a standby's `acked_config_lsn` from a previous tag does not count toward the current push; n_peers=3 with zero covering acks → `-ETIMEDOUT` (the `n_active == 0 → success` short-circuit dies on this branch); self + one covering standby (majority of n_peers=3) → 0; n_peers=1 → immediate 0; update `test_repl_majority.c` configAck cases to the per-push arithmetic
- Integration tests (config_repl.c): `push_config_to_standbys` returns `-ETIMEDOUT` with acks withheld and 0 with majority acks, push count via out-param; on failure the `upgradeConfig` requester receives an error reply (send_upgrade_error shape), `committed_config_tag` is unmoved, and `restart_requested` stays clear; on success the existing reply + restart behavior is unchanged
- Observable completion: new cases FAIL (red) against the current logged-only result (config_repl.c:916-929) and unconditional success ACK (config_repl.c:1062-1073)
- _Requirements: 4.1, 4.2, 4.3, 4.4_
- [ ] 5.2 Implement the config barrier to green
- repl.c: configAck branch takes the in-flight push tag (signature change in src/include/repl.h), counts `acked_config_lsn >= push_tag`, majority `1 + count > n_peers / 2` over configured members, `n_peers <= 1` immediate success
- config_repl.c: `push_config_to_standbys` gains `int *acked_out` and returns the barrier result, passing `new_tag` to the wait; `do_upgrade_config_with_bytes` (config_repl.c:1056-1094 anchor) on non-zero return sends `send_upgrade_error`, skips the success ACK and skips the restart request; `committed_config_tag` advance stays success-branch-only
- Observable completion: all task 5.1 tests PASS; a failed upgrade leaves the admin tool with an explicit error and the primary running on the old committed tag
- _Requirements: 4.1, 4.2, 4.3, 4.4_
- [ ] 6. Idempotency and ordering guards for job transitions
- [ ] 6.1 Write RED tests for mark_job_* guards
- Tests: duplicate FINISH on a DONE job → state unchanged, INFO-level no-op, no AFTER_EVENT hook fired (no WAL entry appended); START on a DONE/EXIT job → rejected with WARN, terminal state preserved; UPDATE attempting to clear terminal bits → rejected; double application of the same transition yields state identical to single application (replay equivalence); fail-after-finish keeps the existing PERR-merge behavior (job.c:702-716 anchor); guard rejections return a code all existing callers treat as success-no-op
- Observable completion: new cases FAIL (red) against the unguarded `mark_job_start/finish/fail/update` (job.c:583/638/696/726 anchors)
- _Requirements: 5.1, 5.2, 5.3_
- [ ] 6.2 Implement the guards to green
- Add transition guards at the top of each `mark_job_*` (under the caller-held job WRITE lock, before BEFORE_EVENT fires): duplicate → log + return no-op; terminal → non-terminal → WARN + reject preserving state; forward transitions unchanged
- Audit all callers (`on_job_*`, `job_ctrl`, `update_job_rusages`) for the new return semantics
- Observable completion: all task 6.1 tests PASS; existing job lifecycle tests green
- _Requirements: 5.1, 5.2, 5.3_
- [ ] 7. Exhaustive reply-site sweep and final validation
- [ ] 7.1 Run the exhaustive xce_reply audit over all control paths
- Grep ALL `xce_reply` call sites under src/controller/ (excluding testsuite/) — research.md lists the design-time population (job.c, queue.c, worker.c, join.c, opupd.c, cluster.c, config_repl.c, ha.c, opget.c, reply.c, task.c); re-run the grep fresh since lines drift
- Classify every site into exactly one category: post-commit confirmation (restructured in task 3 / job submit / config success), receipt-only ACK for fact-shaped delivery (verdicts from task 3.3), or read-only/replication/error reply (no state-changing entries produced)
- Fix any site that confirms a request-shaped state change before commit (a missed handler silently violates umbrella 2.1); commit the audit table (site → category → evidence) in the task completion notes
- Observable completion: zero unclassified or violating reply sites in the recorded audit; any fixes covered by tests added to `test_commit_barrier_rollout.c`
- _Requirements: 2.4, 1.1, 1.4_
- [ ] 7.2 Full regression and barrier-completeness validation
- Build and run the complete controller testsuite with ASAN (`/xce-build controller`); fix regressions from the reply restructure, gate allowlist, configAck hardening, or guards
- End-to-end ordering assertion: across all restructured opcodes in one run, no success reply is observable while the operation's entries exceed `committed_lsn` (integration harness from 3.1 driven as a suite); fact retention test (4.3) and config barrier tests (5.1) green in the same run
- Confirm downstream readiness: classification guard active, sweep audit recorded — the hard precondition for ha-single-leader-role ("barrier before state removal") is satisfied
- Observable completion: full testsuite green under ASAN; audit + ordering evidence recorded
- _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.3, 4.4, 5.1, 5.2, 5.3_