myxce/.kiro/specs/ha-consistency-invariants/design.md

Design: ha-consistency-invariants

Overview

Purpose: This feature gives operators of multi-node XCE controller clusters a uniform Raft+CP consistency contract: every cluster-state-changing operation is durable on a majority before it is confirmed, operations that cannot reach majority are rejected (requests) or retained until committed (facts), and leadership is a single role with no weaker transitional state.

Users: Cluster operators gain predictable failover semantics (no acknowledged change is ever lost); client/API consumers gain honest confirmations; the scheduler is protected from acting on uncommitted state.

Impact: Removes CTRL_PRIMARY_COMMITTING from the HA state machine; generalizes the majority-ACK commit barrier from WAL_JOB_SUBMIT to all state-changing WAL types via a new commit engine; fixes quorum accounting (per-entry ACK verification, catch-up exclusion, TOCTOU closure); adds a commit-watermark egress filter to dataex; converts demotion into truncate-and-self-restart; makes catch-up ACK-driven.

Goals

• Majority ACK as the hard commit barrier for all state-changing operations (committed_lsn ≈ Raft commitIndex)
• Single leader role; post-election write safety via own-term no-op commit (Raft §5.4.2)
• Facts (worker-reported transitions) retained and retried until committed — never dropped while the primary runs and leads
• Cluster-wide single-outcome convergence for uncommitted suffixes (truncate + self-restart on demotion)
• Full fan-out with leader-driven, ACK-continued catch-up

Non-Goals

• Election/vote correctness (raft-library), WAL wire format (ha-wal-replication)
• Read consistency / leases — status queries may observe provisional in-memory state (documented deviation)
• Worker rejoin job-state re-sync (roadmap backlog; designated backstop for facts lost across failover)
• Entity-bearing snapshots; primary step-down on quorum loss; checkpoint replication
• Performance tuning of commit latency (measure only)

Boundary Commitments

This Spec Owns

• Commit semantics: when committed_lsn / committed_config_tag advance, and what confirmation means for every state-changing operation on the primary
• The commit engine (commit.c): per-LSN ACK evaluation, waiter completion, own-term gate, admission cap
• Operation classification (request-shaped vs fact-shaped) and the per-WAL-type write path
• HA leader-role lifecycle simplification (promotion no-op, COMMITTING removal, demotion truncate+restart)
• Quorum accounting used by the dispatch gate and commit evaluation
• The commit-watermark egress rule for scheduler-bound exports
• Quorum-loss observability logging

Out of Boundary

• Vote RPCs, term management, freshness gates (raft-library); WalAckReply wire format and standby apply mechanics (ha-wal-replication)
• Worker protocol changes of any kind (rejoin re-sync is a future spec)
• Snapshot file format (sentinel stays as-is); dual-zarray COW wiring
• apiserver/Go-tier behavior (error codes pass through existing envelopes)

Allowed Dependencies

• repl.c standby registry and acked_lsn per-standby tracking (ha-wal-replication machinery) — the commit engine consumes ACK events, never manages connections
• persist.c WAL append/replay/truncation primitives; persist_recover boot path (reused by self-restart)
• iomgr/vtaskmgr/delaymgr threading rules per ha-async-io.md (ACK processing stays on iomgr; blocking waits only on vtask threads)
• Dependency direction: include/commit.h depends only on libc/stdatomic types. commit.c is called BY persist.c (append), repl.c (ACK events), and ha.c (promotion/demotion); it calls into none of them except via registered callbacks.

Revalidation Triggers

• Any change to acked_lsn semantics or WalAckReply payload (ha-wal-replication) → re-verify commit evaluation
• Worker-rejoin-resync spec landing → re-verify idempotency guards and fact replay ordering
• Entity-bearing snapshots landing → re-evaluate demotion self-restart vs snapshot-based rebuild
• Any new state-changing opcode/WAL type → must be classified request/fact and routed through the commit engine

Architecture

Existing Architecture Analysis

• Single-writer model: only the primary appends WAL entries; standbys ACK (acked_lsn = matchIndex equivalent already exists per standby_conn_t)
• Hooks fire on HOOK_AFTER_EVENT after in-memory mutation — retained as-is under the provisional-state model (in-memory state is a provisional view; the committed log is authoritative; commit gates confirmation and irrevocable egress, not mutation)
• ACKs are decoded on the iomgr thread (peer_ctrl_msg_handler → repl_on_ack) — blocking commit-waiters can always be woken; pool exhaustion is throughput-only
• repl_wait_majority_ack currently counts acked_lsn > 0 (any ACK ever) — replaced by per-entry evaluation in the commit engine
• Technical debt addressed: TOCTOU between dispatch gate and ACK; catch-up standbys counted toward write quorum; config push ACK result ignored; committing_timeout_fn force-promotion

Architecture Pattern & Boundary Map

graph TB
    subgraph clients
        API[apiserver client]
        WK[worker notify]
        SCHED[scheduler]
    end
    subgraph controller_primary
        GATE[ha dispatch gate]
        HOOKS[persist hooks]
        WAL[persist WAL append]
        CE[commit engine]
        REPL[repl broadcast]
        DX[dataex egress filter]
        HA[ha role lifecycle]
    end
    STBY[standbys]

    API --> GATE
    WK --> GATE
    GATE --> HOOKS
    HOOKS --> WAL
    WAL --> CE
    WAL --> REPL
    REPL --> STBY
    STBY -- WalAck on iomgr --> CE
    CE -- committed_lsn --> DX
    DX --> SCHED
    HA -- promote noop / demote restart --> CE

Architecture Integration:

• Selected pattern: single commit engine as the authority for committed_lsn advancement; all write paths converge on it with one of two completion modes
• Existing patterns preserved: hook-based persistence dispatch, full fan-out broadcast, heartbeat-driven repl decision loop (extended with ACK-driven continuation), errnox early-return error style
• New component rationale: commit.c exists because three requirements families (2.x, 3.x, 5.x) are one capability; placing it in repl.c would bloat connection management with consensus logic
• Steering compliance: never block iomgr; blocking waits only on vtask threads with admission cap relative to n_vcpus; TDD with testsuite/ tests first

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Backend / Services	C (gnu17), pthreads, stdatomic	commit engine, HA lifecycle	no new dependencies
Data / Storage	existing WAL segments (persist.c)	log = authoritative state + fact queue	new WAL_NOOP entry type, no schema change
Messaging / Events	existing Cap'n Proto WalAckReply	ACK transport	no .capnp changes

File Structure Plan

Directory Structure

src/controller/
├── commit.c                      # NEW — commit engine: per-LSN ACK evaluation, waiter registry,
│                                 #       own-term gate, admission cap, commit-index advancement
├── include/commit.h              # NEW — commit engine public API (controller-internal)
├── ha.c                          # MODIFIED — promotion/demotion lifecycle, gate, quorum accounting
├── ha.h                          # MODIFIED — remove CTRL_PRIMARY_COMMITTING, add own-term fields
├── persist.c                     # MODIFIED — write-path classification, WAL_NOOP, commit-engine calls
├── repl.c                        # MODIFIED — repl_on_ack feeds commit engine; legacy wait removed
├── repl.h                        # MODIFIED — ack callback signature
├── config_repl.c                 # MODIFIED — config barrier error propagation; ACK-driven catch-up
├── job.c                         # MODIFIED — submit_lsn tagging, mark_job_* idempotency guards
├── dataex.c                      # MODIFIED — egress watermark filter in collect path
│                                 #   (Note: as-built, the filter predicate/wrapper live in a
│                                 #   new file egress.c instead — see ha-convergence-recovery)
└── testsuite/
    ├── test_commit_engine.c      # NEW — per-LSN majority, own-term gate, admission cap, monotonicity
    ├── test_ha_role_lifecycle.c  # NEW — direct promotion, no-op commit, demotion truncate+restart seam
    └── test_egress_filter.c      # NEW — uncommitted submit excluded from collect
src/include/persist.h             # MODIFIED — WAL_NOOP type, classification enum

Modified Files (what and why)

• ha.c/ha.h — delete COMMITTING state, watchdog (committing_timeout_fn), ha_check_majority_convergence, repl_ack_count; promotion appends no-op; ha_has_write_quorum excludes catch-up standbys; demotion path truncates + self-restarts; quorum-loss transition logging
• persist.c / include/persist.h — per-WAL-type classification table drives sync-wait vs fact-register; WAL_NOOP append/replay (replay = no-op); truncation unchanged
• repl.c/repl.h — repl_on_ack forwards (ep, acked_lsn) to commit_on_ack; repl_wait_majority_ack's counting logic retired in favor of commit-engine evaluation (function shell may remain as adapter)
• config_repl.c — push_config_to_standbys returns -ETIMEDOUT on missed majority; caller fails the upgrade request; ha_stream_wal_to_standby continuation on walAck instead of next heartbeat
• job.c — job_t gains submit_lsn; mark_job_* gain state-transition guards (no DONE→RUN, idempotent re-apply)
• dataex.c — collect_jobs (scheduler export path only) skips jobs with submit_lsn > committed_lsn (as-built: this predicate/wrapper live in new file egress.c instead — see the Egress Filter section note below)

System Flows

Request-shaped write (e.g., job submit)

sequenceDiagram
    participant C as client
    participant V as vtask thread
    participant CE as commit engine
    participant R as repl broadcast
    participant S as standby
    participant IO as iomgr thread
    C->>V: state-changing request
    V->>V: gate check incl write quorum
    V->>V: mutate in-memory (provisional)
    V->>CE: append WAL entry lsn=N, mode=WAIT
    CE->>CE: admission cap check
    V->>R: broadcast entry N
    R->>S: wal entry N
    V->>CE: block on commit(N) with timeout
    S->>IO: WalAck lsn>=N
    IO->>CE: commit_on_ack
    CE->>CE: majority for N? advance committed_lsn
    CE-->>V: wake waiter (committed | timeout)
    alt committed
        V->>C: success
    else timeout or quorum lost
        V->>C: error (operation failed)
    end

Key decisions: admission cap rejects (NOT_READY) before blocking when inflight_waits >= n_vcpus - COMMIT_WAIT_RESERVED; majority is evaluated against the entry's LSN (acked_lsn >= N per standby, plus self), re-verified at ACK time — gate-time connectivity never substitutes (4.2, 4.3).

Fact-shaped write (worker-reported transition) — log-is-the-queue

Same path until the commit engine: mode=RETAIN. No blocking wait, no timeout; the vtask returns immediately after append+broadcast. The uncommitted suffix is re-delivered to lagging standbys by catch-up streaming; committed_lsn advances as ACKs arrive. The fact is "pending" purely by virtue of lsn > committed_lsn (2.8). Idempotency guards in mark_job_* make replay/duplicate application safe.

Role lifecycle (after this spec)

stateDiagram-v2
    [*] --> INIT
    INIT --> STANDBY: heartbeat seen
    INIT --> ELECTING: timeout
    ELECTING --> PRIMARY: quorum won, truncate to safe_lsn, append noop
    ELECTING --> STANDBY: lost or higher term
    PRIMARY --> RESTART: higher term seen, truncate above committed_lsn
    STANDBY --> ELECTING: primary lost
    RESTART --> INIT: process self restart via boot recovery

Promotion: raft_cb_become_leader sets CTRL_PRIMARY directly, registers persistence hooks, appends a WAL_NOOP entry of the current term, and broadcasts it. Request-shaped writes are accepted immediately and simply wait behind the contiguous commit index (the no-op commits first); status queries report primary (1.4). No wall-clock promotion/force path exists (1.5, 3.3). (Note: promotion is no longer unconditional — raft_cb_become_leader now runs several eligibility/fencing gates (recovery-truncated, apply-divergent, config serve-fence, etc.) that can refuse promotion and leave the node alive, non-promotable, and retrying instead of becoming primary; added by ha-commit-durability and ha-term-safety, see those specs' design.md.)

Demotion: on observing a higher term while PRIMARY, the node truncates its WAL above committed_lsn, logs the event, and self-restarts; boot recovery (persist_recover + replay of the committed prefix) rebuilds in-memory state, discarding provisional entries (2.5, 2.7). Crash without truncation: the restarted node replays its full local WAL. (Note: the "rejoin as standby ... truncates + self-restarts once" rejoin-reconciliation mechanism described here is superseded by ha-term-safety (task 3.2) — ha_rejoin_reconcile is now a no-op stub; a rejoining node's divergent suffix is instead truncated precisely via the raft library's term-aware on_truncate_from driven off real AppendEntries (term,lsn) matching, not an LSN-watermark self-restart. See ha-term-safety/design.md.)

Requirements Traceability

Requirement	Summary	Components	Flows
1.1, 1.2	direct PRIMARY, single leader role	HA Role Lifecycle	role lifecycle
1.3	no redirect from primary	HA Role Lifecycle (ha_should_redirect)	—
1.4	status reports primary pre-own-term-commit	HA Role Lifecycle	role lifecycle
1.5	no wall-clock leadership change	HA Role Lifecycle (watchdog deleted)	—
2.1	confirm only after majority ACK	Commit Engine, Write-Path Classification	request write
2.2	requester ops fail on commit timeout	Commit Engine (WAIT mode), Write-Path Classification	request write
2.3	no commit-index advance without majority	Commit Engine	request write
2.4	provisional state: no confirm, no irrevocable export	Commit Engine, Egress Filter	request write
2.5	single outcome convergence incl. memory	Demotion Convergence	role lifecycle
2.6	monotonic commit index	Commit Engine	—
2.7	state reconstructible from committed log	Demotion Convergence (boot recovery reuse)	role lifecycle
2.8	facts retained+retried until committed	Commit Engine (RETAIN mode), Idempotency Guards	fact write
3.1	no confirmation before own-term entry commits	Commit Engine own-term gate, no-op at promotion	role lifecycle
3.2	pre-own-term requests: barrier or NOT_READY	Commit Engine own-term gate	request write
3.3	no timeout bypassing 3.1	HA Role Lifecycle (watchdog deleted)	—
4.1	quorum counts only members able to receive	Quorum Accounting	—
4.2	commit requires actual per-entry ACKs	Commit Engine	request write
4.3	post-acceptance quorum shrink → fail	Commit Engine (ACK-time evaluation, timeout)	request write
4.4	never zero-peer majority when n_peers>1	Commit Engine	—
5.1–5.3	config change commit barrier	Config Barrier	—
6.1	replicate to all reachable standbys	existing repl_broadcast (unchanged, verified by test)	—
6.2	catch-up without per-step heartbeat	ACK-Driven Catch-up	—
6.3, 6.4	snapshot fallback, re-include on catch-up	existing paths (unchanged, verified by test)	—
6.5	lag never blocks majority confirmation	Commit Engine (majority threshold only)	—
7.1	reject state changes without quorum	Quorum Accounting + Commit Engine	—
7.2	log quorum-loss enter/exit	Quorum Observability	—
7.3	serve status + replication while degraded	dispatch gate (existing allowlist, verified by test)	—
7.4	auto-resume on quorum restore	Quorum Accounting (stateless gate re-evaluation)	—
7.5	standalone self-quorum unchanged	Commit Engine (n_peers<=1 short-circuit preserved)	—

Components and Interfaces

Component	Layer	Intent	Req Coverage	Key Dependencies	Contracts
Commit Engine	controller/commit.c	authority for commit-index advancement and completion	2.1–2.4, 2.6, 2.8, 3.1, 3.2, 4.2–4.4, 6.5, 7.5	repl ACK events (P0), persist append (P0)	Service, State
Write-Path Classification	controller/persist.c	route each WAL type to WAIT or RETAIN mode	2.1, 2.2, 2.8	Commit Engine (P0), hooks (P0)	Service
HA Role Lifecycle	controller/ha.c	direct promotion + no-op, COMMITTING removal, redirect fix	1.1–1.5, 3.1, 3.3	raft-library callbacks (P0), Commit Engine (P0)	State
Quorum Accounting	controller/ha.c	honest gate-time quorum, catch-up exclusion	4.1, 7.1, 7.4	repl registry (P0)	Service
Config Barrier	controller/config_repl.c	config majority-ACK with error propagation	5.1–5.3	Commit Engine pattern (P1)	Service
Egress Filter	controller/egress.c, job.c	exclude uncommitted submits from scheduler export	2.4	committed_lsn (P0)	Service
Demotion Convergence	controller/ha.c	truncate + self-restart; rejoin reconciliation	2.5, 2.7	persist truncation + recovery (P0)	State
ACK-Driven Catch-up	controller/config_repl.c	continue streaming on walAck	6.2	repl/iomgr (P0)	Event
Quorum Observability	controller/ha.c	log degraded-state transitions	7.2	—	—
Idempotency Guards	controller/job.c	safe duplicate/replay of fact transitions	2.8 (supports 2.5, future resync)	—	Service

Commit Engine (commit.c / include/commit.h)

Field	Detail
Intent	Single authority for per-entry commit evaluation, committed_lsn advancement, and waiter completion
Requirements	2.1–2.4, 2.6, 2.8, 3.1, 3.2, 4.2–4.4, 6.5, 7.5

Responsibilities & Constraints

• Owns advancement of persist->committed_lsn (the atomic stays in persist_t; only the commit engine stores to it after init/truncation paths)
• Commit rule (Raft §5.3): entry N is committed when count(self + standbys with acked_lsn >= N) > n_peers / 2; advancement is contiguous and monotonic (2.6). (Note: superseded by ha-commit-durability — the primary's own self-contribution is now capped at persist_durable_lsn (fdatasync'd locally), not the appended tip, so an entry only in the page cache cannot count self toward majority; see that spec's design.md.)
• Own-term gate: writable_lsn = LSN of the promotion no-op; completion (confirmation) is withheld for any entry until committed_lsn >= writable_lsn (3.1, 3.2)
• Admission cap: at most n_vcpus - COMMIT_WAIT_RESERVED (default reserved = 2) concurrent WAIT-mode blockers; excess requests fail fast with -XCE_ERR_NOT_READY. Expressed relative to n_vcpus only — never absolute (operator tunes --vcpus). (Note: superseded in part by raft-drive-async — when a vtaskmgr pool is wired via commit_set_pool, admission folds onto the pool's single VTASK_LIVENESS_RESERVED reservation instead of the local COMMIT_WAIT_RESERVED constant; the local cap is the fallback only when no pool is wired. See that spec's design.md.)
• n_peers <= 1: self-ACK constitutes majority; WAIT mode returns immediately (7.5). For n_peers > 1, zero peer ACKs never commit (4.4) — the legacy n_active == 0 → success short-circuit is removed

Dependencies

• Inbound: repl_on_ack (iomgr thread) — ACK events (P0); persist_wal_* append paths — entry registration (P0); ha.c — promotion no-op LSN, demotion reset (P0)
• Outbound: none (callbacks only); signals waiters via condvar broadcast with per-LSN predicates

Contracts: Service [x] / State [x]

Service Interface

/* include/commit.h — all functions thread-safe; _on_ack callable from iomgr (non-blocking) */
typedef enum { COMMIT_MODE_WAIT, COMMIT_MODE_RETAIN } commit_mode_t;

int  commit_engine_init(commit_engine_t *ce, persist_t *p, int n_peers, int n_vcpus);
void commit_engine_fini(commit_engine_t *ce);

/* Register entry at append time. RETAIN returns 0 immediately.
 * WAIT blocks up to timeout_ms: 0 = committed; -ETIMEDOUT = not committed
 * (entry remains in log; outcome resolves per 2.5); -XCE_ERR_NOT_READY =
 * admission cap exceeded or own-term gate unmet at entry. */
int  commit_submit(commit_engine_t *ce, uint64_t lsn, commit_mode_t mode,
                   uint32_t timeout_ms);

/* iomgr path: record standby ACK, evaluate majority, advance committed_lsn
 * contiguously, wake satisfied waiters. Never blocks. */
void commit_on_ack(commit_engine_t *ce, int standby_idx, uint64_t acked_lsn);

/* Promotion: set own-term gate to the no-op entry's LSN. */
void commit_set_writable_lsn(commit_engine_t *ce, uint64_t noop_lsn);

/* Quorum-membership change (standby add/remove/catch-up transition):
 * re-evaluate pending entries; standbys in catch-up are excluded. */
void commit_on_membership_change(commit_engine_t *ce);

• Preconditions: entries submitted in LSN order (single-writer guarantees this)
• Postconditions: committed_lsn never decreases; never exceeds highest majority-acked contiguous LSN
• Invariants: no waiter completes successfully while committed_lsn < its lsn or committed_lsn < writable_lsn

State Management

• State: per-standby view delegated to standby_conn_t.acked_lsn (no duplication); internal: sorted matchIndex scratch array, waiter list (lsn, condvar shared, mode), writable_lsn, inflight-wait counter
• Concurrency: one mutex + one condvar (broadcast on committed_lsn advance; waiters check per-LSN predicate); committed_lsn read lock-free via existing atomic

Implementation Notes

• Integration: repl_wait_majority_ack becomes a thin adapter calling commit_submit(WAIT) during migration, then call sites move to the engine directly
• Validation: assert monotonicity under XCE_TESTING; counter for retained-uncommitted entries exposed in logs
• Risks: condvar broadcast storms under high write concurrency — acceptable at current scale; revisit with async continuation (out of scope)

Write-Path Classification (persist.c)

Field	Detail
Intent	Route every state-changing WAL type to its commit mode at the existing hook seam
Requirements	2.1, 2.2, 2.8

Classification (verified against call sites during implementation; table is the contract):

SUPERSEDED IN PART (2026-06-11, cross-spec review): call-site verification during sub-spec generation found WAL_JOB_STOP/RESUME/UPDATE/FAIL originate from worker notify (fact-shaped, RETAIN) — the client-control intent of stop/resume is realized by the handler-level barrier in do_jobcontrol_op per the Reply-Ordering Contract below, which stands unchanged. The verified controlling table lives in ha-commit-barrier-rollout/design.md §Classification (see also its research.md §2). Rows below marked with the original assumption are retained for history:

WAL type	Shape	Mode	Failure behavior
WAL_JOB_SUBMIT	request (client)	WAIT	error to client (existing path)
WAL_JOB_STOP / RESUME	request (client job control)	WAIT	error to client
WAL_QUEUE_CTRL	request (client)	WAIT	error to client
WAL_WORKER_CTRL join/update	request (worker, replied)	WAIT	error reply to worker
WAL_JOB_START / FINISH / FAIL / RUSAGE / CKPT / PREASON	fact (worker notify)	RETAIN	none — retained until committed
WAL_WORKER_CTRL dead	fact (controller-detected)	RETAIN	none
WAL_NOOP	internal (promotion)	RETAIN	none — gates writability when committed
WAL_CHECKPOINT	local-only	none	unchanged (not replicated, by design)

Note (rows added by later specs, not in the original table): WAL_JOB_ARRAY_SUBMIT — request (client array submit), WAIT, single commit barrier per array regardless of size (job-array-single-wal-record task 1.1). WAL_CONFIG_MARKER — RETAIN, the consensus/ordering record for a config change appended to the same unified log (ha-term-safety task 4.2; see also the Config Barrier note below). Current full table verified against wal_classify (persist.c).

Reply-Ordering Contract (commit-first, then reply)

• For every request-shaped operation (job control, queue control, worker join/update), the handler sequence is fixed: validate → mutate (provisional) → WAL append → commit_submit(WAIT) → reply. No success reply may be sent to the requester before commit_submit returns 0; on non-zero return the handler replies with the error (2.1, 2.2).
• Known reply-before-commit call sites to restructure: do_jobcontrol_op (job.c:2198-2223 — line numbers current as of this audit, job.c has grown substantially since spec close) and any queue/worker handlers following the same pattern. handle_jobsubmit (job.c:1412 onward — line numbers current as of this audit) is the reference implementation of the correct ordering. (Note: do_jobcontrol_op is now fixed — it calls persist_commit_barrier before replying, per ha-commit-barrier-rollout Req 1.1-1.4; this row is historical, not an open item.)
• handle_jobdispatch (job.c:1119-1131 — line numbers current as of this audit — replies status=0 to the scheduler before processing allocations) is classified during implementation: if its allocations produce request-shaped state changes, the early reply must move behind the barrier; if its outcomes are fact-shaped (per-job dispatch results delivered via notify_dispatch), the early reply is an ACK-of-receipt and may stand. Task-level verification required; the classification assert enforces the outcome. (Confirmed as-built: outcomes are fact-shaped via notify_dispatch, so the early reply stands unchanged.)

Implementation Notes

• Integration: persist_job_hook_fun / persist_queue_hook_fun / persist_worker_hook_fun consult the table; persist_wal_async survives only as the RETAIN fast path (append + broadcast + register)
• Validation: a debug assert rejects any unclassified WAL type (revalidation trigger for new opcodes); integration tests assert no success reply is observable before committed_lsn covers the operation's entry
• Risks: reply restructuring touches every control handler in job.c/queue paths — mechanical but must be swept exhaustively; a missed site silently violates 2.1

HA Role Lifecycle (ha.c / ha.h)

Field	Detail
Intent	Direct ELECTING→PRIMARY with own-term no-op; remove the transitional state and its timeout
Requirements	1.1–1.5, 3.1, 3.3

Responsibilities & Constraints

• raft_cb_become_leader: truncate to safe_committed_lsn (existing), set CTRL_PRIMARY, register persistence hooks (moved from ha_promote_to_primary), append+broadcast WAL_NOOP, call commit_set_writable_lsn(noop_lsn)
• Deletions: CTRL_PRIMARY_COMMITTING enum value, dispatch-gate COMMITTING branch, committing_timeout_fn + watchdog arming, ha_check_majority_convergence, repl_ack_count / ha_repl_ack_received; ha_is_primary / ha_should_redirect / vote-deny checks reduce to the single PRIMARY role
• Heartbeat re-arm condition drops the COMMITTING term; standby state reports continue to drive the catch-up decision loop unchanged

Contracts: State [x] — role transitions per the lifecycle diagram; no other component may observe COMMITTING after this spec

Implementation Notes

• Integration: legacy/XCE_TESTING election path (ha_elect_evaluate_and_signal) updated symmetrically; cluster_init_phase2 call point unchanged
• Validation: existing convergence tests retargeted from COMMITTING→PRIMARY transition to noop-commit gating
• Risks: test fixtures referencing CTRL_PRIMARY_COMMITTING across the suite — mechanical but broad; sweep required

Quorum Accounting (ha.c)

Summary-only: ha_has_write_quorum counts standbys that are connected AND not in_catch_up (4.1); evaluated at the dispatch gate for fast rejection (7.1) and re-evaluated authoritatively by the commit engine at ACK time (4.2, 4.3); stateless, so quorum restoration auto-resumes acceptance (7.4). Transition logging lives here (see Quorum Observability).

Config Barrier (config_repl.c)

Summary-only: push_config_to_standbys returns -ETIMEDOUT when majority ACK is not reached within the commit timeout; the upgradeConfig caller propagates the error to the requester and fails the upgrade (5.1–5.3). Uses the same majority arithmetic as the commit engine (shared helper). (Note: superseded in part by ha-term-safety task 4.2 — config is no longer a fully parallel, simpler track. config_commit_change now appends a WAL_CONFIG_MARKER consensus record to the same unified (term,lsn) log BEFORE the byte-delivery barrier; the derived config_committed_marker is the commit authority and committed_config_tag is retired to a non-authoritative cache. The byte-delivery majority-ack barrier mechanism itself (push_config_to_standbys, -ETIMEDOUT on failure) is unchanged. See ha-term-safety/design.md.)

Egress Filter (dataex.c, job.c)

Summary-only: job_t gains uint64_t submit_lsn (set when the submit entry is appended). The scheduler export path skips jobs with submit_lsn > committed_lsn (2.4 — prevents dispatch of a job whose entry may never commit). Status/API query paths are intentionally NOT filtered (documented deviation; read consistency out of scope). Replay/standby-restored jobs carry their entry LSN (≤ committed by definition). (Note: the predicate and scheduler-bound collect wrapper this design assigns to dataex.c were factored into a new file egress.c — job_submit_committed() + collect_jobs_committed() — by ha-convergence-recovery, kept out of dataex.c for controller-unit-test linkability. dataex.c's plain collect_jobs remains unfiltered, as intended, for status/API callers. The file's own header comment calls this "the same egress-filter component the design assigns to dataex.c, factored out for testability" — a refinement, not a contradiction.)

Demotion Convergence (ha.c)

Summary-only: on higher-term observation while PRIMARY (raft_cb_become_follower from PRIMARY): disable persistence hooks, truncate WAL above committed_lsn, WARN-log with both LSNs, then self-restart the process (re-exec of saved argv; under XCE_TESTING, a restart hook seam is invoked instead) — this part still matches as-built (ha_schedule_demotion_convergence). Boot recovery replays only the surviving committed prefix → provisional in-memory state is discarded (2.5, 2.7). (Note: the "Rejoin reconciliation" sentence originally here — a standby truncating + self-restarting once on rejoin — is superseded by ha-term-safety task 3.2: ha_rejoin_reconcile is now a no-op stub retained only as the CtrlHelloReply hook that records the learned committed boundary; the actual divergent-suffix reconciliation is owned by the raft library's term-aware on_truncate_from, not a watermark-clamped self-restart. See ha-term-safety/design.md.)

ACK-Driven Catch-up (config_repl.c)

Summary-only: catch-up streaming sends a bounded batch window; receipt of a walAck from a catch-up standby (iomgr) schedules the next batch immediately via a delaymgr task, instead of waiting for the next heartbeat reply (6.2). Heartbeat-driven evaluation remains as the fallback initiator and for snapshot decisions (6.3); repl_complete_catchup re-inclusion unchanged (6.4).

Quorum Observability + Idempotency Guards

Summary-only: quorum-loss/restore transitions WARN-logged once per transition with connected/required counts (7.2). mark_job_start/finish/fail/... gain transition guards: identical-state re-application is a logged no-op; backward transitions (terminal→running) are rejected with a WARN (supports 2.8 retry/replay and the future rejoin-resync).

Data Models

Domain Model (changes only)

• WAL entry (unchanged format; new type WAL_NOOP with empty payload — replay is a no-op, exists purely to commit a current-term entry)
• job_t: + submit_lsn (entry LSN of its submit record) — drives the egress filter; not persisted separately (recomputed at replay from the entry being applied)
• commit_engine_t: in-memory only (commit state is volatile, Raft-conformant — recovered via rejoin reconciliation, never persisted)
• Invariants: committed_lsn monotonic, contiguous; writable_lsn ≤ first confirmable LSN per term; an entry is confirmed ⇔ committed_lsn ≥ lsn ≥ nothing pending below it

Error Handling

Error Strategy

• Request-shaped commit failure → -ETIMEDOUT/-XCE_ERR_NOT_READY propagated via existing errnox early-return pattern to the client reply; no in-memory rollback (provisional model; outcome converges per 2.5)
• Admission-cap rejection and own-term-pending rejection both surface -XCE_ERR_NOT_READY (retriable; distinct log messages)
• Quorum loss: dispatch gate rejects request-shaped ops; fact-shaped ops are still appended and retained (2.8); WARN on transition only (no log storms)
• Demotion/restart: WARN with truncation range before restart; restart failures (exec error) FATAL — the node must not continue as a divergent primary

Monitoring

• Log counters: retained-uncommitted entries (fact backlog depth), inflight WAIT count vs cap, commit latency of the promotion no-op. Metrics export is out of scope (downstream).

Testing Strategy

Unit Tests (testsuite/, XCE_TESTING)

• test_commit_engine.c: per-LSN majority (entry N not committed by ACKs for M<N); contiguous advancement with out-of-order ACKs; monotonicity; own-term gate (no completion before noop LSN commits); admission cap (n_vcpus - reserved enforced, NOT_READY beyond); n_peers<=1 self-quorum; zero-peer-ACK never commits for n_peers>1 (4.4); membership change re-evaluation (4.3)
• test_ha_role_lifecycle.c: ELECTING→PRIMARY direct (no COMMITTING observable); noop appended at promotion; ha_should_redirect/ha_is_primary semantics (1.3, 1.4); no committing watchdog armed (1.5/3.3)
• Quorum accounting: catch-up standby excluded from ha_has_write_quorum (4.1)
• Idempotency guards: duplicate FINISH no-op; DONE→RUN rejected
• test_egress_filter.c: job with submit_lsn > committed_lsn absent from scheduler collect; present after commit

Integration Tests (testsuite/)

• Fact retention: simulate quorum loss → worker-fact WAL types still append (RETAIN), no error, committed after standby ACKs resume (2.8, 7.4)
• Request rejection: job submit and queue ctrl fail with error when majority ACK withheld past timeout; committed_lsn unmoved (2.2, 2.3)
• Config barrier: upgradeConfig fails to requester on withheld ACKs; committed_config_tag unmoved (5.1–5.3)
• Demotion seam: PRIMARY sees higher term → truncation above committed_lsn + restart hook invoked; replay after seam yields committed-only state (2.5, 2.7)
• ACK-driven catch-up: lagging standby receives successive batches without intervening heartbeats (6.2); re-inclusion at parity (6.4)

E2E (live cluster, extends ha-wal-replication task 5.4)

• 3-node: submit jobs → kill primary → promoted standby owns all confirmed jobs and none of the unconfirmed ones; demoted node rejoins converged (single-outcome check, 2.5)
• 3-node: isolate primary from both standbys → submits fail, status still served (7.1, 7.3); heal → submits succeed without intervention (7.4)

Performance & Scalability

• Commit latency target: unchanged from ha-wal-replication baseline (majority RTT + standby fsync); the no-op adds one entry per election
• Throughput under degradation bounded by admission cap — write rejection is deliberate (CP) and n_vcpus-relative; sizing guidance deferred to operator docs (research item 3(e))

Migration Strategy

Phased landing matching the brief's hard ordering (barrier before state removal):

1. Commit engine + per-LSN evaluation behind existing job-submit path (behavioral fix only)
2. Classification rollout: queue/worker/job-control → WAIT; facts → RETAIN; config barrier error propagation
3. Promotion no-op + own-term gate; then delete COMMITTING (+ watchdog, convergence checker); redirect fix
4. Demotion truncate+restart, rejoin reconciliation, egress filter, ACK-driven catch-up, observability Rollback trigger per phase: controller testsuite regression or live-cluster E2E failure; each phase leaves the system no less consistent than before it.