myxce/.kiro/specs/ha-consistency-invariants/brief.md

Brief: ha-consistency-invariants

Governing Model (decision: 2026-06-10)

XCE HA adopts Raft as the consensus model and CP as the CAP stance. This supersedes the earlier "No Raft" steering decision (the single-writer constraint is retained — the controller remains the sole log writer — but log commitment now follows Raft semantics, not just election). All consistency requirements in this spec are deductions from the Raft paper + CP, not ad-hoc rules:

Deduced rule	Raft / CP source
1. No distinct "primary committing" state	Raft has exactly three server states: Follower, Candidate, Leader. A candidate winning an election becomes Leader immediately. Raft's answer to "new leader can't yet serve safely" is not a fourth state — it is the commit rule: a leader may only count own-term entries toward commitment (§5.4.2), conventionally satisfied by committing a no-op entry at term start. The commit barrier subsumes the transitional state.
2. Majority ACK is the hard commit barrier; reject otherwise	Raft commit rule (§5.3): an entry is committed only when the leader has replicated it on a majority; commitIndex advances only on majority match, and the state machine applies entries only once committed. CP: a leader in a minority partition cannot commit — the correct behavior is reject/error to the caller, never local-apply-and-hope.
3. Full-consistency best-effort while running	Raft leaders maintain per-follower progress (nextIndex/matchIndex) and continuously retry AppendEntries to ALL followers (§5.3); majority is the commit threshold, not the fan-out target. Lagging followers catch up via continuous log streaming or InstallSnapshot — driven by the leader, not gated on heartbeat round-trips.

XCE mapping: committed_lsn ≈ Raft commitIndex; WAL entries ≈ log entries; standby ACK ≈ AppendEntries success; snapshot transfer ≈ InstallSnapshot; CTRL_PRIMARY_COMMITTING ≈ a state Raft does not have.

Problem

The HA controller's consistency model is inconsistent across operation types. Job submit goes through a majority-ACK commit barrier, but job state changes, queue control, and worker control are applied locally and replicated fire-and-forget. A primary that has lost quorum can still have in-flight writes "commit" locally, and a newly elected primary force-promotes after a 60s timeout even with zero converged standbys. The ha-config-bootstrap spec explicitly deferred this as the "CP-vs-availability operator decision"; the decision is now made by adopting CP: reject, never degrade.

Current State

Audit of src/controller/ (2026-06-10) found these violations of the Raft+CP deductions:

Rule 1 — drop the PRIMARY_COMMITTING / PRIMARY distinction

• CTRL_PRIMARY_COMMITTING (ha.h:59-65) is a transitional leader state set at ha.c:505 (raft callback) and ha.c:2248 (legacy path), checked in ~7 places. Raft has no such state.
• ha_is_primary() (ha.c:917) returns 0 in COMMITTING; ha_should_redirect() (ha.c:1655) redirects clients to a primary that does not exist yet → redirect loop during the committing window.
• committing_timeout_fn (ha.c:3092-3121) force-promotes to PRIMARY after 60s even with zero converged standbys — accepting writes with no replication (violates the Raft commit rule outright).

Rule 2 — majority-ACK is the hard commit barrier; reject otherwise

• Job state changes (start/finish/fail/stop/resume/update/ckpt/rusage) use persist_wal_async only — local apply + fire-and-forget broadcast, no majority wait (persist.c:1169). Same for queue control (persist.c:1199) and worker control (persist.c:1241). Only WAL_JOB_SUBMIT uses persist_wal_sync_append + repl_wait_majority_ack (persist.c:1003-1049).
• ha_has_write_quorum (ha.c:1507-1525) counts connected standbys including those in catch-up mode, but repl_broadcast skips catch-up standbys — the gate can pass while the write cannot actually reach quorum. (Raft equivalent: matchIndex, not connection count, is what commitment consults.)
• Config push timeout: push_config_to_standbys (config_repl.c:916-928) logs WARN on majority-ACK timeout but returns the ack count; caller (config_repl.c:1057) ignores it — config upgrade proceeds without confirmed majority.
• TOCTOU: quorum checked at dispatch gate only; repl_wait_majority_ack (repl.c:430-434) short-circuits to success when n_active == 0, so a write admitted before mass disconnect commits with primary-only durability.
• Quorum loss while PRIMARY blocks writes at the gate (correct CP direction) but combined with the n_active==0 short-circuit this leaves a hole.
• Apply-before-commit (to verify in requirements phase): Raft applies entries to the state machine only after commit. Audit confirmed committed_lsn advances only on majority ACK for job submit, but whether the in-memory state mutation precedes the ACK on each path needs per-path verification; local WAL append before ACK is fine (that is the Raft log), in-memory apply before commit is not.

Rule 3 — full consistency best-effort while running

• repl_broadcast (repl.c:210-282) already fans out to ALL active standbys — matches Raft.
• Lagging standbys evicted to catch-up mode (pending_ack_count > MAX_PENDING_ACKS) only re-enter via the 3s heartbeat-driven decision loop (ha_dispatch_repl_action_for_standby, ha.c:1373) — catch-up is heartbeat-gated, not leader-driven continuous streaming as Raft prescribes; a busy primary lets catch-up standbys fall arbitrarily behind.

Desired Outcome

1. Single leader state: winning an election transitions directly to CTRL_PRIMARY. CTRL_PRIMARY_COMMITTING, its whitelist branch, and committing_timeout_fn are removed. Post-election write safety comes from the Raft own-term commit rule (the universal commit barrier; optionally an explicit no-op entry committed at term start), not a special state.
2. Every cluster-state-changing operation (job submit, job state change, queue control, worker control, config push) commits only after majority ACK. On timeout or quorum loss the operation is rejected with an error to the caller. committed_lsn / committed_config_tag only advance on real majority — they are XCE's commitIndex and must behave like it.
3. Quorum accounting is honest and ACK-based: ha_has_write_quorum counts only standbys actually reachable for the write; majority is verified at ACK time (matchIndex-style), not just at the dispatch gate; repl_wait_majority_ack never succeeds with zero active standbys when n_peers > 1.
4. Full-consistency best-effort: broadcast remains all-standbys; catch-up streaming is leader-driven and continuous (no heartbeat round-trip gating), with snapshot fallback for standbys behind the oldest retained WAL.

Approach

Chosen: new single spec layered on ha-wal-replication's ACK machinery, with the Raft paper (§5.3, §5.4) as the requirements authority.

The recently built WalAckReply / majority-ack / committed_lsn infrastructure (ha-wal-replication tasks 3.x-5.x) is the enforcement mechanism; this spec generalizes it from WAL_JOB_SUBMIT to all state-changing WAL types, makes rejection (not degradation) the universal timeout behavior, and then deletes the now-redundant COMMITTING state.

Order matters: implement the universal commit barrier (Rule 2) FIRST, then merge the states (Rule 1) — merging first would open a window where a fresh primary accepts unreplicated writes, violating the own-term commit rule.

Rejected alternatives:

• Extending ha-wal-replication in place — scope creep on a mid-implementation spec; state-machine surgery is a different boundary than WAL transport.
• Multi-spec split (state merge / write barrier / catch-up as 3 specs) — the pieces are safety-interlocked (see ordering above); splitting risks landing Rule 1 without Rule 2.

Scope

• In:
  • Route WAL_JOB_* state changes, WAL_QUEUE_CTRL, WAL_WORKER_CTRL through the sync majority-ACK path; reject on timeout.
  • Verify (and fix where violated) apply-after-commit ordering on each write path.
  • Config push: error on majority-ACK timeout, propagate to caller, fail the upgrade request.
  • Fix ha_has_write_quorum catch-up accounting; verify quorum at ACK completion; remove the n_active==0 success short-circuit for n_peers > 1.
  • Remove CTRL_PRIMARY_COMMITTING: direct ELECTING→PRIMARY transition, delete committing whitelist branch and committing_timeout_fn, fix ha_is_primary / ha_should_redirect semantics; evaluate explicit no-op own-term commit at promotion.
  • Leader-driven continuous catch-up streaming (decouple from heartbeat round-trip).
  • Tests: per-WAL-type commit-barrier tests, quorum-loss rejection tests, election→write-availability convergence test, catch-up progress test.
• Out: WAL transport/encoding (ha-wal-replication owns it); election/vote logic (raft-library owns it — already Raft-conformant); single-node (n_peers <= 1) behavior — self-quorum stays as-is (a 1-node Raft cluster's majority is 1); checkpoint replication (deliberately local-only — snapshots are per-node in Raft too); automatic primary step-down on quorum loss (Raft leaders in a minority partition also stay leader but cannot commit — blocking writes satisfies CP; step-down-on-silence is a possible follow-on optimization).

Boundary Candidates

• Commit-barrier generalization (persist.c sync/async split, repl.c ack wait)
• Quorum accounting + TOCTOU closure (ha.c gate, repl.c ack path)
• State-machine simplification (ha.c enum, transitions, timeout removal)
• Leader-driven catch-up (config_repl.c streaming, ha.c decision loop)

Out of Boundary

• Raft vote/election correctness (raft-library)
• WAL wire format and standby apply path (ha-wal-replication)
• Worker/scheduler reconnect-on-failover (worker-join-refactor / P2.5)
• Read consistency / leader leases (Raft §8 read-only optimization — future spec)
• Performance tuning of sync-commit latency (acceptable cost of CP; measure, don't optimize, in this spec)

Upstream / Downstream

• Upstream: ha-wal-replication (ACK machinery, committed_lsn — must complete its remaining tasks first), ha-repl-hardening (payload data plane), raft-library (become_leader callback signature).
• Downstream: read-consistency / lease work; operator-facing degraded-mode metrics; possible primary step-down-on-quorum-loss spec; steering update (product.md decision log — "No Raft" superseded, done at discovery time).

Existing Spec Touchpoints

• Extends behavior introduced by: ha-wal-replication (generalizes its majority-ack from job-submit to all WAL types), ha-config-bootstrap (resolves its documented "commit timeout silently degrades" deviation as REJECT per CP).
• Adjacent: ha-heartbeat-liveness (heartbeat period interacts with catch-up loop), raft-library (election→PRIMARY transition point changes).

Constraints

• C11, vtask async I/O — commit barrier must not block vtask handlers; reuse the existing delaymgr-based repl_wait_majority_ack pattern.
• TDD mandatory: testsuite/ tests before C changes; existing convergence tests in test_ha_connection_reuse.c and ha-wal-replication tests must stay green.
• Single-writer model preserved: the controller remains the sole log writer (XCE's simplification of Raft — no concurrent client proposals); this spec narrows when the single writer may commit, it does not add writers.
• Semantics for n_peers <= 1 (standalone) unchanged: self-quorum, no barrier (consistent with single-node Raft).
• Where this brief and the Raft paper disagree, the Raft paper wins; deviations must be documented in the design with justification.