myxce/.kiro/specs/raft-ha-core-safety/requirements.md

Requirements Document

Introduction

raft-ha-core-safety fixes the pre-existing HA-core consistency, convergence, and liveness failures that the live failover soak (xce-test-ha) reproduces under repeated and chaotic primary failover on an XCE controller cluster. Operators and clients rely on Raft failover to keep committed jobs safe and the cluster available; today, under primary churn, the cluster can silently diverge committed state, fail to recover a healthy quorum, or stall electing a new primary.

All three symptoms reproduce on the pre-raft-drive-async baseline (commit 21e26a1), confirmed by a baseline-vs-current soak comparison, so they are pre-existing HA-core defects independent of the (complete and dormant) raft-drive-async async-drive/offload work. This spec is the gate that must close before raft-drive-async flips its dispatch flags ON in production.

The defects, in protocol terms:

1. Committed-prefix divergence (bug-851; bug-795/828 family). Under chaotic failover, walcompare reports a committed-prefix divergence: one member is missing a committed log entry the majority holds, and silently retains a divergent committed prefix. Known contributors: entry term left at 0 instead of the writing leader's term; two-leaders-per-term windows; and no per-entry commit identity or reconcile path, so a member cannot detect or repair a divergent committed prefix.
2. Standby-rejoin / reconvergence failure (bug-1042 family). A restarted, previously-downed node intermittently fails to return the cluster to a healthy 3-of-3 within bound, even after reaching the leader's log tip at the replication-match level, due to post-recovery log-length/applied-state desync and catch-up apply gaps.
3. Round-N election stall. After several rapid kill/restart rounds the cluster intermittently fails to elect a new primary after a primary kill within the failover window — a liveness regression under churn.

Behavior is verified against raft/raft.tla and the controller TLA+ safety-invariant checklist in src/controller/CLAUDE.md (Election Safety, Leader Completeness, State Machine Safety, Commit Ordering/Durability).

Boundary Context

• In scope: per-entry commit identity and divergence detect/repair; entry term always reflecting the writing leader's term; single-leader-per-term enforcement; deterministic standby-rejoin reconvergence to 3-of-3; election liveness under repeated churn; a deterministic reproduction harness (soak extension + unit/integration) that goes RED before the fix; and a green multi-round chaotic-failover soak as the acceptance gate.
• Out of scope: the raft-drive-async async-drive/offload work (SlowLane, peer-control vtask dispatch, snapshot/config-push offload, endpoint-quiescence). This spec must not change its behavior. New cluster-membership-resize, multi-region, or non-Raft replication features are also out of scope.
• Adjacent expectations: the cluster runs the existing 3-node Raft controller HA topology and persists committed state to a real filesystem. This spec expects xce-test-ha and the controller test harness to be the verification surface; it does not own the async dispatch path or the client-facing API contract beyond preserving committed-job safety.

Requirements

Requirement 1: Committed-prefix safety across members

Objective: As an operator of an XCE controller HA cluster, I want every member to agree on the contents of the committed log prefix, so that no committed job is ever silently lost or diverged under failover.

Acceptance Criteria

1. While the cluster is operating under repeated or chaotic primary failover, the Controller HA subsystem shall keep the committed log prefix identical across all members up to each member's commit point.
2. When a member holds a committed entry at a given log position, the Controller HA subsystem shall guarantee no other member holds a different committed entry at that same position.
3. If a member's committed prefix differs from the majority-agreed committed prefix at any position, then the Controller HA subsystem shall treat that member's committed state as divergent rather than authoritative.
4. The Controller HA subsystem shall satisfy the State Machine Safety and Commit Ordering/Durability invariants in the controller TLA+ safety checklist for every committed entry.

Requirement 2: Per-entry commit identity

Objective: As an operator, I want each log entry to carry an unambiguous identity, so that members can distinguish entries written by different leaders and terms at the same position.

Acceptance Criteria

1. When a leader writes a log entry, the Controller HA subsystem shall stamp that entry with the leader's current term.
2. The Controller HA subsystem shall never persist a committed entry whose term is 0 or otherwise does not identify the writing leader's term.
3. When two entries occupy the same log position, the Controller HA subsystem shall be able to determine from their identity whether they are the same entry or conflicting entries from different leaders.
4. While restoring persisted log entries after restart, the Controller HA subsystem shall preserve each entry's original term identity.

Requirement 3: Divergence detection and repair

Objective: As an operator, I want a member with a divergent committed prefix to detect and repair it, so that the cluster self-heals instead of silently carrying a permanent inconsistency.

Acceptance Criteria

1. When a member rejoins or replicates and its committed prefix conflicts with the majority-agreed prefix, the Controller HA subsystem shall detect the conflict.
2. When a committed-prefix conflict is detected, the Controller HA subsystem shall repair the divergent member's log to match the majority-agreed committed prefix.
3. If a member cannot repair its divergent committed prefix, then the Controller HA subsystem shall surface the unrecoverable divergence rather than continue serving divergent committed state.
4. The Controller HA subsystem shall not discard or overwrite any entry that is part of the majority-agreed committed prefix during repair.

Requirement 4: Single leader per term

Objective: As an operator, I want at most one leader to be able to commit entries in any given term, so that conflicting writes cannot enter the committed prefix.

Acceptance Criteria

1. The Controller HA subsystem shall allow at most one member to act as committing primary within any single term.
2. If a member detects another leader in the same or a higher term, then the Controller HA subsystem shall require that member to stop committing entries in its current term.
3. The Controller HA subsystem shall satisfy the Election Safety and Leader Completeness invariants in the controller TLA+ safety checklist across all failover transitions.

Requirement 5: Standby-rejoin reconvergence to 3-of-3

Objective: As an operator, I want a restarted node to deterministically rejoin and restore a full healthy quorum, so that the cluster regains its fault tolerance after recovery.

Acceptance Criteria

1. When a previously-downed node restarts and reconnects, the Controller HA subsystem shall bring the cluster to a healthy 3-of-3 membership within the bounded recovery window.
2. When a rejoining node reaches the leader's log tip at the replication-match level, the Controller HA subsystem shall also bring that node's applied state and log length into agreement with the leader.
3. If a rejoining node has a catch-up apply gap, then the Controller HA subsystem shall complete the outstanding apply work before reporting the node as healthy.
4. While a node is rejoining, the Controller HA subsystem shall not report the cluster as fully converged until that node's committed and applied state matches the leader.

Requirement 6: Election liveness under churn

Objective: As an operator, I want elections to always complete after a primary failure, so that the cluster never gets permanently stuck without a primary under repeated failover.

Acceptance Criteria

1. When the current primary is killed and a quorum of members remains reachable, the Controller HA subsystem shall elect a new primary at a higher term within the bounded failover window.
2. While the cluster undergoes repeated rapid kill/restart rounds, the Controller HA subsystem shall continue to elect a new primary after each primary loss.
3. If an election round fails to produce a primary, then the Controller HA subsystem shall start a further election rather than remain permanently without a primary.

Requirement 7: Deterministic reproduction harness

Objective: As a developer, I want a deterministic harness that forces the divergence, reconvergence, and election-stall paths, so that each fix is proven by a test that fails before it and passes after.

Acceptance Criteria

1. The reproduction harness shall force the committed-prefix divergence path and fail (RED) on the unfixed baseline.
2. The reproduction harness shall force the standby-rejoin reconvergence-failure path and fail (RED) on the unfixed baseline.
3. The reproduction harness shall force the round-N election-stall path and fail (RED) on the unfixed baseline.
4. When run against the fixed system, the reproduction harness shall pass (GREEN) for the divergence, reconvergence, and election-liveness paths.
5. The reproduction harness shall extend the existing xce-test-ha soak and add unit/integration coverage for the same paths.

Requirement 8: Acceptance soak gate

Objective: As a release owner, I want a green multi-round chaotic-failover soak on a real filesystem before this spec is accepted, so that raft-drive-async may flip its dispatch flags ON only after committed-job safety is proven.

Acceptance Criteria

1. When a chaotic-failover soak of at least 6 rounds runs against a release build on a real filesystem, the Controller HA subsystem shall report zero committed-prefix divergence.
2. When that soak runs, the Controller HA subsystem shall report zero committed-job loss.
3. While the soak runs, the Controller HA subsystem shall reconverge to 3-of-3 and elect a new primary after every primary kill round.
4. The Controller HA subsystem shall not be accepted as the raft-drive-async production flag-flip gate until the soak passes all of the above.