myxce/.kiro/specs/ha-term-safety/requirements.md

Requirements Document

Introduction

This feature closes a Raft safety gap (bug-851) in the XCE HA controller cluster. Under chaotic controller failover (repeated loss of the primary), members can reach a state where every node reports the same committedLsn over a contiguous, CRC-intact WAL, yet the nodes hold different entries at the same LSNs — a silent committed-prefix divergence that loses acknowledged jobs. A clean single failover converges correctly today (validated end-to-end: bugs 848, 849, 850), so the defect is specific to leadership churn.

Root cause has three compounding parts, each addressed by a requirement area below:

1. The election term is not monotonic across the cluster — a leader can act in a term that was never durably propagated to a majority, letting a restarted node re-elect at the same term, so two leaders share one term.
2. The per-entry WAL term is never recorded (always 0), so the WAL carries no Raft term information.
3. Rejoin reconcile cannot perform (term, lsn) log-matching, so divergent same-LSN tails are never truncated and committed advances over entries a majority never held.

The objective is to restore the Raft Log Matching and Leader Completeness guarantees so that any entry reported committed is byte-identical on every member, including across arbitrary failover patterns — without regressing clean-failover behavior or the on-disk WAL format.

The same divergence class latently affects configuration, which today commits through a separate index (committed_config_tag) with no per-entry term. Rather than maintain two consistency domains, this feature unifies the consensus half: a configuration change is recorded as a lightweight in-log marker carrying only a reference to the configuration (version tag + content hash), ordered in the same (term, lsn) log as job state, so configuration inherits the same term-safety and reconcile. The configuration bytes stay out of the WAL records (the marker carries only a reference); the bytes continue to move through the existing out-of-band config-push transport, hash-verified against the committed marker before activation.

The remaining hazard — a configuration upgrade leaving members with divergent materialized state because XCE's state machine is interpreted against the configuration and each member re-derives post-upgrade state by replaying pre-upgrade WAL under the new configuration — requires a full entity-bearing state snapshot at the change boundary, which does not exist yet (snapshots are presently a 16-byte LSN sentinel). That work is descoped to the independent config-upgrade-snapshot spec (depends on entity-bearing snapshots) and is out of scope here.

Boundary Context

• In scope:
  • Recording the live election term in every WAL entry and carrying it through replication and recovery.
  • Making the election term monotonic and durably established before a new leader appends entries (no two leaders per term).
  • (term, lsn) log-matching during rejoin/reconcile, including truncation of divergent local suffixes.
  • Preserving the committed-prefix consistency invariant across arbitrary failover sequences.
  • Unified ordering of configuration changes and job state through the same replicated log, by recording each configuration change as a lightweight in-log marker that references the configuration (version tag + content hash) rather than carrying its bytes, so configuration inherits the same term-safety and reconcile guarantees. Configuration bytes continue to move through the existing out-of-band config-push transport, hash-verified against the committed marker.
  • A regression test that drives chaotic multi-kill failover and asserts cross-member committed-prefix identity (including configuration markers).
• Out of scope (tracked separately):
  • Configuration-upgrade materialized-state consistency (full entity-bearing state snapshot at the change boundary so no member re-derives post-upgrade state) — descoped to the independent config-upgrade-snapshot spec, which depends on entity-bearing snapshots (entity-bearing-snapshots backlog item; today snapshots are a 16-byte LSN sentinel). This spec ships the config marker (consensus) but not the full-snapshot upgrade.
  • Client primary-discovery being positional (bug-854).
  • Catch-up payload drop across multiple WAL segments (bug-847).
  • Each job producing two JOB_SUBMIT WAL entries (bug-845).
  • Snapshot-based prefix backfill / empty-rejoin gap (bug-828, ha-commit-durability).
  • Changes to the controller voting set (adding/removing a controller peer): such membership transitions change the quorum itself and remain governed by ha-config-bootstrap; this spec orders the config-change marker uniformly but does not introduce joint-consensus quorum transitions.
• Adjacent expectations:
  • Relies on the existing election/PreVote machinery (ha-prevote, ha-single-leader-role) and the commit barrier (ha-commit-engine) to provide term and quorum signals; this feature constrains how those signals are recorded and reconciled, and does not redesign the election protocol itself.
  • Relies on persistence recovery (ha-commit-durability) for the recovered durable floor; this feature adds the term dimension to what recovery and reconcile compare.
  • Carries configuration bytes via the existing out-of-band config-push transport, never via WAL entry payloads (the WAL holds only the (term, lsn) marker); the bytes are hash-verified against the committed marker before activation. The follow-on config-upgrade-snapshot spec later supersedes this byte delivery with a full entity-bearing upgrade snapshot once entity-bearing snapshots exist.
  • Realizes its protocol invariants (R1–R6) through the raft library callbacks delivered by the Raft TLA+ Conformance series (raft-log-replication, raft-commit-safety); this spec is the controller-side adapter + config marker + live regression, not a re-implementation of the Raft protocol.
  • Complements ha-consistency-invariants and ha-convergence-recovery by supplying the per-entry term those invariants assume already exists.

Requirements

Requirement 1: Per-entry WAL term recording

Objective: As an operator of an HA controller cluster, I want every WAL entry to record the election term under which it was created, so that entries from different terms are distinguishable and Raft log-matching is possible.

Acceptance Criteria

1. When the primary appends a WAL entry, the Controller shall record in that entry the value of the leader's current election term in effect at append time.
2. When the primary appends its own-term no-op on promotion, the Controller shall record the newly established term in that no-op entry.
3. When the primary replicates a WAL entry to a standby, the Controller shall transmit the entry's recorded term so that the standby persists the leader's term, not a substitute value.
4. While applying a replicated entry, the standby shall persist the entry with the same term the leader recorded for that entry.
5. When a member recovers its WAL at startup, the Controller shall read back each entry's recorded term and make it available to the reconcile and promotion logic.
6. The Controller shall not record a fixed or zero placeholder term for entries created while a valid election term is in effect.
7. The Controller shall preserve the existing on-disk WAL entry header size and field layout, treating the term as a value fix rather than a format change.
8. Where a WAL entry predates this feature and carries a legacy zero term, the Controller shall treat it as belonging to the lowest-known term for matching purposes rather than failing recovery.

Requirement 2: Monotonic, durably-established election term

Objective: As an operator, I want a node to become an acting leader only at a term that is strictly greater than any term already used in the cluster and durably established, so that two leaders never share a term.

Acceptance Criteria

1. When a node wins an election, the Controller shall durably persist the won term locally before it appends any entry as leader.
2. While a node has not propagated its won term to a majority of members, the Controller shall not treat any entry it appends in that term as committed.
3. If a node recovers a stale election term after restart, then the Controller shall not begin or win a new election at a term less than or equal to any term it can observe from a reachable peer or its own persisted WAL.
4. When a node initiates an election, the Controller shall choose a prospective term strictly greater than the highest term it has observed locally and from reachable peers.
5. If two members would otherwise act as leader in the same term, then the Controller shall ensure at most one of them can advance committed in that term and the other steps down without having committed divergent entries.
6. When a member observes a higher term from a peer than its own, the Controller shall adopt the higher term and abandon any leadership it holds at the lower term.

Requirement 3: (term, lsn) log-matching reconcile

Objective: As an operator, I want a rejoining member to converge its WAL to the current leader's log, so that divergent tails created during churn do not persist.

Acceptance Criteria

1. When a member rejoins and synchronizes with the current leader, the Controller shall compare its local WAL to the leader's by (term, lsn) pairs, not by lsn alone.
2. If a member's local entry at a given LSN has a term that does not match the leader's entry at that LSN, then the Controller shall truncate that local entry and every entry after it before accepting the leader's entries.
3. When the Controller truncates divergent local entries, it shall not truncate any entry that is part of the cluster's committed prefix.
4. When reconcile completes, the member's WAL prefix up to the leader's committed point shall be byte-identical to the leader's.
5. If a member cannot reconcile to a committed-consistent state with the leader, then the Controller shall keep that member non-promotable and continue retrying rather than reporting false convergence.
6. While a member holds an unreconciled divergent suffix, the Controller shall not report those divergent entries as committed.

Requirement 4: Committed-prefix consistency invariant

Objective: As a client and operator, I want any entry reported committed to be identical on every member, so that an acknowledged job is never lost or replaced after a failover.

Acceptance Criteria

1. The Controller shall only advance committed to an LSN whose entry is held with a matching (term, lsn) by a majority of members, applying the same rule to job-state entries and configuration-change markers.
2. When any member reports an LSN as committed, the Controller shall ensure the entry at that LSN is byte-identical on every member that reports the same or a higher committed LSN.
3. If an entry was acknowledged to a client as committed, then the Controller shall preserve that exact entry across all subsequent leadership changes.
4. While leadership changes occur, the Controller shall never replace, reorder, or drop an entry within the established committed prefix.
5. When a new leader computes its safe committed point from granted voters, the Controller shall not include an LSN whose (term, lsn) is not corroborated by the required quorum.

Requirement 5: Clean-failover behavior preserved

Objective: As an operator, I want the validated clean-failover and recovery behavior to keep working unchanged, so that this fix introduces no regression.

Acceptance Criteria

1. When a single clean failover occurs (one primary killed, one standby promoted, old primary restarted), the Controller shall converge all members to a byte-identical WAL as it does today.
2. While a promoted standby recovers and advertises its recovered applied LSN, the Controller shall preserve the existing recovery-on-promotion and advertise-real-LSN behavior (bugs 849 and 848).
3. When a standby falls behind and the primary streams catch-up entries, the Controller shall preserve the existing contiguity behavior that refuses a forward gap without dropping live entries (bug 850).
4. The Controller shall not increase committed-acknowledgment latency for the clean, single-leader steady state beyond its current behavior.

Requirement 6: Chaotic-failover regression verification

Objective: As a maintainer, I want an automated test that reproduces the chaotic-failover divergence, so that the fix is proven and protected against regression.

Acceptance Criteria

1. When the regression scenario drives repeated primary kills interleaved with job submissions, the test shall assert that every member's committed WAL prefix is byte-identical.
2. When the test inspects member WALs, it shall compare entries by (term, lsn) and content so that a same-LSN divergence is detected, not masked by equal LSN ranges.
3. If any member holds a divergent entry within the committed prefix, then the test shall fail with the diverging LSN and the per-member entry identities reported.
4. When the regression scenario completes, the test shall confirm that no entry acknowledged as committed during the run is absent from any surviving member.
5. The test shall reproduce the original bug-851 failure on the pre-fix code and pass on the fixed code.
6. When the regression scenario interleaves configuration changes with the chaotic kills, the test shall assert that every member converges to the same committed configuration reference (version tag and content hash).

Requirement 7: Unified configuration ordering with out-of-band configuration data

Objective: As an operator, I want configuration changes ordered and committed through the same replicated log and term machinery as job state, while the configuration bytes stay out of the WAL records (carried by the existing out-of-band configuration transport), so that configuration inherits term-safety and reconcile without bloating the log.

Scope note (descope 2026-06-21): the full materialized-state consistency of a configuration upgrade — establishing a full state snapshot at the change boundary so no member re-derives post-upgrade state — is moved to the independent config-upgrade-snapshot spec, which depends on entity-bearing snapshots (a snapshot that carries entity state does not exist yet; today it is a 16-byte LSN sentinel). This requirement keeps the consensus half (the in-log (term, lsn) marker) and continues to deliver the configuration bytes through the existing out-of-band config-push transport, hash-verified against the marker.

Acceptance Criteria

1. When a configuration change is committed, the Controller shall append a configuration-change marker to the same replicated log used for job state, ordered by the same (term, lsn) sequence.
2. The Controller shall record in each configuration-change marker a reference to the configuration — its version tag and content hash — and shall not place the configuration bytes in the marker.
3. The Controller shall record the live election term in each configuration-change marker, identically to job-state entries (Requirement 1).
4. The Controller shall not store configuration bytes in WAL entry payloads.
5. The Controller shall move the configuration bytes through the existing out-of-band configuration transport (config push), keyed to the marker's version tag and content hash, not through the WAL.
6. When reconcile compares logs by (term, lsn), the Controller shall truncate a divergent configuration-change marker exactly as it truncates a divergent job-state entry.
7. The Controller shall derive the committed configuration from the latest configuration-change marker within the committed log prefix, rather than from a separate configuration commit index.
8. While a configuration-change marker is committed but its referenced configuration bytes are not yet present locally, the Controller shall obtain the bytes through the out-of-band transport and verify them against the marker's content hash before activating the configuration.
9. If a member's locally held configuration bytes do not match the content hash of the latest committed configuration marker, then the Controller shall keep that member from serving under the new configuration and continue retrying rather than activating mismatched configuration.
10. When a member recovers, the Controller shall activate the configuration referenced by the latest committed configuration-change marker, obtaining the bytes from out-of-band storage.
11. The Controller shall preserve the existing majority-acknowledgment requirement for a configuration change, now expressed as the configuration-change marker reaching the committed prefix.