myxce/.kiro/specs/config-upgrade-snapshot/requirements.md

Requirements Document

Introduction

XCE's controller state machine (jobs, queues, hosts, resources) is interpreted against the active configuration. Today a configuration upgrade pushes the new configuration, restarts each controller, and lets each member re-derive its post-upgrade state by replaying its pre-upgrade WAL under the new configuration — with no snapshot pinning an identical materialized state across members. Because replay is configuration-dependent (the lost_and_found queue/host re-homes entities whose configuration entity vanished), members can diverge in materialized state after an upgrade (bug-857), amplified by any WAL tail divergence.

This feature makes a configuration upgrade establish a full, entity-bearing state snapshot at the configuration-change boundary, embed the new configuration in it, and install that same snapshot on every member — so post-upgrade state is identical by construction and no member re-derives it. It was originally Requirement 8 of ha-term-safety and is split out because it has a hard prerequisite (entity-bearing snapshots) that does not yet exist.

Boundary Context

• In scope: a full entity-bearing state snapshot materialized at the configuration-change marker's (term, lsn); embedding the new configuration in that snapshot; installing it on every member via the out-of-band snapshot-install transport; activation gating until the snapshot is installed and hash-verified; the post-upgrade materialized-state + active-configuration consistency invariant; superseding ha-term-safety R7's interim config-push byte delivery for the upgrade case.
• Out of scope:
  • The configuration-change marker and consensus ordering of the change — owned by ha-term-safety (Requirement 7).
  • The entity-serialization machinery that produces an entity-bearing snapshot — owned by entity-bearing-snapshots; this spec consumes it.
  • The Raft protocol (term/log/commit) — owned by the Raft TLA+ Conformance library series.
  • Controller voting-set / membership changes — ha-config-bootstrap.
• Adjacent expectations:
  • HARD dependency on entity-bearing-snapshots: today snapshots are a 16-byte LSN sentinel carrying no entity state; the COW machinery (xzarray_freeze/thaw) is built but dormant. That work must land first.
  • Anchors on the (term, lsn) configuration-change marker from ha-term-safety.
  • Reuses the existing snapshot_transfer_s snapshot-install transport (chunked, (term, lsn)-anchored, hash-verified, install-before-activate).

Requirements

Requirement 1: Full snapshot at the configuration-change boundary

Objective: As an operator, I want a configuration upgrade to materialize a full state snapshot at the change boundary, so that every member adopts the same post-upgrade state rather than re-deriving it.

Acceptance Criteria

1. When a configuration change commits, the Controller shall materialize a full state snapshot at the configuration-change marker's (term, lsn) boundary.
2. The Controller shall install that same snapshot on every member.
3. The Controller shall not require any member to re-derive post-upgrade state by replaying pre-upgrade WAL entries under the new configuration.
4. The Controller shall transfer the upgrade snapshot through the out-of-band snapshot-install path, not through WAL entry payloads.

Requirement 2: Configuration embedded in the upgrade snapshot

Objective: As an operator, I want the new configuration delivered together with the materialized state, so that a member never holds state and configuration from different points.

Acceptance Criteria

1. The Controller shall embed the new configuration in the upgrade snapshot so that installing the snapshot delivers configuration and materialized state as one consistent artifact.
2. The Controller shall make this upgrade snapshot supersede the interim out-of-band config-push byte delivery (ha-term-safety Requirement 7) for the upgrade case.
3. The Controller shall keep the configuration bytes out of WAL entry payloads (the WAL holds only the (term, lsn) configuration-change marker).

Requirement 3: Activation gating

Objective: As an operator, I want a member to serve under a new configuration only after it holds the matching materialized state, so that no member operates on mismatched configuration or state.

Acceptance Criteria

1. While a member has not installed the upgrade snapshot, the Controller shall not let that member serve under or activate the new configuration.
2. When a member installs an upgrade snapshot, the Controller shall verify the embedded configuration against the configuration-change marker's content hash before activating it.
3. If the embedded configuration does not match the marker's content hash, then the Controller shall reject the snapshot and continue retrying rather than activating mismatched configuration.

Requirement 4: Post-upgrade consistency invariant

Objective: As a client and operator, I want every member identical after a configuration upgrade, so that an upgrade can never silently diverge materialized state across members.

Acceptance Criteria

1. When members complete a configuration upgrade, every member's materialized state shall be identical.
2. When members complete a configuration upgrade, every member's active configuration shall be identical.
3. While configuration upgrades interleave with failover, the Controller shall preserve the post-upgrade identical-state invariant across leadership changes.

Requirement 5: Reproduction and regression

Objective: As a maintainer, I want the bug-857 divergence reproduced and guarded, so that the fix is proven.

Acceptance Criteria

1. When the regression scenario interleaves a configuration upgrade with chaotic multi-kill failover, the test shall assert that every member's materialized state and active configuration are identical afterwards.
2. The test shall reproduce the bug-857 materialized-state divergence on the pre-fix code (config-push + per-node replay) and pass on the fixed code.