Rongsong Shen 8ee98beed7 docs(config-upgrade-snapshot): expand spec to 6 requirements/26 ACs

Adds crash-safety, determinism, host-status, and apply-activate
requirements; records research on bug-1418 as a pre-work dependency
(its apply-lag promotion-gate fix is the template for this spec's own
Req3 graceful-degrade gate, and contaminates Req6's chaos-soak signal
if left unfixed first).

2026-07-23 04:29:11 +00:00

4.3 KiB

Raw Blame History

Brief: config-upgrade-snapshot

Problem

XCE's state machine (jobs, queues, hosts, resources) is interpreted against the active configuration. Today a configuration upgrade does config-push + restart + per-node WAL replay under the new config (config_repl.c do_upgrade_config_with_bytes ~1001-1163), with no snapshot pinning an identical post-upgrade state. Because replay is config-dependent (the lost_and_found queue/host re-homes entities whose config entity vanished), members can materialize divergent state after an upgrade — amplified by any WAL tail divergence. Logged as bug-857 (identified via the raft.tla gap analysis; not yet empirically reproduced).

This was originally Requirement 8 of ha-term-safety; it is descoped into this independent spec (2026-06-21) because it has a hard prerequisite that does not exist yet.

Approach

A configuration upgrade establishes a full, entity-bearing state snapshot at the configuration-change boundary (the (term, lsn) of the config-change marker that ha-term-safety introduces), embeds the new configuration in that snapshot, and installs the same snapshot on every member through the out-of-band snapshot-install transport. No member re-derives post-upgrade state by replaying pre-upgrade WAL under the new configuration — installing the snapshot delivers configuration + materialized state together as one consistent artifact. This supersedes ha-term-safety R7's interim config-push byte delivery for the upgrade case.

HARD dependency — entity-bearing snapshots

This spec cannot proceed without entity-bearing snapshots. Today snapshots are a 16-byte LSN sentinel ("XCESNAP\0" + checkpoint_lsn; persist.c write ~1068-1101, load ~1280-1307) carrying zero entity state. The COW machinery needed (xzarray_freeze/thaw, zarray.c:695-757) is built and unit-tested but dormant (kvmgr.c:297 vfreeze=NULL, no production callers) — the roadmap's unstarted entity-bearing-snapshots ("Task 11") backlog item. That work must land first (or be co-scoped). The snapshot transport (snapshot_transfer_s: chunked, (term,lsn)-anchored, hash-verified, install-before-activate) is already reusable; only the entity-bearing payload is missing.

Scope

In: full entity-bearing state snapshot materialized at the config-change marker boundary; embed the new configuration in that snapshot; install on every member via the snapshot-install path; activation gating (a member does not serve under the new configuration until it has installed the upgrade snapshot and verified the embedded configuration against the marker's content hash); post-upgrade consistency invariant (every member's materialized state + active configuration identical); supersede ha-term-safety R7's interim config-push byte delivery for upgrades.
Out: the config-change marker / consensus ordering itself (owned by ha-term-safety R7); the entity serialization machinery (owned by entity-bearing-snapshots — this spec consumes it); the Raft protocol (the Raft TLA+ Conformance library series); controller voting-set/membership changes (ha-config-bootstrap).

Verification

Empirically reproduce bug-857 first (upgrade config during the chaotic-multi-kill repro and diff materialized state across nodes via waldump/state dump) to establish the RED, then prove every member converges to identical materialized state + active config after an upgrade interleaved with failover.

Dependencies

HARD (both now satisfied — 2026-07-21): entity-bearing-snapshots → satisfied by ha-entity-snapshots (21/21 tasks). ha-term-safety (the config-change marker it anchors on) → implementation-complete, GO 2026-07-17. Consumes the snapshot_transfer_s transport. Closes bug-857.

Prepare work (decided 2026-07-21): fix bug-1418 (apply-lag promotion-gate liveness deadlock, still open, no owning spec) before or alongside this spec's implementation. It reliably fires inside the same chaotic-multi-kill soak Requirement 6 needs, so left unfixed it contaminates that test's signal; its own (unimplemented) fix — become primary, self-apply, then serve, instead of hard-blocking promotion — is also the template Requirement 3's activation gate must mirror to avoid reintroducing the same liveness-deadlock shape for materialization-lag. Scope as task 0 / a small prerequisite fix, not optional cleanup.

4.3 KiB Raw Blame History