Rongsong Shen
8ee98beed7
Adds crash-safety, determinism, host-status, and apply-activate requirements; records research on bug-1418 as a pre-work dependency (its apply-lag promotion-gate fix is the template for this spec's own Req3 graceful-degrade gate, and contaminates Req6's chaos-soak signal if left unfixed first).
24 lines
4.3 KiB
Markdown
24 lines
4.3 KiB
Markdown
# Brief: config-upgrade-snapshot
|
|
|
|
## Problem
|
|
XCE's state machine (jobs, queues, hosts, resources) is **interpreted against the active configuration**. Today a configuration upgrade does config-push + restart + **per-node WAL replay under the new config** (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state. Because replay is config-dependent (the `lost_and_found` queue/host re-homes entities whose config entity vanished), members can **materialize divergent state** after an upgrade — amplified by any WAL tail divergence. Logged as **bug-857** (identified via the `raft.tla` gap analysis; not yet empirically reproduced).
|
|
|
|
This was originally Requirement 8 of `ha-term-safety`; it is **descoped into this independent spec** (2026-06-21) because it has a hard prerequisite that does not exist yet.
|
|
|
|
## Approach
|
|
A configuration upgrade establishes a **full, entity-bearing state snapshot at the configuration-change boundary** (the `(term, lsn)` of the config-change marker that `ha-term-safety` introduces), **embeds the new configuration** in that snapshot, and **installs the same snapshot on every member** through the out-of-band snapshot-install transport. No member re-derives post-upgrade state by replaying pre-upgrade WAL under the new configuration — installing the snapshot delivers configuration + materialized state together as one consistent artifact. This supersedes `ha-term-safety` R7's interim config-push byte delivery for the upgrade case.
|
|
|
|
## HARD dependency — entity-bearing snapshots
|
|
This spec **cannot proceed without entity-bearing snapshots**. Today snapshots are a 16-byte LSN sentinel (`"XCESNAP\0"` + checkpoint_lsn; `persist.c` write ~1068-1101, load ~1280-1307) carrying **zero entity state**. The COW machinery needed (`xzarray_freeze`/`thaw`, `zarray.c:695-757`) is **built and unit-tested but dormant** (`kvmgr.c:297` `vfreeze=NULL`, no production callers) — the roadmap's unstarted **`entity-bearing-snapshots` ("Task 11")** backlog item. That work must land first (or be co-scoped). The snapshot *transport* (`snapshot_transfer_s`: chunked, `(term,lsn)`-anchored, hash-verified, install-before-activate) is already reusable; only the entity-bearing *payload* is missing.
|
|
|
|
## Scope
|
|
- **In**: full entity-bearing state snapshot materialized at the config-change marker boundary; embed the new configuration in that snapshot; install on every member via the snapshot-install path; activation gating (a member does not serve under the new configuration until it has installed the upgrade snapshot and verified the embedded configuration against the marker's content hash); post-upgrade consistency invariant (every member's materialized state + active configuration identical); supersede `ha-term-safety` R7's interim config-push byte delivery for upgrades.
|
|
- **Out**: the config-change marker / consensus ordering itself (owned by `ha-term-safety` R7); the entity serialization machinery (owned by `entity-bearing-snapshots` — this spec consumes it); the Raft protocol (the `Raft TLA+ Conformance` library series); controller voting-set/membership changes (`ha-config-bootstrap`).
|
|
|
|
## Verification
|
|
Empirically reproduce bug-857 first (upgrade config during the chaotic-multi-kill repro and diff materialized state across nodes via `waldump`/state dump) to establish the RED, then prove every member converges to identical materialized state + active config after an upgrade interleaved with failover.
|
|
|
|
## Dependencies
|
|
HARD (both now satisfied — 2026-07-21): `entity-bearing-snapshots` → satisfied by `ha-entity-snapshots` (21/21 tasks). `ha-term-safety` (the config-change marker it anchors on) → `implementation-complete`, GO 2026-07-17. Consumes the `snapshot_transfer_s` transport. Closes bug-857.
|
|
|
|
**Prepare work (decided 2026-07-21):** fix **bug-1418** (apply-lag promotion-gate liveness deadlock, still open, no owning spec) before or alongside this spec's implementation. It reliably fires inside the same chaotic-multi-kill soak Requirement 6 needs, so left unfixed it contaminates that test's signal; its own (unimplemented) fix — become primary, self-apply, then serve, instead of hard-blocking promotion — is also the template Requirement 3's activation gate must mirror to avoid reintroducing the same liveness-deadlock shape for materialization-lag. Scope as task 0 / a small prerequisite fix, not optional cleanup.
|