Rongsong Shen 20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.

2026-06-21 14:02:15 +00:00

3.6 KiB

Raw Blame History

Brief: config-upgrade-snapshot

Problem

XCE's state machine (jobs, queues, hosts, resources) is interpreted against the active configuration. Today a configuration upgrade does config-push + restart + per-node WAL replay under the new config (config_repl.c do_upgrade_config_with_bytes ~1001-1163), with no snapshot pinning an identical post-upgrade state. Because replay is config-dependent (the lost_and_found queue/host re-homes entities whose config entity vanished), members can materialize divergent state after an upgrade — amplified by any WAL tail divergence. Logged as bug-857 (identified via the raft.tla gap analysis; not yet empirically reproduced).

This was originally Requirement 8 of ha-term-safety; it is descoped into this independent spec (2026-06-21) because it has a hard prerequisite that does not exist yet.

Approach

A configuration upgrade establishes a full, entity-bearing state snapshot at the configuration-change boundary (the (term, lsn) of the config-change marker that ha-term-safety introduces), embeds the new configuration in that snapshot, and installs the same snapshot on every member through the out-of-band snapshot-install transport. No member re-derives post-upgrade state by replaying pre-upgrade WAL under the new configuration — installing the snapshot delivers configuration + materialized state together as one consistent artifact. This supersedes ha-term-safety R7's interim config-push byte delivery for the upgrade case.

HARD dependency — entity-bearing snapshots

This spec cannot proceed without entity-bearing snapshots. Today snapshots are a 16-byte LSN sentinel ("XCESNAP\0" + checkpoint_lsn; persist.c write ~1068-1101, load ~1280-1307) carrying zero entity state. The COW machinery needed (xzarray_freeze/thaw, zarray.c:695-757) is built and unit-tested but dormant (kvmgr.c:297 vfreeze=NULL, no production callers) — the roadmap's unstarted entity-bearing-snapshots ("Task 11") backlog item. That work must land first (or be co-scoped). The snapshot transport (snapshot_transfer_s: chunked, (term,lsn)-anchored, hash-verified, install-before-activate) is already reusable; only the entity-bearing payload is missing.

Scope

In: full entity-bearing state snapshot materialized at the config-change marker boundary; embed the new configuration in that snapshot; install on every member via the snapshot-install path; activation gating (a member does not serve under the new configuration until it has installed the upgrade snapshot and verified the embedded configuration against the marker's content hash); post-upgrade consistency invariant (every member's materialized state + active configuration identical); supersede ha-term-safety R7's interim config-push byte delivery for upgrades.
Out: the config-change marker / consensus ordering itself (owned by ha-term-safety R7); the entity serialization machinery (owned by entity-bearing-snapshots — this spec consumes it); the Raft protocol (the Raft TLA+ Conformance library series); controller voting-set/membership changes (ha-config-bootstrap).

Verification

Empirically reproduce bug-857 first (upgrade config during the chaotic-multi-kill repro and diff materialized state across nodes via waldump/state dump) to establish the RED, then prove every member converges to identical materialized state + active config after an upgrade interleaved with failover.

Dependencies

HARD: entity-bearing-snapshots (entity payload), ha-term-safety (the config-change marker it anchors on). Consumes the snapshot_transfer_s transport. Closes bug-857.

3.6 KiB Raw Blame History