---
id: "bug-857"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-06-21T10:30:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/config_repl.c do_upgrade_config_with_bytes (~1001-1163)"]
tags: ["config", "snapshot", "ha", "needs-verification", "spec-needed", "descoped", "config-upgrade-snapshot", "entity-bearing-snapshots", "state-divergence", "raft-safety", "upgrade"]
related: ["bug-851", "bug-828"]
legacy: true
---

# bug-857: Config upgrade can leave members with DIVERGENT materialized state: do_upgrade_config_with_bytes…

## Description
Config upgrade can leave members with DIVERGENT materialized state: do_upgrade_config_with_bytes pushes config bytes only then restart_requested; each node re-derives state by replaying its WAL under the NEW config, with no snapshot pinning an identical post-upgrade state. Replay is config-dependent (lost_and_found re-homes orphaned jobs), so nodes can diverge; amplified by bug-851 WAL tail divergence.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
Config is a separate consistency domain (committed_config_tag, no term, tag anti-entropy) outside both the WAL log and raft.c. Upgrade activation = config push + restart + per-node WAL replay under new config; no snapshot transfers an identical materialized state, and state is interpreted against config. Identified via TLA+ gap analysis (config should be log-entry/snapshot-anchored). NOT empirically reproduced yet.

## Fix
FIXED AND VERIFIED LIVE -- config-upgrade-snapshot (the spec this was descoped into) is fully task-complete: every task 1 through 8.9 is done, including task 8.5 ('Reproduce bug-857 on the pre-change behavior and confirm it no longer occurs') and task 8.6 (a live chaotic multi-kill soak, 2026-07-29, confirming zero materialized-state/config divergence after a config upgrade). The fix materializes a full entity-bearing state snapshot at the config-change marker (term,lsn) boundary and installs it on every member, exactly as originally planned. Note: config-upgrade-snapshot/spec.json's own `phase` field was left stale at 'tasks-generated' despite the spec being fully implemented -- corrected separately alongside this entry.

[Original entry]
NOT FIXED. Descoped 2026-06-21 from ha-term-safety R8 into the independent `config-upgrade-snapshot` spec (.kiro/specs/config-upgrade-snapshot/, requirements generated). Fix = upgrade materializes a FULL entity-bearing state snapshot at the config-change marker (term,lsn) boundary, embeds new config, installs on every member; no per-node replay-under-new-config. HARD dependency: entity-bearing-snapshots (snapshots are a 16-byte LSN sentinel today; COW xzarray_freeze/thaw built but dormant). Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state).

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-851, bug-828