feat(config-upgrade-snapshot): add materialization-and-hash-verification activation fence

Adds a new, independent fence (ha_state_t.upgrade_materialization_fence)
distinct from the existing config-byte serve-fence: that fence tracks
whether the locally-held config bytes hash-match the committed marker;
this one tracks whether THIS MEMBER'S OWN upgrade-snapshot
materialization (task 5's marker-triggered checkpoint) has actually
completed for the current committed marker, with the live on-disk
config bytes still hash-verifying against it right now.

upgrade_materialization_verify(c) mirrors config_activate_committed's
shape: reads the current committed marker, checks whether
persist.cfgref_present/cfgref_tag/cfgref_sha already reflect it (stamped
by every checkpoint via scan_latest_committed_config_ref -- proof this
member's own materialization ran for this exact marker), and separately
re-reads the LIVE on-disk config bytes fresh on every call (never a
cached/embedded copy) to hash-verify them too. Both must hold for the
fence to clear. Logs loud only on the 0->1 transition, never
exits/aborts, always fail-loud + fence + stay alive + retriable.
upgrade_materialization_fenced(c) is the trivial atomic-read accessor.

Not yet wired into the dispatch gate, promotion gate, or an automatic
retry trigger -- that's task 6.2. Task 6.1 of 19.

Requirements: 1.8, 3.2, 3.4
This commit is contained in:
Rongsong Shen 2026-07-25 02:51:49 +00:00
parent 1da26b043e
commit 7b507156a0
4 changed files with 509 additions and 1 deletions

View file

@ -53,7 +53,7 @@
- _Depends: 2.1, 2.2, 2.3_
- [ ] 6. Core: activation fence gating serving and promotion on materialization
- [ ] 6.1 (P) Add a materialization-and-hash-verification fence independent of the existing configuration-byte fence
- [x] 6.1 (P) Add a materialization-and-hash-verification fence independent of the existing configuration-byte fence
- Introduce a new fence condition distinct from the existing configuration-byte-mismatch fence, since "bytes don't match" and "materialization for a matching configuration isn't done yet" are different conditions that must be distinguishable
- Verify by reading the current committed configuration-change marker and this member's own materialized-and-verified state for it, always re-reading the live, currently-active configuration bytes on disk on every verification attempt rather than trusting a previously-embedded or cached copy
- Set the fence and log the actionable mismatch loudly on failure; clear it on success; never terminate the process on this condition
@ -149,6 +149,8 @@
## Implementation Notes
- **Task 6.1 process note:** the Claude account's monthly API spend limit was hit mid-task (implementer subagent cut off after landing `ha.c`/`ha.h` but before tests/build). The orchestrator finished this task directly (no subagent implementer/reviewer pair) rather than stopping: verified the partial diff, wrote 4 tests into `test_config_repl.c` (reusing existing `t42_/t43_` fixtures rather than creating `test_upgrade_activation_fence.c` — that file is task 8.3's own boundary per design.md, not 6.1's), built, ran the full suite (1001 Tests 0 Failures 0 Ignored), and self-reviewed the diff in lieu of an independent reviewer subagent. Flagging this so a future session/reviewer knows this task's diff did not go through the normal two-subagent review gate.
- **Task 5 correction to design.md:** design.md's "Marker Apply Trigger" section says "Call `persist_checkpoint()` directly" — this is unsafe as written. `persist_apply_pump_run` holds `apply_pump_lock` across the whole dispatch loop (including the `apply_committed_entry` call), and `persist_checkpoint()` itself re-acquires that same non-recursive mutex — a real self-deadlock on the live pump path (primary AND standby online-apply). Separately, `last_applied_lsn`/replay position only advance AFTER `apply_committed_entry` returns at both call sites, so pinning `checkpoint_lsn` from `last_applied_lsn` inside the marker case would mislabel the checkpoint with the previous entry's LSN, not the marker's own. Fixed via `persist_checkpoint_write_locked(p, checkpoint_lsn)` (no self-lock, explicit LSN param) + `persist_checkpoint_marker_triggered(p, lsn)`, called from `apply_committed_entry`'s new `WAL_CONFIG_MARKER` case with the marker's own `lsn` parameter. design.md should be updated to reflect this; not done as part of task 5 (docs-only, deferred).
- **Task 5 → ha-term-safety/raft-log-replication finding (bug-1466, NOT this spec's to fix):** `persist_write_checkpoint_marker` advances `p->current_lsn` directly, bypassing `ha_primary_log_append`/`raft_log_append`, breaking `ha_primary_log_append`'s own documented `p->current_lsn == s->log_len + 1` invariant (ha.c:4054-4060) once any checkpoint interleaves with primary appends. Pre-existing (periodic checkpoint already did this); task 5 just made the interleaving common instead of rare, surfacing it as real test flakiness for the first time. `ha->raft.log_len` feeds real RequestVote freshness comparisons (ha.c:258-261, 401-406) — NOT advisory-only as bug-1466's original text claimed; severity corrected in `.wolf/buglog.json`. Out of this spec's boundary (Raft log/term safety is `ha-term-safety`/raft-log-replication's territory) — flagging for whoever next owns that area.