diff --git a/.kiro/specs/raft-ha-core-safety/research.md b/.kiro/specs/raft-ha-core-safety/research.md index 2a364d5..9ff6ae4 100644 --- a/.kiro/specs/raft-ha-core-safety/research.md +++ b/.kiro/specs/raft-ha-core-safety/research.md @@ -206,3 +206,19 @@ Code-path mapping (read at the research anchors): - `~/tmp/ha19/` — frozen 3-member deploy with the reproduced LSN 9/10/11 payload divergence (`controller/ctrl-{1,2,3}/data`, `logs/ctrl-1.log`). - `~/tmp/ha19-drive.log` — driver round-by-round output (divergence first seen round 1). - `~/tmp/ha-1p3-run{1,2}.log`, `~/tmp/ha-1p3-seq.log` — the cold-start log-buffer "stall" runs. + +--- + +# Phase-2 interim live validation (post-2.4, real-fs release rebuild) + +Rebuilt the release controller at HEAD 63b9c5a (2.2 payload-aware guard + 2.4 streamer fail-loud) and ran a chaotic soak on real ext4 (`~/tmp/ha-verify-2p4`). The 4-round run **wedged (timed out ~9m40s) on an election stall** before reaching its own verdict — the un-fixed Phase-5 liveness defect. Froze the WALs and measured directly with `walcompare --report`: + +- **Universally-held committed prefix (LSN 1..11) is content-identical across all 3 members** (`walcompare: OK — committed prefixes identical`). +- **No present-but-empty committed divergence** — the bug-1.3 signature (one member `pcrc=0/PLEN=0` PRESENT where the majority holds a full payload) is **absent**. The `pcrc=00000000` entries at LSN 1/8/12/16 are present+identical on ALL members (legitimate empty-payload marker/NOOP entries, same term+crc everywhere). +- Remaining apparent divergences are all `status=MISSING` tail entries (ctrl-1 tip=19, ctrl-2=15, ctrl-3=11) — **catch-up lag** on lagging/killed members, not committed-content corruption. Classifying these as "committed divergence" only happens when an inflated `--committed` boundary over-claims the committed region; at the true majority-held boundary they are uncommitted tail / in-flight catch-up. + +**Conclusion:** the Phase-2 fix empirically eliminates the cross-term payload-loss divergence on the live repro. A *clean end-to-end* `committed_divergence=0` soak verdict is now gated on: +1. **Phase 5 (election liveness under churn)** — the chaotic soak wedges on an election stall before completing, so it never reaches its verdict line. This is the top blocker for the acceptance gate (7.2). +2. **Phase 4 (standby reconvergence)** — lagging members must finish catching up so the tail `MISSING` resolves once the cluster stabilizes. + +**Steering implication:** Phase 3 (snapshot-boundary reconcile) is defense-in-depth for a path NOT implicated in the live repro; Phases 5 and 4 are on the critical path to a clean soak verdict. Consider doing 5 (and 4) before 3.