From d89f7e41bc787bf8d2d527150a8d71e45f50725d Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 27 Jun 2026 15:56:50 +0000 Subject: [PATCH] =?UTF-8?q?research(raft-ha-core-safety):=20Phase-2=20inte?= =?UTF-8?q?rim=20live=20validation=20=E2=80=94=20payload-loss=20eliminated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rebuilt release at HEAD (2.2+2.4), chaotic soak on real ext4. Universally- held committed prefix (1..11) content-identical across all members; the present-but-empty committed divergence (bug-1.3 signature) is ABSENT. The fix eliminates the cross-term payload-loss divergence. Remaining issues: chaotic soak wedges on election stall (Phase 5) before its verdict; tail MISSING = catch-up lag (Phase 4). Clean soak verdict gated on Phase 5/4, not Phase 3 (snapshot, off the critical path). --- .kiro/specs/raft-ha-core-safety/research.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/.kiro/specs/raft-ha-core-safety/research.md b/.kiro/specs/raft-ha-core-safety/research.md index 2a364d5..9ff6ae4 100644 --- a/.kiro/specs/raft-ha-core-safety/research.md +++ b/.kiro/specs/raft-ha-core-safety/research.md @@ -206,3 +206,19 @@ Code-path mapping (read at the research anchors): - `~/tmp/ha19/` — frozen 3-member deploy with the reproduced LSN 9/10/11 payload divergence (`controller/ctrl-{1,2,3}/data`, `logs/ctrl-1.log`). - `~/tmp/ha19-drive.log` — driver round-by-round output (divergence first seen round 1). - `~/tmp/ha-1p3-run{1,2}.log`, `~/tmp/ha-1p3-seq.log` — the cold-start log-buffer "stall" runs. + +--- + +# Phase-2 interim live validation (post-2.4, real-fs release rebuild) + +Rebuilt the release controller at HEAD 63b9c5a (2.2 payload-aware guard + 2.4 streamer fail-loud) and ran a chaotic soak on real ext4 (`~/tmp/ha-verify-2p4`). The 4-round run **wedged (timed out ~9m40s) on an election stall** before reaching its own verdict — the un-fixed Phase-5 liveness defect. Froze the WALs and measured directly with `walcompare --report`: + +- **Universally-held committed prefix (LSN 1..11) is content-identical across all 3 members** (`walcompare: OK — committed prefixes identical`). +- **No present-but-empty committed divergence** — the bug-1.3 signature (one member `pcrc=0/PLEN=0` PRESENT where the majority holds a full payload) is **absent**. The `pcrc=00000000` entries at LSN 1/8/12/16 are present+identical on ALL members (legitimate empty-payload marker/NOOP entries, same term+crc everywhere). +- Remaining apparent divergences are all `status=MISSING` tail entries (ctrl-1 tip=19, ctrl-2=15, ctrl-3=11) — **catch-up lag** on lagging/killed members, not committed-content corruption. Classifying these as "committed divergence" only happens when an inflated `--committed` boundary over-claims the committed region; at the true majority-held boundary they are uncommitted tail / in-flight catch-up. + +**Conclusion:** the Phase-2 fix empirically eliminates the cross-term payload-loss divergence on the live repro. A *clean end-to-end* `committed_divergence=0` soak verdict is now gated on: +1. **Phase 5 (election liveness under churn)** — the chaotic soak wedges on an election stall before completing, so it never reaches its verdict line. This is the top blocker for the acceptance gate (7.2). +2. **Phase 4 (standby reconvergence)** — lagging members must finish catching up so the tail `MISSING` resolves once the cluster stabilizes. + +**Steering implication:** Phase 3 (snapshot-boundary reconcile) is defense-in-depth for a path NOT implicated in the live repro; Phases 5 and 4 are on the critical path to a clean soak verdict. Consider doing 5 (and 4) before 3.