wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

research(raft-ha-core-safety): Phase-2 interim live validation — payload-loss eliminated

Browse source 
Rebuilt release at HEAD (2.2+2.4), chaotic soak on real ext4. Universally-
held committed prefix (1..11) content-identical across all members; the
present-but-empty committed divergence (bug-1.3 signature) is ABSENT. The
fix eliminates the cross-term payload-loss divergence. Remaining issues:
chaotic soak wedges on election stall (Phase 5) before its verdict; tail
MISSING = catch-up lag (Phase 4). Clean soak verdict gated on Phase 5/4,
not Phase 3 (snapshot, off the critical path).
This commit is contained in:
Rongsong Shen 2026-06-27 15:56:50 +00:00
parent 63b9c5a81a
commit d89f7e41bc
1 changed files with 16 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

16
.kiro/specs/raft-ha-core-safety/research.md
View file

@ -206,3 +206,19 @@ Code-path mapping (read at the research anchors):
- `~/tmp/ha19/` — frozen 3-member deploy with the reproduced LSN 9/10/11 payload divergence (`controller/ctrl-{1,2,3}/data`, `logs/ctrl-1.log`).
- `~/tmp/ha19-drive.log` — driver round-by-round output (divergence first seen round 1).
- `~/tmp/ha-1p3-run{1,2}.log`, `~/tmp/ha-1p3-seq.log` — the cold-start log-buffer "stall" runs.
---
# Phase-2 interim live validation (post-2.4, real-fs release rebuild)
Rebuilt the release controller at HEAD 63b9c5a (2.2 payload-aware guard + 2.4 streamer fail-loud) and ran a chaotic soak on real ext4 (`~/tmp/ha-verify-2p4`). The 4-round run **wedged (timed out ~9m40s) on an election stall** before reaching its own verdict — the un-fixed Phase-5 liveness defect. Froze the WALs and measured directly with `walcompare --report`:
- **Universally-held committed prefix (LSN 1..11) is content-identical across all 3 members** (`walcompare: OK — committed prefixes identical`).
- **No present-but-empty committed divergence** — the bug-1.3 signature (one member `pcrc=0/PLEN=0` PRESENT where the majority holds a full payload) is **absent**. The `pcrc=00000000` entries at LSN 1/8/12/16 are present+identical on ALL members (legitimate empty-payload marker/NOOP entries, same term+crc everywhere).
- Remaining apparent divergences are all `status=MISSING` tail entries (ctrl-1 tip=19, ctrl-2=15, ctrl-3=11) — **catch-up lag** on lagging/killed members, not committed-content corruption. Classifying these as "committed divergence" only happens when an inflated `--committed` boundary over-claims the committed region; at the true majority-held boundary they are uncommitted tail / in-flight catch-up.
**Conclusion:** the Phase-2 fix empirically eliminates the cross-term payload-loss divergence on the live repro. A *clean end-to-end* `committed_divergence=0` soak verdict is now gated on:
1. **Phase 5 (election liveness under churn)** — the chaotic soak wedges on an election stall before completing, so it never reaches its verdict line. This is the top blocker for the acceptance gate (7.2).
2. **Phase 4 (standby reconvergence)** — lagging members must finish catching up so the tail `MISSING` resolves once the cluster stabilizes.
**Steering implication:** Phase 3 (snapshot-boundary reconcile) is defense-in-depth for a path NOT implicated in the live repro; Phases 5 and 4 are on the critical path to a clean soak verdict. Consider doing 5 (and 4) before 3.