lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

spec(ha-commit-durability): 6.1 answered — durability loss + liveness are REAL (reproduce off overlay), not an fdatasync artifact; Phase 6 now at 6.2 root-cause

Browse source
This commit is contained in:
Rongsong Shen 2026-06-20 03:19:57 +00:00
parent 2988682bd6
commit 480772c108
2 changed files with 17 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

14
.kiro/specs/ha-commit-durability/research.md
View file

@ -204,3 +204,17 @@ After the `xce-test-e2e-primitives` soak was HARDENED (commit d82cc78: it no lon
2. **Separate LIVENESS concern:** every run shows 2–3 rounds where, after killing the current primary (consistently `ctrl-1`), NO new primary served within 60s (the cluster always eventually recovered). A >60s failover for a single-node kill in a 3-node cluster is poor availability and may share a root cause with the durability race (a behind/just-rejoined node interacting with the promotion fence / freshness floor). Worth a dedicated liveness check in 6.2.
**Phase 6 scope note:** 6.1's reproduction MUST include BOTH the cold-restart-all step AND the rolling-failover rounds (the hardened soak runs both and reports trustworthy per-round verdicts), and should capture the >60s-election cases for the liveness angle.
### UPDATE 2026-06-19c — 6.1 ANSWERED: the loss is REAL, not (only) an overlay artifact
The cold-restart-all / rolling durability loss was initially suspected to be an overlay-`/tmp` `fdatasync`-EIO artifact of the dev sandbox. That is now ruled out as the primary cause:
- **User runs the soak OUTSIDE the container** (real host, real fsync-able fs, NO overlay `fdatasync` EIO) and still observes the DATA LOSS + the liveness failures. So they are genuine, not sandbox `fdatasync` noise.
- In-sandbox confirmation on a **real ext2/3 deploy dir** (`fdatasync` succeeds, ZERO EIO): a 3-round run PASSED clean, but an 8-round run reproduced BOTH a liveness cascade (after killing `ctrl-1` in round 1 the cluster never returned to 3-healthy) AND a cold-restart `expect-jobs` shortfall with a PERSISTENTLY-absent committed job — with **zero `fdatasync` EIO**. (Caveat: the 8-round run's cold-restart followed a degraded run, so the cleanest real-fs evidence is the user's outside-container host.)
**Verdict for 6.1:** the durability loss and the liveness stall reproduce off overlay → REAL controller findings, not an `fdatasync`-EIO artifact. The overlay EIO remains a SEPARATE, sandbox-only confound (it makes the dev-box `controller_unit_test` flaky — see bug-760 — and inflates loss on overlay deploys), but it is not the root cause of the soak loss.
**Phase 6 is now at 6.2 (root-cause).** Two real, non-deterministic findings to root-cause:
- **Durability:** an acked (`jobsubmit ok`) job is PERSISTENTLY absent from a SERVING cluster after failover / cold restart — i.e. bug-661 is NOT fully closed (Phase 5 only exercised rolling single-node kills with the racy soak). Buckets: acked-without-durable-majority (commit-gate timing) / recovery truncates committed entries / behind-then-killed node re-leads with a truncated log.
- **Liveness:** after killing the current primary (consistently `ctrl-1` in these runs), the cluster sometimes takes >60s — or fails entirely — to elect a serving primary, and a restarted node sometimes never returns to a healthy 3-node serving set. Possibly coupled to the promotion-fence / freshness-floor interaction with a just-rejoined node.
The hardened `xce-test-e2e-primitives` soak (trustworthy: liveness vs durability separated, confirm-settle, quiet wait-primary) is the reproduction surface for both.