spec(ha-commit-durability): sharpen bug-795 — durability loss reproduces in rolling failover (trustworthy soak) + >60s election liveness finding

This commit is contained in:
Rongsong Shen 2026-06-20 01:28:26 +00:00
parent d82cc78284
commit c205b5b25d
2 changed files with 143 additions and 3 deletions

View file

@ -195,3 +195,12 @@ No CRITICAL/HIGH residuals. No criterion is **Missing** or **Unknown**: 16/16 ve
- The durable-gating (2.2/2.4) should make `committed_lsn` HOLD if `fdatasync` failed — so an un-fdatasync'd job should NOT have returned `jobsubmit ok`. Yet these jobs DID get `jobsubmit ok`. So either `fdatasync` returned success on overlay without real durability, or a non-durable-gated ack path exists under the total-outage timing, or recovery truncates committed entries.
**Disposition:** the `xce-test-e2e-primitives` soak is the reproduction surface and is working correctly (it detects the loss via exit codes). The fix (if real) belongs HERE. See Phase 6.
### UPDATE 2026-06-19b — trustworthy soak sharpens the finding (rolling-failover loss + liveness)
After the `xce-test-e2e-primitives` soak was HARDENED (commit d82cc78: it no longer conflates liveness with durability — durability is asserted ONLY against a confirmed-serving cluster, with a catch-up-lag confirm re-check; a non-serving cluster is INCONCLUSIVE, never "DATA LOSS"), three 6–10-round live runs give a cleaner signal:
1. **The committed-job loss reproduces in ROLLING single-node failover too — not only cold-restart-all.** One run lost a committed job at round 6 with the cluster CONFIRMED SERVING (`DATA LOSS — persistently absent from a SERVING cluster`); two other runs had zero loss. So the durability race is NON-DETERMINISTIC (~1 in 3 runs here) and broader than the cold-restart-all framing in the original reopen note — it is a general acked-but-not-durable-on-majority race that a kill can expose during normal rolling failover. **This is exactly the bug-661 invariant this spec claimed CLOSED for rolling failover (task 5.2) — it is not fully closed.** (Caveat still applies: sandbox `/tmp` is overlayfs; 6.1 must confirm on a real fsync-able fs.)
2. **Separate LIVENESS concern:** every run shows 2–3 rounds where, after killing the current primary (consistently `ctrl-1`), NO new primary served within 60s (the cluster always eventually recovered). A >60s failover for a single-node kill in a 3-node cluster is poor availability and may share a root cause with the durability race (a behind/just-rejoined node interacting with the promotion fence / freshness floor). Worth a dedicated liveness check in 6.2.
**Phase 6 scope note:** 6.1's reproduction MUST include BOTH the cold-restart-all step AND the rolling-failover rounds (the hardened soak runs both and reports trustworthy per-round verdicts), and should capture the >60s-election cases for the liveness angle.