From 480772c108e56101f81cb9f2850d4997ca796159 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 20 Jun 2026 03:19:57 +0000 Subject: [PATCH] =?UTF-8?q?spec(ha-commit-durability):=206.1=20answered=20?= =?UTF-8?q?=E2=80=94=20durability=20loss=20+=20liveness=20are=20REAL=20(re?= =?UTF-8?q?produce=20off=20overlay),=20not=20an=20fdatasync=20artifact;=20?= =?UTF-8?q?Phase=206=20now=20at=206.2=20root-cause?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .kiro/specs/ha-commit-durability/research.md | 14 ++++++++++++++ .wolf/buglog.json | 6 +++--- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/.kiro/specs/ha-commit-durability/research.md b/.kiro/specs/ha-commit-durability/research.md index cd8c96d..2954ae0 100644 --- a/.kiro/specs/ha-commit-durability/research.md +++ b/.kiro/specs/ha-commit-durability/research.md @@ -204,3 +204,17 @@ After the `xce-test-e2e-primitives` soak was HARDENED (commit d82cc78: it no lon 2. **Separate LIVENESS concern:** every run shows 2–3 rounds where, after killing the current primary (consistently `ctrl-1`), NO new primary served within 60s (the cluster always eventually recovered). A >60s failover for a single-node kill in a 3-node cluster is poor availability and may share a root cause with the durability race (a behind/just-rejoined node interacting with the promotion fence / freshness floor). Worth a dedicated liveness check in 6.2. **Phase 6 scope note:** 6.1's reproduction MUST include BOTH the cold-restart-all step AND the rolling-failover rounds (the hardened soak runs both and reports trustworthy per-round verdicts), and should capture the >60s-election cases for the liveness angle. + +### UPDATE 2026-06-19c — 6.1 ANSWERED: the loss is REAL, not (only) an overlay artifact + +The cold-restart-all / rolling durability loss was initially suspected to be an overlay-`/tmp` `fdatasync`-EIO artifact of the dev sandbox. That is now ruled out as the primary cause: +- **User runs the soak OUTSIDE the container** (real host, real fsync-able fs, NO overlay `fdatasync` EIO) and still observes the DATA LOSS + the liveness failures. So they are genuine, not sandbox `fdatasync` noise. +- In-sandbox confirmation on a **real ext2/3 deploy dir** (`fdatasync` succeeds, ZERO EIO): a 3-round run PASSED clean, but an 8-round run reproduced BOTH a liveness cascade (after killing `ctrl-1` in round 1 the cluster never returned to 3-healthy) AND a cold-restart `expect-jobs` shortfall with a PERSISTENTLY-absent committed job — with **zero `fdatasync` EIO**. (Caveat: the 8-round run's cold-restart followed a degraded run, so the cleanest real-fs evidence is the user's outside-container host.) + +**Verdict for 6.1:** the durability loss and the liveness stall reproduce off overlay → REAL controller findings, not an `fdatasync`-EIO artifact. The overlay EIO remains a SEPARATE, sandbox-only confound (it makes the dev-box `controller_unit_test` flaky — see bug-760 — and inflates loss on overlay deploys), but it is not the root cause of the soak loss. + +**Phase 6 is now at 6.2 (root-cause).** Two real, non-deterministic findings to root-cause: +- **Durability:** an acked (`jobsubmit ok`) job is PERSISTENTLY absent from a SERVING cluster after failover / cold restart — i.e. bug-661 is NOT fully closed (Phase 5 only exercised rolling single-node kills with the racy soak). Buckets: acked-without-durable-majority (commit-gate timing) / recovery truncates committed entries / behind-then-killed node re-leads with a truncated log. +- **Liveness:** after killing the current primary (consistently `ctrl-1` in these runs), the cluster sometimes takes >60s — or fails entirely — to elect a serving primary, and a restarted node sometimes never returns to a healthy 3-node serving set. Possibly coupled to the promotion-fence / freshness-floor interaction with a just-rejoined node. + +The hardened `xce-test-e2e-primitives` soak (trustworthy: liveness vs durability separated, confirm-settle, quiet wait-primary) is the reproduction surface for both. diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 05dbd1b..8f2aed2 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -13003,8 +13003,8 @@ "id": "bug-795", "timestamp": "2026-06-19", "error_message": "Cold-restart-all (kill ALL 3 controllers + restart) non-deterministically loses a committed job: a job that returned 'jobsubmit ok' and survived every rolling single-node failover round vanishes after a full-cluster restart (expect-jobs expected N, actual N-2; expect-job not found). Recovery logs show WAL truncate.", - "file": "src/controller/persist.c (recovery/truncation) + commit/replication path \u2014 UPSTREAM (ha-commit-durability), NOT xce-test-e2e-primitives", - "root_cause": "Commit-durability race under TOTAL-cluster-outage recovery. Acked (committed) entry not durable on a fault-tolerant majority before all nodes die, OR recovery truncates committed entries on cold restart. Non-deterministic (passed 1 run, failed next). Surfaced on overlayfs sandbox (NOT a real fsync-able fs) \u2014 must be reconfirmed on real hardware. UPDATE: with the hardened (trustworthy) soak, the loss ALSO reproduces in ROLLING single-node failover with the cluster confirmed serving (~1/3 runs) \u2014 broader than cold-restart-all; bug-661 not fully closed for rolling failover. Separate finding: >60s post-kill election (liveness) every run, always recovers.", + "file": "src/controller/{persist.c recovery/truncate, commit.c commit-gate, ha.c promotion-fence/election} \u2014 UPSTREAM ha-commit-durability Phase 6.2", + "root_cause": "Commit-durability race under TOTAL-cluster-outage recovery. Acked (committed) entry not durable on a fault-tolerant majority before all nodes die, OR recovery truncates committed entries on cold restart. Non-deterministic (passed 1 run, failed next). Surfaced on overlayfs sandbox (NOT a real fsync-able fs) \u2014 must be reconfirmed on real hardware. UPDATE: with the hardened (trustworthy) soak, the loss ALSO reproduces in ROLLING single-node failover with the cluster confirmed serving (~1/3 runs) \u2014 broader than cold-restart-all; bug-661 not fully closed for rolling failover. Separate finding: >60s post-kill election (liveness) every run, always recovers. CORRECTION: NOT an overlay-fdatasync artifact \u2014 reproduces OUTSIDE the container (user's real host) AND on an in-sandbox real ext2/3 fs with ZERO fdatasync EIO. Real controller durability bug + a coupled liveness stall (>60s / stuck election after killing ctrl-1). Phase 6.1 answered (REAL); now at 6.2 root-cause.", "fix": "NOT FIXED HERE \u2014 out of boundary for xce-test-e2e-primitives (this spec changes no controller/replication code). Routed to spec ha-commit-durability owner. The converted durability soak (src/tools/test-commit-durability.sh, new run_cold_restart_recovery phase) correctly detects it via the exit-code verdict.", "tags": [ "ha-commit-durability", @@ -13024,7 +13024,7 @@ "bug-778", "bug-786" ], - "occurrences": 2, + "occurrences": 3, "last_seen": "2026-06-19" }, {