wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

spec(ha-commit-durability): 6.1 answered — durability loss + liveness are REAL (reproduce off overlay), not an fdatasync artifact; Phase 6 now at 6.2 root-cause

Browse source
This commit is contained in:
Rongsong Shen 2026-06-20 03:19:57 +00:00
parent 2988682bd6
commit 480772c108
2 changed files with 17 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

6
.wolf/buglog.json
View file

@ -13003,8 +13003,8 @@
"id": "bug-795",
"timestamp": "2026-06-19",
"error_message": "Cold-restart-all (kill ALL 3 controllers + restart) non-deterministically loses a committed job: a job that returned 'jobsubmit ok' and survived every rolling single-node failover round vanishes after a full-cluster restart (expect-jobs expected N, actual N-2; expect-job <id> not found). Recovery logs show WAL truncate.",
"file": "src/controller/persist.c (recovery/truncation) + commit/replication path \u2014 UPSTREAM (ha-commit-durability), NOT xce-test-e2e-primitives",
"root_cause": "Commit-durability race under TOTAL-cluster-outage recovery. Acked (committed) entry not durable on a fault-tolerant majority before all nodes die, OR recovery truncates committed entries on cold restart. Non-deterministic (passed 1 run, failed next). Surfaced on overlayfs sandbox (NOT a real fsync-able fs) \u2014 must be reconfirmed on real hardware. UPDATE: with the hardened (trustworthy) soak, the loss ALSO reproduces in ROLLING single-node failover with the cluster confirmed serving (~1/3 runs) \u2014 broader than cold-restart-all; bug-661 not fully closed for rolling failover. Separate finding: >60s post-kill election (liveness) every run, always recovers.",
"file": "src/controller/{persist.c recovery/truncate, commit.c commit-gate, ha.c promotion-fence/election} \u2014 UPSTREAM ha-commit-durability Phase 6.2",
"root_cause": "Commit-durability race under TOTAL-cluster-outage recovery. Acked (committed) entry not durable on a fault-tolerant majority before all nodes die, OR recovery truncates committed entries on cold restart. Non-deterministic (passed 1 run, failed next). Surfaced on overlayfs sandbox (NOT a real fsync-able fs) \u2014 must be reconfirmed on real hardware. UPDATE: with the hardened (trustworthy) soak, the loss ALSO reproduces in ROLLING single-node failover with the cluster confirmed serving (~1/3 runs) \u2014 broader than cold-restart-all; bug-661 not fully closed for rolling failover. Separate finding: >60s post-kill election (liveness) every run, always recovers. CORRECTION: NOT an overlay-fdatasync artifact \u2014 reproduces OUTSIDE the container (user's real host) AND on an in-sandbox real ext2/3 fs with ZERO fdatasync EIO. Real controller durability bug + a coupled liveness stall (>60s / stuck election after killing ctrl-1). Phase 6.1 answered (REAL); now at 6.2 root-cause.",
"fix": "NOT FIXED HERE \u2014 out of boundary for xce-test-e2e-primitives (this spec changes no controller/replication code). Routed to spec ha-commit-durability owner. The converted durability soak (src/tools/test-commit-durability.sh, new run_cold_restart_recovery phase) correctly detects it via the exit-code verdict.",
"tags": [
"ha-commit-durability",
@ -13024,7 +13024,7 @@
"bug-778",
"bug-786"
],
"occurrences": 2,
"occurrences": 3,
"last_seen": "2026-06-19"
},
{