---
id: "bug-795"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-06-19"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 3
files: ["src/controller/{persist.c recovery/truncate", "commit.c commit-gate", "ha.c promotion-fence/election} — UPSTREAM ha-commit-durability Phase 6.2"]
tags: ["ha-commit-durability", "commit-durability", "cold-restart", "total-outage", "wal-truncate", "upstream", "xce-test-e2e-primitives", "needs-real-fs", "rolling-failover", "liveness", "non-deterministic"]
related: ["bug-661", "bug-778", "bug-786"]
fix_commits: ["b77d412"]
legacy: true
---

# bug-795: Cold-restart-all (kill ALL 3 controllers + restart) non-deterministically loses a committed job: a…

## Description
Cold-restart-all (kill ALL 3 controllers + restart) non-deterministically loses a committed job: a job that returned 'jobsubmit ok' and survived every rolling single-node failover round vanishes after a full-cluster restart (expect-jobs expected N, actual N-2; expect-job <id> not found). Recovery logs show WAL truncate.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
Commit-durability race under TOTAL-cluster-outage recovery. Acked (committed) entry not durable on a fault-tolerant majority before all nodes die, OR recovery truncates committed entries on cold restart. Non-deterministic (passed 1 run, failed next). Surfaced on overlayfs sandbox (NOT a real fsync-able fs) — must be reconfirmed on real hardware. UPDATE: with the hardened (trustworthy) soak, the loss ALSO reproduces in ROLLING single-node failover with the cluster confirmed serving (~1/3 runs) — broader than cold-restart-all; bug-661 not fully closed for rolling failover. Separate finding: >60s post-kill election (liveness) every run, always recovers. CORRECTION: NOT an overlay-fdatasync artifact — reproduces OUTSIDE the container (user's real host) AND on an in-sandbox real ext2/3 fs with ZERO fdatasync EIO. Real controller durability bug + a coupled liveness stall (>60s / stuck election after killing ctrl-1). Phase 6.1 answered (REAL); now at 6.2 root-cause.

## Fix
FIXED AND VERIFIED LIVE -- ha-commit-durability was reopened specifically for this bug (Phase 6, tasks 6.1-6.4). Task 6.1 attempted to reproduce the cold-restart-all loss against a REAL fsync-able filesystem (ext4, not this sandbox's overlayfs) and it did not reproduce -- the original loss was an overlayfs fdatasync artifact, not a genuine product defect (tasks 6.2/6.3 root-cause/fix were correctly marked N/A as a direct consequence). Task 6.4 re-validated on real fs and closed bug-795. spec.json's validate_impl (decision: GO, 2026-07-18) independently reconfirms this with a later, fresh live re-validation: a 4-round rolling SIGKILL+restart soak plus a full cold-restart-all cycle, 8/8 PASS, 0 job loss -- all 20 committed jobs survived both rolling failover and cold-restart-all. Distinct root cause from bug-828 (a premature-truncate bug during a live election under rolling single-node failover, fixed separately in ha-term-safety commit b77d412) -- not the same defect, despite early investigation notes grouping them together as 'bug-828/795'. One adjacent, still-open, low-priority item surfaced during the same validation: an extra/duplicate committed job appeared once during the original 2026-07-02 validation run (over-counting, the opposite symptom, not yet root-caused, does not block the GO) -- filed separately, not part of this entry.

[Original entry]
NOT FIXED HERE — out of boundary for xce-test-e2e-primitives (this spec changes no controller/replication code). Routed to spec ha-commit-durability owner. The converted durability soak (src/tools/test-commit-durability.sh, new run_cold_restart_recovery phase) correctly detects it via the exit-code verdict.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-661, bug-778, bug-786