ha-commit-durability: closes the "pending confirmation under simultaneous
total outage" caveat. Reproduced cold-restart-all (kill all 3 members +
restart) 5 times against a live 3-node cluster on genuine ext4 (not the
sandbox's overlayfs) — zero committed-job loss across 40 rolling-failover
rounds + 5 cold-restart-all cycles. Confirms bug-795 was an overlayfs
fdatasync artifact, not a real durability gap; no production fix needed.
Commit Durability is now proven for both rolling single-node failover and
simultaneous total-cluster outage (src/controller/CLAUDE.md updated).
Live-validated and marked complete against a real 3-node cluster:
- controller-status 6.4: off-primary committed fidelity + responder
timestamp, confirmed via a real failover
- ha-heartbeat-committed-propagation 4.1: standby committedLsn tracks the
primary via heartbeat within the 3s interval
- ha-wal-replication 5.4: standby WAL replication verified via committedLsn
match (job-facing ops are correctly primary-only by design; standbys
return E_XCE_STANDBY, not a bug)
- libxce-failover 6.2: live 3-node failover E2E, jobsub raced against a
primary kill resolves to exactly one job, no duplication
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Per design invariant (a controller exits only on crash or admin kill), replace the
self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable
fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual
admin recovery). Records the never-self-terminate invariant as project-wide guidance in
src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.
cleanup.sh is generated by deploy-ha.sh and already does the right thing:
keeps ctrl-1/conf/* and all certs, wipes ctrl-{2..N}/conf, all data/, and logs.
Update skill and CLAUDE.md to use it as the primary clean-restart command
instead of manual rm loops.
- .claude/skills/xce-deploy-ha/SKILL.md: new skill for deploying a
local 3-node XCE HA controller cluster. Covers build prerequisites,
deploy via src/tools/deploy-ha.sh, node startup with -vvv raft trace,
health verification checks, clean restart procedure, and a
troubleshooting table for common failure modes.
- src/controller/CLAUDE.md: add HA Cluster Deployment section with
quick-deploy commands, health check one-liners, and reference to the
xce-deploy-ha skill. Add src/tools/deploy-ha.sh to key files.
