ha-commit-durability: closes the "pending confirmation under simultaneous
total outage" caveat. Reproduced cold-restart-all (kill all 3 members +
restart) 5 times against a live 3-node cluster on genuine ext4 (not the
sandbox's overlayfs) — zero committed-job loss across 40 rolling-failover
rounds + 5 cold-restart-all cycles. Confirms bug-795 was an overlayfs
fdatasync artifact, not a real durability gap; no production fix needed.
Commit Durability is now proven for both rolling single-node failover and
simultaneous total-cluster outage (src/controller/CLAUDE.md updated).
Live-validated and marked complete against a real 3-node cluster:
- controller-status 6.4: off-primary committed fidelity + responder
timestamp, confirmed via a real failover
- ha-heartbeat-committed-propagation 4.1: standby committedLsn tracks the
primary via heartbeat within the 3s interval
- ha-wal-replication 5.4: standby WAL replication verified via committedLsn
match (job-facing ops are correctly primary-only by design; standbys
return E_XCE_STANDBY, not a bug)
- libxce-failover 6.2: live 3-node failover E2E, jobsub raced against a
primary kill resolves to exactly one job, no duplication
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Add ctrlstatus_fmt.h (ctrlstatus_format_time_ms): renders responder_time_ms as
local YYYY-MM-DD HH:MM:SS,mmm, byte-equivalent to the controller log prefix
(tlog/my_tlog_format) so a captured ctrlstatus lines up against ctrl-N.log.
cmd_ctrlstatus prints a responderTime line; TZ-pinned exact-string unit test.
Req 11.2, 11.3. ctrlstatus group passes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Role-branch the committed sourcing: PRIMARY reports its own authoritative
persist.committed_lsn / committed_config_tag; off-primary reports the LEARNED
ha->last_known_committed_lsn / last_known_committed_config_tag instead of the
primary-only raw markers that read 0 on a standby (fixes the applied>committed
inversion). Stamp responder_time_ms from CLOCK_REALTIME for log correlation.
Read-only (Req 10.1-10.4, 11.1). 4 new TDD tests; controller suite 110/0/0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Append-only UInt64 epoch-millis field for the responder snapshot timestamp
(Req 11.1, wire-format setup). Hand-authored xce_ctrlstatus_t typemap gains a
matching responder_time_ms member; codec round-trips it. Controller suite 1/1 OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
