checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.

.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md

@@ -16,19 +16,36 @@ Decoded the raw WAL bytes at `/tmp/xce-ha-deploy/controller/ctrl-{1,2,3}/data/wa
 
 The committed prefix (LSN 2–12 = the lost jobs) is **physically absent** from both live nodes; it exists only on the down node. `snap/` dirs are **empty** on all nodes (no snapshot).
 
+## DECISIVE EVIDENCE 2 — catch-up streaming DROPS payloads (bug-847, REPRODUCED 2026-06-20)
+A live restart of ctrl-1 (kill ctrl-1 → restart) showed it rejoin and catch up to the cluster length `[1..14]` with **correct LSN/type/CRC chain but EMPTY payloads** — `waldump` reports every JOB_SUBMIT as `PLEN=0 PCRC=00000000 jobid=?`, while ctrl-2/ctrl-3 still hold full 872-byte payloads. So the WAL is **structurally intact but content-empty** — silent corruption that passes every check that doesn't decode the payload (the `PCRC` column is what exposed it). This, not just the missing-prefix gap, is a concrete committed-job-loss mechanism: if the empty-payload node ever leads, the empty entries become authoritative.
+
+**Root cause (reproduced in a unit test):** the catch-up batch wire path (`xce_wal_batch_encode_raw`/`wal_entry_to_wire` in `src/lib/request.c`) builds a `WalCatchupEntry` struct-list whose per-element `payload :Data` is allocated via `capn_new_list8(cs=root_seg,…)`. When the batch outgrows its first capnp segment, the per-element payload `Data` pointer is **lost across the segment boundary** → on decode `extract_data_bytes(we.payload)` reads `CAPN_NULL` → `raw_payload_len=0` → the receiver writes a payload-less entry. **`test_wal_batch_roundtrip_large_payloads` (14×872B) FAILS "Expected 872 Was 0"; the 2-entry round-trip PASSES** — the threshold is message SIZE (segment overflow), not entry count. **Live replication (`repl_broadcast`) carries payloads fine — only the catch-up/rejoin path corrupts**, which is exactly why loss surfaces only after a node restarts and catches up. Candidate fix: carry all sub-entry payloads as ONE top-level capnp `Data` blob + lengths (single `Data` round-trips correctly), sidestepping per-element cross-segment pointers. RED test is in `src/lib/testsuite/test_wal_wire.c`.
+
 ## ROOT CAUSE (mechanism, code-grounded)
 A node that (re)joins with an **empty WAL** is brought up to the cluster's **current committed watermark by WAL streaming `[watermark..committed]` — NOT from LSN 1 — and there is no snapshot to backfill `[1..watermark-1]`.** So it ends up with a permanent prefix gap. When the failover test eventually has the only full-log node down while the others rejoined at high watermarks, the committed jobs are gone for good.
 
 Why the `seg-...001` file contains LSN 16: a node booting empty defaults `current_lsn=1` (`persist.c:429`); the first write lazily calls `open_new_segment` which names the file by `current_lsn` → `seg-...001` and **`O_TRUNC`s** it (`persist.c:145`); catch-up then writes entries carrying explicit high LSNs (16…) from offset 0. So "writes from the beginning of the file" = **booted empty + caught up from the watermark**, not an in-place overwrite of recovered data. (`replay_wal` at `persist.c:1422` is correct — it opens existing segments **read-only** and advances `current_lsn` past what it recovers.)
 
 ## OPEN FORKS (what to settle next)
-1. **Test procedure — clean vs preserve.** `scripts/cleanup.sh` does `rm -rf .../data/*`; `scripts/start.sh` only creates `data/` if missing (preserves). The on-disk files (named `001`, starting at 16/13) match the **cleaned** case. **Confirm with the user whether the failover test cleans data dirs before restart or restarts with data intact.**
+1. **Test procedure — clean vs preserve.** `scripts/cleanup.sh` does `rm -rf .../data/*`; `scripts/start.sh` only creates `data/` if missing (preserves). The on-disk files (named `001`, starting at 16/13) match the **cleaned** case. **Confirm with the user whether the failover test cleans data dirs before restart or restarts with data intact.** (Still worth confirming for the exact trigger ordering, but no longer the architectural question — see VERDICT.)
    - If preserved + node still returns gapped → **recovery bug** (`replay_wal` failing to restore).
-   - If cleaned → **catch-up never backfills the prefix + no snapshot** = the architectural gap below.
-2. **Catch-up start point (the actual bug site, NOT yet read).** Trace the `CtrlHello` handshake: does a rejoining standby advertise its **real `applied_lsn`** (fresh ⇒ 0 ⇒ should stream from 1) or a **learned/committed watermark** (⇒ streams from 13/16)? `CtrlHelloRequest.appliedLsn @2` is the value the primary streams `[applied+1..committed]` from. This single field decides the fix: "stream from 1 / install a snapshot first" vs "advertise real applied, not learned committed."
-3. **Snapshots.** `snap/` is empty everywhere. Is snapshot *creation* implemented/triggered in this build, or only `load_snapshot` (`persist.c:1266`, format `"XCESNAP\0"` + 8-byte lsn header — looks like a STUB carrying only an LSN, NO entities)? If snapshots never carry job entities (the "entity-bearing snapshots" backlog item in ha-convergence-recovery), then ANY WAL pruning/empty-rejoin below committed is unrecoverable BY DESIGN. **Strong candidate for the architectural root.**
+   - If cleaned → **catch-up never backfills the prefix + no snapshot** = the architectural gap (CONFIRMED below).
+2. **Catch-up start point — ✅ SETTLED (2026-06-20).** Traced. `handle_ctrlhello` (`config_repl.c:1436`) is connection-setup ONLY; catch-up is driven by the heartbeat decision loop. The standby reports its **REAL `applied_lsn`** — `ha_process_standby_state(ha, peer_idx, sr.config_tag, sr.applied_lsn, …)` at `ha.c:3518`, where `sr.applied_lsn` is the standby's own state-report value (an empty standby reports 0). The decision `ha_evaluate_repl_action` (`ha.c:1811`) is CORRECT: `standby_lsn+1 < oldest_wal_lsn → SEND_SNAPSHOT`, `else standby_lsn < committed → STREAM_WAL`. So advertise is NOT the bug — a fresh standby correctly asks for a snapshot when the primary's WAL no longer starts at 1.
+3. **Snapshots — ✅ SETTLED: THIS IS THE ROOT CAUSE (2026-06-20).** `write_snapshot` (`persist.c:1051`) writes a **16-byte SENTINEL**: `"XCESNAP\0"` (8) + `checkpoint_lsn` (8) — its own comment says *"Full Cap'n Proto serialization is Task 11."* It carries **ZERO job/queue entities.** `load_snapshot` (`persist.c:1267`) just does `p->checkpoint_lsn = p->current_lsn = stored_lsn` — it **advances the watermark past entries it never received, restoring nothing.** And `persist_checkpoint` → `trim_wal_segments` (`persist.c:1090`) **deletes WAL segments below the checkpoint.** So checkpoint = "trim the committed prefix + write an empty backup": a WAL-eating loop with no recoverable safety net. This is the "entity-bearing snapshots" gap (ha-convergence-recovery backlog) — and it makes any empty-rejoin/prefix-trim below committed **unrecoverable BY DESIGN.**
+
+## VERDICT (2026-06-20) — architectural root confirmed, code-grounded end-to-end
+The loss is **not** an advertise/start-point bug and **not** a recovery bug. It is the **absence of entity-bearing snapshots** combined with **prefix-trimming checkpoints**. Full mechanism:
+1. A fresh/empty member (applied_lsn=0), or any member catching up against a primary whose own `oldest_wal_lsn > 1` (its prefix was trimmed by a prior checkpoint).
+2. `ha_evaluate_repl_action` → `SEND_SNAPSHOT` (when `1 < oldest_wal_lsn`) or `STREAM_WAL` from the high watermark.
+3. SEND_SNAPSHOT path: `persist_checkpoint` writes the **sentinel** (no entities) + **trims the prefix WAL**, then ships the sentinel; receiver `persist_load_snapshot` sets `current_lsn = checkpoint_lsn` with **no entity restore**.
+4. STREAM_WAL path: `ha_stream_wal_to_standby` streams `[standby_lsn+1 .. committed]`; `ha_apply_wal_catchup_entry`/`ha_apply_single_entry` **physically persist** those entries (consistent with ctrl-3's on-disk seg holding LSN 13–18) — but the prefix `[1..oldest-1]` is **never streamed and has no snapshot to backfill it.**
+5. Net: the member's watermark reaches `committed` while the committed-prefix job entities are gone. **Permanent loss, BY DESIGN.**
+
+**Fix layer (recommendation — option (b), question-the-architecture):** implement **entity-bearing snapshots** — `write_snapshot` Cap'n-Proto-serializes all live job/queue entities at `checkpoint_lsn`; `load_snapshot` deserializes + `restore_job`/restore_queue them BEFORE advancing `current_lsn`; and `trim_wal_segments` must NOT trim below the LSN covered by a durable entity-bearing snapshot (install-snapshot-before-stream on the receive side). Option (a) "stream `[1..committed]`" only works while some node still retains LSN 1 (false after any trim), so it is a partial mitigation, not the fix. This is the **ha-convergence-recovery** "entity-bearing snapshots" item; route the fix there (or a new `ha-entity-snapshots` spec) under the ha-commit-durability umbrella.
 
 ## SECONDARY FINDINGS
+- **Double `WAL_JOB_SUBMIT` per job (bug-845, 2026-06-20, via waldump).** Every committed job appears as TWO consecutive entries with the SAME jobid (ctrl-1: 2,3 / 4,5 / … ; ctrl-3: 14,15 ; ctrl-2: 17,18). **Ruled out array expansion:** `job_submit` assigns array members DISTINCT ids (`base_id+i`, job.c) and the `Job` capnp has a single jobid field, so identical-jobid pairs are NOT two array tasks — a single submit is double-appending. **CONFIRMED a literal byte-for-byte double-append** by the enhanced waldump (payload-CRC + subtime columns, 2026-06-20): each pair is byte-identical — same PCRC (LSN 2,3 both `6d79074e`; 4,5 `c25ac163`; …) and same `subtime` (e.g. 1781959730), so all 12 ctrl-1 submits = 6 jobs × 2. A failover retry would re-stamp `subtime=time(NULL)` and differ → **retry definitively ruled out; it's the SAME encoded `job_t` written twice in one submit flow.** It is replicated *consistently* (F=…555020288 at 14,15 on BOTH ctrl-1 and ctrl-3; G at 17,18 on BOTH ctrl-2/3), so not itself divergence, but it doubles LSN consumption and is a latent cross-node divergence risk if the duplicate count ever varies by primary. Next: trace `handle_jobsubmit`→`persist_job_hook_fun` AFTER_EVENT loop (jlist holding the job twice, or a double hook-fire).
+- **Raft cross-node consistency check (user request, 2026-06-20).** Per-LSN comparison of the three dumps: at every OVERLAPPING index the entries AGREE — ctrl-1∩ctrl-3 = {13:NOOP, 14:F, 15:F}; ctrl-2∩ctrl-3 = {16:NOOP, 17:G, 18:G}. So **State Machine Safety (no two members hold different entries at the same index) is NOT violated** in this snapshot; no job sits at different LSNs on different nodes. The broken guarantee is **Leader Completeness / durability**: jobid E=…394588160 was committed+acked at LSN 11–12 (present on ctrl-1) but is ABSENT from the surviving majority (ctrl-2/3) — a committed entry lost (exactly the root-cause mechanism: it was below the survivors' empty-rejoin catch-up start, with no entity snapshot to backfill it). Conversely G=…630517760 is on the survivors but absent on ctrl-1, which was already down before G — losing an entry a crashed minority-node never had is Raft-legal. So the cluster ended with two committed histories sharing only the F segment: ctrl-1's {…,E@11-12,F@14-15} vs survivors' {F@14-15,…,G@17-18}.
 - **`term=0` on every WAL entry.** `p->term` (the `hdr.term` source) is **never assigned** the election term anywhere (grep for `p->term =` / `->term =` in persist.c/ha.c returns nothing). So per-entry term is dead. XCE matches logs via the `prev_crc` chain, so possibly benign — but it means **no term-based divergence detection**. Worth a hard look (could let divergent same-LSN entries be treated as matching).
 - **Cluster cannot hold a primary** under the test: peer connect `-111` (ECONNREFUSED), RDHUP, rapid term churn 2→5 in ~7 min, elections logged `safe_committed_lsn=X (from 1 granted voters)`. Logged separately in `ha-convergence-recovery/research.md` (2026-06-20 live finding).
 - **Crash fixed en-route (unrelated):** `xce_strerror_r` global-buffer-overflow on HA error codes — bug-827, fixed in `src/lib/utils.c` (commit 1aa0761).
@@ -45,14 +62,39 @@ The loss **predates** it, so it is **not the origin**. But two real concerns it
 - Deployed binary includes controller-status (responderTime) + ha-heartbeat-committed (the `ha.c:4967 "learned committed_lsn=X from heartbeat"` logs are the consume path).
 
 ## NEXT STEPS
-1. **Build the WAL/snapshot inspection tooling in `xce_test`** (in progress — see plan below) so clean-vs-preserve and gap-location become a 30-second check each run.
+1. **Build the WAL/snapshot inspection tooling in `xce_test`** — ✅ `waldump` DONE (2026-06-20). `snapdump`/`walcompare` still TODO. Gap-location is now a one-liner per run.
 2. **Read the CtrlHello catch-up advertise/start-point** (`config_repl.c` handle_ctrlhello + the standby's CtrlHelloRequest build) — Open Fork #2.
 3. **Confirm whether snapshot creation exists** or is a stub — Open Fork #3 (likely the architectural root: no entity-bearing snapshot ⇒ no prefix backfill).
 4. Decide fix layer: (a) full-WAL stream `[1..committed]` when a node has no snapshot, (b) implement entity-bearing snapshots + install-before-stream, or (c) prevent empty-rejoin from advancing committed without holding the entries. Likely a **question-the-architecture** moment (catch-up assumes a snapshot safety net that does not exist).
 
+## DECISIVE EVIDENCE 3 — bug-851 REPRODUCED & ROOT-CAUSED on an owned cluster (2026-06-21)
+
+Stood up a fresh local 3-node cluster (`/xce-deploy-ha`, binary built with the 848/849/850 fixes) and drove two scenarios with full log access. Artifacts: `repro-851/{ctrl-1,ctrl-2,ctrl-3}.log` + `waldump-ctrl-{1,2,3}.txt`.
+
+**Scenario 1 — clean single failover: PASS (848/849/850 validated end-to-end).**
+6 jobs → identical `1..13` on all 3 (matching PCRCs). Kill primary ctrl-1 → ctrl-3 promoted: `cluster_init_phase2: persist_recover done — recovered durable_floor=13 … applied_lsn=14 advertised` (bug-849 recovery-on-promotion + bug-848 advertise-real-LSN both fire). 4 jobs on new primary → ctrl-2 converges `14..22`. Restart old primary → recovers `1..13`, catches up `14..22`. **Final: all three WALs byte-identical `1..22`, identical PCRC fingerprint `cc5d361e…`, 0 gaps.**
+
+**Scenario 2 — chaotic leadership churn (3× kill-the-primary): FAIL — silent committed-prefix divergence.**
+After 3 rounds, all three report `committedLsn=54`, all `lsn range 1..54`, 0 gaps, CRC intact — **but the logs DIVERGE at LSN 45–54**:
+- ctrl-1: `45=NOOP`, `46..53 = faead246/be0059bf/fe7a3d0b/37d54e4d` → fp `bfeb26ad…`, 25 jobids
+- ctrl-2: `45=NOOP`, `46..53 = d32ac747/97c04cbe/d7ba280a/1e155b4c` (DIFFERENT jobs) → fp `216af434…`, 25 jobids
+- ctrl-3: `45 = 4c6d24e6` (a JOB, no noop), `45..52 = 4c6d24e6/0887af1f/48fdcbab/8152b8ed` → fp `865304ec…`, 26 jobids
+
+Three different "committed" tails, each node reporting them committed. This violates the Raft committed-prefix invariant (a committed entry must be byte-identical on every member).
+
+**ROOT CAUSE (code-grounded, three compounding defects):**
+1. **Term not monotonic across the cluster.** Election term = *locally-persisted* term + 1; PreVote explicitly does NOT persist/propagate ("starting PreVote round … no persist"). Under chaotic kills a leader wins term N, appends its own-term noop + jobs, and is killed **before term N is durably propagated to a majority**. A restarted node recovers a *stale* term and elects at the **same** N+1. Logs show **two leaders at term 3** (ctrl-3 @08:40, ctrl-1 @08:47:39) and **two leaders at term 4** (ctrl-2 @08:47:52, ctrl-3 @08:48:04) — each appending its own-term noop at the **same LSN 45**. Persisted terms ended split: ctrl-1=4, ctrl-2=4, ctrl-3=3.
+2. **The WAL entry term field is DEAD (always 0).** `wal_entry_hdr_t.term` exists (wal_types.h:30) and the replication envelope carries `term` (typemap.h:381), but the writer does `hdr.term = p->term` (persist.c:588/726/1040) and **`p->term` is NEVER assigned from `ha->term` anywhere in the controller** (zero matches). Every entry, every term, persists `term=0`. waldump confirms term=0 cluster-wide.
+3. **No (term,lsn) reconcile → divergent tails never truncated.** Because the WAL has no usable per-entry term, rejoin-reconcile (`ha_convergence.c`) cannot do Raft log-matching: entry 45@ctrl-1 (term 3) and 45@ctrl-2 (term 4) look identical (both term 0, both JOB_SUBMIT/NOOP), so the divergent suffix is taken as already-present and is never overwritten. `ha_compute_safe_committed_lsn` then advances committed (44, 53) "from 1 granted voter" over content that other members never received.
+
+Net: under leadership churn, acked entries written by a short-lived leader become a permanent divergent tail, and `committed` advances over it → silent loss + cross-node divergence. This is distinct from the catch-up payload-drop (bug-847) and the empty-rejoin gap (bug-828); it is a **Raft-safety gap in term establishment + WAL term recording**.
+
+**Keystone fix candidate (smallest correct lever):** stamp `hdr.term` from the live raft term (`ha->term`) on the primary append path + carry it through the replication envelope + persist it, THEN make reconcile truncate any local suffix whose (term,lsn) does not match the leader's — standard Raft log-matching. Must be paired with making the election term durably-established-before-acting (no two leaders per term). **HA-core + persistence → full spec workflow required (per CLAUDE.md); not a quick patch.**
+
+**Secondary (separate) finding — client primary-discovery is positional, not failover-correct.** With `masters = [m0, m1, …]`, the client (`xcecli_ctrlstatus_primary`, client.c:403) only reaches the primary if `m0` IS the primary: a standby answers `primaryName=""` ("unknown" by design, ctrlstatus.c:104) so no redirect hint is captured, and the inner `xcecli_ctrlstatus` SUCCEEDS on the reachable non-primary so transport failover never rotates. Confirmed: `9993`(primary-first)→ok 45ms; `9992,9993`(standby-first)→"no primary"; `9991,9993`(dead-first)→"no primary"; `9993,9991`→ok. Also the dead node returns `ctrlstatus failed: Invalid argument` (bug-846 EINVAL). This is what made earlier remote sessions read as "committed frozen" — the client was pinned to a stale/standby node. Fix options: (a) standby tracks+reports the primary name it heartbeats with (enable redirect), or (b) client rotates to the next master on a reachable-but-non-authoritative reply.
+
 ## TOOLING PLAN (xce_test enhancement — requested by user)
-Add read-only debug commands to `xce_test` (`src/controller/testsuite/test.c`; binary links `persist.c`/`job.c` per `controller/meson.build`, so the capnp `restore_job`/`read_Job` decode and `wal_types.h` are available):
-- `waldump <data_dir|wal_dir|seg_file>` — per node/segment, list every entry `lsn | term | type | payload_len | jobid(submits, via capnp `read_Job`/`decode_Job`)`, flag **LSN gaps** (non-contiguous / missing prefix) and the recovered tip.
-- `snapdump <snap_dir|snap_file>` — decode the `XCESNAP` header (lsn) and report whether it carries entities (it likely does NOT — confirms Fork #3).
-- `walcompare <dirA> <dirB>` — diff two nodes' logs by `lsn → crc/jobid` to flag divergence/gaps instantly.
-Start with `waldump` (highest value; decoder logic already proven in Python). Register in the `client_commands`/`top_commands` table (`test.c:175/207`).
+Read-only debug commands in `xce_test`. **NOTE (correction):** `xce_test` does NOT link `persist.c`/`job.c` — it builds from `test.c` only + `libxce`. But `read_Job` is generated into `controller.capnp.c` which IS compiled into `libxce`, so the capnp decode is reachable; `waldump` adds `c_capnproto_dep`+`zlib_dep` to `xce_test_deps` and ships its own zero-alloc jobid decode (reads `struct Job.jobid` after `read_Job`, no `decode_Job`).
+- `waldump <data_dir|wal_dir|seg_file>` — ✅ **DONE 2026-06-20.** Per node/segment, lists every entry `lsn | term | type | payload_len | jobid`, flags **LSN gaps** (non-contiguous / missing prefix) + the CRC recovered tip; nonzero exit on gap/CRC-break so it doubles as an E2E assertion. Impl: `src/controller/testsuite/waldump.{c,h}` (module), `cmd_waldump` wired into `top_commands` in `test.c`; unit test `test_waldump.c` (5 cases, hand-built raw segments). **Validated live** against `/tmp/xce-ha-deploy/controller/ctrl-{1,2,3}` — reproduced the exact bug table (ctrl-1 full 1–15; ctrl-2 missing prefix 1–15; ctrl-3 missing prefix 1–12). Also surfaced a NEW anomaly: each JOB_SUBMIT appears as a **consecutive duplicate pair** (same jobid at LSN N and N+1) — worth a follow-up.
+- `snapdump <snap_dir|snap_file>` — TODO. Decode the `XCESNAP` header (lsn) and report whether it carries entities (it likely does NOT — confirms Fork #3).
+- `walcompare <dirA> <dirB>` — TODO. Diff two nodes' logs by `lsn → crc/jobid` to flag divergence/gaps instantly.

.kiro/specs/ha-commit-durability/brief.md

@@ -0,0 +1,73 @@
+# Brief: ha-commit-durability
+
+## Problem
+A 3-node HA controller failover test **permanently lost committed jobs**. The client submitted 10 jobs across several forced primary kills; the 7 earliest (JobIds `0-0-1781708993..1781709035`, WAL LSN ≤20) — which had been replicated and majority-ACKed, and survived the *first* failover — were gone by the end, while only the 3 latest survived. Losing an acknowledged, committed job is a correctness (data-loss) failure, the most severe class for this system.
+
+## Current State
+The HA consistency machinery is implemented (commit engine, WAL replication, convergence/truncate-on-demotion, single-leader own-term gate, PreVote, vote-freshness bug-339). But **"committed" does not imply "durable on a majority"**:
+
+- **WAL fsync is asynchronous** — a timer thread (`persist.c:173 fsync_worker`) fdatasyncs every `wal_sync_ms`. `persist_append` does **not** fsync before returning; the standby does **not** fsync before sending its `WalAck`.
+- **The commit engine advances `committed_lsn` on majority *ACK*** (`commit.c:303 commit_on_ack`, `repl_wait_majority_ack`). An ACK only proves the entry reached the OS page cache — it can be durable on **zero** disks at the moment it is declared committed.
+- **Recovery silently truncates to nothing.** When killed nodes restart, `persist_recover` hits a CRC chain break at a zeroed/never-fsync'd region (`ctrl-3.log:1597` `prev_crc=00000000` at offset 40) and stops replay at ~0 entries — but the node still becomes eligible.
+- **A hollow node wins and overwrites the good timeline.** With durability lost on the killed nodes and failover quorum collapsed to `required=1`, the freshness gate cannot fence the stale candidate. ctrl-3 became PRIMARY term=4 advertising `committed_lsn=21` over an empty job table (`ctrl-3.log:1612`); the node that *did* hold the data (ctrl-1, applied_lsn=22) converged **down** to it (`ctrl-3.log:1618`). LSN ≤20 discarded forever.
+
+This violates the Leader-Completeness assumption documented in `src/controller/CLAUDE.md` ("committed entries are on a majority" presumes majority-ACK ⇒ durable-on-a-majority) and the TLA+ mapping `log[i] = WAL on disk`.
+
+## Desired Outcome
+1. An operation reported committed is **durable on a majority of nodes** before it is declared committed — surviving simultaneous crash + restart of any minority.
+2. A node that cannot recover its WAL to a valid committed tip is **fenced from leadership** (fail-loud), never silently truncating to zero and then leading; the cluster never converges onto a timeline below its known committed high-water mark.
+
+## Approach
+Two coupled requirements (chosen scope: *Durability + promotion fence*):
+
+- **R1 Durable commit (fsync-before-ACK):** the standby `fdatasync`s the appended WAL bytes **before** emitting `WalAck`; the primary counts its own vote toward majority only after its local `fdatasync`. `committed_lsn` advances only on a **durable** majority. Preserve throughput via group-commit / batched fdatasync (amortize the sync across pending appends) rather than one sync per entry. Keep `wal_sync_ms` async path only for non-committing/best-effort writes, if any remain.
+- **R2 Recovery & promotion fence:** a WAL CRC/replay failure that prevents reconstructing a contiguous committed tip makes the node **ineligible to become primary** until it has caught up from a healthy peer (FATAL or fence, never silent truncate-to-zero-then-lead). A node must not win election or self-promote advertising a `committed_lsn` below the cluster's known committed high-water mark, even under degraded single-voter quorum.
+
+Honor the user's fail-loud preference: surface the real durability/recovery failure rather than masking it.
+
+## Scope
+- **In**: fsync-before-ACK on the standby apply path and primary self-count; durable-majority gating of `committed_lsn`; group-commit batching to bound the latency cost; fencing an un-recovered/corrupt-WAL node from promotion; preventing a sub-high-water-mark leader from being elected under degraded quorum; unit tests for each; the 3-node live E2E that reproduces this incident (kill-primary-under-load, assert zero committed-job loss).
+- **Out**: WAL wire format / ACK message shape (`ha-wal-replication`); per-LSN majority math (`ha-commit-engine` — consumed, the *durability precondition* of its ACK input is what changes); demotion truncate/self-restart and rejoin reconciliation of an *intact* log (`ha-convergence-recovery`); promotion no-op / own-term gate (`ha-single-leader-role`); raw vote-grant arithmetic (`raft-library` — this spec adds the durability+high-water *precondition*, not new vote math); entity-bearing snapshots (roadmap); client-side failover (`libxce-failover`).
+
+## Boundary Candidates
+- Durable-commit seam: standby `fdatasync`-before-`WalAck` + primary self-fdatasync before majority self-count (the "ACK ⇒ durable" guarantee).
+- Promotion-eligibility seam: recovery-integrity + committed-high-water check gating `raft_cb_become_leader` / PreVote eligibility.
+
+## Out of Boundary
+- Tuning the async `fsync_worker` cadence as a "fix" (it is the defect for committing writes, not a knob to turn).
+- Changing how a majority is *counted* (commit-engine) or how votes are *granted* (raft-library) beyond adding the durability/high-water precondition.
+- Worker-reported fact resync across restart (`worker-rejoin-resync`, roadmap backstop).
+
+## Upstream / Downstream
+- **Upstream**: `ha-commit-engine` (committed_lsn semantics it now gates on durable ACKs), `ha-wal-replication` (ACK/catch-up machinery this makes durability-correct), `persist.c` fsync infrastructure, `ha-prevote` + vote-freshness (bug-339) eligibility path it strengthens.
+- **Downstream**: `ha-convergence-recovery` E2E (its single-outcome guarantee depends on this), `component-failover-adoption` and any real failover deployment.
+
+## Existing Spec Touchpoints
+- **Extends/strengthens**: `ha-wal-replication` (durability precondition on ACK), `ha-commit-engine` (durable-majority), `ha-convergence-recovery` (assumes intact persisted committed state — this spec makes that assumption true).
+- **Adjacent (avoid overlapping)**: `ha-single-leader-role`, `raft-library`, `ha-prevote`.
+
+## Constraints
+- C / gnu17, `src/controller/persist.c`, `commit.c`, `repl.c`, `ha.c`, `cluster.c`; TDD mandatory (tests in `testsuite/` before C changes).
+- Async-I/O rule: never block the iomgr thread — fdatasync must not stall heartbeat/vote handling; use the dedicated persist/fsync thread + group-commit and signal the commit waiter on completion.
+- Must verify against `raft/raft.tla` and the `src/controller/CLAUDE.md` Protocol Verification Checklist (commitIndex monotonic; leader holds all committed entries).
+- Live E2E needs `/xce-deploy-ha` (cluster at `/tmp/xce-ha-deploy`); reproduction harness = `xce_test` driving jobsub + primary kills. An automated soak reproduction now exists at `src/tools/test-commit-durability.sh` (rolling SIGKILL+restart under load, asserts every acked job survives).
+- Performance: durable-majority adds sync latency to the submit path — group-commit is required, not optional.
+
+## Reproduction Findings (2026-06-17, from building the E2E)
+- **bug-661 is a RACE, not a deterministic failure.** `wal_sync_ms` is hardcoded to `DEFAULT_WAL_SYNC_MS = 10` ms (`config.h:86`) with **no CLI/env/config override**. The committed-but-not-fsync'd window is ~10ms, so a single-node SIGKILL+restart only loses a committed job when the kill lands inside that window AND the truncated node re-leads before catch-up repairs its WAL. The soak reproduced real loss once (4 committed jobs) but passed an 8-round run clean — confirming the rarity.
+- **Required test seam (NEW spec item): a configurable fsync interval** (e.g. `--wal-sync-ms`, plumbed into `persist_config_t.wal_sync_ms`). Setting it to several seconds makes every committed entry provably un-fsync'd at kill time, turning the racy soak into a **deterministic** RED. This seam is also the natural way to unit-test fsync-before-ACK. None exists today — add it as part of Requirement 1's testability (cf. the convergence spec's XCE_TESTING restart seam).
+- **Acceptance asymmetry**: a soak FAIL proves the bug; a soak PASS does not prove durability (the race may simply not have fired). After the fix, the soak must pass *deterministically regardless of timing* — that, plus the deterministic `--wal-sync-ms` test, is the real acceptance bar.
+- **Amplifier confirmed**: loss needs a killed+restarted node to re-lead with a truncated WAL *before* replication catch-up repairs it. The fix must therefore also fence a node that has not caught up to the committed high-water mark from leading (Requirement 2).
+
+## Design Decision — durability mechanism (2026-06-17, with user)
+
+**Invariant (user-stated, authoritative):** when the controller replies **ok** to a client, the operation's WAL entry MUST already be persisted (fdatasync'd) on a fault-tolerant majority — self + ⌊N/2⌋ standbys.
+
+**Exact defect located:** `commit.c:majority_candidate_locked` counts the primary's own contribution as `cap = highest_appended_lsn(p)` — every *appended* (page-cache) entry, not every *fdatasync'd* one. On 3 nodes a commit needs self + 1 standby; the standby ACK is durable (`persist_wal_write_replicated:515` fdatasyncs before replying) but the primary's copy is **not** (it relies on the `fsync_worker` timer). So a "committed" entry is durable on **exactly 1 node** → zero durability fault-tolerance. That is bug-661.
+
+**Chosen direction (user): synchronous persistence — eliminate `fsync_worker` as the durability mechanism.**
+- The submit path **already blocks off the iomgr thread** waiting for majority replication (`job.c:568-569`; `persist_commit_barrier → commit_submit(COMMIT_MODE_WAIT)`), so adding the primary's own `fdatasync` on that path is liveness-safe — it does not stall heartbeats.
+- Minimal correctness change: (1) primary `fdatasync`s its entry before its self-contribution covers that LSN (symmetric with the standby path at `persist.c:515`); (2) `commit.c` self-contribution uses a `durable_lsn` (highest fdatasync'd), not `highest_appended_lsn`; (3) reply stays gated on the commit barrier → reply-ok ⇒ durable-majority.
+- **Throughput**: per-entry fdatasync. The dominant amplifier (a job array fanning out to N entries → N fsyncs) is removed by the roadmap item **job-array-single-wal-record** (one WAL record per array → one fsync). Cross-reference that item; this spec assumes it lands or is acceptable interim.
+- **Group-commit IS in scope (revised 2026-06-18 at design-review).** Initially deferred, but `/kiro-validate-design` found the highest-volume WAL path is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage) — not array-collapsible, not covered by job-array-single-record — so a pure per-entry synchronous design penalizes the dominant writes. Decision (with user): repurpose `fsync_worker` into a demand-driven group-commit flusher (batch `fdatasync`, advance `durable_lsn`, wake commit-eval) as part of this spec. The roadmap "WAL group-commit batching" item is absorbed here.
+- **Open for design:** whether to delete the `fsync_worker` thread entirely or retain a minimal best-effort flush for any non-client-acked / local-only writes (e.g. checkpoint markers already fsync inline; audit `wal_classify` for any write that gates no reply). Confirm no client-acked op bypasses the commit barrier.

.kiro/specs/ha-commit-durability/repro-851/ctrl-1.log

@@ -0,0 +1,164 @@
+[2026-06-21 08:37:35,663] [1217413] [  INFO] [                  main] My name is ctrl-1
+[2026-06-21 08:37:35,663] [1217413] [  INFO] [                  main] HA mode: 3 controller candidates, identity=ctrl-1
+[2026-06-21 08:37:35,663] [1217413] [  INFO] [   cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
+[2026-06-21 08:37:35,663] [1217413] [  INFO] [   cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
+[2026-06-21 08:37:35,671] [1217413] [  WARN] [          persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
+[2026-06-21 08:37:35,671] [1217413] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:37:35,671] [1217413] [  INFO] [             myapp_run] Pre-election config loaded: tag=4986624fd309142c
+[2026-06-21 08:37:35,671] [1217413] [  INFO] [          ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=0)
+[2026-06-21 08:37:35,671] [1217413] [  INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
+[2026-06-21 08:37:35,674] [1217413] [  INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
+[2026-06-21 08:37:40,873] [1217413] [  INFO] [   cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
+[2026-06-21 08:37:40,874] [1217413] [  INFO] [       xce_config_init] xce controller is running
+[2026-06-21 08:37:40,874] [1217413] [  INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
+[2026-06-21 08:37:40,874] [1217413] [  INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
+[2026-06-21 08:37:40,874] [1217413] [  INFO] [   cluster_init_phase2] HA: persist_recover done — recovered durable_floor=0 current_lsn=1; applied_lsn=0 advertised for catch-up
+[2026-06-21 08:37:40,874] [1217413] [  INFO] [raft_cb_become_follower] HA FSM (raft): → CTRL_STANDBY (term=1, first_boot=1)
+[2026-06-21 08:37:40,874] [1217413] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:37:50,478] [1217413] [  WARN] [      liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
+[2026-06-21 08:37:50,479] [1217413] [  INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
+[2026-06-21 08:37:50,479] [1217413] [  INFO] [     ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
+[2026-06-21 08:37:50,479] [1217413] [  INFO] [     ha_begin_failover] HA: staggering PreVote start by 8ms (priority=0) to avoid split vote (bug-606)
+[2026-06-21 08:37:50,679] [1217413] [  INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=2, n_peers=3) — staying CTRL_STANDBY, no persist
+[2026-06-21 08:37:50,728] [1217413] [  INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 2)
+[2026-06-21 08:37:50,728] [1217413] [  INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
+[2026-06-21 08:37:50,732] [1217413] [  INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
+[2026-06-21 08:37:50,732] [1217413] [  INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=0 safe_config_tag=5298030106630493228 (from 1 granted voters)
+[2026-06-21 08:37:50,732] [1217413] [  INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=2, first_boot=0)
+[2026-06-21 08:37:50,732] [1217413] [  INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
+[2026-06-21 08:37:50,732] [1217413] [  INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
+[2026-06-21 08:37:50,732] [1217413] [  INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
+[2026-06-21 08:37:50,732] [1217413] [  INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
+[2026-06-21 08:37:50,738] [1217413] [  INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
+[2026-06-21 08:37:50,738] [1217413] [  INFO] [  ha_arm_own_term_gate] HA: own-term noop appended lsn=1, writable_lsn gate armed
+[2026-06-21 08:37:53,924] [1217413] [  INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=2 (committed_tag=5298030106630493228 committed_lsn=1)
+[2026-06-21 08:37:53,925] [1217413] [  INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=2 (committed_tag=5298030106630493228 committed_lsn=1)
+[2026-06-21 08:39:27,321] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=2 < target=5 → streaming [2+1..5]
+[2026-06-21 08:39:27,321] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=3 target_lsn=5
+[2026-06-21 08:39:27,322] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [2+1..5] to standby
+[2026-06-21 08:39:27,522] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=3 < target=7 → streaming [3+1..7]
+[2026-06-21 08:39:27,522] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=4 target_lsn=7
+[2026-06-21 08:39:27,522] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [3+1..7] to standby
+[2026-06-21 08:39:27,722] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=7 < target=11 → streaming [7+1..11]
+[2026-06-21 08:39:27,722] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=8 target_lsn=11
+[2026-06-21 08:39:27,722] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [7+1..11] to standby
+[2026-06-21 08:39:27,922] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=4 < target=13 → streaming [4+1..13]
+[2026-06-21 08:39:27,922] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=5 target_lsn=13
+[2026-06-21 08:39:27,923] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [4+1..13] to standby
+[2026-06-21 08:39:28,123] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=8 < target=13 → streaming [8+1..13]
+[2026-06-21 08:39:28,123] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=9 target_lsn=13
+[2026-06-21 08:39:28,123] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [8+1..13] to standby
+[2026-06-21 08:39:28,323] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=5 < target=13 → streaming [5+1..13]
+[2026-06-21 08:39:28,323] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=6 target_lsn=13
+[2026-06-21 08:39:28,323] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [5+1..13] to standby
+[2026-06-21 08:39:28,523] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=9 < target=13 → streaming [9+1..13]
+[2026-06-21 08:39:28,523] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=10 target_lsn=13
+[2026-06-21 08:39:28,524] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [9+1..13] to standby
+[2026-06-21 08:39:28,724] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=6 < target=13 → streaming [6+1..13]
+[2026-06-21 08:39:28,724] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=7 target_lsn=13
+[2026-06-21 08:39:28,724] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [6+1..13] to standby
+[2026-06-21 08:39:28,924] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=10 < target=13 → streaming [10+1..13]
+[2026-06-21 08:39:28,924] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=11 target_lsn=13
+[2026-06-21 08:39:28,924] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [10+1..13] to standby
+[2026-06-21 08:39:29,125] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=7 < target=13 → streaming [7+1..13]
+[2026-06-21 08:39:29,125] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=8 target_lsn=13
+[2026-06-21 08:39:29,125] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [7+1..13] to standby
+[2026-06-21 08:39:29,325] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=11 < target=13 → streaming [11+1..13]
+[2026-06-21 08:39:29,325] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=12 target_lsn=13
+[2026-06-21 08:39:29,325] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [11+1..13] to standby
+[2026-06-21 08:39:29,525] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=8 < target=13 → streaming [8+1..13]
+[2026-06-21 08:39:29,525] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=9 target_lsn=13
+[2026-06-21 08:39:29,526] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [8+1..13] to standby
+[2026-06-21 08:39:29,726] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=12 < target=13 → streaming [12+1..13]
+[2026-06-21 08:39:29,726] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=13 target_lsn=13
+[2026-06-21 08:39:29,726] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [12+1..13] to standby
+[2026-06-21 08:39:29,926] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=9 < target=13 → streaming [9+1..13]
+[2026-06-21 08:39:29,926] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=10 target_lsn=13
+[2026-06-21 08:39:29,926] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [9+1..13] to standby
+[2026-06-21 08:39:30,326] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=10 < target=13 → streaming [10+1..13]
+[2026-06-21 08:39:30,326] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=11 target_lsn=13
+[2026-06-21 08:39:30,327] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [10+1..13] to standby
+[2026-06-21 08:39:30,527] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=11 < target=13 → streaming [11+1..13]
+[2026-06-21 08:39:30,527] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=12 target_lsn=13
+[2026-06-21 08:39:30,527] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [11+1..13] to standby
+[2026-06-21 08:39:30,727] [1217413] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=12 < target=13 → streaming [12+1..13]
+[2026-06-21 08:39:30,727] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=13 target_lsn=13
+[2026-06-21 08:39:30,727] [1217413] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [12+1..13] to standby
+[2026-06-21 08:43:44,321] [1224672] [  INFO] [                  main] My name is ctrl-1
+[2026-06-21 08:43:44,321] [1224672] [  INFO] [                  main] HA mode: 3 controller candidates, identity=ctrl-1
+[2026-06-21 08:43:44,321] [1224672] [  INFO] [   cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
+[2026-06-21 08:43:44,321] [1224672] [  INFO] [   cluster_init_phase1] HA: restored term=2 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
+[2026-06-21 08:43:44,327] [1224672] [  WARN] [          persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
+[2026-06-21 08:43:44,328] [1224672] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:43:44,328] [1224672] [  INFO] [             myapp_run] Pre-election config loaded: tag=4986624fd309142c
+[2026-06-21 08:43:44,328] [1224672] [  INFO] [          ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=2)
+[2026-06-21 08:43:44,328] [1224672] [  INFO] [             myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-1, voted_term=2, raft_term=2, primary_fd=-1) — skipping own election
+[2026-06-21 08:43:44,328] [1224672] [  INFO] [   cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
+[2026-06-21 08:43:44,328] [1224672] [  INFO] [       xce_config_init] xce controller is running
+[2026-06-21 08:43:44,328] [1224672] [  INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
+[2026-06-21 08:43:44,328] [1224672] [  INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
+[2026-06-21 08:43:44,328] [1224672] [  INFO] [   cluster_init_phase2] HA: persist_recover done — recovered durable_floor=13 current_lsn=14; applied_lsn=13 advertised for catch-up
+[2026-06-21 08:43:44,328] [1224672] [  INFO] [             myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
+[2026-06-21 08:43:44,328] [1224672] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:43:45,250] [1224672] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:43:45,250] [1224672] [  INFO] [      handle_ctrlhello] HA: primary control channel established (fd=6)
+[2026-06-21 08:47:39,017] [1224672] [  WARN] [      liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
+[2026-06-21 08:47:39,017] [1224672] [  INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
+[2026-06-21 08:47:39,017] [1224672] [  INFO] [     ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
+[2026-06-21 08:47:39,017] [1224672] [  INFO] [     ha_begin_failover] HA: staggering PreVote start by 122ms (priority=0) to avoid split vote (bug-606)
+[2026-06-21 08:47:39,217] [1224672] [  INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=3, n_peers=3) — staying CTRL_STANDBY, no persist
+[2026-06-21 08:47:39,262] [1224672] [  INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 3)
+[2026-06-21 08:47:39,265] [1224672] [  INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
+[2026-06-21 08:47:39,267] [1224672] [  INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
+[2026-06-21 08:47:39,267] [1224672] [  INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=44 safe_config_tag=5298030106630493228 (from 1 granted voters)
+[2026-06-21 08:47:39,267] [1224672] [  INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=3, first_boot=0)
+[2026-06-21 08:47:39,267] [1224672] [  INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
+[2026-06-21 08:47:39,267] [1224672] [  INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
+[2026-06-21 08:47:39,267] [1224672] [  INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
+[2026-06-21 08:47:39,267] [1224672] [  INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
+[2026-06-21 08:47:39,272] [1224672] [  INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
+[2026-06-21 08:47:39,272] [1224672] [  INFO] [  ha_arm_own_term_gate] HA: own-term noop appended lsn=45, writable_lsn gate armed
+[2026-06-21 08:47:41,419] [1224672] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=46 < target=48 → streaming [46+1..48]
+[2026-06-21 08:47:41,419] [1224672] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=47 target_lsn=48
+[2026-06-21 08:47:41,420] [1224672] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [46+1..48] to standby
+[2026-06-21 08:47:41,620] [1224672] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=48 < target=53 → streaming [48+1..53]
+[2026-06-21 08:47:41,620] [1224672] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=49 target_lsn=53
+[2026-06-21 08:47:41,620] [1224672] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [48+1..53] to standby
+[2026-06-21 08:47:47,832] [1228504] [  INFO] [                  main] My name is ctrl-1
+[2026-06-21 08:47:47,832] [1228504] [  INFO] [                  main] HA mode: 3 controller candidates, identity=ctrl-1
+[2026-06-21 08:47:47,832] [1228504] [  INFO] [   cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
+[2026-06-21 08:47:47,832] [1228504] [  INFO] [   cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
+[2026-06-21 08:47:47,839] [1228504] [  WARN] [          persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
+[2026-06-21 08:47:47,840] [1228504] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:47:47,840] [1228504] [  INFO] [             myapp_run] Pre-election config loaded: tag=4986624fd309142c
+[2026-06-21 08:47:47,840] [1228504] [  INFO] [          ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=3)
+[2026-06-21 08:47:47,840] [1228504] [  INFO] [             myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-1, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
+[2026-06-21 08:47:47,840] [1228504] [  INFO] [   cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
+[2026-06-21 08:47:47,840] [1228504] [  INFO] [       xce_config_init] xce controller is running
+[2026-06-21 08:47:47,840] [1228504] [  INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
+[2026-06-21 08:47:47,840] [1228504] [  INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
+[2026-06-21 08:47:47,841] [1228504] [  INFO] [   cluster_init_phase2] HA: persist_recover done — recovered durable_floor=53 current_lsn=54; applied_lsn=53 advertised for catch-up
+[2026-06-21 08:47:47,841] [1228504] [  INFO] [             myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
+[2026-06-21 08:47:47,841] [1228504] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:47:52,855] [1228504] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:47:52,855] [1228504] [  INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
+[2026-06-21 08:47:52,855] [1228504] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:47:52,855] [1228504] [  INFO] [      handle_ctrlhello] HA: primary control channel established (fd=8)
+[2026-06-21 08:48:04,047] [1228504] [  WARN] [      liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
+[2026-06-21 08:48:04,047] [1228504] [  INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
+[2026-06-21 08:48:04,047] [1228504] [  INFO] [     ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
+[2026-06-21 08:48:04,047] [1228504] [  INFO] [     ha_begin_failover] HA: staggering PreVote start by 159ms (priority=0) to avoid split vote (bug-606)
+[2026-06-21 08:48:04,248] [1228504] [  INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=4, n_peers=3) — staying CTRL_STANDBY, no persist
+[2026-06-21 08:48:04,295] [1228504] [  INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 4)
+[2026-06-21 08:48:04,295] [1228504] [  INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
+[2026-06-21 08:48:04,300] [1228504] [  INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
+[2026-06-21 08:48:04,300] [1228504] [  INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=53 safe_config_tag=5298030106630493228 (from 1 granted voters)
+[2026-06-21 08:48:04,300] [1228504] [  INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=4, first_boot=0)
+[2026-06-21 08:48:04,300] [1228504] [  INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
+[2026-06-21 08:48:04,300] [1228504] [  INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
+[2026-06-21 08:48:04,300] [1228504] [  INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
+[2026-06-21 08:48:04,300] [1228504] [  INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
+[2026-06-21 08:48:04,305] [1228504] [  INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
+[2026-06-21 08:48:04,307] [1228504] [  INFO] [  ha_arm_own_term_gate] HA: own-term noop appended lsn=54, writable_lsn gate armed
+[2026-06-21 08:48:07,493] [1228504] [  INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=55 (committed_tag=5298030106630493228 committed_lsn=54)
+[2026-06-21 08:48:07,493] [1228504] [  INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=55 (committed_tag=5298030106630493228 committed_lsn=54)

.kiro/specs/ha-commit-durability/repro-851/ctrl-2.log

@@ -0,0 +1,99 @@
+[2026-06-21 08:37:36,163] [1217460] [  INFO] [                  main] My name is ctrl-2
+[2026-06-21 08:37:36,164] [1217460] [  INFO] [                  main] HA mode: 3 controller candidates, identity=ctrl-2
+[2026-06-21 08:37:36,164] [1217460] [  INFO] [   cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
+[2026-06-21 08:37:36,164] [1217460] [  INFO] [   cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
+[2026-06-21 08:37:36,173] [1217460] [  WARN] [          persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
+[2026-06-21 08:37:36,173] [1217460] [  INFO] [   load_primary_config] No config at '/tmp/xce-deploy/controller/ctrl-2/conf/current' (rc=-2) — will fetch from primary after election
+[2026-06-21 08:37:36,173] [1217460] [  INFO] [          ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=0)
+[2026-06-21 08:37:36,673] [1217460] [  INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
+[2026-06-21 08:37:36,673] [1217460] [  INFO] [ha_start_election_async] ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only, not starting own election
+[2026-06-21 08:37:36,673] [1217460] [ ERROR] [   cluster_init_phase2] cluster_init_phase2: cluster_config not loaded
+[2026-06-21 08:37:36,673] [1217460] [  WARN] [             myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; falling back to election-path recovery
+[2026-06-21 08:37:36,673] [1217460] [  INFO] [             myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
+[2026-06-21 08:37:36,673] [1217460] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:37:46,177] [1217460] [  WARN] [      liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
+[2026-06-21 08:37:46,177] [1217460] [  INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
+[2026-06-21 08:37:46,177] [1217460] [  INFO] [ha_on_primary_ctrl_fd_rdhup] ha_on_primary_ctrl_fd_rdhup: no local config (tag=0, n_peers=3) — voter only, not starting PreVote/RDHUP re-election; awaiting configured primary (ha-config-bootstrap Req 2.8)
+[2026-06-21 08:37:50,743] [1217460] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:37:50,743] [1217460] [  INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
+[2026-06-21 08:37:50,743] [1217460] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:37:50,743] [1217460] [  INFO] [      handle_ctrlhello] HA: primary control channel established (fd=7)
+[2026-06-21 08:40:02,029] [1217460] [  WARN] [      liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
+[2026-06-21 08:40:02,029] [1217460] [  INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
+[2026-06-21 08:40:02,029] [1217460] [  INFO] [     ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
+[2026-06-21 08:40:02,029] [1217460] [  INFO] [     ha_begin_failover] HA: staggering PreVote start by 585ms (priority=1) to avoid split vote (bug-606)
+[2026-06-21 08:40:02,629] [1217460] [  INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=1, n_peers=3) — staying CTRL_STANDBY, no persist
+[2026-06-21 08:40:03,270] [1217460] [  INFO] [ha_on_heartbeat_received] HA: primary heartbeat arrived mid-PreVote — cancelling the round and re-attaching as CTRL_STANDBY (leader alive, no term change) (Req 4.2)
+[2026-06-21 08:40:03,273] [1217460] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:40:03,273] [1217460] [  INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
+[2026-06-21 08:40:03,273] [1217460] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:40:03,273] [1217460] [  INFO] [      handle_ctrlhello] HA: primary control channel established (fd=9)
+[2026-06-21 08:46:07,141] [1227110] [  INFO] [                  main] My name is ctrl-2
+[2026-06-21 08:46:07,141] [1227110] [  INFO] [                  main] HA mode: 3 controller candidates, identity=ctrl-2
+[2026-06-21 08:46:07,141] [1227110] [  INFO] [   cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
+[2026-06-21 08:46:07,141] [1227110] [  INFO] [   cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
+[2026-06-21 08:46:07,149] [1227110] [  WARN] [          persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
+[2026-06-21 08:46:07,149] [1227110] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:46:07,149] [1227110] [  INFO] [             myapp_run] Pre-election config loaded: tag=4986624fd309142c
+[2026-06-21 08:46:07,149] [1227110] [  INFO] [          ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=3)
+[2026-06-21 08:46:07,649] [1227110] [  INFO] [             myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-3, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
+[2026-06-21 08:46:07,649] [1227110] [  INFO] [   cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
+[2026-06-21 08:46:07,649] [1227110] [  INFO] [       xce_config_init] xce controller is running
+[2026-06-21 08:46:07,649] [1227110] [  INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
+[2026-06-21 08:46:07,649] [1227110] [  INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
+[2026-06-21 08:46:07,650] [1227110] [  INFO] [   cluster_init_phase2] HA: persist_recover done — recovered durable_floor=22 current_lsn=23; applied_lsn=22 advertised for catch-up
+[2026-06-21 08:46:07,650] [1227110] [  INFO] [             myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
+[2026-06-21 08:46:07,650] [1227110] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:46:09,319] [1227110] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:46:09,319] [1227110] [  INFO] [      handle_ctrlhello] HA: primary control channel established (fd=6)
+[2026-06-21 08:47:38,988] [1227110] [  WARN] [      liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
+[2026-06-21 08:47:38,988] [1227110] [  INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
+[2026-06-21 08:47:38,988] [1227110] [  INFO] [     ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
+[2026-06-21 08:47:38,988] [1227110] [  INFO] [     ha_begin_failover] HA: staggering PreVote start by 556ms (priority=1) to avoid split vote (bug-606)
+[2026-06-21 08:47:39,276] [1227110] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:47:39,276] [1227110] [  INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
+[2026-06-21 08:47:39,277] [1227110] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:47:39,277] [1227110] [  INFO] [      handle_ctrlhello] HA: primary control channel established (fd=8)
+[2026-06-21 08:47:39,588] [1227110] [  INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)
+[2026-06-21 08:47:51,992] [1227110] [  WARN] [      liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
+[2026-06-21 08:47:51,992] [1227110] [  INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
+[2026-06-21 08:47:51,992] [1227110] [  INFO] [     ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
+[2026-06-21 08:47:51,992] [1227110] [  INFO] [     ha_begin_failover] HA: staggering PreVote start by 662ms (priority=1) to avoid split vote (bug-606)
+[2026-06-21 08:47:52,793] [1227110] [  INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=4, n_peers=3) — staying CTRL_STANDBY, no persist
+[2026-06-21 08:47:52,840] [1227110] [  INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 4)
+[2026-06-21 08:47:52,843] [1227110] [  INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
+[2026-06-21 08:47:52,845] [1227110] [  INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
+[2026-06-21 08:47:52,845] [1227110] [  INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=44 safe_config_tag=5298030106630493228 (from 1 granted voters)
+[2026-06-21 08:47:52,845] [1227110] [  INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=4, first_boot=0)
+[2026-06-21 08:47:52,845] [1227110] [  INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
+[2026-06-21 08:47:52,845] [1227110] [  INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9991)
+[2026-06-21 08:47:52,845] [1227110] [  INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
+[2026-06-21 08:47:52,845] [1227110] [  INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
+[2026-06-21 08:47:52,850] [1227110] [  INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
+[2026-06-21 08:47:52,850] [1227110] [  INFO] [  ha_arm_own_term_gate] HA: own-term noop appended lsn=45, writable_lsn gate armed
+[2026-06-21 08:47:53,395] [1227110] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=48 < target=49 → streaming [48+1..49]
+[2026-06-21 08:47:53,395] [1227110] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=49 target_lsn=49
+[2026-06-21 08:47:53,395] [1227110] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [48+1..49] to standby
+[2026-06-21 08:47:53,595] [1227110] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=49 < target=53 → streaming [49+1..53]
+[2026-06-21 08:47:53,595] [1227110] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=50 target_lsn=53
+[2026-06-21 08:47:53,596] [1227110] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [49+1..53] to standby
+[2026-06-21 08:47:59,844] [1228672] [  INFO] [                  main] My name is ctrl-2
+[2026-06-21 08:47:59,844] [1228672] [  INFO] [                  main] HA mode: 3 controller candidates, identity=ctrl-2
+[2026-06-21 08:47:59,844] [1228672] [  INFO] [   cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
+[2026-06-21 08:47:59,845] [1228672] [  INFO] [   cluster_init_phase1] HA: restored term=4 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
+[2026-06-21 08:47:59,854] [1228672] [  WARN] [          persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
+[2026-06-21 08:47:59,854] [1228672] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:47:59,854] [1228672] [  INFO] [             myapp_run] Pre-election config loaded: tag=4986624fd309142c
+[2026-06-21 08:47:59,854] [1228672] [  INFO] [          ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=4)
+[2026-06-21 08:48:00,354] [1228672] [  INFO] [             myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-2, voted_term=4, raft_term=4, primary_fd=-1) — skipping own election
+[2026-06-21 08:48:00,354] [1228672] [  INFO] [   cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
+[2026-06-21 08:48:00,355] [1228672] [  INFO] [       xce_config_init] xce controller is running
+[2026-06-21 08:48:00,355] [1228672] [  INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
+[2026-06-21 08:48:00,355] [1228672] [  INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
+[2026-06-21 08:48:00,357] [1228672] [  INFO] [   cluster_init_phase2] HA: persist_recover done — recovered durable_floor=53 current_lsn=54; applied_lsn=53 advertised for catch-up
+[2026-06-21 08:48:00,357] [1228672] [  INFO] [             myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
+[2026-06-21 08:48:00,357] [1228672] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:48:04,312] [1228672] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:48:04,312] [1228672] [  INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
+[2026-06-21 08:48:04,312] [1228672] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:48:04,312] [1228672] [  INFO] [      handle_ctrlhello] HA: primary control channel established (fd=7)

.kiro/specs/ha-commit-durability/repro-851/ctrl-3.log

@@ -0,0 +1,213 @@
+[2026-06-21 08:37:36,665] [1217472] [  INFO] [                  main] My name is ctrl-3
+[2026-06-21 08:37:36,665] [1217472] [  INFO] [                  main] HA mode: 3 controller candidates, identity=ctrl-3
+[2026-06-21 08:37:36,666] [1217472] [  INFO] [   cluster_init_phase1] HA peers: 3 candidates, my_priority=2 (name=ctrl-3)
+[2026-06-21 08:37:36,666] [1217472] [  INFO] [   cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-3/data/ha-term.bin
+[2026-06-21 08:37:36,672] [1217472] [  WARN] [          persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-3/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
+[2026-06-21 08:37:36,672] [1217472] [  INFO] [   load_primary_config] No config at '/tmp/xce-deploy/controller/ctrl-3/conf/current' (rc=-2) — will fetch from primary after election
+[2026-06-21 08:37:36,672] [1217472] [  INFO] [          ha_init_raft] HA: raft initialized (n_peers=3, self=2, term=0)
+[2026-06-21 08:37:37,672] [1217472] [  INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
+[2026-06-21 08:37:37,672] [1217472] [  INFO] [ha_start_election_async] ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only, not starting own election
+[2026-06-21 08:37:37,672] [1217472] [ ERROR] [   cluster_init_phase2] cluster_init_phase2: cluster_config not loaded
+[2026-06-21 08:37:37,672] [1217472] [  WARN] [             myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; falling back to election-path recovery
+[2026-06-21 08:37:37,672] [1217472] [  INFO] [             myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
+[2026-06-21 08:37:37,672] [1217472] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:37:47,276] [1217472] [  WARN] [      liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
+[2026-06-21 08:37:47,276] [1217472] [  INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
+[2026-06-21 08:37:47,276] [1217472] [  INFO] [ha_on_primary_ctrl_fd_rdhup] ha_on_primary_ctrl_fd_rdhup: no local config (tag=0, n_peers=3) — voter only, not starting PreVote/RDHUP re-election; awaiting configured primary (ha-config-bootstrap Req 2.8)
+[2026-06-21 08:37:50,743] [1217472] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:37:50,743] [1217472] [  INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
+[2026-06-21 08:37:50,743] [1217472] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:37:50,743] [1217472] [  INFO] [      handle_ctrlhello] HA: primary control channel established (fd=7)
+[2026-06-21 08:40:01,930] [1217472] [  WARN] [      liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
+[2026-06-21 08:40:01,930] [1217472] [  INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
+[2026-06-21 08:40:01,930] [1217472] [  INFO] [     ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
+[2026-06-21 08:40:01,930] [1217472] [  INFO] [     ha_begin_failover] HA: staggering PreVote start by 1150ms (priority=2) to avoid split vote (bug-606)
+[2026-06-21 08:40:02,636] [1217472] [  INFO] [         myreq_handler] HA: PreVote DENIED candidate='ctrl-2' term=1 reason=stale_term (my_term=0, floor tag=5298030106630493228 lsn=13)
+[2026-06-21 08:40:03,130] [1217472] [  INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=3, n_peers=3) — staying CTRL_STANDBY, no persist
+[2026-06-21 08:40:03,176] [1217472] [  INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 3)
+[2026-06-21 08:40:03,176] [1217472] [  INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
+[2026-06-21 08:40:03,180] [1217472] [  INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
+[2026-06-21 08:40:03,223] [1217472] [  INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=13 safe_config_tag=5298030106630493228 (from 1 granted voters)
+[2026-06-21 08:40:03,223] [1217472] [  INFO] [   cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
+[2026-06-21 08:40:03,223] [1217472] [  INFO] [       xce_config_init] xce controller is running
+[2026-06-21 08:40:03,223] [1217472] [  INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
+[2026-06-21 08:40:03,223] [1217472] [  INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
+[2026-06-21 08:40:03,224] [1217472] [  INFO] [   cluster_init_phase2] HA: persist_recover done — recovered durable_floor=13 current_lsn=14; applied_lsn=14 advertised for catch-up
+[2026-06-21 08:40:03,224] [1217472] [  INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=3, first_boot=1)
+[2026-06-21 08:40:03,224] [1217472] [  INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
+[2026-06-21 08:40:03,224] [1217472] [  INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 0, localhost:9992)
+[2026-06-21 08:40:03,229] [1217472] [  INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
+[2026-06-21 08:40:03,230] [1217472] [  INFO] [  ha_arm_own_term_gate] HA: own-term noop appended lsn=14, writable_lsn gate armed
+[2026-06-21 08:40:06,333] [1217472] [  INFO] [    ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 1)
+[2026-06-21 08:40:06,333] [1217472] [  WARN] [    ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
+[2026-06-21 08:40:06,374] [1217472] [  INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=15 (committed_tag=5298030106630493228 committed_lsn=14)
+[2026-06-21 08:40:33,348] [1217472] [  INFO] [    ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 10)
+[2026-06-21 08:40:33,348] [1217472] [  WARN] [    ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
+[2026-06-21 08:41:03,364] [1217472] [  INFO] [    ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 20)
+[2026-06-21 08:41:03,364] [1217472] [  WARN] [    ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
+[2026-06-21 08:41:33,382] [1217472] [  INFO] [    ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 30)
+[2026-06-21 08:41:33,382] [1217472] [  WARN] [    ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
+[2026-06-21 08:42:03,397] [1217472] [  INFO] [    ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 40)
+[2026-06-21 08:42:03,397] [1217472] [  WARN] [    ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
+[2026-06-21 08:42:33,411] [1217472] [  INFO] [    ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 50)
+[2026-06-21 08:42:33,411] [1217472] [  WARN] [    ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
+[2026-06-21 08:43:03,425] [1217472] [  INFO] [    ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 60)
+[2026-06-21 08:43:03,425] [1217472] [  WARN] [    ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
+[2026-06-21 08:43:33,240] [1217472] [  INFO] [    ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 70)
+[2026-06-21 08:43:33,240] [1217472] [  WARN] [    ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
+[2026-06-21 08:43:45,247] [1217472] [  INFO] [    ha_connect_to_peer] HA: connecting outbound to peer[0] (localhost:9991)
+[2026-06-21 08:43:45,250] [1217472] [  INFO] [  on_standby_connected] HA: outbound standby connection to peer[0] established, ctrlhello sent
+[2026-06-21 08:43:45,250] [1217472] [  INFO] [  on_standby_connected] HA: repl registered late-joiner standby peer[0] (slot 1, localhost:9991)
+[2026-06-21 08:43:48,290] [1217472] [  INFO] [ha_dispatch_repl_action_for_standby] HA repl: standby[0] lsn=13 < committed_lsn=22 → stream WAL batches
+[2026-06-21 08:43:48,290] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=14 target_lsn=22
+[2026-06-21 08:43:48,290] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [13+1..22] to standby
+[2026-06-21 08:43:48,449] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=14 < target=22 → streaming [14+1..22]
+[2026-06-21 08:43:48,449] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=15 target_lsn=22
+[2026-06-21 08:43:48,449] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [14+1..22] to standby
+[2026-06-21 08:43:48,649] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=15 < target=22 → streaming [15+1..22]
+[2026-06-21 08:43:48,649] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=16 target_lsn=22
+[2026-06-21 08:43:48,649] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [15+1..22] to standby
+[2026-06-21 08:43:48,849] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=16 < target=22 → streaming [16+1..22]
+[2026-06-21 08:43:48,849] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=17 target_lsn=22
+[2026-06-21 08:43:48,850] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [16+1..22] to standby
+[2026-06-21 08:43:49,050] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=17 < target=22 → streaming [17+1..22]
+[2026-06-21 08:43:49,050] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=18 target_lsn=22
+[2026-06-21 08:43:49,050] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [17+1..22] to standby
+[2026-06-21 08:43:49,250] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=18 < target=22 → streaming [18+1..22]
+[2026-06-21 08:43:49,250] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=19 target_lsn=22
+[2026-06-21 08:43:49,251] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [18+1..22] to standby
+[2026-06-21 08:43:49,451] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=19 < target=22 → streaming [19+1..22]
+[2026-06-21 08:43:49,451] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=20 target_lsn=22
+[2026-06-21 08:43:49,451] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [19+1..22] to standby
+[2026-06-21 08:43:49,651] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=20 < target=22 → streaming [20+1..22]
+[2026-06-21 08:43:49,651] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=21 target_lsn=22
+[2026-06-21 08:43:49,651] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [20+1..22] to standby
+[2026-06-21 08:43:49,851] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=21 < target=22 → streaming [21+1..22]
+[2026-06-21 08:43:49,851] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=22 target_lsn=22
+[2026-06-21 08:43:49,852] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [21+1..22] to standby
+[2026-06-21 08:43:51,294] [1217472] [  INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=23 (committed_tag=5298030106630493228 committed_lsn=22)
+[2026-06-21 08:45:24,291] [1217472] [  INFO] [    ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 1)
+[2026-06-21 08:45:24,292] [1217472] [  INFO] [    ha_connect_to_peer] HA: connecting outbound to peer[0] (localhost:9991)
+[2026-06-21 08:45:24,292] [1217472] [  INFO] [    ha_connect_to_peer] HA: peer[1] (localhost) not connected — initiating outbound connect (attempt 1)
+[2026-06-21 08:45:24,292] [1217472] [  WARN] [    ha_connect_to_peer] HA: connect to peer[1] (localhost:9992) failed: -111
+[2026-06-21 08:45:24,294] [1217472] [  INFO] [  on_standby_connected] HA: outbound standby connection to peer[0] established, ctrlhello sent
+[2026-06-21 08:45:24,294] [1217472] [  INFO] [  on_standby_connected] HA: repl registered late-joiner standby peer[0] (slot 0, localhost:9991)
+[2026-06-21 08:45:51,305] [1217472] [  INFO] [    ha_connect_to_peer] HA: peer[1] (localhost) not connected — initiating outbound connect (attempt 10)
+[2026-06-21 08:45:51,305] [1217472] [  WARN] [    ha_connect_to_peer] HA: connect to peer[1] (localhost:9992) failed: -111
+[2026-06-21 08:46:07,913] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=35 < target=36 → streaming [35+1..36]
+[2026-06-21 08:46:07,913] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=36 target_lsn=36
+[2026-06-21 08:46:07,913] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [35+1..36] to standby
+[2026-06-21 08:46:09,314] [1217472] [  INFO] [    ha_connect_to_peer] HA: connecting outbound to peer[1] (localhost:9992)
+[2026-06-21 08:46:09,319] [1217472] [  INFO] [  on_standby_connected] HA: outbound standby connection to peer[1] established, ctrlhello sent
+[2026-06-21 08:46:09,319] [1217472] [  INFO] [  on_standby_connected] HA: repl registered late-joiner standby peer[1] (slot 2, localhost:9992)
+[2026-06-21 08:46:12,358] [1217472] [  INFO] [ha_dispatch_repl_action_for_standby] HA repl: standby[1] lsn=22 < committed_lsn=44 → stream WAL batches
+[2026-06-21 08:46:12,358] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=23 target_lsn=44
+[2026-06-21 08:46:12,358] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [22+1..44] to standby
+[2026-06-21 08:46:12,516] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=23 < target=44 → streaming [23+1..44]
+[2026-06-21 08:46:12,516] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=24 target_lsn=44
+[2026-06-21 08:46:12,516] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [23+1..44] to standby
+[2026-06-21 08:46:12,716] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=24 < target=44 → streaming [24+1..44]
+[2026-06-21 08:46:12,716] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=25 target_lsn=44
+[2026-06-21 08:46:12,717] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [24+1..44] to standby
+[2026-06-21 08:46:12,917] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=25 < target=44 → streaming [25+1..44]
+[2026-06-21 08:46:12,917] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=26 target_lsn=44
+[2026-06-21 08:46:12,917] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [25+1..44] to standby
+[2026-06-21 08:46:13,117] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=26 < target=44 → streaming [26+1..44]
+[2026-06-21 08:46:13,118] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=27 target_lsn=44
+[2026-06-21 08:46:13,118] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [26+1..44] to standby
+[2026-06-21 08:46:13,318] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=27 < target=44 → streaming [27+1..44]
+[2026-06-21 08:46:13,318] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=28 target_lsn=44
+[2026-06-21 08:46:13,319] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [27+1..44] to standby
+[2026-06-21 08:46:13,519] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=28 < target=44 → streaming [28+1..44]
+[2026-06-21 08:46:13,519] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=29 target_lsn=44
+[2026-06-21 08:46:13,519] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [28+1..44] to standby
+[2026-06-21 08:46:13,719] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=29 < target=44 → streaming [29+1..44]
+[2026-06-21 08:46:13,719] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=30 target_lsn=44
+[2026-06-21 08:46:13,720] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [29+1..44] to standby
+[2026-06-21 08:46:13,920] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=30 < target=44 → streaming [30+1..44]
+[2026-06-21 08:46:13,920] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=31 target_lsn=44
+[2026-06-21 08:46:13,920] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [30+1..44] to standby
+[2026-06-21 08:46:14,121] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=31 < target=44 → streaming [31+1..44]
+[2026-06-21 08:46:14,121] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=32 target_lsn=44
+[2026-06-21 08:46:14,121] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [31+1..44] to standby
+[2026-06-21 08:46:14,321] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=32 < target=44 → streaming [32+1..44]
+[2026-06-21 08:46:14,321] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=33 target_lsn=44
+[2026-06-21 08:46:14,321] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [32+1..44] to standby
+[2026-06-21 08:46:14,522] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=33 < target=44 → streaming [33+1..44]
+[2026-06-21 08:46:14,522] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=34 target_lsn=44
+[2026-06-21 08:46:14,522] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [33+1..44] to standby
+[2026-06-21 08:46:14,722] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=34 < target=44 → streaming [34+1..44]
+[2026-06-21 08:46:14,722] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=35 target_lsn=44
+[2026-06-21 08:46:14,722] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [34+1..44] to standby
+[2026-06-21 08:46:14,922] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=35 < target=44 → streaming [35+1..44]
+[2026-06-21 08:46:14,922] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=36 target_lsn=44
+[2026-06-21 08:46:14,923] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [35+1..44] to standby
+[2026-06-21 08:46:15,123] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=36 < target=44 → streaming [36+1..44]
+[2026-06-21 08:46:15,123] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=37 target_lsn=44
+[2026-06-21 08:46:15,123] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [36+1..44] to standby
+[2026-06-21 08:46:15,323] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=37 < target=44 → streaming [37+1..44]
+[2026-06-21 08:46:15,323] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=38 target_lsn=44
+[2026-06-21 08:46:15,323] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [37+1..44] to standby
+[2026-06-21 08:46:15,565] [1217472] [  INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=45 (committed_tag=5298030106630493228 committed_lsn=44)
+[2026-06-21 08:46:15,724] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=38 < target=44 → streaming [38+1..44]
+[2026-06-21 08:46:15,724] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=39 target_lsn=44
+[2026-06-21 08:46:15,724] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [38+1..44] to standby
+[2026-06-21 08:46:15,924] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=39 < target=44 → streaming [39+1..44]
+[2026-06-21 08:46:15,924] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=40 target_lsn=44
+[2026-06-21 08:46:15,924] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [39+1..44] to standby
+[2026-06-21 08:46:16,124] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=40 < target=44 → streaming [40+1..44]
+[2026-06-21 08:46:16,124] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=41 target_lsn=44
+[2026-06-21 08:46:16,125] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [40+1..44] to standby
+[2026-06-21 08:46:16,325] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=41 < target=44 → streaming [41+1..44]
+[2026-06-21 08:46:16,325] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=42 target_lsn=44
+[2026-06-21 08:46:16,325] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [41+1..44] to standby
+[2026-06-21 08:46:16,525] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=42 < target=44 → streaming [42+1..44]
+[2026-06-21 08:46:16,525] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=43 target_lsn=44
+[2026-06-21 08:46:16,526] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [42+1..44] to standby
+[2026-06-21 08:46:16,726] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=43 < target=44 → streaming [43+1..44]
+[2026-06-21 08:46:16,726] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=44 target_lsn=44
+[2026-06-21 08:46:16,726] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [43+1..44] to standby
+[2026-06-21 08:47:29,557] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=45 < target=48 → streaming [45+1..48]
+[2026-06-21 08:47:29,557] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=46 target_lsn=48
+[2026-06-21 08:47:29,558] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [45+1..48] to standby
+[2026-06-21 08:47:29,758] [1217472] [  INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=46 < target=51 → streaming [46+1..51]
+[2026-06-21 08:47:29,758] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=47 target_lsn=51
+[2026-06-21 08:47:29,758] [1217472] [  INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [46+1..51] to standby
+[2026-06-21 08:47:36,064] [1228337] [  INFO] [                  main] My name is ctrl-3
+[2026-06-21 08:47:36,064] [1228337] [  INFO] [                  main] HA mode: 3 controller candidates, identity=ctrl-3
+[2026-06-21 08:47:36,065] [1228337] [  INFO] [   cluster_init_phase1] HA peers: 3 candidates, my_priority=2 (name=ctrl-3)
+[2026-06-21 08:47:36,065] [1228337] [  INFO] [   cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-3/data/ha-term.bin
+[2026-06-21 08:47:36,071] [1228337] [  WARN] [          persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-3/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
+[2026-06-21 08:47:36,072] [1228337] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:47:36,072] [1228337] [  INFO] [             myapp_run] Pre-election config loaded: tag=4986624fd309142c
+[2026-06-21 08:47:36,072] [1228337] [  INFO] [          ha_init_raft] HA: raft initialized (n_peers=3, self=2, term=3)
+[2026-06-21 08:47:37,072] [1228337] [  INFO] [             myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-3, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
+[2026-06-21 08:47:37,072] [1228337] [  INFO] [   cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
+[2026-06-21 08:47:37,072] [1228337] [  INFO] [       xce_config_init] xce controller is running
+[2026-06-21 08:47:37,072] [1228337] [  INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
+[2026-06-21 08:47:37,072] [1228337] [  INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
+[2026-06-21 08:47:37,074] [1228337] [  INFO] [   cluster_init_phase2] HA: persist_recover done — recovered durable_floor=52 current_lsn=53; applied_lsn=52 advertised for catch-up
+[2026-06-21 08:47:37,074] [1228337] [  INFO] [             myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
+[2026-06-21 08:47:37,074] [1228337] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:47:39,277] [1228337] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:47:39,277] [1228337] [  INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
+[2026-06-21 08:47:39,277] [1228337] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:47:39,277] [1228337] [  INFO] [      handle_ctrlhello] HA: primary control channel established (fd=7)
+[2026-06-21 08:47:52,078] [1228337] [  WARN] [      liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
+[2026-06-21 08:47:52,078] [1228337] [  INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
+[2026-06-21 08:47:52,078] [1228337] [  INFO] [     ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
+[2026-06-21 08:47:52,078] [1228337] [  INFO] [     ha_begin_failover] HA: staggering PreVote start by 1088ms (priority=2) to avoid split vote (bug-606)
+[2026-06-21 08:47:52,855] [1228337] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:47:52,855] [1228337] [  INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
+[2026-06-21 08:47:52,855] [1228337] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:47:52,855] [1228337] [  INFO] [      handle_ctrlhello] HA: primary control channel established (fd=7)
+[2026-06-21 08:47:53,279] [1228337] [  INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)
+[2026-06-21 08:48:04,083] [1228337] [  WARN] [      liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
+[2026-06-21 08:48:04,083] [1228337] [  INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
+[2026-06-21 08:48:04,083] [1228337] [  INFO] [     ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
+[2026-06-21 08:48:04,083] [1228337] [  INFO] [     ha_begin_failover] HA: staggering PreVote start by 1092ms (priority=2) to avoid split vote (bug-606)
+[2026-06-21 08:48:04,312] [1228337] [  INFO] [   load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
+[2026-06-21 08:48:04,312] [1228337] [  INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
+[2026-06-21 08:48:04,312] [1228337] [  INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
+[2026-06-21 08:48:04,312] [1228337] [  INFO] [      handle_ctrlhello] HA: primary control channel established (fd=7)
+[2026-06-21 08:48:05,284] [1228337] [  INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)

.kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-1.txt

@@ -0,0 +1,57 @@
+         1 |     0 | NOOP        |     0 | 00000000 | 
+         2 |     0 | JOB_SUBMIT  |  3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
+         3 |     0 | JOB_SUBMIT  |  3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
+         4 |     0 | JOB_SUBMIT  |  3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
+         5 |     0 | JOB_SUBMIT  |  3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
+         6 |     0 | JOB_SUBMIT  |  3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
+         7 |     0 | JOB_SUBMIT  |  3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
+         8 |     0 | JOB_SUBMIT  |  3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
+         9 |     0 | JOB_SUBMIT  |  3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
+        10 |     0 | JOB_SUBMIT  |  3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
+        11 |     0 | JOB_SUBMIT  |  3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
+        12 |     0 | JOB_SUBMIT  |  3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
+        13 |     0 | JOB_SUBMIT  |  3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
+        14 |     0 | NOOP        |     0 | 00000000 | 
+        15 |     0 | JOB_SUBMIT  |  3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
+        16 |     0 | JOB_SUBMIT  |  3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
+        17 |     0 | JOB_SUBMIT  |  3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
+        18 |     0 | JOB_SUBMIT  |  3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
+        19 |     0 | JOB_SUBMIT  |  3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
+        20 |     0 | JOB_SUBMIT  |  3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
+        21 |     0 | JOB_SUBMIT  |  3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
+        22 |     0 | JOB_SUBMIT  |  3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
+        23 |     0 | JOB_SUBMIT  |  3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
+        24 |     0 | JOB_SUBMIT  |  3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
+        25 |     0 | JOB_SUBMIT  |  3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
+        26 |     0 | JOB_SUBMIT  |  3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
+        27 |     0 | JOB_SUBMIT  |  3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
+        28 |     0 | JOB_SUBMIT  |  3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
+        29 |     0 | JOB_SUBMIT  |  3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
+        30 |     0 | JOB_SUBMIT  |  3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
+        31 |     0 | JOB_SUBMIT  |  3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
+        32 |     0 | JOB_SUBMIT  |  3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
+        33 |     0 | JOB_SUBMIT  |  3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
+        34 |     0 | JOB_SUBMIT  |  3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
+        35 |     0 | JOB_SUBMIT  |  3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
+        36 |     0 | JOB_SUBMIT  |  3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
+        37 |     0 | JOB_SUBMIT  |  3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
+        38 |     0 | JOB_SUBMIT  |  3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
+        39 |     0 | JOB_SUBMIT  |  3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
+        40 |     0 | JOB_SUBMIT  |  3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
+        41 |     0 | JOB_SUBMIT  |  3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
+        42 |     0 | JOB_SUBMIT  |  3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
+        43 |     0 | JOB_SUBMIT  |  3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
+        44 |     0 | JOB_SUBMIT  |  3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
+        45 |     0 | NOOP        |     0 | 00000000 | 
+        46 |     0 | JOB_SUBMIT  |  3952 | faead246 | jobid=1868595630964736 subtime=1782031661 state=1
+        47 |     0 | JOB_SUBMIT  |  3952 | faead246 | jobid=1868595630964736 subtime=1782031661 state=1
+        48 |     0 | JOB_SUBMIT  |  3952 | be0059bf | jobid=1868595630964737 subtime=1782031661 state=1
+        49 |     0 | JOB_SUBMIT  |  3952 | be0059bf | jobid=1868595630964737 subtime=1782031661 state=1
+        50 |     0 | JOB_SUBMIT  |  3952 | fe7a3d0b | jobid=1868595630964738 subtime=1782031661 state=1
+        51 |     0 | JOB_SUBMIT  |  3952 | fe7a3d0b | jobid=1868595630964738 subtime=1782031661 state=1
+        52 |     0 | JOB_SUBMIT  |  3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1
+        53 |     0 | JOB_SUBMIT  |  3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1
+        54 |     0 | NOOP        |     0 | 00000000 | 
+  segments: 3  entries: 54  submits: 50
+  lsn range: 1..54   recovered tip: 54
+  gaps: 0   crc chain: intact

.kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-2.txt

@@ -0,0 +1,57 @@
+         1 |     0 | NOOP        |     0 | 00000000 | 
+         2 |     0 | JOB_SUBMIT  |  3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
+         3 |     0 | JOB_SUBMIT  |  3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
+         4 |     0 | JOB_SUBMIT  |  3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
+         5 |     0 | JOB_SUBMIT  |  3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
+         6 |     0 | JOB_SUBMIT  |  3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
+         7 |     0 | JOB_SUBMIT  |  3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
+         8 |     0 | JOB_SUBMIT  |  3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
+         9 |     0 | JOB_SUBMIT  |  3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
+        10 |     0 | JOB_SUBMIT  |  3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
+        11 |     0 | JOB_SUBMIT  |  3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
+        12 |     0 | JOB_SUBMIT  |  3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
+        13 |     0 | JOB_SUBMIT  |  3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
+        14 |     0 | NOOP        |     0 | 00000000 | 
+        15 |     0 | JOB_SUBMIT  |  3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
+        16 |     0 | JOB_SUBMIT  |  3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
+        17 |     0 | JOB_SUBMIT  |  3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
+        18 |     0 | JOB_SUBMIT  |  3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
+        19 |     0 | JOB_SUBMIT  |  3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
+        20 |     0 | JOB_SUBMIT  |  3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
+        21 |     0 | JOB_SUBMIT  |  3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
+        22 |     0 | JOB_SUBMIT  |  3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
+        23 |     0 | JOB_SUBMIT  |  3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
+        24 |     0 | JOB_SUBMIT  |  3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
+        25 |     0 | JOB_SUBMIT  |  3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
+        26 |     0 | JOB_SUBMIT  |  3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
+        27 |     0 | JOB_SUBMIT  |  3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
+        28 |     0 | JOB_SUBMIT  |  3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
+        29 |     0 | JOB_SUBMIT  |  3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
+        30 |     0 | JOB_SUBMIT  |  3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
+        31 |     0 | JOB_SUBMIT  |  3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
+        32 |     0 | JOB_SUBMIT  |  3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
+        33 |     0 | JOB_SUBMIT  |  3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
+        34 |     0 | JOB_SUBMIT  |  3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
+        35 |     0 | JOB_SUBMIT  |  3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
+        36 |     0 | JOB_SUBMIT  |  3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
+        37 |     0 | JOB_SUBMIT  |  3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
+        38 |     0 | JOB_SUBMIT  |  3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
+        39 |     0 | JOB_SUBMIT  |  3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
+        40 |     0 | JOB_SUBMIT  |  3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
+        41 |     0 | JOB_SUBMIT  |  3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
+        42 |     0 | JOB_SUBMIT  |  3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
+        43 |     0 | JOB_SUBMIT  |  3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
+        44 |     0 | JOB_SUBMIT  |  3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
+        45 |     0 | NOOP        |     0 | 00000000 | 
+        46 |     0 | JOB_SUBMIT  |  3952 | d32ac747 | jobid=1868595643547648 subtime=1782031673 state=1
+        47 |     0 | JOB_SUBMIT  |  3952 | d32ac747 | jobid=1868595643547648 subtime=1782031673 state=1
+        48 |     0 | JOB_SUBMIT  |  3952 | 97c04cbe | jobid=1868595643547649 subtime=1782031673 state=1
+        49 |     0 | JOB_SUBMIT  |  3952 | 97c04cbe | jobid=1868595643547649 subtime=1782031673 state=1
+        50 |     0 | JOB_SUBMIT  |  3952 | d7ba280a | jobid=1868595643547650 subtime=1782031673 state=1
+        51 |     0 | JOB_SUBMIT  |  3952 | d7ba280a | jobid=1868595643547650 subtime=1782031673 state=1
+        52 |     0 | JOB_SUBMIT  |  3952 | 1e155b4c | jobid=1868595643547651 subtime=1782031673 state=1
+        53 |     0 | JOB_SUBMIT  |  3952 | 1e155b4c | jobid=1868595643547651 subtime=1782031673 state=1
+        54 |     0 | NOOP        |     0 | 00000000 | 
+  segments: 3  entries: 54  submits: 50
+  lsn range: 1..54   recovered tip: 54
+  gaps: 0   crc chain: intact

.kiro/specs/ha-commit-durability/repro-851/waldump-ctrl-3.txt

@@ -0,0 +1,57 @@
+         1 |     0 | NOOP        |     0 | 00000000 | 
+         2 |     0 | JOB_SUBMIT  |  3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
+         3 |     0 | JOB_SUBMIT  |  3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
+         4 |     0 | JOB_SUBMIT  |  3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
+         5 |     0 | JOB_SUBMIT  |  3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
+         6 |     0 | JOB_SUBMIT  |  3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
+         7 |     0 | JOB_SUBMIT  |  3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
+         8 |     0 | JOB_SUBMIT  |  3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
+         9 |     0 | JOB_SUBMIT  |  3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
+        10 |     0 | JOB_SUBMIT  |  3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
+        11 |     0 | JOB_SUBMIT  |  3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
+        12 |     0 | JOB_SUBMIT  |  3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
+        13 |     0 | JOB_SUBMIT  |  3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
+        14 |     0 | NOOP        |     0 | 00000000 | 
+        15 |     0 | JOB_SUBMIT  |  3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
+        16 |     0 | JOB_SUBMIT  |  3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
+        17 |     0 | JOB_SUBMIT  |  3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
+        18 |     0 | JOB_SUBMIT  |  3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
+        19 |     0 | JOB_SUBMIT  |  3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
+        20 |     0 | JOB_SUBMIT  |  3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
+        21 |     0 | JOB_SUBMIT  |  3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
+        22 |     0 | JOB_SUBMIT  |  3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
+        23 |     0 | JOB_SUBMIT  |  3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
+        24 |     0 | JOB_SUBMIT  |  3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
+        25 |     0 | JOB_SUBMIT  |  3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
+        26 |     0 | JOB_SUBMIT  |  3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
+        27 |     0 | JOB_SUBMIT  |  3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
+        28 |     0 | JOB_SUBMIT  |  3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
+        29 |     0 | JOB_SUBMIT  |  3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
+        30 |     0 | JOB_SUBMIT  |  3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
+        31 |     0 | JOB_SUBMIT  |  3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
+        32 |     0 | JOB_SUBMIT  |  3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
+        33 |     0 | JOB_SUBMIT  |  3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
+        34 |     0 | JOB_SUBMIT  |  3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
+        35 |     0 | JOB_SUBMIT  |  3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
+        36 |     0 | JOB_SUBMIT  |  3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
+        37 |     0 | JOB_SUBMIT  |  3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
+        38 |     0 | JOB_SUBMIT  |  3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
+        39 |     0 | JOB_SUBMIT  |  3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
+        40 |     0 | JOB_SUBMIT  |  3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
+        41 |     0 | JOB_SUBMIT  |  3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
+        42 |     0 | JOB_SUBMIT  |  3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
+        43 |     0 | JOB_SUBMIT  |  3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
+        44 |     0 | JOB_SUBMIT  |  3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
+        45 |     0 | JOB_SUBMIT  |  3952 | 4c6d24e6 | jobid=1868595618381824 subtime=1782031649 state=1
+        46 |     0 | JOB_SUBMIT  |  3952 | 4c6d24e6 | jobid=1868595618381824 subtime=1782031649 state=1
+        47 |     0 | JOB_SUBMIT  |  3952 | 0887af1f | jobid=1868595618381825 subtime=1782031649 state=1
+        48 |     0 | JOB_SUBMIT  |  3952 | 0887af1f | jobid=1868595618381825 subtime=1782031649 state=1
+        49 |     0 | JOB_SUBMIT  |  3952 | 48fdcbab | jobid=1868595618381826 subtime=1782031649 state=1
+        50 |     0 | JOB_SUBMIT  |  3952 | 48fdcbab | jobid=1868595618381826 subtime=1782031649 state=1
+        51 |     0 | JOB_SUBMIT  |  3952 | 8152b8ed | jobid=1868595618381827 subtime=1782031649 state=1
+        52 |     0 | JOB_SUBMIT  |  3952 | 8152b8ed | jobid=1868595618381827 subtime=1782031649 state=1
+        53 |     0 | JOB_SUBMIT  |  3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1
+        54 |     0 | NOOP        |     0 | 00000000 | 
+  segments: 2  entries: 54  submits: 51
+  lsn range: 1..54   recovered tip: 54
+  gaps: 0   crc chain: intact