feat(ha-commit-durability): bounded-window catch-up before FATAL — fence no longer FATALs synchronously (task 4.4)

.kiro/specs/ha-commit-durability/tasks.md

@@ -98,6 +98,13 @@
   - _Requirements: 2.2, 2.5, 2.6_
   - _Depends: 4.2, 3.1, 3.2, 2.2_
 
+- [x] 4.4 Bounded-window catch-up before FATAL (fence does not FATAL immediately)
+  - The promotion gate (4.3) currently FATALs a recovery-truncated/behind member immediately at the promotion instant when no granting voter has a live endpoint AT THAT INSTANT — a transient no-peer snapshot during aggressive failover. Per the 3.2 decision table (rows 3/5) a member with a (momentarily) reachable peer must FENCE→catch-up→ELIGIBLE, NEVER immediate-FATAL; FATAL is reached ONLY after a BOUNDED failure to reach a committed-consistent state. Implement the deferred bounded-window catch-up: on FENCE (or would-be-FATAL no-peer), keep the member alive in `in_catch_up`, schedule a one-shot delaymgr timer for `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS`, and reach the 3.3 FATAL ONLY from that timer if the member is STILL recovery-truncated AND durable_floor < cluster HWM AND no healthy peer / no catch-up progress when it fires. Re-derive eligibility (release the fence) when catch-up brings durable_floor ≥ HWM. Never FATAL synchronously in `raft_cb_become_leader`; never fence indefinitely.
+  - Observable: a recovery-truncated node whose peers are transiently unreachable FENCES (stays alive) and is NOT FATAL'd; once a healthy peer is reachable and catch-up passes the committed boundary it regains eligibility; FATAL fires only after the bounded window with no peer/progress (unit-tested via the `persist_fatal_hook` + a short window). The rolling SIGKILL soak (task 5.2) keeps the cluster available (always re-elects) with zero acknowledged-job loss.
+  - _Requirements: 2.2, 2.4, 2.6_
+  - _Depends: 4.3, 3.3_
+  - _Boundary: PromotionFence_
+
 - [ ] 5. Integration and validation
 - [ ] 5.1 Wire durable-majority commit end to end
   - Connect the durable-LSN self-count and the commit barrier so the client reply is released only after the entry is durable on a majority; assert liveness (the flusher does not stall heartbeat/election handling).
@@ -135,3 +142,4 @@
 - 4.2: Implements the 4.1 formula. New `ha_candidate_advertise_lsn(persist*)` = bare `persist_durable_floor`. CANDIDATE sites advertise bare durable_floor for the LSN (tag still via `ha_vote_self_freshness(...,&tag,NULL)`): ha.c:~3690 (real election), ha.c:~3771 (PreVote), AND myapp.c:~234 (boot election — a THIRD candidate site the brief didn't name; fixed too, else a boot-time truncated candidate over-advertises). VOTER floor: cluster.c:~383 (real) + cluster.c:~471 (PreVote) feed `local_lsn = persist_durable_floor(&c->persist)` into `ha_vote_self_freshness` → floor = `max(durable_floor, learned_committed)`. Comparison unchanged (term-then-index). `ha.c:2243` legacy `handle_request_vote` is XCE_TESTING-only/param-driven — left unchanged. `safe_committed_lsn` UNTOUCHED (voter-reply-sourced). 3 tests in test_ha_vote_gate.c prove: candidate advertises bare durable_floor (not stale learned); voter durable-floor denies truncated; longest-durable grantable / no livelock. bug-339 regression test fixture primed `durable_lsn=10` (model-preserving). INFO: truncated-deny proven at the freshness-comparison-expression level shared by both vote+PreVote paths (no dedicated live PreVote-handler truncation driver — adequate; nice-to-have for 5.x). GOTCHA for future election tests: the freshness floor now reads `persist_durable_floor`, so any vote/election fixture that sets `applied_lsn`/`last_known_committed_lsn` but not durable state will see floor/advertise=0 — prime `durable_lsn` to the modeled tip.
 - 4.3: Pre-promotion eligibility gate in `raft_cb_become_leader` (ha.c, post-truncate / pre-CTRL_PRIMARY, returns on refuse). `ha_promotion_gate_decision` (pure, delegates to 3.3's `persist_recover_decision` — table is single source of truth) + `ha_evaluate_promotion_fence` (live: ELIGIBLE clears `in_catch_up`, FENCE sets it, FATAL → 3.3 `persist_recover_fatal_if_unrecoverable`). REFUSE if `persist_recovery_truncated` OR `persist_durable_floor < promotion_safe_lsn` (the existing voter-reply-derived `safe_committed_lsn`, read AFTER `persist_truncate_wal_above` so it reflects 2.2's clamp). `peer_reachable` = some granting voter has a live endpoint (`ectx->votes[i].vote_granted && .ep != NULL`) — so a multi-node election winner ALWAYS has a peer ⇒ FENCE (never premature FATAL, honoring rows 3/5); FATAL only fires no-peer (single-node corrupt = genuinely unrecoverable, row 4). Reuses `in_catch_up` (NOT a parallel fence); release on a later promotion attempt once caught up (durable_floor ≥ HWM, not truncated) → ELIGIBLE clears the fence (Req 2.6, re-evaluation model). `safe_committed_lsn` arithmetic UNTOUCHED. Turned 3.2 rows 1/2/3/5/6 GREEN (0 ignored boot-policy rows remain) + minimal-quorum test (Req 2.5). DEFERRED to 5.1: the live bounded-window catch-up TIMER (grant `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS` to find a peer before FATAL) — here `catch_up_succeeded=false` at the gate, so it FENCEs (peer) or FATALs (no peer) immediately; fail-loud-conservative, never silently leads hollow. NOTE: reviewed in MAIN CONTEXT (review subagent hit a spend limit) — suite green ×1 (Fail:0, all 9 boot-policy/gate tests PASS, test_fact_hook_append passed too since this shell's /tmp is fsync-able). Phase 4 COMPLETE.
 - 5.1: INTEGRATION verification — the end-to-end durable-majority path was ALREADY wired by 2.1/2.2/2.4/4.x (no production gap found); 5.1 adds 3 integration tests + a tiny `#ifdef XCE_TESTING` liveness-observability seam (`persist_test_last_fdatasync_thread`, nm-excluded from prod). `test_durable_majority_commit_end_to_end`: holds durability off (forced fsync-fail), delivers BOTH standby ACKs, asserts committed_lsn/WAIT do NOT release (ACK-majority alone insufficient) — then flush makes self durable ⇒ committed advances + WAIT releases rc=0; asserts `committed_lsn <= durable_lsn`. RED proof: injecting the pre-2.2 `highest_appended_lsn` cap trips `assert(cand <= persist_durable_lsn)`. `test_sole_durable_contributor_loss_survives` (Req 1.5/2.5): property-asserted (committed ⇒ ≥⌊N/2⌋+1 durable copies, two persist dirs, drop one) — LIVE failover proof is 5.2. `test_durability_work_off_iomgr_preserves_liveness` (Req 1.6): seam records the flusher thread id; asserts fdatasync runs off the caller/iomgr thread + heartbeat reads stay <50ms during flushes. GOTCHA: a real persist with registered standbys needs the `XCE_TESTING repl_broadcast_send_hook` (drop) installed before append, else dummy endpoints trip an ASAN overflow in `dataq_enq`. CAVEATS (for 5.3/validation): (1) meson `Fail:0` masks Unity sub-group failures — gate on `grep -c :FAIL: testlog.txt`==0; (2) run on a fsync-able fs (the pre-existing test_fact_hook_append EIO failure is /tmp-overlay-only). 4.3 catch-up-timer-before-FATAL still a follow-up. Reviewed in MAIN CONTEXT (subagent spend limit).
+- 4.4: Closed the 5.2-soak liveness gap. `raft_cb_become_leader` NO LONGER FATALs synchronously — on FENCE / would-be-FATAL (truncated or durable_floor < HWM) it calls `ha_arm_promotion_fence_timer` (sets `in_catch_up`, schedules a one-shot `delay_task_ms(&c->delaymgr, ha_promotion_fence_window_ms(), 1, promotion_fence_timeout_fn, ctx)`) and RETURNS without becoming primary. The timer callback (`promotion_fence_timeout_fn`, ha.c:702) re-reads LIVE state after the window: caught-up (`!truncated && durable_floor>=HWM`) → release fence, NO FATAL (Req 2.6); peer reachable (`repl_active_standby_count>0`) OR durable_floor advanced past the arm-time baseline → progress possible, leave fenced (existing catch-up machinery / next promotion-gate ELIGIBLE releases it), NO FATAL; ONLY still-truncated/behind + no-peer + no-progress → 3.3 `persist_recover_fatal_if_unrecoverable` (Req 2.4). UAF-safe cancel (`ha_cancel_promotion_fence_timer`: cancelled flag + atomic-exchange the stored ctx; handle cleared before the cancelled-check, mirrors `raft_election_timeout_fn`); cancelled in `ha_fini`; fail-loud if `delay_task_ms` itself fails (never fence indefinitely). Test window seam `ha_set_promotion_fence_window_ms` + driver `ha_arm_promotion_fence_timer_for_test`. 3 tests (defers-fatal-to-timer, released-on-catchup-no-fatal, cancel-prevents-fatal) with RED proof (re-injecting sync FATAL → 3 fails). Reviewed MAIN CONTEXT (subagent spend limit): suite :FAIL:==0, /tmp fsync-able this run. NOTE: this is the bounded-window timer 4.3/5.1 deferred; with it, 5.2's soak should keep the cluster available (truncated rejoiners catch up, not die).