diff --git a/.kiro/specs/ha-commit-durability/tasks.md b/.kiro/specs/ha-commit-durability/tasks.md index 76a5e23..394a557 100644 --- a/.kiro/specs/ha-commit-durability/tasks.md +++ b/.kiro/specs/ha-commit-durability/tasks.md @@ -98,6 +98,13 @@ - _Requirements: 2.2, 2.5, 2.6_ - _Depends: 4.2, 3.1, 3.2, 2.2_ +- [x] 4.4 Bounded-window catch-up before FATAL (fence does not FATAL immediately) + - The promotion gate (4.3) currently FATALs a recovery-truncated/behind member immediately at the promotion instant when no granting voter has a live endpoint AT THAT INSTANT — a transient no-peer snapshot during aggressive failover. Per the 3.2 decision table (rows 3/5) a member with a (momentarily) reachable peer must FENCE→catch-up→ELIGIBLE, NEVER immediate-FATAL; FATAL is reached ONLY after a BOUNDED failure to reach a committed-consistent state. Implement the deferred bounded-window catch-up: on FENCE (or would-be-FATAL no-peer), keep the member alive in `in_catch_up`, schedule a one-shot delaymgr timer for `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS`, and reach the 3.3 FATAL ONLY from that timer if the member is STILL recovery-truncated AND durable_floor < cluster HWM AND no healthy peer / no catch-up progress when it fires. Re-derive eligibility (release the fence) when catch-up brings durable_floor ≥ HWM. Never FATAL synchronously in `raft_cb_become_leader`; never fence indefinitely. + - Observable: a recovery-truncated node whose peers are transiently unreachable FENCES (stays alive) and is NOT FATAL'd; once a healthy peer is reachable and catch-up passes the committed boundary it regains eligibility; FATAL fires only after the bounded window with no peer/progress (unit-tested via the `persist_fatal_hook` + a short window). The rolling SIGKILL soak (task 5.2) keeps the cluster available (always re-elects) with zero acknowledged-job loss. + - _Requirements: 2.2, 2.4, 2.6_ + - _Depends: 4.3, 3.3_ + - _Boundary: PromotionFence_ + - [ ] 5. Integration and validation - [ ] 5.1 Wire durable-majority commit end to end - Connect the durable-LSN self-count and the commit barrier so the client reply is released only after the entry is durable on a majority; assert liveness (the flusher does not stall heartbeat/election handling). @@ -135,3 +142,4 @@ - 4.2: Implements the 4.1 formula. New `ha_candidate_advertise_lsn(persist*)` = bare `persist_durable_floor`. CANDIDATE sites advertise bare durable_floor for the LSN (tag still via `ha_vote_self_freshness(...,&tag,NULL)`): ha.c:~3690 (real election), ha.c:~3771 (PreVote), AND myapp.c:~234 (boot election — a THIRD candidate site the brief didn't name; fixed too, else a boot-time truncated candidate over-advertises). VOTER floor: cluster.c:~383 (real) + cluster.c:~471 (PreVote) feed `local_lsn = persist_durable_floor(&c->persist)` into `ha_vote_self_freshness` → floor = `max(durable_floor, learned_committed)`. Comparison unchanged (term-then-index). `ha.c:2243` legacy `handle_request_vote` is XCE_TESTING-only/param-driven — left unchanged. `safe_committed_lsn` UNTOUCHED (voter-reply-sourced). 3 tests in test_ha_vote_gate.c prove: candidate advertises bare durable_floor (not stale learned); voter durable-floor denies truncated; longest-durable grantable / no livelock. bug-339 regression test fixture primed `durable_lsn=10` (model-preserving). INFO: truncated-deny proven at the freshness-comparison-expression level shared by both vote+PreVote paths (no dedicated live PreVote-handler truncation driver — adequate; nice-to-have for 5.x). GOTCHA for future election tests: the freshness floor now reads `persist_durable_floor`, so any vote/election fixture that sets `applied_lsn`/`last_known_committed_lsn` but not durable state will see floor/advertise=0 — prime `durable_lsn` to the modeled tip. - 4.3: Pre-promotion eligibility gate in `raft_cb_become_leader` (ha.c, post-truncate / pre-CTRL_PRIMARY, returns on refuse). `ha_promotion_gate_decision` (pure, delegates to 3.3's `persist_recover_decision` — table is single source of truth) + `ha_evaluate_promotion_fence` (live: ELIGIBLE clears `in_catch_up`, FENCE sets it, FATAL → 3.3 `persist_recover_fatal_if_unrecoverable`). REFUSE if `persist_recovery_truncated` OR `persist_durable_floor < promotion_safe_lsn` (the existing voter-reply-derived `safe_committed_lsn`, read AFTER `persist_truncate_wal_above` so it reflects 2.2's clamp). `peer_reachable` = some granting voter has a live endpoint (`ectx->votes[i].vote_granted && .ep != NULL`) — so a multi-node election winner ALWAYS has a peer ⇒ FENCE (never premature FATAL, honoring rows 3/5); FATAL only fires no-peer (single-node corrupt = genuinely unrecoverable, row 4). Reuses `in_catch_up` (NOT a parallel fence); release on a later promotion attempt once caught up (durable_floor ≥ HWM, not truncated) → ELIGIBLE clears the fence (Req 2.6, re-evaluation model). `safe_committed_lsn` arithmetic UNTOUCHED. Turned 3.2 rows 1/2/3/5/6 GREEN (0 ignored boot-policy rows remain) + minimal-quorum test (Req 2.5). DEFERRED to 5.1: the live bounded-window catch-up TIMER (grant `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS` to find a peer before FATAL) — here `catch_up_succeeded=false` at the gate, so it FENCEs (peer) or FATALs (no peer) immediately; fail-loud-conservative, never silently leads hollow. NOTE: reviewed in MAIN CONTEXT (review subagent hit a spend limit) — suite green ×1 (Fail:0, all 9 boot-policy/gate tests PASS, test_fact_hook_append passed too since this shell's /tmp is fsync-able). Phase 4 COMPLETE. - 5.1: INTEGRATION verification — the end-to-end durable-majority path was ALREADY wired by 2.1/2.2/2.4/4.x (no production gap found); 5.1 adds 3 integration tests + a tiny `#ifdef XCE_TESTING` liveness-observability seam (`persist_test_last_fdatasync_thread`, nm-excluded from prod). `test_durable_majority_commit_end_to_end`: holds durability off (forced fsync-fail), delivers BOTH standby ACKs, asserts committed_lsn/WAIT do NOT release (ACK-majority alone insufficient) — then flush makes self durable ⇒ committed advances + WAIT releases rc=0; asserts `committed_lsn <= durable_lsn`. RED proof: injecting the pre-2.2 `highest_appended_lsn` cap trips `assert(cand <= persist_durable_lsn)`. `test_sole_durable_contributor_loss_survives` (Req 1.5/2.5): property-asserted (committed ⇒ ≥⌊N/2⌋+1 durable copies, two persist dirs, drop one) — LIVE failover proof is 5.2. `test_durability_work_off_iomgr_preserves_liveness` (Req 1.6): seam records the flusher thread id; asserts fdatasync runs off the caller/iomgr thread + heartbeat reads stay <50ms during flushes. GOTCHA: a real persist with registered standbys needs the `XCE_TESTING repl_broadcast_send_hook` (drop) installed before append, else dummy endpoints trip an ASAN overflow in `dataq_enq`. CAVEATS (for 5.3/validation): (1) meson `Fail:0` masks Unity sub-group failures — gate on `grep -c :FAIL: testlog.txt`==0; (2) run on a fsync-able fs (the pre-existing test_fact_hook_append EIO failure is /tmp-overlay-only). 4.3 catch-up-timer-before-FATAL still a follow-up. Reviewed in MAIN CONTEXT (subagent spend limit). +- 4.4: Closed the 5.2-soak liveness gap. `raft_cb_become_leader` NO LONGER FATALs synchronously — on FENCE / would-be-FATAL (truncated or durable_floor < HWM) it calls `ha_arm_promotion_fence_timer` (sets `in_catch_up`, schedules a one-shot `delay_task_ms(&c->delaymgr, ha_promotion_fence_window_ms(), 1, promotion_fence_timeout_fn, ctx)`) and RETURNS without becoming primary. The timer callback (`promotion_fence_timeout_fn`, ha.c:702) re-reads LIVE state after the window: caught-up (`!truncated && durable_floor>=HWM`) → release fence, NO FATAL (Req 2.6); peer reachable (`repl_active_standby_count>0`) OR durable_floor advanced past the arm-time baseline → progress possible, leave fenced (existing catch-up machinery / next promotion-gate ELIGIBLE releases it), NO FATAL; ONLY still-truncated/behind + no-peer + no-progress → 3.3 `persist_recover_fatal_if_unrecoverable` (Req 2.4). UAF-safe cancel (`ha_cancel_promotion_fence_timer`: cancelled flag + atomic-exchange the stored ctx; handle cleared before the cancelled-check, mirrors `raft_election_timeout_fn`); cancelled in `ha_fini`; fail-loud if `delay_task_ms` itself fails (never fence indefinitely). Test window seam `ha_set_promotion_fence_window_ms` + driver `ha_arm_promotion_fence_timer_for_test`. 3 tests (defers-fatal-to-timer, released-on-catchup-no-fatal, cancel-prevents-fatal) with RED proof (re-injecting sync FATAL → 3 fails). Reviewed MAIN CONTEXT (subagent spend limit): suite :FAIL:==0, /tmp fsync-able this run. NOTE: this is the bounded-window timer 4.3/5.1 deferred; with it, 5.2's soak should keep the cluster available (truncated rejoiners catch up, not die). diff --git a/src/controller/ha.c b/src/controller/ha.c index c3811eb..29b28fb 100644 --- a/src/controller/ha.c +++ b/src/controller/ha.c @@ -623,6 +623,226 @@ recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha, return d; } +/* ── ha-commit-durability task 4.4: bounded-window catch-up before FATAL ───── + * + * The 4.3 gate above can DECIDE FATAL synchronously, but a member that won an + * election with no peer reachable AT THE PROMOTION INSTANT may simply be seeing + * a transient no-peer snapshot during aggressive failover (task-5.2 soak: a + * recovery-truncated winner self-FATAL'd immediately and repeatedly until + * quorum was lost — an availability collapse, not durable loss). Per the 3.2 + * decision table the FATAL terminal must be reached ONLY after a BOUNDED + * failure: the member is fenced (in_catch_up), given DEFAULT_RECOVERY_CATCHUP_- + * WINDOW_MS to find a peer / make catch-up progress, and FATAL'd from a one-shot + * timer only if it is STILL hollow with no peer and no progress when the window + * elapses. + * + * The window is a settable seam so the timer test runs fast. Default = + * DEFAULT_RECOVERY_CATCHUP_WINDOW_MS (persist.h). */ +static _Atomic uint32_t s_promotion_fence_window_ms = + DEFAULT_RECOVERY_CATCHUP_WINDOW_MS; + +uint32_t ha_promotion_fence_window_ms(void) +{ + uint32_t w = atomic_load_explicit(&s_promotion_fence_window_ms, + memory_order_acquire); + return w ? w : DEFAULT_RECOVERY_CATCHUP_WINDOW_MS; +} + +#ifdef XCE_TESTING +void ha_set_promotion_fence_window_ms(uint32_t ms) +{ + atomic_store_explicit(&s_promotion_fence_window_ms, ms, + memory_order_release); +} +#endif + +/* promotion_fence_ctx_t: delaymgr context for the bounded catch-up window. + * Captures the cluster committed high-water mark (target_hwm) and the durable + * floor AT SCHEDULE TIME (baseline_durable_floor) so the timer can detect + * catch-up progress (durable_floor advanced toward the HWM). `cancelled` is set + * by ha_cancel_promotion_fence_timer (recovery / shutdown) so a fired-but- + * cancelled callback is a safe no-op. */ +typedef struct { + _Atomic int cancelled; + ha_state_t *ha; + cluster_t *c; + bool was_truncated; /* recovery-truncated at schedule time */ + uint64_t target_hwm; /* cluster committed high-water mark */ + uint64_t baseline_durable_floor; /* durable floor at schedule time */ +} promotion_fence_ctx_t; + +/* ha_cancel_promotion_fence_timer: cancel a pending bounded-window FATAL timer. + * Idempotent; safe to call when no timer is armed. Used when the member catches + * up (fence released) or when the controller is shutting down — so neither a + * caught-up nor a stopped member ever reaches the deferred FATAL. */ +void ha_cancel_promotion_fence_timer(ha_state_t *ha) +{ + if (!ha) return; + promotion_fence_ctx_t *t = (promotion_fence_ctx_t *)atomic_exchange_explicit( + &ha->promotion_fence_ctx, NULL, memory_order_acq_rel); + if (t) + atomic_store_explicit(&t->cancelled, 1, memory_order_release); +} + +/* promotion_fence_timeout_fn: the bounded-window timer callback. Owns its ctx — + * frees it on exit. Re-evaluates the LIVE recovery/durability state and reaches + * the 3.3 FATAL ONLY on a real bounded failure (Req 2.4): + * - still recovery-truncated/behind AND no healthy peer reachable AND no + * catch-up progress (durable_floor has not advanced past the baseline) + * → persist_recover_fatal_if_unrecoverable (fail loud). + * - caught up (NOT truncated AND durable_floor >= target_hwm) → release the + * in_catch_up fence; do NOT FATAL (Req 2.6 — eligible again). + * - a healthy peer is reachable now, OR catch-up is making progress but is not + * yet complete → leave the member fenced under the existing catch-up + * machinery and do NOT FATAL while progress is possible. The single bounded + * window has elapsed, so we do not re-arm: a member with a live peer is no + * longer in the "no peer + no progress" bounded-failure state the FATAL + * guards, and the existing catch-up / re-election path will release the + * fence (ha_evaluate_promotion_fence → ELIGIBLE) when it completes. */ +static int promotion_fence_timeout_fn(void *arg) +{ + promotion_fence_ctx_t *t = (promotion_fence_ctx_t *)arg; + ha_state_t *ha = t->ha; + cluster_t *c = t->c; + + /* Clear the stored handle before checking cancelled so a concurrent + * ha_cancel_promotion_fence_timer is a safe no-op (matches the + * raft_election_timeout_fn convention). */ + promotion_fence_ctx_t *cur = (promotion_fence_ctx_t *)atomic_load_explicit( + &ha->promotion_fence_ctx, memory_order_acquire); + if (cur == t) + atomic_store_explicit(&ha->promotion_fence_ctx, NULL, + memory_order_release); + + if (atomic_load_explicit(&t->cancelled, memory_order_acquire)) { + free(t); + return 0; + } + + if (!c) { free(t); return 0; } /* defensive: no persist to evaluate */ + + /* Re-read the LIVE state after the bounded window. */ + bool truncated = persist_recovery_truncated(&c->persist); + uint64_t durable_floor = persist_durable_floor(&c->persist); + bool below_hwm = durable_floor < t->target_hwm; + bool made_progress = durable_floor > t->baseline_durable_floor; + bool peer_reachable = (ha->repl != NULL) + ? (repl_active_standby_count(ha->repl) > 0) + : false; + + /* Caught up: not truncated AND durable position reached the boundary → + * release the fence; the member is eligible on its next election (Req 2.6). */ + if (!truncated && !below_hwm) { + atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release); + INFO("HA: promotion fence RELEASED after catch-up — durable_floor=%" + PRIu64 " >= cluster HWM=%" PRIu64 " (ha-commit-durability Req 2.6)", + durable_floor, t->target_hwm); + free(t); + return 0; + } + + /* Still hollow, but a peer is reachable or catch-up advanced the durable + * floor: progress is possible — do NOT FATAL. Leave the member fenced + * (in_catch_up stays set) under the existing catch-up machinery; the bounded + * window has elapsed so we do not re-arm. A member with a live peer is no + * longer in the no-peer bounded-failure state the FATAL guards. */ + if (peer_reachable || made_progress) { + INFO("HA: promotion fence catch-up window elapsed but progress is " + "possible (peer_reachable=%d durable_floor=%" PRIu64 " baseline=%" + PRIu64 " HWM=%" PRIu64 ") — remaining fenced, NOT FATAL " + "(ha-commit-durability Req 2.2)", + peer_reachable, durable_floor, t->baseline_durable_floor, + t->target_hwm); + free(t); + return 0; + } + + /* Bounded failure: STILL truncated/behind AND no healthy peer AND no + * catch-up progress after the window → fail loud (Req 2.4). This is the + * ONLY path that reaches the 3.3 FATAL — never synchronously at promotion. */ + { + recover_policy_inputs_t in = { + .verdict = truncated ? RECOVER_TRUNCATED + : RECOVER_CLEAN, + .durable_below_hwm = below_hwm, + .healthy_peer_reachable = false, + .catch_up_succeeded = false, + .durable_floor_lsn = durable_floor, + .cluster_hwm_lsn = t->target_hwm, + }; + ERROR("HA: promotion catch-up window (%u ms) elapsed with no peer and " + "no progress — recovery_truncated=%d durable_floor=%" PRIu64 + " < cluster HWM=%" PRIu64 "; failing loud rather than serving " + "divergent state (ha-commit-durability Req 2.4)", + ha_promotion_fence_window_ms(), truncated, durable_floor, + t->target_hwm); + persist_recover_fatal_if_unrecoverable(&in); + } + + free(t); + return 0; +} + +/* ha_arm_promotion_fence_timer: fence the member (set in_catch_up) and schedule + * the one-shot bounded-window FATAL timer. Cancels any previously-armed timer + * first so a re-evaluation never leaks or double-fires. Returns 0 on success. + * On a failure to schedule the timer it FATALs immediately via the 3.3 action so + * the bounded-failure terminal is never silently lost (a member that could not + * arm its own catch-up deadline must not run on indefinitely). */ +static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c, + bool truncated, uint64_t durable_floor, + uint64_t target_hwm) +{ + /* Fence first (Req 2.2): refuse promotion while hollow. */ + atomic_store_explicit(&ha->in_catch_up, 1, memory_order_release); + + /* Cancel a stale timer (e.g. a prior promotion attempt). */ + ha_cancel_promotion_fence_timer(ha); + + promotion_fence_ctx_t *t = calloc(1, sizeof(*t)); + if (!t) + goto fail_loud; + + atomic_init(&t->cancelled, 0); + t->ha = ha; + t->c = c; + t->was_truncated = truncated; + t->target_hwm = target_hwm; + t->baseline_durable_floor = durable_floor; + + uint32_t window_ms = ha_promotion_fence_window_ms(); + int rc = delay_task_ms(&c->delaymgr, (int)window_ms, 1, + promotion_fence_timeout_fn, t); + if (rc < 0) { + WARN("ha_arm_promotion_fence_timer: delay_task_ms failed: %d", rc); + free(t); + goto fail_loud; + } + + atomic_store_explicit(&ha->promotion_fence_ctx, t, memory_order_release); + INFO("HA: promotion FENCED + bounded catch-up window armed (%u ms) — " + "recovery_truncated=%d durable_floor=%" PRIu64 " < cluster HWM=%" PRIu64 + "; FATAL deferred to the timer (ha-commit-durability Req 2.2/2.4)", + window_ms, truncated, durable_floor, target_hwm); + return; + +fail_loud: + { + /* Could not arm the bounded deadline — reach the FATAL terminal now + * rather than fence indefinitely. */ + recover_policy_inputs_t in = { + .verdict = truncated ? RECOVER_TRUNCATED + : RECOVER_CLEAN, + .durable_below_hwm = (durable_floor < target_hwm), + .healthy_peer_reachable = false, + .catch_up_succeeded = false, + .durable_floor_lsn = durable_floor, + .cluster_hwm_lsn = target_hwm, + }; + persist_recover_fatal_if_unrecoverable(&in); + } +} + /* * raft_cb_become_leader: node won the election — enter CTRL_PRIMARY directly. * @@ -722,30 +942,41 @@ static void raft_cb_become_leader(void *ctx, uint64_t term) } /* catch_up_succeeded is false at the promotion-eligibility evaluation: - * the live bounded-window catch-up loop is wired by task 5.1; here the - * gate refuses (FENCE) or fails loud (FATAL when no peer) immediately, - * never silently proceeding while hollow. */ - recover_decision_t gate = ha_evaluate_promotion_fence( - ha, truncated, durable_floor, promotion_safe_lsn, + * a hollow member never silently proceeds to leadership. We evaluate + * the PURE 3.2/3.3 decision here and DEFER its terminal — task 4.4 does + * NOT reach the 3.3 FATAL synchronously at the promotion instant. A + * FENCE or a would-be-FATAL (truncated/behind + no peer right now) both + * fence the member (in_catch_up) and arm the bounded catch-up window; + * the FATAL is reached ONLY from that timer after a real bounded failure + * (still hollow + no peer + no progress). This avoids the task-5.2 soak + * collapse where a transient no-peer snapshot during aggressive failover + * self-FATAL'd a recovering winner repeatedly until quorum was lost. */ + recover_decision_t gate = ha_promotion_gate_decision( + truncated, durable_floor, promotion_safe_lsn, peer_reachable, /*catch_up_succeeded=*/false); - if (gate != RECOVER_DECISION_ELIGIBLE) { - /* FENCE: refuse promotion — do NOT become CTRL_PRIMARY. The - * in_catch_up fence is set; the member catches up from a healthy - * peer and re-attempts promotion later (Req 2.2, 2.6). - * FATAL already terminated in production (or fired the test hook); - * still bail out here so a test-hooked FATAL does not fall through - * to authoritative leadership. */ + if (gate == RECOVER_DECISION_ELIGIBLE) { + /* ELIGIBLE: release the fence (a caught-up member regains + * eligibility, Req 2.6) and cancel any pending bounded-window FATAL + * timer — a member that is now promotable must never be FATAL'd by a + * stale timer. Proceed to authoritative promotion below. */ + atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release); + ha_cancel_promotion_fence_timer(ha); + } else { + /* FENCE or would-be-FATAL: refuse promotion — do NOT become + * CTRL_PRIMARY. Fence the member and arm the bounded catch-up + * window; the 3.3 FATAL is deferred to the timer (Req 2.2/2.4). + * Never FATAL synchronously, never fence indefinitely. */ WARN("HA: promotion REFUSED (decision=%d) — recovery_truncated=%d " "durable_floor=%" PRIu64 " cluster_hwm=%" PRIu64 - " peer_reachable=%d; remaining non-primary until caught up " + " peer_reachable=%d; fencing + arming bounded catch-up window " "(ha-commit-durability Req 2.2/2.4)", (int)gate, truncated, durable_floor, promotion_safe_lsn, peer_reachable); + ha_arm_promotion_fence_timer(ha, c, truncated, durable_floor, + promotion_safe_lsn); return; } - /* ELIGIBLE: clear of the fence (release on a caught-up member, Req 2.6) - * — proceed to authoritative promotion below. */ } int is_first_boot = !ha->phase2_done; @@ -1230,6 +1461,13 @@ void ha_fini(ha_state_t *ha) { if (!ha) return; + + /* ha-commit-durability task 4.4: cancel any pending bounded-window FATAL + * timer so a stopped member never reaches the deferred FATAL. The delaymgr + * task ctx is freed by promotion_fence_timeout_fn when it fires; cancelling + * makes that firing a no-op (it observes cancelled=1 and frees). */ + ha_cancel_promotion_fence_timer(ha); + pthread_mutex_destroy(&ha->lock); sem_destroy(&ha->heartbeat_seen); @@ -3600,6 +3838,20 @@ void raft_cb_become_leader_for_test(ha_state_t *ha, uint64_t term) raft_cb_become_leader((void *)ha, term); } +#ifdef XCE_TESTING +/* ha_arm_promotion_fence_timer_for_test (ha-commit-durability task 4.4): + * expose the file-static bounded-window arm path so the durability suite can + * drive the exact deferral wired into raft_cb_become_leader without staging a + * full election. Production never references this symbol (XCE_TESTING only). */ +void ha_arm_promotion_fence_timer_for_test(cluster_t *c, bool truncated, + uint64_t durable_floor, + uint64_t target_hwm) +{ + ha_arm_promotion_fence_timer(&c->ha, c, truncated, durable_floor, + target_hwm); +} +#endif + /* * raft_cb_become_follower_for_test: drive the follower path without a live * cluster. Passes ha as ctx; sets ha->raft_c = NULL to suppress production diff --git a/src/controller/ha.h b/src/controller/ha.h index 432ac6f..1dd8d6d 100644 --- a/src/controller/ha.h +++ b/src/controller/ha.h @@ -220,6 +220,23 @@ typedef struct { * once per edge (no log storms across * repeated dispatch-gate rejections). * 0 at init (cleared by ha_init memset). */ + /* ── New field added by ha-commit-durability spec (Task 4.4) ──────── */ + void * _Atomic promotion_fence_ctx; /* promotion_fence_ctx_t* for the + * bounded-window catch-up timer (Req + * 2.2/2.4): a recovery-truncated/behind + * member that wins an election with no + * reachable peer is NOT FATAL'd at the + * promotion instant; it is fenced + * (in_catch_up) and this one-shot timer + * is armed for DEFAULT_RECOVERY_CATCHUP_- + * WINDOW_MS. The 3.3 FATAL is reached + * ONLY from that timer iff still + * hollow + no peer + no catch-up + * progress when it fires; cleared/ + * cancelled on recovery or shutdown so a + * caught-up or stopped member never + * FATALs. NULL at init / when no fence + * timer is pending. */ } ha_state_t; /* ── Public API ────────────────────────────────────────────── */ @@ -541,6 +558,37 @@ recover_decision_t ha_evaluate_promotion_fence(ha_state_t *ha, bool peer_reachable, bool catch_up_succeeded); +/* ── ha-commit-durability task 4.4: bounded-window catch-up before FATAL ───── + * + * The promotion gate (4.3) no longer reaches the 3.3 FATAL synchronously at the + * promotion instant. When a recovery-truncated / behind member wins an election + * with no reachable peer AT THAT INSTANT (a transient no-peer snapshot during + * aggressive failover), it FENCES (sets in_catch_up, does NOT become primary) + * and arms a one-shot delaymgr timer for DEFAULT_RECOVERY_CATCHUP_WINDOW_MS. + * The 3.3 FATAL is reached ONLY from that timer, and ONLY if the member is STILL + * recovery-truncated/behind AND no healthy peer is reachable AND no catch-up + * progress was made when it fires (Req 2.4). Catch-up that brings durable_floor + * >= HWM releases the fence and cancels the timer (Req 2.6); shutdown cancels it + * too — never a synchronous FATAL, never an indefinite fence. + * + * ha_promotion_fence_window_ms: the bounded catch-up window in milliseconds. + * Defaults to DEFAULT_RECOVERY_CATCHUP_WINDOW_MS; tests override it (via + * ha_set_promotion_fence_window_ms) to a small value so the timer fires fast. */ +uint32_t ha_promotion_fence_window_ms(void); + +/* ha_cancel_promotion_fence_timer: cancel a pending bounded-window FATAL timer + * (idempotent; no-op when none is armed). Called when the member catches up + * (fence released) or on shutdown, so neither a caught-up nor a stopped member + * ever reaches the deferred FATAL. */ +void ha_cancel_promotion_fence_timer(ha_state_t *ha); + +#ifdef XCE_TESTING +/* Test seam: override the bounded catch-up window so the deferred-FATAL timer + * fires deterministically fast. 0 restores the DEFAULT_RECOVERY_CATCHUP_WINDOW_MS + * default. XCE_TESTING only — compiled out of production. */ +void ha_set_promotion_fence_window_ms(uint32_t ms); +#endif + /* ── Task 3.2: Unified replication decision enum and evaluator ────────────── */ /* diff --git a/src/controller/testsuite/test_commit_durability.c b/src/controller/testsuite/test_commit_durability.c index dd9dc30..702aff0 100644 --- a/src/controller/testsuite/test_commit_durability.c +++ b/src/controller/testsuite/test_commit_durability.c @@ -12,11 +12,13 @@ #include "commit.h" #include "config.h" +#include "delay.h" /* task 4.4: delaymgr for the bounded-window timer */ #include "ha.h" /* task 4.3: ha_promotion_gate_decision / fence */ #include "persist.h" #include "repl.h" /* task 5.1: repl_add_standby + standby ACK feed */ #include "typemap.h" /* task 5.1: wal_catchup_entry_t (send hook) */ #include "wal_types.h" +#include "xce/cluster_def.h" /* task 4.4: cluster_t (delaymgr + persist + ha) */ /* ------------------------------------------------------------------ * * ha-commit-durability — task 1.2: durability-verification test seam * @@ -2001,6 +2003,237 @@ static void test_promotion_gate_hollow_refused_at_minimal_quorum(void) printf("test_promotion_gate_hollow_refused_at_minimal_quorum: PASS\n"); } +/* ── task 4.4: bounded-window catch-up before FATAL ────────────────────────── + * + * The 4.3 gate decided FATAL synchronously at the promotion instant when a + * recovery-truncated/behind member had no reachable peer right then. Task 4.4 + * defers that FATAL to a one-shot bounded-window timer: the member is fenced + * (in_catch_up) and given DEFAULT_RECOVERY_CATCHUP_WINDOW_MS to find a peer / + * make progress; the 3.3 FATAL is reached ONLY from the timer after a real + * bounded failure (still hollow + no peer + no progress). + * + * These tests drive ha_arm_promotion_fence_timer (the production deferral wired + * into raft_cb_become_leader) through a REAL delaymgr with a short window seam, + * so the timer fires fast and deterministically. A minimal cluster_t carries + * the delaymgr + persist + ha exactly as production does. */ + +/* The delaymgr runner used by production (run_delay_tasks in cluster.c) is not + * linked into the unit-test target; provide the same trivial pass-through. */ +static int dur_run_delay_task(delay_task_entry_fn_t entry, void* arg) +{ + return entry(arg); +} + +/* Build a minimal cluster_t (calloc'd) with a started delaymgr and a recovered + * persist whose recovery verdict + durable floor are set directly, so the + * promotion fence can be evaluated without a full election. truncated → the + * recovery_truncated flag; durable_floor → the durable watermark the gate reads. + * ha.repl stays NULL so "no healthy peer reachable" holds in the timer. */ +static cluster_t* dur_make_fence_cluster(const char* tmpdir, bool truncated, + uint64_t durable_floor) +{ + cluster_t* c = calloc(1, sizeof(*c)); + if (!c) return NULL; + + persist_config_t cfg = persist_config_defaults(); + cfg.wal_sync_ms = 5000; + cfg.snap_keep_n = 2; + persist_set_workdir(&c->persist, tmpdir, "ctrl-test"); + if (persist_init(&c->persist, &cfg, NULL) != 0) { free(c); return NULL; } + if (persist_start(&c->persist) != 0) { + persist_fini(&c->persist); free(c); return NULL; + } + /* Set the recovered state the gate reads (mirrors a real persist_recover). */ + c->persist.recovery_verdict = truncated ? RECOVER_TRUNCATED : RECOVER_CLEAN; + atomic_store_explicit(&c->persist.durable_lsn, durable_floor, + memory_order_release); + + ha_config_t hcfg = dur_ha_cfg(); + if (ha_init(&c->ha, &hcfg, 0) != 0) { + persist_stop(&c->persist); persist_fini(&c->persist); free(c); + return NULL; + } + c->ha.raft_c = c; + c->ha.repl = NULL; /* no standby registry → no healthy peer reachable */ + + if (delaymgr_init(&c->delaymgr, 5 /* precision_ms */, dur_run_delay_task) != 0) { + ha_fini(&c->ha); + persist_stop(&c->persist); persist_fini(&c->persist); free(c); + return NULL; + } + if (delaymgr_start(&c->delaymgr) != 0) { + delaymgr_fini(&c->delaymgr); + ha_fini(&c->ha); + persist_stop(&c->persist); persist_fini(&c->persist); free(c); + return NULL; + } + return c; +} + +static void dur_destroy_fence_cluster(cluster_t* c) +{ + if (!c) return; + /* Cancel any pending fence timer BEFORE tearing down the delaymgr so a + * still-armed callback cannot fire against freed state. */ + ha_cancel_promotion_fence_timer(&c->ha); + delaymgr_stop(&c->delaymgr); + delaymgr_fini(&c->delaymgr); + ha_fini(&c->ha); + persist_stop(&c->persist); + persist_fini(&c->persist); + free(c); +} + +/* Spin until the fatal hook fires or a timeout elapses (the delaymgr fires the + * one-shot timer on its own worker thread). */ +static bool dur_wait_fatal(int expect_calls, int timeout_ms) +{ + int waited = 0; + while (g_fatal_hook_calls < expect_calls && waited < timeout_ms) { + usleep(2000); + waited += 2; + } + return g_fatal_hook_calls >= expect_calls; +} + +/* 4.4-RED carrier + core deferral: a recovery-truncated/behind member with NO + * reachable peer must NOT FATAL synchronously at the promotion instant — it + * fences (in_catch_up set, process alive) and arms a bounded-window timer; the + * 3.3 FATAL fires ONLY from that timer after the window with no peer/progress. + * + * RED (pre-4.4): the gate routed a no-peer truncated member straight through + * ha_evaluate_promotion_fence → persist_recover_fatal_if_unrecoverable, so the + * fatal hook was called SYNCHRONOUSLY (count==1 with zero delay) — there was no + * bounded window, which is exactly the task-5.2 soak collapse. (Req 2.2, 2.4) */ +static void test_promotion_fence_defers_fatal_to_bounded_timer(void) +{ + char tmp[256]; + strcpy(tmp, "/tmp/xce_fence_defer_XXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(tmp)); + + cluster_t* c = dur_make_fence_cluster(tmp, /*truncated=*/true, + /*durable_floor=*/4); + TEST_ASSERT_NOT_NULL(c); + + g_fatal_hook_calls = 0; + persist_fatal_hook = dur_fatal_hook; + ha_set_promotion_fence_window_ms(40); /* short bounded window for the test */ + + /* Arm the fence exactly as raft_cb_become_leader does for a truncated, + * below-HWM, no-peer winner. This must NOT FATAL synchronously. */ + extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t, + uint64_t); + ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true, + /*durable_floor=*/4, /*hwm=*/10); + + TEST_ASSERT_EQUAL_INT_MESSAGE( + 0, g_fatal_hook_calls, + "4.4: the promotion gate must NOT FATAL synchronously — FATAL is " + "deferred to the bounded-window timer (Req 2.2/2.4)"); + TEST_ASSERT_EQUAL_INT_MESSAGE( + 1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire), + "4.4: a fenced member is held in_catch_up (refused promotion), alive"); + + /* Now let the bounded window elapse: still truncated + no peer + no + * progress → the 3.3 FATAL fires from the timer (exactly once). */ + TEST_ASSERT_TRUE_MESSAGE( + dur_wait_fatal(1, 1000), + "4.4: the deferred FATAL must fire from the timer after the bounded " + "window with no peer and no progress (Req 2.4)"); + TEST_ASSERT_EQUAL_INT_MESSAGE( + 1, g_fatal_hook_calls, + "4.4: the deferred FATAL fires exactly once from the timer"); + + persist_fatal_hook = NULL; + ha_set_promotion_fence_window_ms(0); /* restore default */ + dur_destroy_fence_cluster(c); + printf("test_promotion_fence_defers_fatal_to_bounded_timer: PASS\n"); +} + +/* 4.4 Req 2.6: catch-up within the window → fence released, FATAL NOT called. + * Arm the fence, then simulate catch-up (clear truncated + advance durable_floor + * past the HWM) before the timer fires; when the timer fires it observes a + * caught-up member, releases in_catch_up, and does NOT FATAL. */ +static void test_promotion_fence_released_on_catchup_no_fatal(void) +{ + char tmp[256]; + strcpy(tmp, "/tmp/xce_fence_catchup_XXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(tmp)); + + cluster_t* c = dur_make_fence_cluster(tmp, /*truncated=*/true, + /*durable_floor=*/4); + TEST_ASSERT_NOT_NULL(c); + + g_fatal_hook_calls = 0; + persist_fatal_hook = dur_fatal_hook; + ha_set_promotion_fence_window_ms(120); /* generous so we can mutate first */ + + extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t, + uint64_t); + ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true, + /*durable_floor=*/4, /*hwm=*/10); + TEST_ASSERT_EQUAL_INT(0, g_fatal_hook_calls); + TEST_ASSERT_EQUAL_INT( + 1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire)); + + /* Simulate catch-up completing: log now intact and durable past the HWM. */ + c->persist.recovery_verdict = RECOVER_CLEAN; + atomic_store_explicit(&c->persist.durable_lsn, 10, memory_order_release); + + /* Wait long enough for the window to fire; assert it did NOT FATAL and the + * fence was released (Req 2.6). */ + usleep(300 * 1000); + TEST_ASSERT_EQUAL_INT_MESSAGE( + 0, g_fatal_hook_calls, + "4.4: a member that caught up within the window must NOT FATAL (Req 2.6)"); + TEST_ASSERT_EQUAL_INT_MESSAGE( + 0, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire), + "4.4: catch-up within the window RELEASES the in_catch_up fence (Req 2.6)"); + + persist_fatal_hook = NULL; + ha_set_promotion_fence_window_ms(0); + dur_destroy_fence_cluster(c); + printf("test_promotion_fence_released_on_catchup_no_fatal: PASS\n"); +} + +/* 4.4: cancelling the timer (shutdown / recovery) before it fires → NO FATAL. + * Proves the deferred FATAL never fires after a clean cancel — a stopped member + * must not self-terminate, and there is no use-after-free of the timer ctx. */ +static void test_promotion_fence_cancel_prevents_fatal(void) +{ + char tmp[256]; + strcpy(tmp, "/tmp/xce_fence_cancel_XXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(tmp)); + + cluster_t* c = dur_make_fence_cluster(tmp, /*truncated=*/true, + /*durable_floor=*/4); + TEST_ASSERT_NOT_NULL(c); + + g_fatal_hook_calls = 0; + persist_fatal_hook = dur_fatal_hook; + ha_set_promotion_fence_window_ms(60); + + extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t, + uint64_t); + ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true, + /*durable_floor=*/4, /*hwm=*/10); + TEST_ASSERT_EQUAL_INT(0, g_fatal_hook_calls); + + /* Cancel before the window elapses. */ + ha_cancel_promotion_fence_timer(&c->ha); + + /* Wait past the window — the cancelled timer must be a no-op, NO FATAL. */ + usleep(200 * 1000); + TEST_ASSERT_EQUAL_INT_MESSAGE( + 0, g_fatal_hook_calls, + "4.4: a cancelled fence timer must NEVER FATAL (shutdown/recovery)"); + + persist_fatal_hook = NULL; + ha_set_promotion_fence_window_ms(0); + dur_destroy_fence_cluster(c); + printf("test_promotion_fence_cancel_prevents_fatal: PASS\n"); +} + #endif /* XCE_TESTING */ /* ------------------------------------------------------------------ */ @@ -2052,6 +2285,12 @@ void test_commit_durability(void) RUN_TEST(test_boot_policy_row6_empty_first_boot_eligible); /* task 4.3 — hollow node refused even at minimal quorum (Req 2.5). */ RUN_TEST(test_promotion_gate_hollow_refused_at_minimal_quorum); + /* task 4.4 — bounded-window catch-up before FATAL (no synchronous FATAL; + * deferred-timer FATAL only after a real bounded failure; fence released on + * catch-up; cancelled timer never FATALs). (Req 2.2, 2.4, 2.6) */ + RUN_TEST(test_promotion_fence_defers_fatal_to_bounded_timer); + RUN_TEST(test_promotion_fence_released_on_catchup_no_fatal); + RUN_TEST(test_promotion_fence_cancel_prevents_fatal); #else printf("test_commit_durability: skipped (no XCE_TESTING)\n"); #endif