From 01419e131029d8e3f3cd011096a700925723c300 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 30 Jun 2026 02:03:50 +0000 Subject: [PATCH] fix(raft-ha-core-safety): suppress spurious PreVote failover on transient RDHUP (task 5.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A recovered standby whose control channel flaps (transient RDHUP) while a current-term primary is alive treated each flap as "primary died" and launched an unwinnable PreVote failover (peers see the live primary and deny, the round times out, the channel re-establishes, repeat ~15s) — so the node prevote-spun forever instead of rejoining, wedging the chaotic soak (research.md §"Phase 4/5"). Add a freshness guard at the RDHUP failover-initiation path (ha_on_primary_ctrl_fd_rdhup): when CTRL_STANDBY and ha_heartbeat_is_fresh, stay CTRL_STANDBY, re-arm the liveness watchdog (production), and await the primary's reconnect instead of campaigning — the same guard ha_failover_stagger_fn already applies on the deferred PreVote path. Genuine loss (stale heartbeat) falls through to failover unchanged; no term bump / no role change, so Election Safety is untouched. Keeping the standby attached lets catch-up complete, which clears in_catch_up and lets votes flow again — so this one guard also resolves the downstream catching-up-node vote-deny deadlock (obs 1240). Flips task 5.1's CORE RED to GREEN: test_ha_core_safety 17 Tests / 0 Failures (spurious launches=0/5; genuine-loss final_term=8; Election Safety A=3/B=0 → 1 primary). Zero failing suites binary-wide; LSan unchanged vs baseline. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC --- .kiro/specs/raft-ha-core-safety/tasks.md | 4 ++- src/controller/ha.c | 32 ++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/.kiro/specs/raft-ha-core-safety/tasks.md b/.kiro/specs/raft-ha-core-safety/tasks.md index 9ab67bb..2ca192c 100644 --- a/.kiro/specs/raft-ha-core-safety/tasks.md +++ b/.kiro/specs/raft-ha-core-safety/tasks.md @@ -103,7 +103,7 @@ - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3, 7.3_ - _Boundary: Election Liveness_ -- [ ] 5.2 Suppress unwinnable failover on transient RDHUP; keep genuine-loss liveness and single-leader safety +- [x] 5.2 Suppress unwinnable failover on transient RDHUP; keep genuine-loss liveness and single-leader safety - On a control-channel RDHUP, attempt reconnect + catch-up first, and/or back off failover when a node's own PreVotes repeatedly fail to reach quorum because a current-term leader exists — only escalate to a real election when the primary is genuinely unreachable to the quorum. Preserve: genuine primary loss still elects a higher-term primary (no permanent stall while quorum reachable); step-down-on-higher-term paired with election restart; at-most-one-committing-primary-per-term - Observable completion: 5.1 spurious-failover case passes (a recovered standby rejoins a live primary instead of prevote-spinning); genuine-loss liveness still completes each round; the Election Safety assertion still holds - _Requirements: 4.1, 4.2, 4.3, 6.1, 6.2, 6.3_ @@ -149,4 +149,6 @@ - **Task 5.1 — RED lands (3 cases in `test_ha_core_safety.c`).** CORE RED `test_5_1_spurious_failover_suppressed_while_primary_alive` FAILs on baseline (`Expected 0 Was 5`): it drives the REAL `ha_on_primary_ctrl_fd_rdhup` (unit-test `c==NULL` path) 5× with a FRESH heartbeat (`ha_heartbeat_is_fresh()==1`) and counts PreVote launches — baseline launches one unwinnable round per flap. **Root-cause asymmetry the fix targets:** the DEFERRED PreVote path (`ha_failover_stagger_fn`, ha.c ~6664) ALREADY guards with `ha_heartbeat_is_fresh` and skips, but the RDHUP-INITIATION path (`ha_on_primary_ctrl_fd_rdhup`→`ha_begin_failover`→`ha_start_prevote_async` synchronous/`c==NULL` branch) has NO such guard. **Task 5.2 GREEN = add the `ha_heartbeat_is_fresh` guard at the RDHUP initiation path** (reconnect+rejoin instead of campaigning) WITHOUT regressing the two PASSing pins: `test_5_1_genuine_loss_elects_and_advances_term` (genuine loss — stale heartbeat — must STILL launch; failed round re-arms; rapid rounds advance term) guards against over-suppression, and `test_5_1_election_safety_one_primary_per_term` (vote-once via the real `handle_request_vote`, A=3/B=0 → ≤1 primary/term) is the falsifiable Election Safety check mapped to raft.tla. Note: `handle_request_vote` already denies same-term challengers when `primary_ctrl_fd>=0 && ha_heartbeat_is_fresh` — the VOTE path is safe; only failover-INITIATION is not. +- **Task 5.2 — GREEN (the one-guard fix, `ha.c:ha_on_primary_ctrl_fd_rdhup`).** Added the spurious-failover-suppression guard between the fd-close and the failover INFO: `if (rdhup_role == CTRL_STANDBY && ha_heartbeat_is_fresh(ha)) { log; if (c) ha_arm_heartbeat_timeout_vtask(ha,c); return; }`. A transient RDHUP while a current-term primary is alive (fresh heartbeat) now stays CTRL_STANDBY and awaits reconnect instead of launching an unwinnable PreVote — the same guard `ha_failover_stagger_fn` already applies on the deferred path. Genuine loss (stale heartbeat) falls through unchanged; no term bump / no role change, so Election Safety is untouched. The watchdog re-arm (production-only, `c!=NULL`) resets `last_activity` to now for a fresh grace window, so a primary that truly never heartbeats again still fires a real failover after `heartbeat_timeout_ms`. **Why one guard suffices over the `in_catch_up`-denies-PreVote facet (obs 1240):** that deny is correct Raft (Req 2.5) and only deadlocks if catch-up never finishes — suppressing the flap keeps the standby attached so catch-up completes, `in_catch_up` clears, and votes flow. Validation: `test_ha_core_safety` 17 Tests / 0 Failures (spurious `launches=0/5` GREEN; genuine-loss `final_term=8`; Election Safety A=3/B=0→1 primary); zero failing suites binary-wide; LSan unchanged (174379 B / 1553 allocs = pre-fix baseline — suppression path allocates a liveness ctx only under real `c`, untouched by the c==NULL unit tests). Live ≥6-round chaotic soak proof is task 7.2. + - **Task 2.4 — root-cause source fix + a deferral.** `ha_stream_wal_to_standby` now reads the entry body via `stream_read_entry_body` (bounded retry, max 4 no-progress attempts, partial-read-accumulating, no sleep); on persistent short `pread`/`malloc`-fail it emits a loud ERROR and ABORTS the stream (flushes the valid contiguous prefix, drops the unreadable entry, returns `-EIO`) instead of shipping `PLEN=0`. The incomplete stream is re-driven by the ACK-driven catch-up continuation. Combined with the 2.2 self-healing guard, this closes the live cross-term payload-loss divergence. **Deferred:** the structural retirement of the term-blind `ha_apply_single_entry` bypass — its empty-over-full safety is enforced by the 2.2 guard + pinned by two passing tests, so it cannot corrupt committed state; the structural removal is tracked to task 6.2. Live ≥6-round soak proof is task 7.2. diff --git a/src/controller/ha.c b/src/controller/ha.c index b7434c3..8acc889 100644 --- a/src/controller/ha.c +++ b/src/controller/ha.c @@ -6722,6 +6722,38 @@ void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c) ha->primary_ctrl_fd = -1; } + /* raft-ha-core-safety task 5.2: SPURIOUS-FAILOVER SUPPRESSION. + * + * A TRANSIENT control-channel RDHUP while the current-term primary is still + * alive (a heartbeat arrived within heartbeat_timeout_ms) must NOT launch an + * unwinnable PreVote failover. The live stall (research.md §"Phase 4/5"): a + * recovered standby whose channel to the live primary flaps treats each + * RDHUP as "primary died" → PreVote → peers see the primary alive and deny → + * round times out → reconnect → repeat ~15s, so the node prevote-spins + * forever instead of rejoining. Mirror the guard ha_failover_stagger_fn + * already applies on the DEFERRED path (a flap = the primary re-established + * the channel, so ha_heartbeat_is_fresh is the right liveness signal — + * the same one handle_request_vote uses to deny same-term challengers). + * + * Stay CTRL_STANDBY and await the primary's reconnect (handle_ctrlhello); + * re-arm the liveness watchdog so a GENUINE subsequent loss is still caught + * (the watchdog resets last_activity to now, granting a fresh grace window — + * if the primary truly never heartbeats again it fires a real failover after + * heartbeat_timeout_ms). GENUINE loss (stale heartbeat) falls through to the + * failover path below unchanged; Election Safety is unaffected (no term bump, + * no role change here). */ + ctrl_role_t rdhup_role = (ctrl_role_t)atomic_load_explicit(&ha->role, + memory_order_acquire); + if (rdhup_role == CTRL_STANDBY && ha_heartbeat_is_fresh(ha)) { + INFO("HA: primary_ctrl_fd RDHUP but the primary is alive (fresh " + "heartbeat) — suppressing PreVote failover, staying CTRL_STANDBY " + "and awaiting reconnect (spurious-failover suppression, " + "raft-ha-core-safety 5.2)"); + if (c) + ha_arm_heartbeat_timeout_vtask(ha, c); + return; + } + /* ha-prevote task 4.2: the failover trigger NO LONGER transitions to * CTRL_ELECTING or truncates the WAL inline. It starts a PreVote round and * STAYS CTRL_STANDBY; the destructive truncate + CTRL_ELECTING sequence is