forked from lei.chen/myxce
fix(raft-ha-core-safety): suppress spurious PreVote failover on transient RDHUP (task 5.2)
A recovered standby whose control channel flaps (transient RDHUP) while a current-term primary is alive treated each flap as "primary died" and launched an unwinnable PreVote failover (peers see the live primary and deny, the round times out, the channel re-establishes, repeat ~15s) — so the node prevote-spun forever instead of rejoining, wedging the chaotic soak (research.md §"Phase 4/5"). Add a freshness guard at the RDHUP failover-initiation path (ha_on_primary_ctrl_fd_rdhup): when CTRL_STANDBY and ha_heartbeat_is_fresh, stay CTRL_STANDBY, re-arm the liveness watchdog (production), and await the primary's reconnect instead of campaigning — the same guard ha_failover_stagger_fn already applies on the deferred PreVote path. Genuine loss (stale heartbeat) falls through to failover unchanged; no term bump / no role change, so Election Safety is untouched. Keeping the standby attached lets catch-up complete, which clears in_catch_up and lets votes flow again — so this one guard also resolves the downstream catching-up-node vote-deny deadlock (obs 1240). Flips task 5.1's CORE RED to GREEN: test_ha_core_safety 17 Tests / 0 Failures (spurious launches=0/5; genuine-loss final_term=8; Election Safety A=3/B=0 → 1 primary). Zero failing suites binary-wide; LSan unchanged vs baseline. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
This commit is contained in:
parent
806257afcf
commit
01419e1310
2 changed files with 35 additions and 1 deletions
|
|
@ -103,7 +103,7 @@
|
|||
- _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3, 7.3_
|
||||
- _Boundary: Election Liveness_
|
||||
|
||||
- [ ] 5.2 Suppress unwinnable failover on transient RDHUP; keep genuine-loss liveness and single-leader safety
|
||||
- [x] 5.2 Suppress unwinnable failover on transient RDHUP; keep genuine-loss liveness and single-leader safety
|
||||
- On a control-channel RDHUP, attempt reconnect + catch-up first, and/or back off failover when a node's own PreVotes repeatedly fail to reach quorum because a current-term leader exists — only escalate to a real election when the primary is genuinely unreachable to the quorum. Preserve: genuine primary loss still elects a higher-term primary (no permanent stall while quorum reachable); step-down-on-higher-term paired with election restart; at-most-one-committing-primary-per-term
|
||||
- Observable completion: 5.1 spurious-failover case passes (a recovered standby rejoins a live primary instead of prevote-spinning); genuine-loss liveness still completes each round; the Election Safety assertion still holds
|
||||
- _Requirements: 4.1, 4.2, 4.3, 6.1, 6.2, 6.3_
|
||||
|
|
@ -149,4 +149,6 @@
|
|||
|
||||
- **Task 5.1 — RED lands (3 cases in `test_ha_core_safety.c`).** CORE RED `test_5_1_spurious_failover_suppressed_while_primary_alive` FAILs on baseline (`Expected 0 Was 5`): it drives the REAL `ha_on_primary_ctrl_fd_rdhup` (unit-test `c==NULL` path) 5× with a FRESH heartbeat (`ha_heartbeat_is_fresh()==1`) and counts PreVote launches — baseline launches one unwinnable round per flap. **Root-cause asymmetry the fix targets:** the DEFERRED PreVote path (`ha_failover_stagger_fn`, ha.c ~6664) ALREADY guards with `ha_heartbeat_is_fresh` and skips, but the RDHUP-INITIATION path (`ha_on_primary_ctrl_fd_rdhup`→`ha_begin_failover`→`ha_start_prevote_async` synchronous/`c==NULL` branch) has NO such guard. **Task 5.2 GREEN = add the `ha_heartbeat_is_fresh` guard at the RDHUP initiation path** (reconnect+rejoin instead of campaigning) WITHOUT regressing the two PASSing pins: `test_5_1_genuine_loss_elects_and_advances_term` (genuine loss — stale heartbeat — must STILL launch; failed round re-arms; rapid rounds advance term) guards against over-suppression, and `test_5_1_election_safety_one_primary_per_term` (vote-once via the real `handle_request_vote`, A=3/B=0 → ≤1 primary/term) is the falsifiable Election Safety check mapped to raft.tla. Note: `handle_request_vote` already denies same-term challengers when `primary_ctrl_fd>=0 && ha_heartbeat_is_fresh` — the VOTE path is safe; only failover-INITIATION is not.
|
||||
|
||||
- **Task 5.2 — GREEN (the one-guard fix, `ha.c:ha_on_primary_ctrl_fd_rdhup`).** Added the spurious-failover-suppression guard between the fd-close and the failover INFO: `if (rdhup_role == CTRL_STANDBY && ha_heartbeat_is_fresh(ha)) { log; if (c) ha_arm_heartbeat_timeout_vtask(ha,c); return; }`. A transient RDHUP while a current-term primary is alive (fresh heartbeat) now stays CTRL_STANDBY and awaits reconnect instead of launching an unwinnable PreVote — the same guard `ha_failover_stagger_fn` already applies on the deferred path. Genuine loss (stale heartbeat) falls through unchanged; no term bump / no role change, so Election Safety is untouched. The watchdog re-arm (production-only, `c!=NULL`) resets `last_activity` to now for a fresh grace window, so a primary that truly never heartbeats again still fires a real failover after `heartbeat_timeout_ms`. **Why one guard suffices over the `in_catch_up`-denies-PreVote facet (obs 1240):** that deny is correct Raft (Req 2.5) and only deadlocks if catch-up never finishes — suppressing the flap keeps the standby attached so catch-up completes, `in_catch_up` clears, and votes flow. Validation: `test_ha_core_safety` 17 Tests / 0 Failures (spurious `launches=0/5` GREEN; genuine-loss `final_term=8`; Election Safety A=3/B=0→1 primary); zero failing suites binary-wide; LSan unchanged (174379 B / 1553 allocs = pre-fix baseline — suppression path allocates a liveness ctx only under real `c`, untouched by the c==NULL unit tests). Live ≥6-round chaotic soak proof is task 7.2.
|
||||
|
||||
- **Task 2.4 — root-cause source fix + a deferral.** `ha_stream_wal_to_standby` now reads the entry body via `stream_read_entry_body` (bounded retry, max 4 no-progress attempts, partial-read-accumulating, no sleep); on persistent short `pread`/`malloc`-fail it emits a loud ERROR and ABORTS the stream (flushes the valid contiguous prefix, drops the unreadable entry, returns `-EIO`) instead of shipping `PLEN=0`. The incomplete stream is re-driven by the ACK-driven catch-up continuation. Combined with the 2.2 self-healing guard, this closes the live cross-term payload-loss divergence. **Deferred:** the structural retirement of the term-blind `ha_apply_single_entry` bypass — its empty-over-full safety is enforced by the 2.2 guard + pinned by two passing tests, so it cannot corrupt committed state; the structural removal is tracked to task 6.2. Live ≥6-round soak proof is task 7.2.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue