feat(ha-commit-durability): durable-backed vote/PreVote freshness floor + bare-durable candidate advertise (task 4.2)

This commit is contained in:
Rongsong Shen 2026-06-18 09:02:11 +00:00
parent 4b7457144c
commit 3049b620f3
7 changed files with 282 additions and 26 deletions

View file

@ -85,7 +85,7 @@
- _Requirements: 2.3_
- _Boundary: FreshnessFloor_
- [ ] 4.2 Derive the vote and PreVote freshness floor from durable WAL state
- [x] 4.2 Derive the vote and PreVote freshness floor from durable WAL state
- Implement the pinned formula so a candidate cannot advertise, and a voter cannot grant against, a position beyond durable WAL state; the post-election safe-commit point continues to come from voter replies, never a stale ctrlhello.
- Observable: a node restarted with a truncated log is denied (vote and PreVote), while the node holding the longest durable log still wins — no livelock.
- Note: 4.2 and 4.3 both edit the same election code (`cluster.c` vote/PreVote and `ha.c` promotion); they are sequential (4.2 before 4.3), not parallel.
@ -132,3 +132,4 @@
- 3.3: FATAL branch = pure decision `persist_recover_decision(recover_policy_inputs_t) → recover_decision_t {ELIGIBLE, FENCE, FATAL}` (persist.c) + action `persist_recover_fatal_if_unrecoverable()` which logs the truncation boundary (`durable_floor_lsn`/`cluster_hwm_lsn` + window ms) at ERROR then terminates via `persist_fatal_hook` (extern `void(*)(const char*)`, NULL in prod → `tlog_exit(); abort()`; mirrors `ha_fatal_hook`; tests install it to capture instead of abort). FATAL ONLY for bounded-unrecoverable + NO healthy peer (rows 4/4b); a reachable peer NEVER FATALs (gated behind `if (!healthy_peer_reachable)`); eligibility rows 1/2/6 never FATAL. Bound = FINITE `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS=30000` (persist.h) — keys off the `catch_up_succeeded` bounded-window result, never an indefinite wait. Turned 3.2 skeleton rows 4/4b GREEN + added `test_boot_policy_peer_reachable_never_fatal`; rows 1/2/3/5/6 stay IGNORE (FENCE→ELIGIBLE is 4.3). DEFERRED to 4.3/5.x: wiring this decision into the live boot/promotion path (3.3 is the unit-tested decision + terminate only; no ha.c/cluster.c edits).
- 3.4: On recovery, `persist_recover` publishes the durable floor = highest intact replayed LSN (recovered WAL tip = `current_lsn-1` after `replay_wal`) into `durable_lsn` via `atomic_store` (ASSIGN, NOT max — so a truncating restart lands at the LOWER intact tip, never over-advertising; this is the crux of Req 2.3). Added `persist_durable_floor()` (NULL-safe thin accessor) as the seam task 4.2 consumes. Floor is the durable LAST-LOG INDEX (WAL tip), NOT the commit position (`committed_lsn`) and NOT a volatile in-memory value. Invariant-safe (recovery/init-time; recovered entries are CRC-validated on-disk = genuinely durable; preserves `durable_lsn <= current_lsn-1`). 4.2 (ha.c/cluster.c) CONSUMES this floor in `ha_vote_self_freshness` — 3.4 only PUBLISHES (persistence-only). Phase 3 (recovery integrity) COMPLETE.
- 4.1: ANALYSIS-ONLY (no .c/.h). Pinned freshness-floor formula in design.md §"Freshness-Floor Formula (task 4.1)" — THE CONTRACT 4.2 implements. CANDIDATE advertises BARE `cand_lsn = persist_durable_floor(p)` (durable WAL tip) — sites ha.c:3660-3666 (real-vote) + ha.c:3735-3741 (PreVote) must call `persist_durable_floor` DIRECTLY, BYPASSING `ha_vote_self_freshness` on the LSN axis (that fn unconditionally folds in `last_known_committed_lsn` → `max(durable_floor, learned)` = the truncated-candidate-wins bug). VOTER floor = `max(durable_floor, last_known_committed_lsn)` — sites cluster.c:388-395/471-475 + ha.c:2243 keep `ha_vote_self_freshness` with `local_lsn=durable_floor` (the learned_committed backstop stays on the VOTER axis only). Grant iff `cand_tag > floor_tag OR (==tag AND cand_lsn >= floor_lsn)` (term-then-index, matches raft.tla `LastTerm`/`mlastLogIndex |-> Len(log)`). `cand_tag`/`floor_tag` = `max(local_tag, learned_tag)`. SAFETY (Leader Completeness): quorum intersection Q_L∩Q_v ⇒ truncated candidate (durable tip < committed L) denied. LIVENESS (no bug-339 livelock): rests on D* (max durable tip in reachable majority); `durable_floor(v) ≤ D*` and `learned_committed(v) ≤ L ≤ D*`, so the most-durable candidate is always grantable. `safe_committed_lsn` STAYS voter-reply-sourced (unchanged). CAVEAT: raft.tla validates election/freshness LOGIC only, NOT durability (no TLC run — no toolchain; conceptual alignment) — durability verified by 2.x/3.4 tests. (1 remediation: candidate-vs-voter contradiction + §5b example fixed.)
- 4.2: Implements the 4.1 formula. New `ha_candidate_advertise_lsn(persist*)` = bare `persist_durable_floor`. CANDIDATE sites advertise bare durable_floor for the LSN (tag still via `ha_vote_self_freshness(...,&tag,NULL)`): ha.c:~3690 (real election), ha.c:~3771 (PreVote), AND myapp.c:~234 (boot election — a THIRD candidate site the brief didn't name; fixed too, else a boot-time truncated candidate over-advertises). VOTER floor: cluster.c:~383 (real) + cluster.c:~471 (PreVote) feed `local_lsn = persist_durable_floor(&c->persist)` into `ha_vote_self_freshness` → floor = `max(durable_floor, learned_committed)`. Comparison unchanged (term-then-index). `ha.c:2243` legacy `handle_request_vote` is XCE_TESTING-only/param-driven — left unchanged. `safe_committed_lsn` UNTOUCHED (voter-reply-sourced). 3 tests in test_ha_vote_gate.c prove: candidate advertises bare durable_floor (not stale learned); voter durable-floor denies truncated; longest-durable grantable / no livelock. bug-339 regression test fixture primed `durable_lsn=10` (model-preserving). INFO: truncated-deny proven at the freshness-comparison-expression level shared by both vote+PreVote paths (no dedicated live PreVote-handler truncation driver — adequate; nice-to-have for 5.x). GOTCHA for future election tests: the freshness floor now reads `persist_durable_floor`, so any vote/election fixture that sets `applied_lsn`/`last_known_committed_lsn` but not durable state will see floor/advertise=0 — prime `durable_lsn` to the modeled tip.