forked from lei.chen/myxce
test(ha-term-safety): controller Election-Safety regression for bug-1095/1096 + absorb into spec
Adds two controller-level regression tests driving the production raft_on_vote_reply
via ha->raft (test_ha_election.c): a duplicated vote-grant from one peer must not
double-count toward quorum (bug-1095) and a stale-term reply must be dropped
(bug-1096) — both asserted on role/promotion (no two leaders per term). Formally
absorbs bug-1095/1096 into the ha-term-safety spec (Implementation Note) mapping to
its Requirement 1 / Objective; the raft.c fix (b5b7e01) is owned there with
protocol-suite regression at both the libxce (test_vote_reply_*) and controller
(test_ha_election) levels. No production source changed.
This commit is contained in:
parent
95629797f8
commit
13c43c23a4
2 changed files with 171 additions and 0 deletions
|
|
@ -222,3 +222,8 @@
|
|||
|
||||
- **3.4 done — bug-990 CLOSED (the live-send cutover).** ROOT CAUSE: `repl_broadcast` (the LIVE replicator, persist.c:747) built `wal_catchup_entry_t` without setting `entry.term`, so the live broadcast wire-encoded term=0 even though primary `hdr.term=p->term`; standbys took the legacy term-0 direct-apply and persisted term=0. FIX (MINIMAL cutover, reviewer-justified over the fuller raft_send_append_entries rewrite — the requirement set is met by carrying the term so entries auto-route through the already-wired 3.2 intake + 3.3 ACK): `repl_broadcast` now takes `prev_log_term` + sets `entry.term=hdr->term`/`prev_log_index=lsn-1`/`prev_log_term`; `persist_wal_async` sources prev_log_term via NEW lock-free `entry_term_locked(p, lsn-1)` (predecessor term, under p->lock; lsn==1→(0,0)); catch-up streamer `ha_stream_wal_to_standby` (config_repl.c) carries term+prev tuple per entry (ascending (lsn,term) tracker + conservative segment-skip so the FIRST streamed entry's prev references what the standby already holds); `ha_apply_wal_catchup_entry` (ha.c) routes EVERY term≠0 entry (single + each batch sub-entry, in order, guarded raft_c==c) through `raft_on_append_entries_request`→on_persist_entry (leader term stamped) — retiring the term-0 direct-apply for live/catch-up; `wal_entry_to_wire` (request.c) encodes prevLog{Index,Term}+leaderCommit PER SUB-ENTRY. 0-entry heartbeat still routes through the library for UpdateTerm (persists nothing). bug-850 contiguity + persist-before-ACK + commit-barrier + bug-960 peer-resolution all preserved. Live commit chain now genuinely end-to-end (term-carrying send → standby on_persist_entry + ACK → ha_raft_on_repl_ack → raft_advance_commit → on_commit_advanced → barrier). 2 new tests (send: `Expected 7 Was 0`; receive via the REAL `ha_apply_wal_catchup_entry` entry point: `Expected 7 Was 3`), RED independently reproduced by the reviewer. 175 Tests 0 Failures. Reviewer APPROVED.
|
||||
- **REMAINING (ENV-only, NOT code):** the definitive LIVE proof — clean failover → byte-identical `(term,lsn)` WALs via `clean_failover_preservation.sh XCE_FAILOVER_STRICT_TERM=1` — could NOT run here: this overlayfs sandbox can't complete a failover on baseline OR the change (split-vote / `fdatasync errno=5`, bug-924). The fix is proven at the integration level (real receive path, real wire shape). 5.2/5.4 + the live half of 5.5 need a REAL (non-overlayfs) fs to run the harness; the harness + STRICT_TERM gate are ready and should flip GREEN there.
|
||||
|
||||
- **bug-1095 + bug-1096 ABSORBED (Requirement 1 / Objective "two leaders never share a term"; bug-851 family) — owned here, fixed cross-boundary in libxce `raft.c` with user authorization.** _Surfaced by the raft-sim-conformance oracle (task 4.2), absorbed into ha-term-safety because the defect is an Election-Safety violation in the controller's SOLE production promotion driver: `vote_reply_dispatch` (ha.c ~4336) → `raft_on_vote_reply(&ha->raft, idx, vote_granted, term)` (ha.c ~4361)._
|
||||
- **The defect:** `raft_on_vote_reply` tallied vote grants with a plain counter that ignored `from_idx` and had no stale-term drop. (1) **bug-1095** — a duplicated vote-response (legal `DuplicateMessage`) from ONE peer double-counted toward quorum, so two candidates could each cross majority and become **leader at the SAME term**. (2) **bug-1096** — a grant carrying `mterm < currentTerm` (a stale, prior-election reply) was counted toward the CURRENT election, again admitting a spurious quorum. Both break Election Safety (`raft.tla` `votesGranted` is a SET; `DropStaleResponse` discards `mterm < currentTerm`).
|
||||
- **The fix (commit `b5b7e01`, libxce `raft.c` only — NOT touched by this task):** added a per-peer `vote_replies[RAFT_MAX_PEERS]` idempotency set (a duplicate `from_idx` is dropped before the tally) + a `term < s->term` `DropStaleResponse` that discards the stale reply WITHOUT stepping the candidate down — mirroring the already-present pre-vote dedup idiom (`pre_vote_replies[]`).
|
||||
- **Regression coverage (BOTH levels):** protocol unit — libxce `test_vote_reply_duplicate_grant_not_double_counted` + `test_vote_reply_stale_term_dropped` (`src/lib/testsuite/test_raft.c`); controller level — NEW `test_vote_reply_duplicate_grant_does_not_promote` + `test_vote_reply_stale_term_does_not_count_or_promote` (`src/controller/testsuite/test_ha_election.c`). The controller tests drive the REAL `raft_on_vote_reply` against the controller's own `ha->raft` instance (re-inited to a 5-node cluster; `xce_vote_reply_hook` NULL, `raft_c` NULL so the callbacks take their test-mode paths and `raft_cb_become_leader` still sets `ha->role = CTRL_PRIMARY`) and assert on the resulting **role / promotion** (Election Safety), not an internal counter — a duplicate grant must NOT promote at 2-of-5, and a stale grant must NOT count or promote (and must NOT step the candidate down). Both PASS over the fixed library; they would FAIL without the dedup / stale-drop. `test_ha_election` group: 0 failures.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue