feat(ha-term-safety): retire term-0 repl_broadcast live send; carry leader term on replication (task 3.4)

Closes bug-990: the live replicator repl_broadcast wire-encoded term=0
(entry.term was never set), so standbys persisted hdr.term=0 and took the
legacy direct-apply path instead of the library (term,lsn) intake. Now the
live broadcast and catch-up streamer carry the leader's real term + AE prev
tuple; ha_apply_wal_catchup_entry routes every term-carrying entry through
raft_on_append_entries_request -> on_persist_entry (leader term stamped),
retiring the term-0 direct-apply. Standbys now persist the leader's real term
and reconcile by (term,lsn); the live commit chain is end-to-end.

848/849/850 + persist-before-ACK + commit-barrier + bug-960 preserved.
Proven at integration level; the definitive live STRICT_TERM failover run is
env-blocked here (bug-924 overlayfs) and recommended on a real fs.
This commit is contained in:
Rongsong Shen 2026-06-22 15:45:14 +00:00
parent 87242aa8e9
commit efe621c124
11 changed files with 307 additions and 45 deletions

View file

@ -84,7 +84,7 @@
- _Requirements: 4.1, 4.5_
- _Depends: 2.5, 3.2_
- [ ] 3.4 Retire the term-0 `repl_broadcast` live send and drive the library send loop on the live primary (the deferred live-send cutover)
- [x] 3.4 Retire the term-0 `repl_broadcast` live send and drive the library send loop on the live primary (the deferred live-send cutover)
- Make the LIVE primary replicate via the library send (`raft_send_append_entries`/the library's per-peer send) so `on_send_append_entries` (task 2.3) fires on the real replication path, carrying the leader's term; retire/replace the term-0 `repl_broadcast` (`persist.c:747`) as the live replicator so standbys no longer receive term-0 entries. Standbys must intake every replicated entry through `raft_on_append_entries_request` (task 3.2 route) and persist `hdr.term = leader_term` (not 0); retire the legacy direct-apply branch (`ha_apply_wal_catchup_entry` gated `entry->term != 0`) as the live intake. Preserve persist-before-ACK + the commit barrier + 848/849/850 + the bug-850 contiguity guard; keep `next_index[]`/`match_index[]` driven by the ACK path (3.3) so commit advances live.
- Observable completion: on a live cluster a primary `jobsub`-driven append replicates a term-CARRYING AppendEntries; standbys persist the leader's real term (recovered `hdr.term == leader_term`, not 0); a clean failover converges to byte-identical `(term,lsn)` WALs (closes the bug-990 / R5.1 term-identity gap); the library `commit_index` advances from live quorum ACKs.
- _Requirements: 1.1, 1.2, 1.3, 1.4, 2.5, 4.1_
@ -124,7 +124,7 @@
- Observable completion: the clean-failover test converges to identical `(term,lsn)` WALs and the 848/849/850 behaviors remain green
- _Requirements: 5.1, 5.2, 5.3, 5.4_
- _Depends: 3.3_
- _Blocked: bug-990 — the term-identity half of R5.1 cannot be GREEN because the LIVE replication SEND is still term-0 `repl_broadcast` (persist.c:747); the library send loop (`on_send_append_entries`→`repl_send_append_entries`) has ZERO live callers (no live `raft_send_append_entries` driver), so standbys persist hdr.term=0 and walcompare only passes by masking via legacy-0→WAL_LEGACY_TERM on a fresh single-term cluster. This is the live-send retirement DEFERRED across 3.2→3.3→"5.2/5.4" but never given an implementation task (5.2/5.4 are testsuite-boundary verification tasks that cannot contain the production cutover over repl.c/persist.c/ha.c). NEEDS a new impl task (e.g. "3.4 Retire term-0 repl_broadcast + drive the library send on the live primary"). Live deploy IS feasible here; harness `clean_failover_preservation.sh` is committed + runs GREEN in PAYLOAD mode (guards 848/849/850 + no job loss) and flips to the term gate via XCE_FAILOVER_STRICT_TERM=1 once the cutover lands._
- _Blocked (ENV-only — code gap CLOSED): bug-990 (the term-0 live-send gap) is FIXED by task 3.4 — the live broadcast + catch-up now carry the leader's term and standbys persist it, proven at the integration level (real receive path). The remaining blocker is purely the EXECUTION ENV: this overlayfs sandbox cannot complete a live failover (split-vote / `fdatasync errno=5`, bug-924) on baseline OR the fixed code, so the definitive `clean_failover_preservation.sh XCE_FAILOVER_STRICT_TERM=1` byte-identical-(term,lsn) assertion can't be run here. RUN the committed harness on a REAL (non-overlayfs) fs to close: it should pass PAYLOAD (848/849/850 + no job loss) AND STRICT_TERM now that 3.4 landed._
- [x] 5.3 Cross-member `walcompare` divergence tool
- Build a tool that diffs member WALs by `(lsn → term, pcrc)` and exits non-zero on any committed-prefix divergence, reporting the diverging LSN and per-member entry identities; reuse the existing WAL decoder
@ -137,7 +137,7 @@
- Observable completion: the regression passes on the fixed code and fails on pre-fix code; no acked-committed job is absent; committed config reference identical across members
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 7.11_
- _Depends: 5.2, 5.3, 4.3_
- _Blocked: transitively on bug-990 (same live-send retirement gap as 5.2) — the R6 headline asserts `(term,lsn,pcrc)` committed-prefix identity, which the term-0 live send cannot satisfy. Unblocks once the live-send cutover lands and 5.2 passes._
- _Blocked (ENV-only — code gap CLOSED by 3.4): the R6 headline `(term,lsn,pcrc)` committed-prefix identity is now satisfiable (3.4 fixed bug-990). Remaining blocker is the same real-fs live-run env as 5.2 (bug-924 overlayfs can't run the chaotic-multi-kill harness here). Author/run on a real fs; pass an explicit `--committed N` to walcompare (5.3 advisory) + `expect-job`._
- [ ] 5.5 Full controller build + suite green under ASAN
- Run `/xce-build controller`; the controller compiles and links (build restored) and all controller unit/integration tests plus the adapter tests pass under ASAN with no regression in election/durability behavior
@ -212,3 +212,6 @@
- **5.3 done.** `walcompare` tool: `xce_test walcompare [--committed N] <memberA> <memberB> [...]` + reusable `walcompare()` (walcompare.c/h). Diffs members by `(lsn → term, pcrc)` over the committed prefix; pcrc = payload-only crc32 (NOT header) so same-LSN/diff-content divergences surface; legacy-0→WAL_LEGACY_TERM. Exit 0 identical / WALCMP_DIVERGENT(1) on committed-prefix divergence (reports diverging LSN + per-member term/pcrc/present-or-MISSING for ALL N members) / negative errno on load error. REUSES the canonical waldump decoder via a NEW `waldump_path_visit` per-entry visitor seam (`waldump_path` is now a thin wrapper — existing waldump behavior + tests unchanged, NO 5th WAL parser). 11 tests; reviewer APPROVED (RED empirically reproduced by neutering compare; CLI hand-verified).
- **ADVISORY for 5.4 (reviewer MEDIUM):** the DEFAULT committed boundary = `min(recovered_tip)` across members. A member that diverged then has a CRC-broken/short tail carries a LOWER recovered_tip → lowers the shared boundary → could MASK a divergence above it. 5.4 MUST pass an EXPLICIT `--committed N` (highest acked LSN), and pair walcompare with `expect-job $jobid` (R6.4) which independently catches a lost committed job.
- **3.4 done — bug-990 CLOSED (the live-send cutover).** ROOT CAUSE: `repl_broadcast` (the LIVE replicator, persist.c:747) built `wal_catchup_entry_t` without setting `entry.term`, so the live broadcast wire-encoded term=0 even though primary `hdr.term=p->term`; standbys took the legacy term-0 direct-apply and persisted term=0. FIX (MINIMAL cutover, reviewer-justified over the fuller raft_send_append_entries rewrite — the requirement set is met by carrying the term so entries auto-route through the already-wired 3.2 intake + 3.3 ACK): `repl_broadcast` now takes `prev_log_term` + sets `entry.term=hdr->term`/`prev_log_index=lsn-1`/`prev_log_term`; `persist_wal_async` sources prev_log_term via NEW lock-free `entry_term_locked(p, lsn-1)` (predecessor term, under p->lock; lsn==1→(0,0)); catch-up streamer `ha_stream_wal_to_standby` (config_repl.c) carries term+prev tuple per entry (ascending (lsn,term) tracker + conservative segment-skip so the FIRST streamed entry's prev references what the standby already holds); `ha_apply_wal_catchup_entry` (ha.c) routes EVERY term≠0 entry (single + each batch sub-entry, in order, guarded raft_c==c) through `raft_on_append_entries_request`→on_persist_entry (leader term stamped) — retiring the term-0 direct-apply for live/catch-up; `wal_entry_to_wire` (request.c) encodes prevLog{Index,Term}+leaderCommit PER SUB-ENTRY. 0-entry heartbeat still routes through the library for UpdateTerm (persists nothing). bug-850 contiguity + persist-before-ACK + commit-barrier + bug-960 peer-resolution all preserved. Live commit chain now genuinely end-to-end (term-carrying send → standby on_persist_entry + ACK → ha_raft_on_repl_ack → raft_advance_commit → on_commit_advanced → barrier). 2 new tests (send: `Expected 7 Was 0`; receive via the REAL `ha_apply_wal_catchup_entry` entry point: `Expected 7 Was 3`), RED independently reproduced by the reviewer. 175 Tests 0 Failures. Reviewer APPROVED.
- **REMAINING (ENV-only, NOT code):** the definitive LIVE proof — clean failover → byte-identical `(term,lsn)` WALs via `clean_failover_preservation.sh XCE_FAILOVER_STRICT_TERM=1` — could NOT run here: this overlayfs sandbox can't complete a failover on baseline OR the change (split-vote / `fdatasync errno=5`, bug-924). The fix is proven at the integration level (real receive path, real wire shape). 5.2/5.4 + the live half of 5.5 need a REAL (non-overlayfs) fs to run the harness; the harness + STRICT_TERM gate are ready and should flip GREEN there.