fix(ha): close two independent chaotic-failover bugs — term-desync liveness stall and bug-847 WAL batch payload loss

Found via extended chaotic /xce-test-ha soaking on top of the bug-1419 fix
(same session). Both bugs independently caused the 10-round chaotic soak to
fail; sequential and chaotic now pass cleanly and repeatably.

1. Term-desync liveness deadlock (new): a standby that loses a near-
   simultaneous election race to a peer can legitimately end up at a HIGHER
   term than the peer that actually won and became primary. The primary's
   now-stale-term AppendEntries/heartbeats are correctly rejected by the raft
   library, but every one of those rejected messages was ALSO resetting the
   standby's own liveness deadline (ha_on_heartbeat_received), so its
   liveness timeout never fired and it could never start a fresh,
   higher-term election to let the cluster self-heal — a permanent stall.
   Fix: ha_msg_term_is_stale() gates the liveness reset (and
   ha_discover_primary step-down) on the incoming message's own term stamp,
   skipping it for provably-stale messages, in cluster.c's wal_catchup_entry
   and nop/heartbeat handlers.

2. bug-847 (root-caused 2026-06-30, never fixed): a per-sub-entry `payload`
   Data blob in a multi-entry WAL catch-up batch loses its capnp pointer once
   the message crosses a segment boundary — the receiver decodes
   raw_payload_len=0 for entries at/after the boundary and silently persists
   empty WAL entries (structurally intact: correct LSN/term/CRC chain, empty
   content). Only surfaces when a node rejoins and catches up a real batch,
   which is why it needed a live chaotic soak (not the 2-entry unit test) to
   reproduce. Fix: carry every sub-entry's payload concatenated in ONE
   top-level batchPayloads blob instead of one Data blob per entry — a
   single top-level blob does not exhibit the cross-segment loss. Sliced
   back apart on decode using each sub-entry's own wirePayloadLen, a plain
   scalar that (like lsn/entryType/term) decodes via the generic generated
   decode path and was never subject to the bug. Confirmed via targeted
   diagnostic logging that the corrupted entry arrives already-empty over
   the wire (not a receive-side race), and via a faithful reproduction at
   live JOB_SUBMIT payload size (5×3560B) after the original 2-entry/
   14×872B tests turned out not to cross a segment boundary in this
   environment.

Incidental fixes found and fixed along the way:
- A real self-deadlock (persist_entry_term() re-acquiring p->lock from
  inside a caller that already held it) introduced by an initial, ultimately
  unnecessary WAL-read locking fix attempted before bug-847 was pinned down;
  fixed via a new persist_entry_term_locked() lock-free counterpart. Left in
  place as a genuine (if incidental) hardening of ha_stream_wal_to_standby's
  read consistency.
- .claude/skills/xce-test-ha/SKILL.md's `context: fork` frontmatter caused
  the skill's own forked sub-session teardown to SIGHUP the whole
  xce_controller process group mid-soak, killing all 3 nodes silently with
  no error logged, whenever the soak was launched via the Skill tool itself
  (setsid/nohup inside run-ha-soak.sh only protects against a controlling
  terminal's own SIGHUP, not a parent forked-session's teardown). Removed;
  added explicit guidance to launch long soaks via direct Bash from the main
  session instead.

Verified: full 78-group unit regression sweep clean (0 failures) after each
change; two consecutive clean 10-round chaotic soaks and one clean 10-round
sequential soak on the final binary (zero committed divergence, zero job
loss, all 3 nodes led in both modes).
This commit is contained in:
Rongsong Shen 2026-07-08 15:21:24 +00:00
parent 11244dd654
commit 0941cf9649
11 changed files with 295 additions and 47 deletions

View file

@ -598,6 +598,13 @@ void persist_set_term(persist_t* p, uint64_t term);
* Safe on a NULL p (returns 0). */
uint64_t persist_entry_term(const persist_t* p, uint64_t lsn);
/* Lock-free counterpart of persist_entry_term for a caller that ALREADY holds
* p->lock (calling persist_entry_term there would re-acquire the same
* non-recursive mutex and self-deadlock — raft-ha-core-safety task 2.5). Same
* absent/legacy-0 semantics, just without the lock/unlock. Caller MUST hold
* p->lock; safe on a NULL p (returns 0). */
uint64_t persist_entry_term_locked(const persist_t* p, uint64_t lsn);
/* raft-ha-core-safety task 3.2 (Req 3.1/5.1) — Snapshot Boundary Reconcile.
*
* Authoritatively overwrite the per-LSN term index entry at `checkpoint_lsn`

View file

@ -377,6 +377,24 @@ struct wal_catchup_entry {
* 0 = unset (legacy sender) → fall back to the stored term. Wire field
* leaderCurrentTerm @8. */
uint64_t leader_current_term;
/* bug-847: this sub-entry's payload length, carried as a plain scalar (wire
* field wirePayloadLen @9) so it decodes via the generic generated decode
* exactly like lsn/entry_type/term — never subject to the per-element Data
* cross-segment loss. Set by the encoder for every xce_wal_batch_encode_raw
* send; used on decode to slice this entry's bytes out of the TOP-LEVEL
* container's batch_payloads blob (below). */
uint32_t wire_payload_len;
/* bug-847: generic decoded placeholder for wire field batchPayloads @10 —
* same broken-for-raw-bytes limitation as `payload` above; kept NULL, do
* not use. Only meaningful on the TOP-LEVEL container (never on a nested
* sub-entry). The real bytes are hand-extracted by xce_decode() into
* raw_batch_payloads/raw_batch_payloads_len below, then sliced per
* sub-entry using each entry's wire_payload_len into that entry's OWN
* raw_payload/raw_payload_len (so every consumer keeps reading
* raw_payload exactly as before — this is purely a wire-transport fix). */
xce_data_t *batch_payloads;
uint8_t *raw_batch_payloads;
size_t raw_batch_payloads_len;
};
/* ── Persistence Cap'n Proto mapped types (P0-9) ─────────────────────