lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): close two independent chaotic-failover bugs — term-desync liveness stall and bug-847 WAL batch payload loss

Browse source 
Found via extended chaotic /xce-test-ha soaking on top of the bug-1419 fix
(same session). Both bugs independently caused the 10-round chaotic soak to
fail; sequential and chaotic now pass cleanly and repeatably.

1. Term-desync liveness deadlock (new): a standby that loses a near-
   simultaneous election race to a peer can legitimately end up at a HIGHER
   term than the peer that actually won and became primary. The primary's
   now-stale-term AppendEntries/heartbeats are correctly rejected by the raft
   library, but every one of those rejected messages was ALSO resetting the
   standby's own liveness deadline (ha_on_heartbeat_received), so its
   liveness timeout never fired and it could never start a fresh,
   higher-term election to let the cluster self-heal — a permanent stall.
   Fix: ha_msg_term_is_stale() gates the liveness reset (and
   ha_discover_primary step-down) on the incoming message's own term stamp,
   skipping it for provably-stale messages, in cluster.c's wal_catchup_entry
   and nop/heartbeat handlers.

2. bug-847 (root-caused 2026-06-30, never fixed): a per-sub-entry `payload`
   Data blob in a multi-entry WAL catch-up batch loses its capnp pointer once
   the message crosses a segment boundary — the receiver decodes
   raw_payload_len=0 for entries at/after the boundary and silently persists
   empty WAL entries (structurally intact: correct LSN/term/CRC chain, empty
   content). Only surfaces when a node rejoins and catches up a real batch,
   which is why it needed a live chaotic soak (not the 2-entry unit test) to
   reproduce. Fix: carry every sub-entry's payload concatenated in ONE
   top-level batchPayloads blob instead of one Data blob per entry — a
   single top-level blob does not exhibit the cross-segment loss. Sliced
   back apart on decode using each sub-entry's own wirePayloadLen, a plain
   scalar that (like lsn/entryType/term) decodes via the generic generated
   decode path and was never subject to the bug. Confirmed via targeted
   diagnostic logging that the corrupted entry arrives already-empty over
   the wire (not a receive-side race), and via a faithful reproduction at
   live JOB_SUBMIT payload size (5×3560B) after the original 2-entry/
   14×872B tests turned out not to cross a segment boundary in this
   environment.

Incidental fixes found and fixed along the way:
- A real self-deadlock (persist_entry_term() re-acquiring p->lock from
  inside a caller that already held it) introduced by an initial, ultimately
  unnecessary WAL-read locking fix attempted before bug-847 was pinned down;
  fixed via a new persist_entry_term_locked() lock-free counterpart. Left in
  place as a genuine (if incidental) hardening of ha_stream_wal_to_standby's
  read consistency.
- .claude/skills/xce-test-ha/SKILL.md's `context: fork` frontmatter caused
  the skill's own forked sub-session teardown to SIGHUP the whole
  xce_controller process group mid-soak, killing all 3 nodes silently with
  no error logged, whenever the soak was launched via the Skill tool itself
  (setsid/nohup inside run-ha-soak.sh only protects against a controlling
  terminal's own SIGHUP, not a parent forked-session's teardown). Removed;
  added explicit guidance to launch long soaks via direct Bash from the main
  session instead.

Verified: full 78-group unit regression sweep clean (0 failures) after each
change; two consecutive clean 10-round chaotic soaks and one clean 10-round
sequential soak on the final binary (zero committed divergence, zero job
loss, all 3 nodes led in both modes).
This commit is contained in:
Rongsong Shen 2026-07-08 15:21:24 +00:00
parent 11244dd654
commit 0941cf9649
11 changed files with 295 additions and 47 deletions
Download patch file Download diff file Expand all files Collapse all files

89
src/lib/request.c
View file

@ -202,6 +202,15 @@ int xce_decode(io_packet_t* pkt, xce_request_t* req)
extract_data_bytes(WalCatchupEntry_get_payload(b_ptr),
&batch->raw_payload_len);
/* bug-847: the top-level batch_payloads blob — every sub-entry's
* payload concatenated in ONE Data field (see wal_entry_to_wire /
* xce_wal_batch_encode_raw). A single top-level Data blob does not
* exhibit the per-element cross-segment pointer loss that scattered
* per-sub-entry Data blobs did. */
batch->raw_batch_payloads =
extract_data_bytes(WalCatchupEntry_get_batchPayloads(b_ptr),
&batch->raw_batch_payloads_len);
/* ha-term-safety task 3.2: the top-level container's AppendEntries
* metadata (term, prevLogIndex, prevLogTerm, leaderCommit) is decoded
* into batch->{term,prev_log_index,prev_log_term,leader_commit} by the
@ -209,18 +218,40 @@ int xce_decode(io_packet_t* pkt, xce_request_t* req)
* the Data payload needs manual extraction (above/below). The standby
* intake (ha_consume_append_entries) reads those struct fields directly. */
/* Batch sub-entries: each carries its own lsn/entryType/term (scalar,
* decoded) + payload (Data, extracted here). */
/* Batch sub-entries: each carries its own lsn/entryType/term/
* wire_payload_len (scalar, decoded — bug-847 immune) + payload. Slice
* this entry's bytes out of the top-level batch_payloads blob using
* wire_payload_len and a running cumulative offset (entries stream in
* the same order they were concatenated on encode). Falls back to the
* legacy per-element Data field only when no batch_payloads blob was
* sent at all (defensive — every current sender populates it). */
WalCatchupEntry_list wlist = WalCatchupEntry_get_entries(b_ptr);
int wn = (wlist.p.type != CAPN_NULL) ? capn_len(wlist) : 0;
size_t batch_off = 0;
for (int i = 0; i < wn && i < batch->n_entries; i++) {
if (!batch->entries || !batch->entries[i])
continue;
struct WalCatchupEntry we;
get_WalCatchupEntry(&we, wlist, i);
batch->entries[i]->raw_payload =
extract_data_bytes(we.payload,
&batch->entries[i]->raw_payload_len);
if (batch->raw_batch_payloads != NULL) {
uint32_t plen = batch->entries[i]->wire_payload_len;
if (plen > 0 &&
batch_off + plen <= batch->raw_batch_payloads_len) {
uint8_t *slice = (uint8_t *)malloc(plen);
if (slice) {
memcpy(slice, batch->raw_batch_payloads + batch_off,
plen);
batch->entries[i]->raw_payload = slice;
batch->entries[i]->raw_payload_len = plen;
}
batch_off += plen;
}
} else {
batch->entries[i]->raw_payload =
extract_data_bytes(we.payload,
&batch->entries[i]->raw_payload_len);
}
}
}
@ -302,6 +333,9 @@ void xce_free_request(xce_request_t *req)
free(batch->raw_payload);
batch->raw_payload = NULL;
batch->raw_payload_len = 0;
free(batch->raw_batch_payloads);
batch->raw_batch_payloads = NULL;
batch->raw_batch_payloads_len = 0;
for (int i = 0; i < batch->n_entries; i++) {
if (!batch->entries || !batch->entries[i])
continue;
@ -491,11 +525,14 @@ static void wal_entry_to_wire(struct capn_segment *cs,
* the AppendEntries header term so historical (entry_term < leader_term)
* catch-up entries are not stale-rejected. */
w->leaderCurrentTerm = e->leader_current_term;
if (e->raw_payload && e->raw_payload_len > 0) {
capn_list8 pl = capn_new_list8(cs, (int)e->raw_payload_len);
capn_setv8(pl, 0, e->raw_payload, (int)e->raw_payload_len);
w->payload.p = pl.p;
}
/* bug-847: do NOT write this sub-entry's payload as its own per-element
* Data blob — that is exactly the shape that loses its pointer once a
* multi-entry batch crosses a capnp segment boundary. The caller
* (xce_wal_batch_encode_raw) instead carries every sub-entry's bytes
* concatenated in ONE top-level batchPayloads blob; this scalar length
* is all a sub-entry needs to find its slice on decode. */
(void)cs;
w->wirePayloadLen = (uint32_t)e->raw_payload_len;
}
/*
@ -549,7 +586,7 @@ int xce_wal_batch_encode_raw(wal_catchup_entry_t **entries, int n_entries,
top.leaderCurrentTerm = entries[0]->leader_current_term;
/* Batch container: entries list carries the per-entry payload(s) (empty for
* a heartbeat). Each sub-entry carries lsn/entryType/term/payload. */
* a heartbeat). Each sub-entry carries lsn/entryType/term/wirePayloadLen. */
WalCatchupEntry_list lst = new_WalCatchupEntry_list(cs, n_entries);
for (int i = 0; i < n_entries; i++) {
struct WalCatchupEntry w;
@ -558,6 +595,36 @@ int xce_wal_batch_encode_raw(wal_catchup_entry_t **entries, int n_entries,
}
top.entries = lst;
/* bug-847: carry every sub-entry's payload concatenated in ONE top-level
* Data blob instead of one Data blob per sub-entry — a single blob does
* not exhibit the cross-segment pointer loss that scattered per-element
* Data blobs do in a multi-entry batch. Decode slices this back apart
* per entry using each entry's own wirePayloadLen (set above). */
{
size_t total = 0;
for (int i = 0; i < n_entries; i++)
total += entries[i]->raw_payload_len;
if (total > 0) {
uint8_t *concat = (uint8_t *)malloc(total);
if (!concat) {
capn_free(&c);
return -ENOMEM;
}
size_t off = 0;
for (int i = 0; i < n_entries; i++) {
if (entries[i]->raw_payload && entries[i]->raw_payload_len > 0) {
memcpy(concat + off, entries[i]->raw_payload,
entries[i]->raw_payload_len);
off += entries[i]->raw_payload_len;
}
}
capn_list8 pl = capn_new_list8(cs, (int)total);
capn_setv8(pl, 0, concat, (int)total);
top.batchPayloads.p = pl.p;
free(concat);
}
}
tp = new_WalCatchupEntry(cs);
write_WalCatchupEntry(&top, tp);

24
src/lib/testsuite/test_wal_wire.c
View file

@ -90,21 +90,19 @@ static void test_wal_batch_roundtrip_payload_bytes(void)
printf("test_wal_batch_roundtrip_payload_bytes: PASS\n");
}
/* Production-scale batch: many entries each carrying a full-size (872-byte)
* capnp-Job-sized payload — mirrors the live catch-up stream where ctrl-1
* rejoined and received header-only entries (PLEN=0). The small 2-entry
* round-trip passes; this reproduces whatever the larger/multi-segment batch
* does on the wire (bug-847). */
/* Production-scale batch: several entries each carrying a full-size
* (3560-byte) JOB_SUBMIT-sized payload — mirrors the live catch-up stream
* that reproduced bug-847 (a rejoining node persisted header-only entries,
* PLEN=0, for a mid-batch LSN range). The small 2-entry round-trip above
* fits in a single capnp segment and always passed; this size crosses a
* segment boundary, which used to lose the per-entry payload Data pointer on
* the wire. Fixed by carrying every sub-entry's payload concatenated in one
* top-level batch_payloads blob instead of one Data blob per entry (see
* wal_entry_to_wire / xce_wal_batch_encode_raw in request.c) — a single Data
* blob does not exhibit the cross-segment loss. */
static void test_wal_batch_roundtrip_large_payloads(void)
{
/* KNOWN-FAILING REPRODUCER for bug-847: a production-scale batch that spans
* more than one capnp segment loses its per-entry payload Data pointers on
* the wire (decode returns raw_payload_len=0; it also leaks the orphaned
* decode allocations under ASAN). IGNORE'd so the suite stays green; REMOVE
* this TEST_IGNORE line when bug-847 is fixed and assert the payloads
* survive. Decisive evidence: this fails "Expected 872 Was 0" while the
* 2-entry round-trip above passes. */
enum { N = 14, PSZ = 872 };
enum { N = 5, PSZ = 3560 };
wal_catchup_entry_t e[N];
wal_catchup_entry_t *entries[N];
uint8_t* payloads[N];