fix(ha): close two independent chaotic-failover bugs — term-desync liveness stall and bug-847 WAL batch payload loss
Found via extended chaotic /xce-test-ha soaking on top of the bug-1419 fix (same session). Both bugs independently caused the 10-round chaotic soak to fail; sequential and chaotic now pass cleanly and repeatably. 1. Term-desync liveness deadlock (new): a standby that loses a near- simultaneous election race to a peer can legitimately end up at a HIGHER term than the peer that actually won and became primary. The primary's now-stale-term AppendEntries/heartbeats are correctly rejected by the raft library, but every one of those rejected messages was ALSO resetting the standby's own liveness deadline (ha_on_heartbeat_received), so its liveness timeout never fired and it could never start a fresh, higher-term election to let the cluster self-heal — a permanent stall. Fix: ha_msg_term_is_stale() gates the liveness reset (and ha_discover_primary step-down) on the incoming message's own term stamp, skipping it for provably-stale messages, in cluster.c's wal_catchup_entry and nop/heartbeat handlers. 2. bug-847 (root-caused 2026-06-30, never fixed): a per-sub-entry `payload` Data blob in a multi-entry WAL catch-up batch loses its capnp pointer once the message crosses a segment boundary — the receiver decodes raw_payload_len=0 for entries at/after the boundary and silently persists empty WAL entries (structurally intact: correct LSN/term/CRC chain, empty content). Only surfaces when a node rejoins and catches up a real batch, which is why it needed a live chaotic soak (not the 2-entry unit test) to reproduce. Fix: carry every sub-entry's payload concatenated in ONE top-level batchPayloads blob instead of one Data blob per entry — a single top-level blob does not exhibit the cross-segment loss. Sliced back apart on decode using each sub-entry's own wirePayloadLen, a plain scalar that (like lsn/entryType/term) decodes via the generic generated decode path and was never subject to the bug. Confirmed via targeted diagnostic logging that the corrupted entry arrives already-empty over the wire (not a receive-side race), and via a faithful reproduction at live JOB_SUBMIT payload size (5×3560B) after the original 2-entry/ 14×872B tests turned out not to cross a segment boundary in this environment. Incidental fixes found and fixed along the way: - A real self-deadlock (persist_entry_term() re-acquiring p->lock from inside a caller that already held it) introduced by an initial, ultimately unnecessary WAL-read locking fix attempted before bug-847 was pinned down; fixed via a new persist_entry_term_locked() lock-free counterpart. Left in place as a genuine (if incidental) hardening of ha_stream_wal_to_standby's read consistency. - .claude/skills/xce-test-ha/SKILL.md's `context: fork` frontmatter caused the skill's own forked sub-session teardown to SIGHUP the whole xce_controller process group mid-soak, killing all 3 nodes silently with no error logged, whenever the soak was launched via the Skill tool itself (setsid/nohup inside run-ha-soak.sh only protects against a controlling terminal's own SIGHUP, not a parent forked-session's teardown). Removed; added explicit guidance to launch long soaks via direct Bash from the main session instead. Verified: full 78-group unit regression sweep clean (0 failures) after each change; two consecutive clean 10-round chaotic soaks and one clean 10-round sequential soak on the final binary (zero committed divergence, zero job loss, all 3 nodes led in both modes).
This commit is contained in:
parent
11244dd654
commit
0941cf9649
11 changed files with 295 additions and 47 deletions
|
|
@ -172,8 +172,15 @@ static int handle_wal_catchup_entry(cluster_t *c, endpoint_t *ep,
|
|||
return 0;
|
||||
|
||||
/* Req 1.3 (ha-heartbeat-liveness): any msg from primary resets standby
|
||||
* liveness and steps down from CTRL_ELECTING. */
|
||||
{
|
||||
* liveness and steps down from CTRL_ELECTING. Skipped when the entry's
|
||||
* own term stamp proves it is already stale (bug-1418 family): a
|
||||
* stale-term AppendEntries is about to be rejected by the raft library
|
||||
* below anyway, and letting it ALSO reset liveness would mean a standby
|
||||
* that legitimately outraced its own peer to a higher term (a rare but
|
||||
* valid election-timing outcome) never times out to start a fresh,
|
||||
* higher-term election — permanently deadlocking the cluster on a
|
||||
* primary whose term the standby correctly knows is stale. */
|
||||
if (!ha_msg_term_is_stale(&c->ha, entry->term)) {
|
||||
ctrl_role_t role = (ctrl_role_t)atomic_load_explicit(
|
||||
&c->ha.role, memory_order_acquire);
|
||||
if (role == CTRL_STANDBY)
|
||||
|
|
@ -324,14 +331,25 @@ static int myreq_handler(vtask_t* task)
|
|||
ha_nop_reply_state_report(&c->ha, c->myconf, &sr, &reply);
|
||||
xce_reply(ep, c->myhost, &reply);
|
||||
/* Req 1.3: any NOP from primary resets liveness deadline on standby.
|
||||
* Also steps down from CTRL_ELECTING if a live primary is discovered. */
|
||||
* Also steps down from CTRL_ELECTING if a live primary is discovered.
|
||||
* Skipped when the heartbeat's own term stamp proves it is already
|
||||
* stale (bug-1418 family) -- see the identical guard's doc comment
|
||||
* in handle_wal_catchup_entry above. A bodyless nop (no HeartbeatMsg)
|
||||
* carries no term, so ha_msg_term_is_stale's msg_term==0 case
|
||||
* preserves the existing unconditional behavior for it. */
|
||||
{
|
||||
ctrl_role_t role = (ctrl_role_t)atomic_load_explicit(
|
||||
&c->ha.role, memory_order_acquire);
|
||||
if (role == CTRL_STANDBY)
|
||||
ha_on_heartbeat_received(&c->ha);
|
||||
else if (role == CTRL_ELECTING)
|
||||
ha_discover_primary(&c->ha, c);
|
||||
uint64_t hb_term = (request.body_which == Request_body_heartbeatmsg
|
||||
&& request.body.heartbeatmsg)
|
||||
? request.body.heartbeatmsg->term
|
||||
: 0;
|
||||
if (!ha_msg_term_is_stale(&c->ha, hb_term)) {
|
||||
ctrl_role_t role = (ctrl_role_t)atomic_load_explicit(
|
||||
&c->ha.role, memory_order_acquire);
|
||||
if (role == CTRL_STANDBY)
|
||||
ha_on_heartbeat_received(&c->ha);
|
||||
else if (role == CTRL_ELECTING)
|
||||
ha_discover_primary(&c->ha, c);
|
||||
}
|
||||
}
|
||||
/* ha-heartbeat-committed-propagation task 2.2: after the unchanged
|
||||
* liveness/reply path above, a heartbeat that carries a HeartbeatMsg
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue