lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-ha-core-safety): streamer fail-loud bounded-retry, never ship empty (task 2.4)

Browse source 
Root-cause source fix: ha_stream_wal_to_standby reads the entry body via
stream_read_entry_body (bounded retry max 4, no sleep); on persistent short
pread/malloc-fail it fails loud + ABORTS the stream (flush valid prefix,
drop unreadable entry, -EIO) instead of shipping PLEN=0. Incomplete stream
re-driven by ACK-driven catch-up. With the 2.2 guard this closes the live
cross-term payload-loss divergence. 2.3 CORE RED -> GREEN (fail-loud: no
empty persisted; + retry-recover test ships full 64). Term-blind structural
retirement deferred to 6.2 (empty-over-full safety enforced by 2.2 guard +
pinned). 12/12 core suite; siblings green; no regression introduced.
Phase 2 complete.
This commit is contained in:
Rongsong Shen 2026-06-27 15:42:37 +00:00
parent ae8291ed56
commit 63b9c5a81a
3 changed files with 210 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

129
src/controller/config_repl.c
View file

@ -2043,6 +2043,73 @@ static int send_wal_batch(endpoint_t *ep,
return 0;
}
/* Maximum bounded re-attempts when an entry body read makes no forward progress
* (a zero/EOF/EINTR short read or an allocation failure). Small and finite so the
* loop can never busy-spin; a regular-file body completes in one or two reads, so
* this only ever spends attempts on a genuinely failing read. No sleep/backoff —
* a transient cause (EINTR, partial read) recovers on the next immediate attempt;
* a persistent cause exhausts the budget and is surfaced loud (never shipped
* empty). raft-ha-core-safety task 2.4. */
#define WAL_STREAM_BODY_READ_MAX_ATTEMPTS 4
/* stream_read_entry_body: read a WAL entry body of `payload_len` bytes at
* `body_off` into a freshly malloc'd buffer, accumulating partial reads and
* bounded-retrying a no-progress read. On success returns 0 with *out_buf set to
* a heap buffer the caller owns (free()); on persistent failure returns negative
* (-ENOMEM / -EIO) with *out_buf == NULL.
*
* raft-ha-core-safety task 2.4 — the SEND-side payload-integrity fix. The
* baseline freed the buffer and shipped the entry with PLEN=0 on a short
* pread/malloc-fail, silently transmitting a non-empty COMMITTED entry as empty
* (the live LSN-9 divergence). This helper instead fails LOUD so the caller can
* abort the stream rather than degrade the entry to empty. A short read is not
* inherently fatal: pread may legitimately return fewer bytes, so we re-read from
* the advanced offset (forward progress never spends an attempt); only a read
* that returns <= 0 consumes the bounded retry budget. */
static int stream_read_entry_body(int fd, uint32_t body_off,
uint32_t payload_len, uint64_t lsn,
uint64_t stored_term, uint64_t leader_term,
uint8_t **out_buf)
{
*out_buf = NULL;
uint8_t *pbuf = NULL;
for (int a = 0; a < WAL_STREAM_BODY_READ_MAX_ATTEMPTS && !pbuf; a++)
pbuf = malloc(payload_len);
if (!pbuf)
return -ENOMEM;
uint32_t got = 0;
int attempts = 0;
while (got < payload_len) {
ssize_t pr = pread(fd, pbuf + got, payload_len - got,
(off_t)(body_off + got));
#ifdef XCE_TESTING
/* raft-ha-core-safety task 2.3/2.4 fault-injection seam: a test may force
* this read to look SHORT (pr=0) to exercise the fail-loud + bounded-retry
* path. NULL in production — no behaviour change. The hook fires on every
* attempt, so a permanent fault exhausts the budget (fail loud) while a
* transient one (hook returns 0 after N) recovers. */
if (ha_stream_body_read_fault_hook != NULL &&
ha_stream_body_read_fault_hook(lsn, stored_term, leader_term,
payload_len))
pr = 0;
#endif
if (pr > 0) {
got += (uint32_t)pr; /* forward progress — does not spend an attempt */
continue;
}
/* pr <= 0: EOF/zero/EINTR/error — bounded retry, no busy-spin. */
if (++attempts >= WAL_STREAM_BODY_READ_MAX_ATTEMPTS) {
free(pbuf);
return -EIO;
}
}
*out_buf = pbuf;
return 0;
}
int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
const persist_t *p, uint64_t leader_term,
uint64_t standby_lsn, uint64_t committed_lsn)
@ -2166,30 +2233,48 @@ int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
entry->leader_current_term = leader_term;
if (hdr.payload_len > 0) {
uint8_t *pbuf = malloc(hdr.payload_len);
if (pbuf) {
ssize_t pr = pread(fd, pbuf, hdr.payload_len,
(off_t)(off + sizeof(hdr)));
#ifdef XCE_TESTING
/* raft-ha-core-safety task 2.3 fault-injection seam: a
* test may force this body read to look SHORT (the same
* outcome as the malloc-fail tail above) to prove the
* baseline ships a non-empty entry as PLEN=0. NULL in
* production — no behaviour change. Task 2.4 owns the
* fix (fail loud + bounded retry); this seam ships no fix. */
if (ha_stream_body_read_fault_hook != NULL &&
ha_stream_body_read_fault_hook(hdr.lsn, entry->term,
leader_term,
hdr.payload_len))
pr = 0; /* simulate a short read */
#endif
if (pr == (ssize_t)hdr.payload_len) {
entry->raw_payload = pbuf;
entry->raw_payload_len = hdr.payload_len;
} else {
free(pbuf);
/* raft-ha-core-safety task 2.4: read the body with bounded
* retry; a non-empty COMMITTED entry must NEVER be shipped
* empty (the live LSN-9 send-side payload loss). */
uint8_t *pbuf = NULL;
int rrc = stream_read_entry_body(
fd, off + (uint32_t)sizeof(hdr), hdr.payload_len,
hdr.lsn, entry->term, leader_term, &pbuf);
if (rrc != 0) {
/* FAIL LOUD: the body could not be read in full after the
* bounded retry. Do NOT degrade the entry to PLEN=0 —
* abort the stream here. Flush the valid prefix already
* batched (entries strictly below this LSN remain correct
* and contiguous), drop THIS entry, and return an error so
* the ACK-driven catch-up continuation re-streams from this
* LSN on the next cycle. The standby simply does not
* advance past the gap — it never receives a committed
* entry as empty. The controller stays alive and
* self-heals (CLAUDE.md process-lifecycle invariant:
* fail-loud = refuse to ship divergent state, not exit). */
ERROR("ha_stream_wal_to_standby: entry body read FAILED "
"for lsn=%" PRIu64 " payload_len=%u stored_term=%"
PRIu64 " leader_term=%" PRIu64 " rc=%d after %d "
"attempts — ABORTING stream, NOT shipping empty (a "
"non-empty committed entry must never be transmitted "
"as PLEN=0; catch-up will re-stream from this LSN)",
hdr.lsn, hdr.payload_len, entry->term, leader_term,
rrc, WAL_STREAM_BODY_READ_MAX_ATTEMPTS);
free(entry);
if (batch_n > 0) {
send_wal_batch(ep, batch_entries, batch_n);
for (int bi = 0; bi < batch_n; bi++) {
free(batch_entries[bi]->raw_payload);
free(batch_entries[bi]);
}
batch_n = 0;
batch_bytes = 0;
}
close(fd);
return -EIO;
}
entry->raw_payload = pbuf;
entry->raw_payload_len = hdr.payload_len;
}
batch_entries[batch_n] = entry;