wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix bug-1060 bug-1067

Browse source
This commit is contained in:
Rongsong Shen 2026-06-24 17:05:43 +08:00
parent 6067ba6c94
commit 61e1391cd0
28 changed files with 7021 additions and 2926 deletions
Download patch file Download diff file Expand all files Collapse all files

7
src/controller/CLAUDE.md
View file

@ -70,7 +70,7 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft
**Election Safety** (≤1 leader per term): Enforced by `votedFor` — a server grants at most one vote per term (`raft_on_vote_request` line 145-151). A candidate needs majority votes; each server votes at most once per term.
**Leader Completeness** (committed entries survive): Freshness gate in `RequestVote` — candidate must have `lastLogTerm ≥ voter` AND `lastLogIndex ≥ voter`. Since committed entries are on a majority, the new leader must have all committed entries. `persist_truncate_wal_above(safe_committed_lsn)` removes any divergent entries above the safe commit point.
**Leader Completeness** (committed entries survive): Freshness gate in `RequestVote` — candidate must have `lastLogTerm ≥ voter` AND `lastLogIndex ≥ voter`. Since committed entries are on a majority, the new leader must have all committed entries. The post-election truncation removes only a divergent tail **above the leader's own durable log**: the cutoff is `ha_promotion_truncate_floor(safe_committed_lsn, durable_tip) = max(safe_committed_lsn, persist_durable_floor)`, so `persist_truncate_wal_above` is a no-op on the leader's own durable entries. **A new leader MUST NEVER truncate its own log** — entries it holds durably were acked by it (as a standby) and so were committed by a prior leader; truncating to the bare majority element destroys them (bug-1060: granted `{self=47, peer=44}` → `safe=44` cut the leader's own committed 45..47 → divergent WALs + permanent prevote deadlock).
**State Machine Safety** (no divergent entries): Monotonic `committed_lsn` + truncation on promotion. Entries at or below `committed_lsn` are identical across all servers.
@ -88,7 +88,7 @@ After winning election, the new primary determines `safe_committed_lsn` from vot
safe_committed_lsn = max(lsn : count of voters with peerLastLogLsn >= lsn) + 1 >= majority)
```
The `RequestVoteReply` message already carries `peerConfigTag` and `peerLastLogLsn` from each voter. The winning candidate uses these (from `ha_elect_ctx_t.votes[]`) to compute `safe_committed_lsn` immediately in `raft_cb_become_leader`, then calls `persist_truncate_wal_above(safe_committed_lsn)`.
The `RequestVoteReply` message already carries `peerConfigTag` and `peerLastLogLsn` from each voter. The winning candidate uses these (from `ha_elect_ctx_t.votes[]`) to compute `safe_committed_lsn` immediately in `raft_cb_become_leader`, then truncates at `ha_promotion_truncate_floor(safe_committed_lsn, persist_durable_floor)` — i.e. `max(safe_committed_lsn, own durable tip)`, so it never truncates the leader's own durable log (bug-1060 Leader Completeness; `safe_committed_lsn` is still used as the commit lower-bound and the promotion-gate HWM).
**Never** use `last_known_committed_lsn` from the previous primary's ctrlhello for truncation — it may be stale (the old primary may have committed entries after the last ctrlhello was sent).
@ -100,8 +100,9 @@ When modifying HA protocol code (election, replication, heartbeat), verify:
- [ ] `commitIndex` never decreases (monotonic)
- [ ] A leader only commits entries from its current term (or equivalent via per-entry majority ack)
- [ ] `safe_committed_lsn` is derived from voter replies, not from stale ctrlhello
- [ ] WAL truncation on promotion only removes entries above the safe commit point
- [ ] WAL truncation on promotion never falls below the leader's OWN durable tip — cutoff = `max(safe_committed_lsn, persist_durable_floor)` (`ha_promotion_truncate_floor`); a new leader never truncates its own committed log (bug-1060)
- [ ] Freshness gate in RequestVote: candidate must have `lastLogTerm >= voter` AND `lastLogIndex >= voter`
- [ ] AppendEntries carries TWO distinct terms: the HEADER term = leader's CURRENT term (UpdateTerm/stale-check) and each ENTRY's STORED term (logOk/persist). For catch-up of historical entries they DIFFER — the AE header term must be the leader's current term (`wal_catchup_entry_t.leader_current_term`), never the entry's stored term, or the library stale-rejects every cross-term entry (bug-1067)
- [ ] `committed_lsn` advances only as a contiguous LSN-ordered prefix — no out-of-order or per-entry commit
- [ ] Durability flushing makes a contiguous prefix durable; `durable_lsn` is a prefix watermark (never a per-entry set)
- [ ] Causal dependencies are appended in order (cause before effect, e.g. submit before start)

5
src/controller/cluster.c
View file

@ -1372,6 +1372,11 @@ int cluster_init_phase2(cluster_t* c)
* receiver overwrites already-consistent records instead of fetching only
* the missing suffix (Raft incremental catch-up). */
ha_sync_applied_lsn_from_recovery(&c->ha, &c->persist);
/* bug-1042: also raise the raft LIBRARY's log_len/commit_index to the
* recovered position. ha_init_raft ran BEFORE this recovery on the standby-
* rejoin boot, so without this the library stays at log_len=0 and rejects
* every AppendEntries (logOk fail) — wedging the rejoined member forever. */
ha_sync_raft_log_from_recovery(&c->ha, &c->persist);
INFO("HA: persist_recover done — recovered durable_floor=%" PRIu64
" current_lsn=%" PRIu64 "; applied_lsn=%" PRIu64
" advertised for catch-up",

38
src/controller/config_repl.c
View file

@ -2014,6 +2014,7 @@ static int send_wal_batch(endpoint_t *ep,
}
int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
const persist_t *p, uint64_t leader_term,
uint64_t standby_lsn, uint64_t committed_lsn)
{
if (!wal_path)
@ -2102,9 +2103,11 @@ int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
batch_bytes = 0;
}
/* Advance the prev tracker even on the OOM-skip so a later
* entry's prev tuple stays correct. */
* entry's prev tuple stays correct. bug-1028: term from the
* authoritative index, not raw hdr.term (see main tracker). */
prev_streamed_lsn = hdr.lsn;
prev_streamed_term = hdr.term;
prev_streamed_term = p ? persist_entry_term(p, hdr.lsn)
: hdr.term;
off += (uint32_t)(sizeof(hdr) + hdr.payload_len);
continue;
}
@ -2119,9 +2122,18 @@ int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
* the (lsn,term) of the immediately preceding WAL entry (0/0
* before the first), which IS the predecessor in contiguous LSN
* order. */
entry->term = hdr.term;
/* bug-1028: own term from the authoritative per-LSN index, not
* raw hdr.term. prev_streamed_term is already normalised above. */
entry->term = p ? persist_entry_term(p, hdr.lsn)
: hdr.term;
entry->prev_log_index = prev_streamed_lsn;
entry->prev_log_term = prev_streamed_term;
/* bug-1067: stamp the leader's CURRENT term as the AppendEntries
* header term. entry->term above is the entry's STORED term (may
* be a PRIOR leader's term for a historical catch-up entry); the
* standby intake must check the leader's current term, not the
* stored term, or it stale-rejects every cross-term entry. */
entry->leader_current_term = leader_term;
if (hdr.payload_len > 0) {
uint8_t *pbuf = malloc(hdr.payload_len);
@ -2157,9 +2169,16 @@ int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
* at/below standby_lsn that are not sent — so the first streamed
* entry's prev tuple references the entry the standby already holds
* (prev_log_index == standby_lsn). (Skips on the OOM path above set
* it inline before continuing.) */
* it inline before continuing.)
*
* bug-1028: source the term from the AUTHORITATIVE per-LSN term index
* (persist_entry_term), NOT the raw on-disk hdr.term. The standby's
* logOk and the library send path both normalise via persist_entry_term
* (legacy-0 → WAL_LEGACY_TERM); sending raw hdr.term (0 for a legacy
* entry) makes prev_log_term diverge from the standby's term → logOk
* rejects the catch-up forever. p may be NULL only in count-only tests. */
prev_streamed_lsn = hdr.lsn;
prev_streamed_term = hdr.term;
prev_streamed_term = p ? persist_entry_term(p, hdr.lsn) : hdr.term;
off += (uint32_t)(sizeof(hdr) + hdr.payload_len);
}
close(fd);
@ -2240,9 +2259,14 @@ static void catchup_stream_range(cluster_t *c, repl_ep_t ep,
return;
}
#endif
if (c && c->persist.wal_path[0])
if (c && c->persist.wal_path[0]) {
/* bug-1067: pass the leader's CURRENT term so historical catch-up entries
* carry the AE header term, not their (lower) stored term. */
uint64_t leader_term =
atomic_load_explicit(&c->ha.term, memory_order_acquire);
ha_stream_wal_to_standby((endpoint_t *)ep, c->persist.wal_path,
standby_lsn, target);
&c->persist, leader_term, standby_lsn, target);
}
}
/* ha_catchup_continuation_ctx_t — heap context for the one-shot delaymgr task.

23
src/controller/config_repl.h
View file

@ -11,6 +11,12 @@
#include "request.h"
#include "wal_types.h" /* WAL_CONFIG_MARKER, WAL_CONFIG_MARKER_PAYLOAD_LEN */
/* Forward decl: ha_stream_wal_to_standby takes a persist handle by pointer only
* (for the authoritative per-LSN term index via persist_entry_term). Avoids a
* heavy persist.h include in this header. */
struct persist_s;
typedef struct persist_s persist_t;
/* Max config blob size — reject before allocating */
#define CONF_MAX_SIZE (1 * 1024 * 1024) /* 1 MiB */
@ -362,16 +368,31 @@ int handle_upgrade_config(cluster_t *c, endpoint_t *ep, xce_request_t *req);
* Parameters:
* ep — outbound endpoint to the standby
* wal_path — directory containing WAL segment files (seg-*.wal)
* p — persist handle for the AUTHORITATIVE per-LSN term index
* (persist_entry_term). Each streamed entry's term and the
* AppendEntries prev_log_term MUST come from this index, NOT
* the raw on-disk hdr.term — the standby's logOk and the
* library send path both use persist_entry_term, which maps a
* legacy-0 entry to WAL_LEGACY_TERM. Sourcing raw hdr.term (0)
* for a legacy entry diverges from the standby's normalized
* term (1) → logOk rejects forever (bug-1028). May be NULL only
* in count-only unit tests that do not assert terms.
* standby_lsn — standby's last applied LSN (stream starts at standby_lsn+1)
* committed_lsn— primary's committed LSN (stream ends at committed_lsn inclusive)
*
* Returns 0 on success; negative on internal error (e.g., NULL ep).
* Requirements: 5.1, 5.2, 5.4, 5.5 (ha-repl-hardening spec)
* Requirements: 5.1, 5.2, 5.4, 5.5 (ha-repl-hardening spec); bug-1028.
* Boundary: WAL Streaming (config_repl.c)
*/
#define WAL_STREAM_BATCH_ENTRIES 256
#define WAL_STREAM_BATCH_BYTES 65536
/* bug-1067: leader_term is the streaming leader's CURRENT term — stamped onto
* each streamed entry's leader_current_term so the standby intake uses it as the
* AppendEntries header term (UpdateTerm/stale-check), distinct from each entry's
* stored term (logOk/persist). Historical catch-up entries have a stored term
* BELOW leader_term; using the stored term as the AE term stale-rejected them. */
int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
const persist_t *p, uint64_t leader_term,
uint64_t standby_lsn, uint64_t committed_lsn);
#define SNAPSHOT_CHUNK_SIZE 65536

114
src/controller/ha.c
View file

@ -568,6 +568,18 @@ void ha_compute_safe_committed_lsn(ha_elect_ctx_t *ctx,
" (from %d granted voters)", *safe_lsn_out, *safe_tag_out, n_granted);
}
/* ── bug-1060: never truncate the new leader's own durable log ───────────────
*
* The post-election WAL truncation cutoff must never fall below the leader's own
* durable tip, or it destroys entries committed by a prior leader (which this
* node had acked as a standby, and so are committed) — a Raft §5.4 Leader
* Completeness violation that diverges WALs and deadlocks the next election.
* See ha.h for the full rationale. */
uint64_t ha_promotion_truncate_floor(uint64_t safe_lsn, uint64_t durable_tip)
{
return safe_lsn > durable_tip ? safe_lsn : durable_tip;
}
/* ── ha-commit-durability task 4.3: pre-promotion eligibility gate ───────────
*
* ha_promotion_gate_decision: pure decision (Req 2.2, 2.5, 2.6). Build the Boot
@ -939,9 +951,22 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
promotion_safe_lsn = safe_lsn;
/* Store results for use in WAL truncation and post-election commit.
* safe_lsn is the highest LSN on a majority; entries above it
* may not have been committed on all nodes and must be truncated. */
* may not have been committed on all nodes and must be truncated.
*
* bug-1060 (Leader Completeness, Raft §5.4): the truncation cutoff is
* clamped to the leader's OWN durable tip so it never destroys entries
* this node holds durably — those were acked by this node as a standby
* and so were committed by the prior leader. Truncating to the bare
* majority element (which can sit BELOW the leader's durable tip when
* a slower peer is in the granting majority) diverged WALs and dead-
* locked the next election. With the clamp, persist_truncate_wal_above
* is a no-op on the leader's durable log; safe_lsn only governs removal
* of a non-durable divergent tail strictly above the durable tip. */
if (c) {
persist_truncate_wal_above(&c->persist, safe_lsn);
uint64_t durable_tip = persist_durable_floor(&c->persist);
uint64_t trunc_floor =
ha_promotion_truncate_floor(safe_lsn, durable_tip);
persist_truncate_wal_above(&c->persist, trunc_floor);
}
/* Task 4.2 (ha-repl-hardening): set committed_lsn = safe_lsn.
* persist_truncate_wal_above already sets c->persist.committed_lsn
@ -2128,6 +2153,21 @@ static int ha_apply_term_entry_via_library(ha_state_t *ha, persist_t *p,
* e->lsn + 2 and SKIPS the real next-needed entry e->lsn + 1, wedging
* logOk and stalling commit. */
(void)before_term;
/* bug-1043: the library route (on_persist_entry → persist_wal_write_
* replicated) writes the entry to the WAL but does NOT apply it to the
* in-memory entity state (job/queue/worker tables). The legacy direct-
* apply path (ha_apply_single_entry) called apply_wal_entry after the
* persist; the term-safe library route dropped it. So a standby that
* caught up entries 8..N via replication held them on disk but NOT in the
* queryable job table — on promotion `joblist`/`expect-job` could not
* find a committed job (it IS on disk; walcompare passes), looking like a
* committed-job loss. Re-apply here, mirroring the legacy path. Idempotent
* on a re-stream (restore_job upserts by jobid). WAL_NOOP / WAL_CONFIG_
* MARKER carry no entity payload — apply_wal_entry no-ops on them. */
apply_wal_entry(p, (uint32_t)e->entry_type, e->lsn,
e->raw_payload, (uint32_t)e->raw_payload_len);
ha_advance_applied_lsn(ha, e->lsn);
ha_send_wal_ack_reply(ha, ep, c, e->lsn);
return 1;
@ -2323,9 +2363,16 @@ int ha_consume_append_entries(cluster_t *c, wal_catchup_entry_t *entry,
const void *payload = e->raw_payload;
uint32_t plen = (uint32_t)e->raw_payload_len;
entry_term = e->term;
/* ae_term for UpdateTerm is the entry's term (the leader appends at its
* current term, so entry_term == leader term for a live append). */
ae_term = entry_term;
/* bug-1067: the AppendEntries HEADER term (UpdateTerm/stale-check) is the
* leader's CURRENT term, NOT the entry's stored term. For a LIVE append
* the two are equal (the leader appends at its current term), but for a
* CATCH-UP entry written by a PRIOR leader the stored term is LOWER than
* the current leader's term — using it as the AE term made the library
* stale-reject (term < s->term) every historical entry, wedging cross-term
* catch-up forever. The streamer stamps leader_current_term = ha->term;
* fall back to the stored term only when it is unset (legacy sender). */
ae_term = (e->leader_current_term != 0) ? e->leader_current_term
: entry_term;
blob_len = sizeof(raft_entry_blob_t) + plen;
uint8_t *buf = (blob_len <= sizeof(stackbuf)) ? stackbuf
@ -2680,10 +2727,19 @@ int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
if (c && c->persist.wal_path[0]) {
extern int ha_stream_wal_to_standby(endpoint_t *ep,
const char *wal_path,
const persist_t *p,
uint64_t leader_term,
uint64_t standby_lsn,
uint64_t committed_lsn);
ha_stream_wal_to_standby(ep, c->persist.wal_path,
standby_lsn, committed_lsn);
/* bug-1028: pass &c->persist so streamed entry terms come from the
* authoritative per-LSN index (persist_entry_term), matching the
* standby's logOk + the library send path.
* bug-1067: pass the leader's CURRENT term so historical entries
* carry the AE header term (not their lower stored term). */
uint64_t leader_term =
atomic_load_explicit(&c->ha.term, memory_order_acquire);
ha_stream_wal_to_standby(ep, c->persist.wal_path, &c->persist,
leader_term, standby_lsn, committed_lsn);
}
break;
@ -3521,6 +3577,50 @@ void ha_sync_applied_lsn_from_recovery(ha_state_t *ha, persist_t *p)
memory_order_release);
}
/* bug-1042: re-sync the raft library's log_len / commit_index / term from the
* recovered WAL AFTER persist_recover.
*
* Init-ordering hazard: ha_init_raft (myapp.c, before the election) sources
* raft.initial_log_len from persist_durable_floor — but on a STANDBY-REJOIN boot
* persist_recover has not run yet at that point, so the library comes up with
* log_len=0. After cluster_init_phase2 runs persist_recover (recovering e.g.
* durable_floor=7), the library still thinks its log is EMPTY. Every incoming
* AppendEntries then fails logOk (prevLogIndex>=1 against log_len=0 → reject),
* so the rejoined member is PERMANENTLY stuck as a non-converging standby and
* the primary re-streams [tip+1..committed] every heartbeat forever.
*
* Fix: after recovery, raise the library's log_len to the recovered WAL tip and
* its commit_index to the recovered durable floor (clamped to log_len, honoring
* the commit_index<=log_len invariant), and align raft.term with the persisted
* term so the per-LSN (term,index) reconcile reads consistent state. Monotonic:
* never lowers an already-higher library position (e.g. if catch-up advanced it
* before this runs). The library stores no entry bytes — on_get_entry_term reads
* the recovered per-LSN term index — so raising log_len is sufficient for logOk. */
void ha_sync_raft_log_from_recovery(ha_state_t *ha, persist_t *p)
{
if (!ha || ha->raft_c == NULL || !p)
return;
uint64_t recovered_tip = persist_durable_floor(p);
if (recovered_tip > ha->raft.log_len)
ha->raft.log_len = recovered_tip;
/* commit_index advances to the recovered durable floor but never past the
* log tip (library invariant commit_index <= log_len), and never decreases. */
uint64_t recovered_commit = recovered_tip;
if (recovered_commit > ha->raft.log_len)
recovered_commit = ha->raft.log_len;
if (recovered_commit > ha->raft.commit_index)
ha->raft.commit_index = recovered_commit;
/* Align the library term with the persisted/restored HA term so the
* (term,lsn) reconcile and own-term gate read a consistent value after a
* rejoin (ha_restore_term already loaded ha->term before ha_init_raft). */
uint64_t ha_term = atomic_load_explicit(&ha->term, memory_order_acquire);
if (ha_term > ha->raft.term)
ha->raft.term = ha_term;
}
/* ── Vote Handler (Req 2.2–2.6, 6.1, 6.2, 6.4, 9.1) ────────────────────────
*
* handle_request_vote: process an incoming OP_requestVote RPC.

30
src/controller/ha.h
View file

@ -599,6 +599,25 @@ void ha_compute_safe_committed_lsn(ha_elect_ctx_t *ctx,
uint64_t *safe_lsn_out,
uint64_t *safe_tag_out);
/* ── bug-1060: Leader Completeness guard for promotion-time WAL truncation ────
*
* ha_promotion_truncate_floor: clamp the post-election WAL truncation cutoff so
* a newly-elected leader NEVER truncates entries it holds durably.
*
* Returns max(safe_lsn, durable_tip). safe_lsn is the majority-threshold commit
* point from the vote replies (ha_compute_safe_committed_lsn); durable_tip is the
* leader's own durable WAL tip (persist_durable_floor). Because the cutoff is
* never below the leader's durable tip, persist_truncate_wal_above is a no-op on
* the leader's own durable log — preserving every entry committed by a prior
* leader (Raft §5.4 Leader Completeness). safe_lsn only governs removal of a
* divergent, non-durable tail STRICTLY ABOVE the leader's durable tip.
*
* Root cause it fixes: truncating to the bare majority element destroyed
* committed entries the new leader had acked as a standby (granted voters
* {self=47, peer=44} → safe_lsn=44 cut the leader's own committed 45..47),
* producing divergent WALs and a permanent prevote deadlock. */
uint64_t ha_promotion_truncate_floor(uint64_t safe_lsn, uint64_t durable_tip);
/* ── ha-commit-durability task 4.3: pre-promotion eligibility gate ──────────
*
* ha_promotion_gate_decision: PURE pre-promotion eligibility decision (Req 2.2,
@ -957,6 +976,17 @@ uint64_t ha_candidate_advertise_lsn(persist_t *p);
* already-higher applied_lsn (monotonic). */
void ha_sync_applied_lsn_from_recovery(ha_state_t *ha, persist_t *p);
/* ha_sync_raft_log_from_recovery (bug-1042) — after local WAL recovery on a
* (re)start, raise the raft LIBRARY's log_len / commit_index / term to the
* recovered position. ha_init_raft (before the election) sources the library's
* initial_log_len from persist_durable_floor, but on a standby-rejoin boot
* persist_recover runs LATER (cluster_init_phase2), so the library would
* otherwise stay at log_len=0 and reject every incoming AppendEntries (logOk
* fails for prevLogIndex>=1), wedging the rejoined member as a permanently non-
* converging standby. Call this right after persist_recover. Monotonic + NULL-
* safe; clamps commit_index<=log_len. */
void ha_sync_raft_log_from_recovery(ha_state_t *ha, persist_t *p);
/*
* ha_elect_evaluate_and_signal: evaluate accumulated ha_elect_ctx_t votes.
* Sets ctx->result (1=primary, 0=standby) and posts ctx->done semaphore.

6
src/controller/testsuite/test_coldstart_lsn0.c
View file

@ -336,7 +336,8 @@ static void test_h4_empty_standby_receives_lsn1_and_lsn2_no_hole(void)
/* Empty fresh standby = standby_lsn 0; primary committed_lsn = 2 (holds
* [1,2]). Drive the REAL streamer. */
int rc = ha_stream_wal_to_standby(/*ep=*/NULL, tmpdir,
int rc = ha_stream_wal_to_standby(/*ep=*/NULL, tmpdir, /*p=*/NULL,
/*leader_term=*/7,
/*standby_lsn=*/0,
/*committed_lsn=*/2);
xce_wal_stream_request_hook = NULL;
@ -384,7 +385,8 @@ static void test_h4_control_standby_with_lsn1_gets_only_lsn2(void)
h4_capture_reset();
/* standby_lsn = 1 (already holds the first entry) catching up to committed 2. */
int rc = ha_stream_wal_to_standby(/*ep=*/NULL, tmpdir,
int rc = ha_stream_wal_to_standby(/*ep=*/NULL, tmpdir, /*p=*/NULL,
/*leader_term=*/7,
/*standby_lsn=*/1,
/*committed_lsn=*/2);
xce_wal_stream_request_hook = NULL;

93
src/controller/testsuite/test_config_repl.c
View file

@ -2586,7 +2586,8 @@ static void test_wal_stream_500_entries_two_batches(void)
g_wal_stream_last_n_entries = 0;
/* Stream entries [1..500]: standby_lsn=0, committed_lsn=500 */
rc = ha_stream_wal_to_standby(/*ep=*/NULL, tmpdir,
rc = ha_stream_wal_to_standby(/*ep=*/NULL, tmpdir, /*p=*/NULL,
/*leader_term=*/7,
/*standby_lsn=*/0,
/*committed_lsn=*/500);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc, "ha_stream_wal_to_standby failed");
@ -2627,7 +2628,8 @@ static void test_wal_stream_partial_batch(void)
g_wal_stream_last_n_entries = 0;
/* Stream entries [6..14]: standby_lsn=5, committed_lsn=14 */
rc = ha_stream_wal_to_standby(/*ep=*/NULL, tmpdir,
rc = ha_stream_wal_to_standby(/*ep=*/NULL, tmpdir, /*p=*/NULL,
/*leader_term=*/7,
/*standby_lsn=*/5,
/*committed_lsn=*/14);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc, "ha_stream_wal_to_standby failed");
@ -2661,7 +2663,8 @@ static void test_wal_stream_empty_range_no_batches(void)
g_wal_stream_total_entries = 0;
/* standby_lsn == committed_lsn: nothing to stream */
int rc = ha_stream_wal_to_standby(/*ep=*/NULL, tmpdir,
int rc = ha_stream_wal_to_standby(/*ep=*/NULL, tmpdir, /*p=*/NULL,
/*leader_term=*/7,
/*standby_lsn=*/100,
/*committed_lsn=*/100);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc, "Empty range must return 0");
@ -2676,6 +2679,89 @@ static void test_wal_stream_empty_range_no_batches(void)
printf("test_wal_stream_empty_range_no_batches: PASS\n");
}
/*
* bug-1028 — the catch-up streamer must source each streamed entry's term and
* the AppendEntries prev_log_term from the AUTHORITATIVE per-LSN term index
* (persist_entry_term), NOT the raw on-disk hdr.term. A legacy-0 entry has
* hdr.term==0 on disk but persist_entry_term maps it to WAL_LEGACY_TERM(1) —
* which is exactly what the standby's logOk and the library send path compute.
* If the streamer sends the raw 0, prev_log_term(0) != standby's term(1) → logOk
* rejects the new-term entry forever (the "stream WAL batches" wedge).
*
* Fixture: LSN 1 appended at term 0 (legacy), LSN 2 at term 3. Streaming
* (standby_lsn=1, committed=2] must emit LSN 2 with term=3 and prev_log_term=1
* (NOT 0). RED before the fix (streamer sent hdr.term=0 for the predecessor).
*/
static uint64_t g_cap_term_for_lsn2 = 0;
static uint64_t g_cap_prevterm_for_lsn2 = 0;
static int g_cap_saw_lsn2 = 0;
static int capture_terms_wal_stream_hook(endpoint_t *ep, xce_request_t *req)
{
(void)ep;
if (!req || !req->body.walcatchupentrymsg) return 0;
wal_catchup_entry_t *batch =
(wal_catchup_entry_t *)req->body.walcatchupentrymsg;
for (int i = 0; i < batch->n_entries; i++) {
wal_catchup_entry_t *e = batch->entries[i];
if (e && e->lsn == 2) {
g_cap_saw_lsn2 = 1;
g_cap_term_for_lsn2 = e->term;
g_cap_prevterm_for_lsn2 = e->prev_log_term;
}
}
return 0;
}
static void test_wal_stream_term_from_index_not_raw_hdr(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_cr_streamterm_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
cluster_t *c = t42_make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
/* LSN 1: legacy entry (term 0 on disk → persist_entry_term maps to 1). */
persist_set_term(&c->persist, 0);
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(&c->persist, WAL_JOB_UPDATE,
NULL, 0));
/* LSN 2: real new-term entry (term 3). */
persist_set_term(&c->persist, 3);
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(&c->persist, WAL_JOB_UPDATE,
NULL, 0));
/* Sanity: the authoritative index normalises legacy-0 → 1, keeps 3. */
TEST_ASSERT_EQUAL_UINT64(WAL_LEGACY_TERM, persist_entry_term(&c->persist, 1));
TEST_ASSERT_EQUAL_UINT64(3, persist_entry_term(&c->persist, 2));
xce_wal_stream_request_hook = capture_terms_wal_stream_hook;
g_cap_saw_lsn2 = 0; g_cap_term_for_lsn2 = 0; g_cap_prevterm_for_lsn2 = 0;
/* Standby holds LSN 1, catching up to committed 2 — streams just LSN 2,
* whose prev tuple references LSN 1 (the legacy entry). */
int rc = ha_stream_wal_to_standby(/*ep=*/NULL, c->persist.wal_path,
&c->persist,
/*leader_term=*/atomic_load_explicit(
&c->ha.term, memory_order_acquire),
/*standby_lsn=*/1, /*committed_lsn=*/2);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc, "ha_stream_wal_to_standby failed");
xce_wal_stream_request_hook = NULL;
TEST_ASSERT_TRUE_MESSAGE(g_cap_saw_lsn2, "LSN 2 must be streamed");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(3, g_cap_term_for_lsn2,
"streamed entry term must come from persist_entry_term (3)");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(WAL_LEGACY_TERM, g_cap_prevterm_for_lsn2,
"bug-1028: prev_log_term must be the index-normalised legacy term (1), "
"NOT raw on-disk hdr.term (0) — else the standby's logOk rejects forever");
t42_teardown(c);
char cmd[512];
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp);
system(cmd);
printf("test_wal_stream_term_from_index_not_raw_hdr: PASS\n");
}
/* ── Task 3.5: Snapshot chunk transfer ──────────────────────── */
static snapshot_transfer_t g_snaptest_chunks[16];
@ -3363,6 +3449,7 @@ void test_config_repl(void)
RUN_TEST(test_wal_stream_500_entries_two_batches);
RUN_TEST(test_wal_stream_partial_batch);
RUN_TEST(test_wal_stream_empty_range_no_batches);
RUN_TEST(test_wal_stream_term_from_index_not_raw_hdr); /* bug-1028 */
/* Task 3.5: snapshot chunk transfer (sender side) */
RUN_TEST(test_snapshot_transfer_two_chunks);
RUN_TEST(test_snapshot_transfer_small_less_than_one_chunk);

57
src/controller/testsuite/test_ha_election.c
View file

@ -1001,6 +1001,60 @@ static void test_safe_commit_reflects_granting_voter_lsn(void)
printf("test_safe_commit_reflects_granting_voter_lsn: PASS\n");
}
/* ── bug-1060: a new leader must NEVER truncate its own committed entries ─────
*
* Leader Completeness (Raft §5.4): a server that wins an election must already
* hold every entry committed by a prior leader, and it must never truncate its
* own log to "catch down" to a slower majority.
*
* Production root cause (bug-1060): raft_cb_become_leader truncated its WAL down
* to ha_compute_safe_committed_lsn()'s majority-threshold element. When the new
* leader durably holds entries ABOVE that element (it acked them as a standby
* under the prior primary — so they were committed), truncating to the majority
* element physically destroyed those committed entries and rewound commitIndex.
* e.g. granted voters {self=47, peer=44} → median safe_lsn=44, but self holds
* committed entries 45..47 → truncate(44) deletes 45..47. The leader then
* wrote its leadership NOP at LSN 45 (term+1) while the diverged peers still
* held the old job at LSN 45 → permanent prevote deadlock + committed-job loss.
*
* Invariant under test: the promotion truncation floor is clamped to the new
* leader's own durable tip, so persist_truncate_wal_above is a no-op on the
* leader's durable log. safe_lsn only governs removal of entries STRICTLY ABOVE
* the leader's durable tip (non-durable, divergent tail).
*
* RED before the fix: ha_promotion_truncate_floor does not exist (undefined
* reference).
*/
static void test_promotion_never_truncates_own_committed_entries(void)
{
/* The bug-1060 case: majority element (44) is BELOW the leader's own durable
* tip (47). The truncation floor must clamp UP to the durable tip so the
* leader's committed entries 45..47 are never cut. */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
47ULL, ha_promotion_truncate_floor(/*safe_lsn=*/44ULL, /*durable_tip=*/47ULL),
"truncation floor must never fall below the leader's own durable tip "
"(Leader Completeness: a new leader never truncates its own committed log)");
/* Equal: floor stays at the durable tip (no truncation). */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
47ULL, ha_promotion_truncate_floor(47ULL, 47ULL),
"equal safe_lsn and durable tip → floor at the tip (no-op truncation)");
/* safe_lsn ABOVE the durable tip: only a divergent tail strictly above the
* leader's durable log can be removed — safe_lsn governs there. */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
50ULL, ha_promotion_truncate_floor(50ULL, 47ULL),
"when safe_lsn exceeds the durable tip, the floor follows safe_lsn "
"(removes only the non-durable divergent tail above the leader's log)");
/* Single-node / fresh boot: durable tip 0 → floor follows safe_lsn. */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
0ULL, ha_promotion_truncate_floor(0ULL, 0ULL),
"empty log: floor 0 (nothing to truncate)");
printf("test_promotion_never_truncates_own_committed_entries: PASS\n");
}
/* ── Test suite entry point ───────────────────────────────────────────────── */
void test_ha_election(void)
@ -1059,4 +1113,7 @@ void test_ha_election(void)
/* bug-828 — safe-commit-from-votes must reflect the granting voter LSN */
RUN_TEST(test_safe_commit_reflects_granting_voter_lsn);
/* bug-1060 — a new leader must never truncate its own committed entries */
RUN_TEST(test_promotion_never_truncates_own_committed_entries);
}

70
src/controller/testsuite/test_persist_recovery.c
View file

@ -309,6 +309,75 @@ static void test_recovered_member_advertises_real_applied_lsn(void)
printf("test_recovered_member_advertises_real_applied_lsn: PASS\n");
}
/* ------------------------------------------------------------------ */
/* bug-1042: after recovery, the raft LIBRARY's log_len/commit_index */
/* must be raised to the recovered position. ha_init_raft runs BEFORE */
/* persist_recover on a standby-rejoin boot, leaving the library at */
/* log_len=0; without ha_sync_raft_log_from_recovery the library then */
/* rejects every AppendEntries (logOk fail for prevLogIndex>=1) and */
/* the rejoined member never converges. */
/* ------------------------------------------------------------------ */
static void test_recovered_member_syncs_raft_log_len(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_rec1042_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
/* Phase 1: write WAL entries, flush, stop. */
{
persist_t* p = make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
for (int i = 0; i < 5; i++)
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_UPDATE, "z", 1));
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
teardown_persist(p);
}
/* Phase 2: restart — recover the WAL, then mimic the init-ordering hazard:
* the raft library was init'd with log_len=0 BEFORE recovery ran. The sync
* must raise the library's log_len to the recovered durable tip. */
{
persist_t* p2 = make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p2);
TEST_ASSERT_EQUAL_INT(0, persist_recover(p2));
uint64_t recovered_floor = persist_durable_floor(p2);
TEST_ASSERT_TRUE(recovered_floor > 0);
ha_state_t ha;
memset(&ha, 0, sizeof(ha));
/* Library came up empty (ha_init_raft before recovery). */
ha.raft.log_len = 0;
ha.raft.commit_index = 0;
ha.raft.term = 0;
atomic_store_explicit(&ha.term, 4ULL, memory_order_release);
/* The helper requires a non-NULL raft_c back-ref (production wiring). */
ha.raft_c = (cluster_t*)0x1; /* non-NULL sentinel; helper never derefs it */
ha_sync_raft_log_from_recovery(&ha, p2);
/* The bug: log_len used to stay 0 → logOk rejects every AppendEntries. */
TEST_ASSERT_EQUAL_UINT64(recovered_floor, ha.raft.log_len);
/* commit_index advanced to the durable floor, clamped to log_len. */
TEST_ASSERT_EQUAL_UINT64(recovered_floor, ha.raft.commit_index);
TEST_ASSERT_TRUE(ha.raft.commit_index <= ha.raft.log_len);
/* term aligned to the persisted/restored HA term. */
TEST_ASSERT_EQUAL_UINT64(4ULL, ha.raft.term);
/* Monotonic: never lowers an already-higher library position. */
ha.raft.log_len = recovered_floor + 100;
ha.raft.commit_index = recovered_floor + 50;
ha_sync_raft_log_from_recovery(&ha, p2);
TEST_ASSERT_EQUAL_UINT64(recovered_floor + 100, ha.raft.log_len);
TEST_ASSERT_EQUAL_UINT64(recovered_floor + 50, ha.raft.commit_index);
teardown_persist(p2);
}
printf("test_recovered_member_syncs_raft_log_len: PASS\n");
}
void test_persist_recovery(void)
{
RUN_TEST(test_recover_from_empty_state);
@ -317,4 +386,5 @@ void test_persist_recovery(void)
RUN_TEST(test_recover_rejects_corrupt_snapshot);
RUN_TEST(test_recover_picks_latest_snapshot);
RUN_TEST(test_recovered_member_advertises_real_applied_lsn);
RUN_TEST(test_recovered_member_syncs_raft_log_len);
}

137
src/controller/testsuite/test_term_safety_adapter.c
View file

@ -1529,6 +1529,141 @@ static void test_catchup_applied_lsn_is_last_held_no_skip(void)
printf("test_catchup_applied_lsn_is_last_held_no_skip: PASS\n");
}
/* ================================================================== */
/* bug-1067 — cross-term catch-up: the AppendEntries header term is the */
/* leader's CURRENT term, NOT the streamed entry's stored term */
/* ================================================================== */
/* Root cause: ha_consume_append_entries used the entry's stored term (entry_term)
* as the AppendEntries header term. For LIVE appends that is the leader's current
* term (leader appends at its term), but for CATCH-UP streaming of HISTORICAL
* entries written by a PRIOR leader, entry_term < the current leader's term. The
* library's stale-term check (term < s->term → ae_reject_stale) then permanently
* rejected every historical entry, so the lagging standby could never catch up
* (standby_reported_lsn never advanced; the streamer retransmitted forever).
*
* The fix carries the leader's current term out-of-band per entry
* (wal_catchup_entry_t.leader_current_term, WalCatchupEntry.leaderCurrentTerm @8),
* set by the streamer to ha->term, and uses THAT as the AE header term while the
* entry's stored term remains the per-entry term for logOk / on_persist_entry.
*
* RED before the fix: the leader_current_term field does not exist (compile
* error); once added but before the receiver reads it, ae_term == entry_term (4) <
* s->term (5) → stale-reject → LSN 2 never persists. */
static void test_intake_crossterm_catchup_uses_leader_current_term(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_xterm_catchup_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
/* Standby has already adopted the current leader's term (5) from heartbeats. */
make_follower(c, /*term*/5, /*n_peers*/3);
/* Local log holds a HISTORICAL entry LSN1@term4 (written by the prior leader
* at term 4, replicated to this standby). */
uint8_t blob[64];
size_t n = pack_entry(blob, WAL_JOB_SUBMIT, "a", 1);
TEST_ASSERT_EQUAL_INT(0, ha_cb_persist_entry_for_test(&c->ha, 1, 4, blob, n));
c->ha.raft.log_len = 1;
TEST_ASSERT_EQUAL_UINT64(4, persist_entry_term(&c->persist, 1));
/* The CURRENT leader (term 5) streams the historical entry LSN2@term4 as a
* catch-up sub-entry: the entry's STORED term is 4, but the AppendEntries
* header term is the leader's CURRENT term 5 (leader_current_term). prev tuple
* (1,4) matches the local log so logOk holds. */
uint8_t lpay[1] = { 'b' };
wal_catchup_entry_t sub;
memset(&sub, 0, sizeof(sub));
sub.lsn = 2;
sub.entry_type = WAL_JOB_SUBMIT;
sub.term = 4; /* the entry's STORED term (prior leader) */
sub.leader_current_term = 5; /* the streaming leader's CURRENT term */
sub.prev_log_index = 1;
sub.prev_log_term = 4;
sub.raw_payload = lpay;
sub.raw_payload_len = sizeof(lpay);
wal_catchup_entry_t* sub_arr[1] = { &sub };
wal_catchup_entry_t batch;
memset(&batch, 0, sizeof(batch));
batch.n_entries = 1;
batch.entries = sub_arr;
batch.term = 4;
batch.leader_current_term = 5;
batch.prev_log_index = 1;
batch.prev_log_term = 4;
int rc = ha_apply_wal_catchup_entry(&c->ha, &c->persist, &batch, /*ep*/NULL, c);
TEST_ASSERT_EQUAL_INT(0, rc);
/* THE bug-1067 assertion: the historical entry is ACCEPTED (not stale-rejected)
* and persisted at its stored term 4. Pre-fix it was rejected (absent). */
TEST_ASSERT_EQUAL_UINT64(4, persist_entry_term(&c->persist, 2));
/* The agreed prefix (LSN1@term4) is untouched. */
TEST_ASSERT_EQUAL_UINT64(4, persist_entry_term(&c->persist, 1));
/* The leader's current term (5) equals our term — no spurious term change. */
TEST_ASSERT_EQUAL_UINT64(5, c->ha.raft.term);
teardown_cluster(c);
printf("test_intake_crossterm_catchup_uses_leader_current_term: PASS\n");
}
/* Safety guard: a GENUINELY stale leader (leader_current_term < our term) must
* STILL be rejected — the fix must not open a hole letting a deposed leader's
* divergent entries through. */
static void test_intake_stale_leader_term_still_rejected(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_stale_leader_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_follower(c, /*term*/5, /*n_peers*/3);
uint8_t blob[64];
size_t n = pack_entry(blob, WAL_JOB_SUBMIT, "a", 1);
TEST_ASSERT_EQUAL_INT(0, ha_cb_persist_entry_for_test(&c->ha, 1, 4, blob, n));
c->ha.raft.log_len = 1;
/* A deposed leader at term 3 (< our term 5) tries to stream. Even though the
* entry's stored term and prev tuple look locally plausible, the AE header
* term (leader_current_term=3) is stale → reject. */
uint8_t lpay[1] = { 'z' };
wal_catchup_entry_t sub;
memset(&sub, 0, sizeof(sub));
sub.lsn = 2;
sub.entry_type = WAL_JOB_SUBMIT;
sub.term = 3;
sub.leader_current_term = 3; /* STALE leader term < our term 5 */
sub.prev_log_index = 1;
sub.prev_log_term = 4;
sub.raw_payload = lpay;
sub.raw_payload_len = sizeof(lpay);
wal_catchup_entry_t* sub_arr[1] = { &sub };
wal_catchup_entry_t batch;
memset(&batch, 0, sizeof(batch));
batch.n_entries = 1;
batch.entries = sub_arr;
batch.term = 3;
batch.leader_current_term = 3;
batch.prev_log_index = 1;
batch.prev_log_term = 4;
int rc = ha_apply_wal_catchup_entry(&c->ha, &c->persist, &batch, /*ep*/NULL, c);
TEST_ASSERT_EQUAL_INT(0, rc);
/* Stale leader's entry was REJECTED — LSN 2 absent, our term unchanged. */
TEST_ASSERT_EQUAL_UINT64(0, persist_entry_term(&c->persist, 2));
TEST_ASSERT_EQUAL_UINT64(5, c->ha.raft.term);
teardown_cluster(c);
printf("test_intake_stale_leader_term_still_rejected: PASS\n");
}
/* ================================================================== */
/* Task 3.3 — drive raft_on_append_entries_reply + raft_advance_commit */
/* on the replication ACK path */
@ -2065,6 +2200,8 @@ void test_term_safety_adapter(void)
RUN_TEST(test_intake_logmismatch_rejected_no_false_convergence);
RUN_TEST(test_live_broadcast_receive_persists_leader_term);
RUN_TEST(test_catchup_applied_lsn_is_last_held_no_skip);
RUN_TEST(test_intake_crossterm_catchup_uses_leader_current_term);
RUN_TEST(test_intake_stale_leader_term_still_rejected);
RUN_TEST(test_repl_ack_currentterm_quorum_advances_commit);
RUN_TEST(test_repl_ack_priorterm_quorum_holds_commit);
RUN_TEST(test_repl_ack_staleterm_reply_dropped);

17
src/controller/testsuite/waldump.c
View file

@ -14,6 +14,7 @@
#include "config.h" /* WAL_HDR_CRC_SIZE, WAL_CRC_NONE, MAX_WAL_PAYLOAD_BYTES */
#include "controller.capnp.h" /* Job decode for WAL_JOB_SUBMIT */
#include "utils.h" /* fmt_jobid — human-readable JobId formatting */
#include "wal_types.h"
#define WAL_SEG_FMT "%s/seg-%016" PRIx64 ".wal"
@ -303,8 +304,9 @@ static void scan_segment(const char* seg_path, FILE* out,
}
}
/* Per-entry line, with jobid/subtime for submits. */
char info[96] = "";
/* Per-entry line, with jobid/subtime for submits. Sized to hold the
* formatted JobId (fmt_jobid, up to ~40 chars) + subtime + state. */
char info[192] = "";
int is_dup = 0;
int dup_identical = 0;
if (hdr.type == WAL_JOB_SUBMIT) {
@ -312,9 +314,16 @@ static void scan_segment(const char* seg_path, FILE* out,
uint64_t jobid = 0;
if (decode_submit_info(payload, hdr.payload_len, &si) == 0) {
jobid = si.jobid;
/* Render the JobId in the same human-readable form the client
* (joblist / jobsub) prints — fmt_jobid (utils.c) — so a waldump
* jobid can be eyeballed against an `expect-job <id>` / joblist
* entry without converting the raw uint64. */
char jidStr[128];
if (fmt_jobid(jidStr, sizeof(jidStr), si.jobid) < 0)
snprintf(jidStr, sizeof(jidStr), "%" PRIu64, si.jobid);
snprintf(info, sizeof(info),
"jobid=%" PRIu64 " subtime=%" PRIu64 " state=%u",
si.jobid, si.subtime, si.state);
"jobid=%s subtime=%" PRIu64 " state=%u",
jidStr, si.subtime, si.state);
} else {
snprintf(info, sizeof(info), "jobid=?");
}