lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha-term-safety): bug-828 partial — revive dead safe-commit-from-votes + advertise real durable tip

Browse source 
Clean failover was truncating committed jobs because the post-election
safe-commit-from-votes computation was dead: vote_reply_dispatch decoded each
granting voter's advertised peerLastLogLsn but discarded it, so ha_compute_safe
_committed_lsn saw only the self-vote (the 'from 1 granted voters' fingerprint)
and truncated to self_lsn.

Part 1: vote_reply_dispatch now records the granting voter's (lsn,tag,grant) into
ctx->votes[] via a record-only ha_record_vote_reply seam (no double-promotion;
raft_on_vote_reply stays the sole production promotion driver) — so safe-commit
reflects the true voted majority.
Part 2: the candidate/voter advertise the real DURABLE (fsync'd) tip; persist_wal
_write_replicated advances durable_lsn after its fail-loud inline fdatasync.

Reviewer-verified SAFE (advertises only durable positions; safe-commit bounded by
the durable majority, preserving more committed entries, never marking a
non-majority-durable LSN safe). 177 unit tests; new test_ha_vote_gate + election
tests green.

PARTIAL: does NOT yet make the live clean-failover pass. A deeper slice of
bug-828/795 remains — a catch-up'd standby still under-advertises its durable
position in the live run (the documented architectural catch-up/durability gap),
so a job-less node can still win and truncate. Tracked for a dedicated effort.
This commit is contained in:
Rongsong Shen 2026-06-23 03:13:02 +00:00
parent 14c41cb298
commit 91b9a445f0
6 changed files with 313 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

77
src/controller/testsuite/test_ha_election.c
View file

@ -116,6 +116,17 @@ void handle_request_vote_reply(ha_state_t *ha, ha_elect_ctx_t *ctx,
int vote_granted,
uint64_t peer_config_tag, uint64_t peer_lsn);
/* bug-828: record-only vote-reply seam. Records ctx->votes[peer_idx] (granted
* flag + peer_tag + peer_lsn) under ctx->lock with the idempotency guard, but
* does NOT call ha_elect_evaluate_and_signal (the raft library owns promotion
* in production). This is the helper the production vote_reply_dispatch path
* now invokes so the granting voter's advertised LSN reaches the election ctx
* and ha_compute_safe_committed_lsn can compute the true majority-held LSN. */
void ha_record_vote_reply(ha_state_t *ha, ha_elect_ctx_t *ctx,
int peer_idx, uint64_t term,
int vote_granted,
uint64_t peer_config_tag, uint64_t peer_lsn);
/* ── Minimal ha_config_t used across all election tests ─────────────────── */
static ha_config_t g_elect_cfg = {
@ -927,6 +938,69 @@ static void test_reply_handler_all_denied_posts_done_on_last_reply(void)
printf("test_reply_handler_all_denied_posts_done_on_last_reply: PASS\n");
}
/* ── bug-828: safe-commit-from-votes must reflect the voted majority ─────────
*
* Production root cause: vote_reply_dispatch decoded the granting voter's
* advertised LSN but discarded it, so ctx->votes[i].vote_granted stayed 0 and
* ha_compute_safe_committed_lsn saw only the self-vote (n_granted=1) →
* safe_committed_lsn = self_lsn. A standby promoting with self_lsn=1 then
* truncated committed entries 2..5.
*
* This test drives the record-only seam (ha_record_vote_reply) that the
* production path now invokes: two peers GRANT advertising peer_lsn=5; the
* candidate's own self_lsn=1. ha_compute_safe_committed_lsn must return the
* majority-held LSN (5), NOT self_lsn (1).
*
* RED before the fix: ha_record_vote_reply does not exist (undefined reference);
* and even if grants were not recorded, safe_committed would be self_lsn=1.
*/
static void test_safe_commit_reflects_granting_voter_lsn(void)
{
/* 3-node cluster: n_peers=2 other peers. */
ha_state_t ha;
ha_init(&ha, &g_elect_cfg, 0);
atomic_store_explicit(&ha.term, 1ULL, memory_order_release);
ha_elect_ctx_t *ctx = alloc_elect_ctx(2);
/* Candidate's own (boot) position: durable floor 1, the catch-up node case. */
ctx->self_lsn = 1ULL;
ctx->self_tag = 100ULL;
ctx->votes[0].my_tag = 100ULL;
ctx->votes[0].my_lsn = 1ULL;
/* Both peers GRANT, each advertising a durable position of LSN 5
* (they hold the committed prefix). Record via the production seam. */
ha_record_vote_reply(&ha, ctx, 0, 1ULL, 1, 100ULL, 5ULL);
ha_record_vote_reply(&ha, ctx, 1, 1ULL, 1, 100ULL, 5ULL);
/* The recording must NOT have signalled the legacy promotion path —
* ha_record_vote_reply records only; the raft library owns promotion.
* ctx->done must remain unposted (no double-promotion). */
int rc = sem_trywait(&ctx->done);
TEST_ASSERT_EQUAL_INT_MESSAGE(-1, rc,
"ha_record_vote_reply must NOT post ctx->done (record-only; no double-promotion)");
TEST_ASSERT_EQUAL_INT(EAGAIN, errno);
/* Grants must be recorded into the election context. */
TEST_ASSERT_EQUAL_INT(1, ctx->votes[0].vote_granted);
TEST_ASSERT_EQUAL_UINT64(5ULL, ctx->votes[0].peer_lsn);
TEST_ASSERT_EQUAL_INT(1, ctx->votes[1].vote_granted);
TEST_ASSERT_EQUAL_UINT64(5ULL, ctx->votes[1].peer_lsn);
/* The binding assertion: safe_committed_lsn = majority-held LSN = 5,
* NOT self_lsn (1). granted={self=1, peer0=5, peer1=5} → median = 5. */
uint64_t safe_lsn = 0;
uint64_t safe_tag = 0;
ha_compute_safe_committed_lsn(ctx, &safe_lsn, &safe_tag);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(5ULL, safe_lsn,
"safe_committed_lsn must reflect the majority-durable LSN (5), not self_lsn (1)");
free_elect_ctx(ctx);
ha_fini(&ha);
printf("test_safe_commit_reflects_granting_voter_lsn: PASS\n");
}
/* ── Test suite entry point ───────────────────────────────────────────────── */
void test_ha_election(void)
@ -982,4 +1056,7 @@ void test_ha_election(void)
RUN_TEST(test_reply_handler_quorum_reached_posts_done);
RUN_TEST(test_reply_handler_partial_replies_do_not_post_done);
RUN_TEST(test_reply_handler_all_denied_posts_done_on_last_reply);
/* bug-828 — safe-commit-from-votes must reflect the granting voter LSN */
RUN_TEST(test_safe_commit_reflects_granting_voter_lsn);
}