fix(controller): clamp committed_config_tag to own loaded config on promotion

bug-1462: raft_cb_become_leader baselined committed_config_tag to the
majority-safe floor across voters, correct for WAL LSN safety but wrong
for config tags — a leader's own actively-loaded config isn't
speculative like an unacknowledged WAL entry. On a fresh deploy every
peer reports tag=0 except the new primary, so the floor stuck at 0
forever and late-joining standbys never received a config push.

ha_promotion_config_tag_floor() clamps to max(safe_tag, self_tag),
mirroring bug-1060's WAL-truncation clamp pattern. Validated with
20-round sequential (63 jobs, zero loss) and 20-round chaotic
overlapping-kill (zero committed divergence, zero committed job loss)
HA failover soaks; unit suite 977/977 green.
This commit is contained in:
Rongsong Shen 2026-07-23 04:23:50 +00:00
parent ee8473bd99
commit 0649e17f62
3 changed files with 106 additions and 2 deletions

View file

@ -1216,6 +1216,69 @@ static void test_promotion_never_truncates_own_committed_entries(void)
printf("test_promotion_never_truncates_own_committed_entries: PASS\n");
}
/* ── bug-1462: a new leader must never floor its OWN config below what it
* already has loaded ───────────────────────────────────────────────────────
*
* ha_compute_safe_committed_lsn's majority-floor math is correct for WAL LSN
* (an unacknowledged entry is genuinely speculative) but wrong for config_tag:
* the new primary's own actively-loaded config is not speculative the way an
* unacknowledged WAL entry is.
*
* Production root cause (bug-1462): on a cluster where no live admin config
* push has ever happened (deploy-ha.sh seeds cluster.bin on disk pre-boot),
* every peer voter reports config_tag=0 while the new primary's own self_tag
* is a real, non-zero, currently-loaded tag. The majority floor picks 0, so
* raft_cb_become_leader baselines ha->committed_config_tag at 0. The one-shot
* promotion push still gets already-connected standbys the real config (it
* reads myconf directly), but any LATE-joining standby depends entirely on
* the heartbeat-driven `standby_tag < committed_tag` decision loop — which
* can never fire again for the rest of the term because committed_tag is
* stuck at 0. Config replication silently stops while WAL/job replication
* (which floors correctly, since LSN 0 IS legitimately unacknowledged)
* continues to work — exactly the observed symptom.
*
* Fix (mirrors bug-1060's ha_promotion_truncate_floor precedent): clamp the
* post-election config_tag baseline to max(majority_safe_tag, self_tag).
*
* RED before the fix: ha_promotion_config_tag_floor does not exist (undefined
* reference).
*/
static void test_promotion_config_tag_never_drops_below_own_loaded_config(void)
{
/* The bug-1462 case: majority floor (0) is BELOW the leader's own loaded
* tag (0x860cfd4f59ef9381) — no peer has ever received a live config push,
* but the leader itself demonstrably has one. The floor must clamp UP to
* self_tag so the heartbeat decision loop can still push it to late
* joiners for the rest of the term. */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
0x860cfd4f59ef9381ULL,
ha_promotion_config_tag_floor(/*safe_tag=*/0ULL,
/*self_tag=*/0x860cfd4f59ef9381ULL),
"config_tag floor must never fall below the leader's own loaded config "
"(a node's actively-loaded config is not speculative like an "
"unacknowledged WAL entry)");
/* Equal: floor stays at self_tag (no-op). */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
100ULL, ha_promotion_config_tag_floor(100ULL, 100ULL),
"equal safe_tag and self_tag -> floor at that tag");
/* safe_tag ABOVE self_tag: a majority genuinely holds a newer config than
* this node loaded (e.g. this node was behind before winning) — follow
* the majority-safe tag, same as the LSN case follows safe_lsn above the
* durable tip. */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
200ULL, ha_promotion_config_tag_floor(200ULL, 100ULL),
"when safe_tag exceeds self_tag, the floor follows the majority-safe tag");
/* Fresh boot / never configured: both zero -> floor 0. */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
0ULL, ha_promotion_config_tag_floor(0ULL, 0ULL),
"no config anywhere yet: floor 0");
printf("test_promotion_config_tag_never_drops_below_own_loaded_config: PASS\n");
}
/* ── Test suite entry point ───────────────────────────────────────────────── */
void test_ha_election(void)
@ -1252,6 +1315,10 @@ void test_ha_election(void)
/* Req 2.8 — zero config_tag candidate must not win when peer has config */
RUN_TEST(test_zero_config_tag_candidate_does_not_win_when_peer_has_config);
/* bug-1462 — post-election config_tag floor must never drop below the
* leader's own loaded config */
RUN_TEST(test_promotion_config_tag_never_drops_below_own_loaded_config);
/* Req 2.9 — single-node self-election */
RUN_TEST(test_single_node_self_elects_as_primary);