lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): scan standby_endpoints by capacity, not count (bug-1474)

Browse source 
Root cause of the "spurious duplicate reconnect" (angle a): zarray_del()
NULLs the slot and decrements count without compacting, so a live entry
can sit at any index < caps. Every standby_endpoints loop bounded by
`count` missed it. Live evidence (--config-upgrade-wal-trim soak,
2026-09-26): ctrl-2 won, promoted vote eps [ctrl-1 (dead), ctrl-3]; the
dead one's removal left [NULL, ctrl-3] with count 1, so the primary

- stopped heartbeating ctrl-3 (primary_heartbeat_fn's send loop),
- re-dialed it ("peer[2] not connected -- initiating outbound connect"
  9s later, for a peer that never dropped) and registered a second,
  "late-joiner" channel that burned a MAX_STANDBYS repl slot,
- could not evict the stale channel (on_standby_connected's dedup scan),
- and skipped it in config pushes and the ctrlhello broadcast, where a
  hole below count also handed endpoint_ref() a NULL.

The re-dial landed in the freed slot 0, which is what made ctrl-3 visible
again -- the "spurious" reconnect was masking the lost heartbeats.

All index scans of standby_endpoints (ha.c heartbeat, sweep, dedup, two
removals; cluster.c removal and ctrlhello; config_repl.c push collection)
now run to `caps` and skip NULL slots; the push collects exactly the
non-NULL entries. `count` stays where it means "how many connected".

New test_reconnect_dedups_channel_behind_deleted_slot (RED before: the
stale channel behind the hole was never evicted). Controller suite 1241/0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-27 06:01:11 +00:00
parent 255b0faad1
commit c3a17b19ab
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
4 changed files with 67 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

45
src/controller/testsuite/test_ha_liveness.c
View file

@ -1563,6 +1563,50 @@ static void test_reconnect_preserves_other_peer_channel(void)
printf("test_reconnect_preserves_other_peer_channel: PASS\n");
}
/* bug-1474 (angle a, root cause): zarray_del() leaves a NULL hole and
* decrements count WITHOUT compacting, so a live entry can sit at an index
* >= count. Every standby_endpoints loop bounded by `count` skipped it: after
* peer 1's disconnect removed index 0, peer 2's channel at index 1 became
* invisible -- the primary stopped heartbeating it, the reconnect sweep
* re-dialed it (the "spurious duplicate reconnect"), and this dedup could not
* evict the stale channel, leaving two live channels for one peer. */
static void test_reconnect_dedups_channel_behind_deleted_slot(void)
{
cluster_t c;
myconf_t conf;
own_setup_primary(&c, &conf);
atomic_store(&ha_standby_evicted_count_for_test, 0);
endpoint_t *peer1 = own_make_standby_ep(&c.ha, &c, 1);
zarray_store(c.ha.standby_endpoints, peer1); /* index 0 */
endpoint_t *peer2_old = own_make_standby_ep(&c.ha, &c, 2);
zarray_store(c.ha.standby_endpoints, peer2_old); /* index 1 */
/* peer 1 disconnects: its slot becomes a hole, count 2 -> 1 */
for (int i = 0; i < c.ha.standby_endpoints->caps; i++) {
if (zarray_get(c.ha.standby_endpoints, i) == peer1) {
zarray_del(c.ha.standby_endpoints, i);
break;
}
}
endpoint_t *peer2_new = own_make_standby_ep(&c.ha, &c, 2);
on_standby_connected_for_test(peer2_new);
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
atomic_load(&ha_standby_evicted_count_for_test),
"bug-1474: the stale channel for peer 2 (behind the deleted slot) "
"must be found and evicted");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, c.ha.standby_endpoints->count,
"bug-1474: exactly one live channel per peer");
standby_ep_err_handler_primary_for_test(peer1);
standby_ep_err_handler_primary_for_test(peer2_new);
fini_ha_primary_with_standby_eps(&c.ha);
}
/* ═══════════════════════════════════════════════════════════════════════════
* raft-drive-async Task 3.1 — PeerCtrlDispatch
*
@ -2419,6 +2463,7 @@ void test_ha_liveness(void)
RUN_TEST(test_reconnect_resolves_dual_channel_same_peer);
RUN_TEST(test_reconnect_resolves_dual_channel_promoted_vote_ep);
RUN_TEST(test_reconnect_preserves_other_peer_channel);
RUN_TEST(test_reconnect_dedups_channel_behind_deleted_slot);
/* raft-drive-async Task 2.3 — reconnect-offload INTEGRATION (Req 2.1, 2.2, 2.5) */
RUN_TEST(test_reconnect_offload_sustains_survivor_heartbeats);
RUN_TEST(test_reconnect_inline_stalls_survivor_heartbeats_RED);