fix(controller): scan standby_endpoints by capacity, not count (bug-1474)
Root cause of the "spurious duplicate reconnect" (angle a): zarray_del()
NULLs the slot and decrements count without compacting, so a live entry
can sit at any index < caps. Every standby_endpoints loop bounded by
`count` missed it. Live evidence (--config-upgrade-wal-trim soak,
2026-09-26): ctrl-2 won, promoted vote eps [ctrl-1 (dead), ctrl-3]; the
dead one's removal left [NULL, ctrl-3] with count 1, so the primary
- stopped heartbeating ctrl-3 (primary_heartbeat_fn's send loop),
- re-dialed it ("peer[2] not connected -- initiating outbound connect"
9s later, for a peer that never dropped) and registered a second,
"late-joiner" channel that burned a MAX_STANDBYS repl slot,
- could not evict the stale channel (on_standby_connected's dedup scan),
- and skipped it in config pushes and the ctrlhello broadcast, where a
hole below count also handed endpoint_ref() a NULL.
The re-dial landed in the freed slot 0, which is what made ctrl-3 visible
again -- the "spurious" reconnect was masking the lost heartbeats.
All index scans of standby_endpoints (ha.c heartbeat, sweep, dedup, two
removals; cluster.c removal and ctrlhello; config_repl.c push collection)
now run to `caps` and skip NULL slots; the push collects exactly the
non-NULL entries. `count` stays where it means "how many connected".
New test_reconnect_dedups_channel_behind_deleted_slot (RED before: the
stale channel behind the hole was never evicted). Controller suite 1241/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
255b0faad1
commit
c3a17b19ab
4 changed files with 67 additions and 9 deletions
|
|
@ -88,7 +88,7 @@ static void handle_endpoint_error(endpoint_t* ep)
|
|||
if (c->ha.standby_endpoints) {
|
||||
pthread_mutex_lock(&c->ha.lock);
|
||||
for (int idx = 0;
|
||||
idx < c->ha.standby_endpoints->count; idx++) {
|
||||
idx < c->ha.standby_endpoints->caps; idx++) {
|
||||
endpoint_t *se = (endpoint_t *)zarray_get(
|
||||
c->ha.standby_endpoints, idx);
|
||||
if (se == ep) {
|
||||
|
|
@ -2424,7 +2424,8 @@ void ha_send_ctrlhello_to_all_standbys(cluster_t *c)
|
|||
char *my_name = c->myconf ? c->myconf->name : "";
|
||||
|
||||
pthread_mutex_lock(&ha->lock);
|
||||
int n = ha->standby_endpoints->count;
|
||||
/* bug-1474: zarray_del leaves holes, so scan caps, not count. */
|
||||
int n = ha->standby_endpoints->caps;
|
||||
for (int i = 0; i < n; i++) {
|
||||
endpoint_t *ep = (endpoint_t *)zarray_get(ha->standby_endpoints, i);
|
||||
if (!ep) continue;
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue