lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): scan standby_endpoints by capacity, not count (bug-1474)

Browse source 
Root cause of the "spurious duplicate reconnect" (angle a): zarray_del()
NULLs the slot and decrements count without compacting, so a live entry
can sit at any index < caps. Every standby_endpoints loop bounded by
`count` missed it. Live evidence (--config-upgrade-wal-trim soak,
2026-09-26): ctrl-2 won, promoted vote eps [ctrl-1 (dead), ctrl-3]; the
dead one's removal left [NULL, ctrl-3] with count 1, so the primary

- stopped heartbeating ctrl-3 (primary_heartbeat_fn's send loop),
- re-dialed it ("peer[2] not connected -- initiating outbound connect"
  9s later, for a peer that never dropped) and registered a second,
  "late-joiner" channel that burned a MAX_STANDBYS repl slot,
- could not evict the stale channel (on_standby_connected's dedup scan),
- and skipped it in config pushes and the ctrlhello broadcast, where a
  hole below count also handed endpoint_ref() a NULL.

The re-dial landed in the freed slot 0, which is what made ctrl-3 visible
again -- the "spurious" reconnect was masking the lost heartbeats.

All index scans of standby_endpoints (ha.c heartbeat, sweep, dedup, two
removals; cluster.c removal and ctrlhello; config_repl.c push collection)
now run to `caps` and skip NULL slots; the push collects exactly the
non-NULL entries. `count` stays where it means "how many connected".

New test_reconnect_dedups_channel_behind_deleted_slot (RED before: the
stale channel behind the hole was never evicted). Controller suite 1241/0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-27 06:01:11 +00:00
parent 255b0faad1
commit c3a17b19ab
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
4 changed files with 67 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

11
src/controller/ha.c
View file

@ -7775,7 +7775,7 @@ static void vote_ep_handle_promoted_disconnect(ha_state_t *ha, endpoint_t *ep,
/* Promoted connection dropped — remove from standby_endpoints */
if (ha->standby_endpoints) {
pthread_mutex_lock(&ha->lock);
for (int i = 0; i < ha->standby_endpoints->count; i++) {
for (int i = 0; i < ha->standby_endpoints->caps; i++) { /* bug-1474 */
if (zarray_get(ha->standby_endpoints, i) == ep) {
zarray_del(ha->standby_endpoints, i);
break;
@ -9464,7 +9464,7 @@ static void on_standby_connected(endpoint_t *ep)
int votes_idx_for_new = votes_idx_for_peer(ha, sctx->peer_idx);
pthread_mutex_lock(&ha->lock);
int already = 0;
for (int i = 0; i < ha->standby_endpoints->count; i++) {
for (int i = 0; i < ha->standby_endpoints->caps; i++) { /* bug-1474 */
endpoint_t *cur = (endpoint_t *)zarray_get(ha->standby_endpoints, i);
if (cur == ep) { already = 1; continue; }
/* Different ep, same peer → stale duplicate to evict. */
@ -9599,7 +9599,7 @@ static int standby_ep_err_handler_primary(void *ep_raw)
if (ha->standby_endpoints) {
pthread_mutex_lock(&ha->lock);
for (int i = 0; i < ha->standby_endpoints->count; i++) {
for (int i = 0; i < ha->standby_endpoints->caps; i++) { /* bug-1474 */
if (zarray_get(ha->standby_endpoints, i) == ep) {
zarray_del(ha->standby_endpoints, i);
break;
@ -9959,7 +9959,8 @@ static int primary_heartbeat_fn(void *arg)
int n_evicted = 0;
pthread_mutex_lock(&ha->lock);
int n = ha->standby_endpoints->count;
/* bug-1474: zarray_del leaves holes, so scan caps, not count. */
int n = ha->standby_endpoints->caps;
for (int i = 0; i < n; i++) {
endpoint_t *ep = (endpoint_t *)zarray_get(ha->standby_endpoints, i);
if (!ep) continue;
@ -10045,7 +10046,7 @@ static int primary_heartbeat_fn(void *arg)
int found = 0;
if (ha->standby_endpoints) {
pthread_mutex_lock(&ha->lock);
for (int j = 0; j < ha->standby_endpoints->count; j++) {
for (int j = 0; j < ha->standby_endpoints->caps; j++) { /* bug-1474 */
endpoint_t *sep = (endpoint_t *)zarray_get(
ha->standby_endpoints, j);
if (!sep || !sep->data) continue;