lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): probe a late-joining standby at once so its catch-up starts without waiting a heartbeat (bug-1913)

Browse source 
After an election where the vote connections are not promoted, both
standbys register later via on_standby_connected; the own-term no-op was
appended before any standby existed, and catch-up waited for heartbeat-
driven state reports (config push one 3 s cycle, WAL stream the next), so
the no-op committed ~6 s late and every client write in that window
failed -110.

ha_send_catchup_probe() sends the ordinary heartbeat request (same fields
as primary_heartbeat_fn; encode + enqueue only; primary only) at
registration and once after the standby's first accepted config-push ACK,
so the decision loop's next action follows at ~RTT. The decision loop,
its one-action-per-report pacing, commit/majority rules and the dispatch
gate are unchanged; WAL streaming still runs only on the vtask. The
one-shot arm is cleared on disconnect and on every become_leader /
become_follower.

Tests: test_late_joiner_catchup_kick.c (6). Controller suite 1285/0.
Live: catch-up starts 0-6 ms after a late registration (was up to ~6 s);
/xce-test-ha sequential 20 rounds, chaotic 6 rounds and sequential 8
rounds under 40-spinner load all PASS with zero job loss.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-03 23:18:10 +00:00
parent 6d9ef3d024
commit 2aefa939db
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
5 changed files with 485 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

123
src/controller/ha.c
View file

@ -541,7 +541,20 @@ static void ha_arm_own_term_gate(cluster_t *c);
int ha_heartbeat_is_fresh(ha_state_t *ha); /* exposed for PreVote denial-reason logging (cluster.c) */
static int primary_heartbeat_fn(void *arg);
static void on_standby_connected(endpoint_t *ep);
/* bug-1913: drop every armed catch-up probe. An arm belongs to one standby
* registration under one leadership; a role or term change, or the
* standby's disconnect, ends that registration, so a stale arm must not
* fire an extra probe on some later, unrelated config-push ACK. */
static void ha_clear_catchup_kicks(ha_state_t *ha)
{
for (int i = 0; i < MAX_HA_CANDIDATES; i++)
atomic_store_explicit(&ha->catchup_kick_armed[i], 0,
memory_order_release);
}
static int standby_ep_err_handler_primary(void *ep_raw);
static int ha_send_catchup_probe(cluster_t *c, endpoint_t *ep, int peer_idx,
const char *why);
/* ── ha-elect-ctx-lifecycle task 2.1: election carrier reference-counted
* lifetime ──────────────────────────────────────────────────────────────
@ -1631,6 +1644,7 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
/* Sync ha->term from raft term (raft owns term increments) */
atomic_store_explicit(&ha->term, term, memory_order_release);
ha_clear_catchup_kicks(ha); /* bug-1913: arms are per leadership */
/* The cluster committed high-water mark for this promotion — derived from
* the voter replies (NOT a stale ctrlhello). The task-4.3 pre-promotion
@ -2180,6 +2194,7 @@ static void raft_cb_become_follower(void *ctx, uint64_t term, int leader_idx)
/* Sync ha->term */
atomic_store_explicit(&ha->term, term, memory_order_release);
ha_clear_catchup_kicks(ha); /* bug-1913: arms are per leadership */
int is_first_boot = !ha->phase2_done;
@ -7031,6 +7046,28 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx,
repl_on_config_ack(&c->persist.repl, ep, acked_tag);
TRACE("repl_ack_decode_and_dispatch: config ACK from standby"
" tag=%016" PRIx64 " accepted=1", acked_tag);
/* bug-1913: completion-driven catch-up for a freshly
* registered standby. The decision loop takes one action
* per state report (config before WAL); the next report used
* to come only from the next heartbeat tick, so the WAL
* stream carrying the own-term no-op trailed the config push
* by a full heartbeat cycle. Now that the standby confirmed
* the config, request its state report once. One-shot per
* registration (catchup_kick_armed), so a standby whose
* reported tag stays behind cannot loop probe → push → ACK;
* it falls back to the periodic heartbeat. Runs on the
* peer-control vtask; the probe is a non-blocking enqueue. */
int kick_idx = ha_repl_ack_abs_peer_idx(
ha, /*ep_is_vote*/ vctx->ctx != NULL, vctx->peer_idx);
if (kick_idx >= 0 && kick_idx < MAX_HA_CANDIDATES) {
int armed = 1;
if (atomic_compare_exchange_strong_explicit(
&ha->catchup_kick_armed[kick_idx], &armed, 0,
memory_order_acq_rel, memory_order_acquire))
(void)ha_send_catchup_probe(c, ep, kick_idx,
"config push acked");
}
} else if (!accepted) {
WARN("repl_ack_decode_and_dispatch: config push rejected by"
" standby (tag=%016" PRIx64 ") — ignoring", acked_tag);
@ -9666,6 +9703,86 @@ static void on_standby_connected(endpoint_t *ep)
/* Set state-aware dispatcher for subsequent messages on this endpoint */
ep->msg_handler = peer_ctrl_msg_handler;
/* bug-1913: request the new standby's state report NOW instead of on the
* next heartbeat tick. A standby registering here (late-joiner election:
* no vote ep was promoted; or a reconnect) missed every entry broadcast
* while it was unregistered — in particular the own-term WAL_NOOP, which
* repl_broadcast dropped ("no standbys registered") and repl_add_standby
* does not resend. Only the decision loop (heartbeat reply → state
* report → ha_dispatch_repl_action_for_standby) catches it up, and the
* first report used to wait for the next tick (last_push_time = now above
* can push it to the tick after), keeping committed_lsn below the no-op
* — and every client write NACKed — for up to two heartbeat cycles.
* Same non-blocking heartbeat send the tick uses (enqueue only — allowed
* on this iomgr callback, like the ctrlhello above); sent after
* msg_handler is set so the reply is routed to the decision loop. Also
* arm the one-shot ACK-driven probe (repl_ack_decode_and_dispatch's
* config-ACK branch) so the WAL stream follows a config push at ~RTT. */
if (sctx->peer_idx >= 0 && sctx->peer_idx < MAX_HA_CANDIDATES)
atomic_store_explicit(&ha->catchup_kick_armed[sctx->peer_idx], 1,
memory_order_release);
(void)ha_send_catchup_probe(c, ep, sctx->peer_idx, "standby registered");
}
/*
* ha_send_catchup_probe — bug-1913: send one out-of-cycle heartbeat to a
* standby so its reply (RequestReply.stateReport) drives the replication
* decision loop (repl_ack_decode_and_dispatch → ha_dispatch_repl_action_for_
* standby) now rather than on the next periodic tick.
*
* It is exactly the heartbeat primary_heartbeat_fn sends — the primary's own
* current term / committed_lsn / committed_config_tag / elected_at_ms, read
* atomically — so the standby handles it like any heartbeat; no new message,
* opcode or decision path. Non-blocking: xce_ha_heartbeat_req only encodes
* and enqueues on the endpoint's send queue (safe on iomgr and vtask
* threads). Only a CTRL_PRIMARY heartbeats its standbys; a node demoted
* before this runs sends nothing. Returns the send rc, -EPERM when not
* primary, -EINVAL on bad args. Failure is logged and otherwise harmless:
* the periodic heartbeat remains the fallback driver.
*/
static int ha_send_catchup_probe(cluster_t *c, endpoint_t *ep, int peer_idx,
const char *why)
{
if (!c || !ep)
return -EINVAL;
ha_state_t *ha = &c->ha;
if ((ctrl_role_t)atomic_load_explicit(&ha->role, memory_order_acquire)
!= CTRL_PRIMARY)
return -EPERM;
uint64_t term = atomic_load_explicit(&ha->term, memory_order_acquire);
uint64_t committed_lsn =
atomic_load_explicit(&c->persist.committed_lsn, memory_order_acquire);
uint64_t committed_tag =
atomic_load_explicit(&ha->committed_config_tag, memory_order_acquire);
uint64_t elected_at_ms =
atomic_load_explicit(&ha->last_known_elected_at_ms,
memory_order_acquire);
int rc;
#ifdef XCE_TESTING
if (ha_heartbeat_req_hook)
rc = ha_heartbeat_req_hook(ep, term, committed_lsn, committed_tag,
elected_at_ms);
else if (ha_nop_req_hook)
rc = ha_nop_req_hook(ep);
else
rc = xce_ha_heartbeat_req(ep, term, committed_lsn, committed_tag,
elected_at_ms);
#else
rc = xce_ha_heartbeat_req(ep, term, committed_lsn, committed_tag,
elected_at_ms);
#endif
if (rc < 0)
WARN("HA: catch-up probe (%s) to standby peer[%d] failed: %d — "
"periodic heartbeat remains the fallback", why, peer_idx, rc);
else
INFO("HA: catch-up probe (%s) sent to standby peer[%d] "
"(term=%" PRIu64 " committed_lsn=%" PRIu64 ")",
why, peer_idx, term, committed_lsn);
return rc;
}
/*
@ -9710,6 +9827,12 @@ static int standby_ep_err_handler_primary(void *ep_raw)
if (c)
repl_remove_standby(&c->persist.repl, ep);
/* bug-1913: this registration is over; its catch-up arm must not fire
* on a later, unrelated config-push ACK. */
if (sctx->peer_idx >= 0 && sctx->peer_idx < MAX_HA_CANDIDATES)
atomic_store_explicit(&ha->catchup_kick_armed[sctx->peer_idx], 0,
memory_order_release);
/* bug-1839: do not free sctx / null ep->data synchronously here — a
* peer-control vtask for this same ep may already be queued or in
* flight (peer_ctrl_msg_handler_inline reads ep->data), same hazard