lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): standby dispatch-gate rejections now redirect to a known primary (bug-1700, bug-1701)

Browse source 
xceadmin scheduler record status/start/stop against a standby failed
with the bare "controller returned status -10014" (E_XCE_STANDBY) with
no redirect, forcing the caller to guess the primary manually.

bug-1700: myreq_handler's generic ha_dispatch_gate rejection path
(cluster.c) now calls fill_redirect_reply() when rc == -XCE_ERR_STANDBY,
mirroring the same helper OPCode_join's handler already uses --
-XCE_ERR_NOT_READY (CTRL_INIT/CTRL_ELECTING, no settled primary to
redirect to) is deliberately excluded. Landing this alone had zero
observable effect live: ha_state_t.primary_host/primary_port were never
written by any production path on a standby (only test fixtures set
them) -- fill_redirect_reply() always saw an empty/zero redirect target
in a real deployment.

bug-1701: closes that data gap with a pure local lookup, no new wire
field. handle_ctrlhello() (config_repl.c) now resolves the ctrlhello
sender's name against this node's own statically-known peers_cfg
(parsed once from --peers at boot) and populates primary_host/port at
the exact moment a standby first learns who primary is.

Live-verified end to end on a fresh 3-node cluster: xceadmin scheduler
record status against any standby now prints "redirected: <standby> ->
<primary>" and returns the real status, confirmed again after a
re-election changed which node was primary.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-27 08:25:55 +00:00
parent ac1e312e2f
commit f886c80930
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
4 changed files with 305 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

54
src/controller/config_repl.c
View file

@ -2164,6 +2164,9 @@ char *sanitise_cn(const char *src, size_t maxlen)
* Steps:
* 1. Validate request body
* 2. Set primary_ctrl_fd = ep->fd (first ctrlhello from winner, idempotent)
* 2b. (bug-1701) Look up the sender's name against our own static
* peers_cfg candidate list and copy its host:port into
* ha->primary_host/primary_port
* 3. Call ha_on_heartbeat_received (resets liveness watchdog)
* 4. Arm ha_arm_heartbeat_timeout_vtask
*/
@ -2183,6 +2186,57 @@ int handle_ctrlhello(cluster_t *c, endpoint_t *ep, xce_request_t *req)
memory_order_acquire) == CTRL_STANDBY &&
c->ha.primary_ctrl_fd == -1) {
c->ha.primary_ctrl_fd = ep->fd;
/* bug-1701: ha->primary_host/primary_port (aliased into a redirect
* reply by fill_redirect_reply(), join.c:27) were never written by
* any production path, so bug-1700's redirect fix was a live no-op.
* Fix is a pure LOCAL lookup — no wire-protocol change needed: every
* node already knows every candidate's host:port from boot
* (c->myconf->peers_cfg, parsed once from --peers). This is the
* moment a standby first learns "I am now connected to my primary";
* req->from->name carries the sender's name= identity (set for
* every incoming request at request.c:379, "req->from = ep->peer;"),
* which is the SAME name used in --peers name=host:port. Match it
* against our own candidate list (mirrors the lookup-by-name pattern
* already used for voted_for_name in ha_init_raft, ha.c:7822-7829)
* and copy that candidate's hostname/port. On no match (real config
* mismatch) or a NULL req->from/name, leave primary_host/primary_port
* at their previous value — a stale-but-previously-correct target is
* safer than wiping a working redirect on a transient miss — and log
* a WARN so a genuine mismatch is operator-visible.
*
* No explicit clear is needed on primary loss: ha_on_primary_ctrl_fd_
* rdhup (ha.c) resets primary_ctrl_fd back to -1, so the NEXT real
* ctrlhello (from the same or a new primary) re-satisfies this same
* idempotency guard and re-runs this lookup naturally.
*
* Locking: ha.h's `lock` field doc ("Protects primary_host/port +
* voted_for_term/name") and existing writers of the other field it
* covers (voted_for_name, ha.c:2304-2332, 5559-5593) take
* c->ha.lock around the write — matched here for the same fields. */
if (req->from && req->from->name && c->myconf) {
const xce_cluster_config_t *pcfg = &c->myconf->peers_cfg;
int matched = 0;
for (int i = 0; i < pcfg->candidates_count; i++) {
if (strcmp(pcfg->candidates[i].name, req->from->name) == 0) {
pthread_mutex_lock(&c->ha.lock);
strncpy(c->ha.primary_host, pcfg->candidates[i].hostname,
sizeof(c->ha.primary_host) - 1);
c->ha.primary_host[sizeof(c->ha.primary_host) - 1] = '\0';
c->ha.primary_port = (uint32_t)pcfg->candidates[i].port;
pthread_mutex_unlock(&c->ha.lock);
matched = 1;
break;
}
}
if (!matched) {
WARN("handle_ctrlhello: sender '%s' does not match any "
"configured peer candidate in --peers -- "
"primary_host/primary_port left unchanged (possible "
"config mismatch)", req->from->name);
}
}
ha_on_heartbeat_received(&c->ha);
ha_arm_heartbeat_timeout_vtask(&c->ha, c);
INFO("HA: primary control channel established (fd=%d)", ep->fd);