Rongsong Shen
aa7fc65a36
bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its term-1 election before its peers ran; connect failures count as "no" votes (bug-1423), so it lost at once and became a follower with no leader. ctrl-2/ctrl-3 hold no config and are voters only (by design), so nothing retried until ctrl-1's 10 s standby liveness timeout: every cold start had no primary for ~10 s. - ha_note_leaderless() marks a config-holding node that becomes a follower with no leader known; the liveness poll then retries the election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to heartbeat_timeout_ms. Primary contact or becoming leader clears it. - ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity then only holds the watchdog's arm time, and the failover gates took it for a live primary. Live cold start: primary 1.7 s after the first node started (was ~10 s). bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a standby-only counter a primary never advances, so every primary showed "appliedLsn: 0". A primary now reports persist->last_applied_lsn. The controller-status design doc's "already correct in every role" note is corrected. Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c (RED: expected 4900, was 0); controller suite 1256/0. HA soaks: sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero job loss). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
360 lines
18 KiB
C
360 lines
18 KiB
C
/* ctrlstatus.c — controller status query handler (controller-status spec).
|
|
*
|
|
* Implements the read-only `ctrlstatus` opcode (an XCE analog of LSF `lsid`):
|
|
* snapshot the responding node's local HA/config state into a CtrlStatus reply
|
|
* and send it. Answerable in every role (the dispatch-gate allow-listing for
|
|
* that is task 2.2); this file only builds and sends the reply.
|
|
*
|
|
* Split into a pure snapshot (ctrlstatus_fill) + a thin send wrapper
|
|
* (handle_ctrlstatus), mirroring ha_nop_reply_state_report, so the field
|
|
* sourcing is unit-testable without an endpoint/socket.
|
|
*
|
|
* Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 5.3,
|
|
* 6.1, 6.2, 6.3, 6.4 (controller-status spec, task 2.1).
|
|
*/
|
|
#include "config.h"
|
|
|
|
#include <stdatomic.h>
|
|
#include <stdlib.h>
|
|
#include <string.h>
|
|
#include <time.h>
|
|
|
|
#include "cluster.h"
|
|
#include "ha.h"
|
|
#include "persist.h"
|
|
#include "utils.h" /* WARN */
|
|
#include "xce/cluster_def.h"
|
|
|
|
/*
|
|
* ctrlstatus_fill — snapshot local HA/config state into *st.
|
|
*
|
|
* Read-only: every field comes from an atomic load, an immutable config read,
|
|
* or a lock-guarded copy of primary_host/port. No HA, persist, or config state
|
|
* is modified (Req 1.2).
|
|
*
|
|
* Field sourcing:
|
|
* responderName / clusterName — own peerinfo / loaded cluster config
|
|
* responderRole / term — ha->role / ha->term atomics (Req 3.2, 6.1)
|
|
* primaryName — derived from role + voted_for_name (NOT the
|
|
* vestigial, never-written ha->primary_host):
|
|
* role==CTRL_PRIMARY → responder's OWN name
|
|
* (this node IS the primary; Req 2.1)
|
|
* role==CTRL_STANDBY → ha->voted_for_name
|
|
* (the elected leader it follows; Req 2.3),
|
|
* but only if it is a real peer name —
|
|
* empty or "self" → empty (Req 2.2)
|
|
* else (CTRL_ELECTING/CTRL_INIT) → empty —
|
|
* no confirmed primary (Req 2.2)
|
|
* read under ha->lock (guards voted_for_name);
|
|
* empty string ⇒ client renders "unknown"
|
|
* (Req 1.3)
|
|
* configVersion — loaded config_version_tag (Req 5.1)
|
|
* configChecksum — lowercase-hex of myconf->config_sha256[32]
|
|
* (64-char Text; Req 5.2 content checksum,
|
|
* 5.3 divergence detection) — see NOTE below
|
|
* committedConfigVersion — ha->committed_config_tag (Req 5.1)
|
|
* committedLsn / appliedLsn — persist->committed_lsn / ha->applied_lsn (6.2)
|
|
* fenced — in_catch_up || recovery_unrecoverable (6.3)
|
|
* peers / peerCount — myconf->peers_cfg.candidates (Req 4.1, 4.2)
|
|
*
|
|
* NOTE (configChecksum): the schema field is `configChecksum @6: Text` (it was
|
|
* originally `Data`, but `Data` maps to the project's structured `xce_data_t` —
|
|
* a tagged union over job/worker/resource/resval table payloads, NOT a capnp
|
|
* byte blob, so it cannot carry the raw 32-byte digest). We therefore carry the
|
|
* full SHA-256 content checksum as a 64-char lowercase-hex `Text` string, which
|
|
* the codec encodes/decodes like any other Text member (clusterName/primaryName)
|
|
* and the client renders as hex anyway. This gives Req 5.2 (content checksum)
|
|
* and Req 5.3 (same-version divergence detection — identical config bytes on two
|
|
* nodes yield the same hex; differing bytes differ). configVersion /
|
|
* committedConfigVersion (the 8-byte SHA-256 prefix tag) carry the version-level
|
|
* identity for Req 5.1. The hex string is heap-allocated here and freed in
|
|
* ctrlstatus_free_fields().
|
|
*/
|
|
void ctrlstatus_fill(cluster_t* c, xce_ctrlstatus_t* st)
|
|
{
|
|
ha_state_t* ha = &c->ha;
|
|
myconf_t* mc = c->myconf;
|
|
|
|
memset(st, 0, sizeof(*st));
|
|
|
|
/* responder identity + cluster name */
|
|
st->responderName = (c->myhost && c->myhost->name[0] != '\0')
|
|
? c->myhost->name
|
|
: (mc && mc->name) ? mc->name
|
|
: "";
|
|
st->clusterName = (mc && mc->cluster_config && mc->cluster_config->clustername)
|
|
? mc->cluster_config->clustername
|
|
: "";
|
|
|
|
/* role / term (atomics) */
|
|
st->responderRole = (uint8_t)atomic_load_explicit(&ha->role,
|
|
memory_order_acquire);
|
|
st->term = atomic_load_explicit(&ha->term, memory_order_acquire);
|
|
|
|
/* current primary's election time (Req 12.1, 12.2, 12.5): a single
|
|
* unconditional read of last_known_elected_at_ms -- no role-branch. Both
|
|
* promotion roads self-stamp this field at the instant they commit to
|
|
* CTRL_PRIMARY (task 7.2), and it is unconditionally learned off-primary
|
|
* via heartbeat (task 7.3), so one atomic load correctly serves both the
|
|
* primary (self-stamped) and standby (heartbeat-learned) case -- unlike
|
|
* committedLsn/committedConfigVersion above, there is no separate
|
|
* primary-only marker to branch on here. "Unknown" display gating (Req
|
|
* 12.3) is a client-side print-layer decision (task 7.5), not this
|
|
* handler's concern -- mirrors how st->term above is also read
|
|
* unconditionally regardless of whether a primary is currently known. */
|
|
st->elected_at_ms = atomic_load_explicit(&ha->last_known_elected_at_ms,
|
|
memory_order_acquire);
|
|
|
|
/* primary identity (Req 2.1/2.2/2.3). Only the PRIMARY can authoritatively
|
|
* name the primary — itself. The controller does NOT track the current
|
|
* primary's NAME on a standby: ha->primary_host is vestigial (never written)
|
|
* and ha->voted_for_name is "who I last voted for" (frequently this node
|
|
* itself), NOT the elected leader — using it makes a standby misreport
|
|
* itself as primary and misdirects the authoritative redirect. So a non-
|
|
* primary responder reports an empty primaryName ("unknown"); the client's
|
|
* authoritative path (xcecli_ctrlstatus_primary) then reaches the real
|
|
* primary by master rotation and gets the authoritative self-report. */
|
|
if (st->responderRole == (uint8_t)CTRL_PRIMARY) {
|
|
st->primaryName = strdup(st->responderName); /* this node IS the primary */
|
|
} else {
|
|
st->primaryName = strdup(""); /* unknown — not reliably known off-primary */
|
|
}
|
|
|
|
/* responder snapshot timestamp (Req 11.1): CLOCK_REALTIME (wall clock, NOT
|
|
* CLOCK_MONOTONIC which is used for heartbeat liveness) at the sample
|
|
* moment, in epoch millis, so an operator can line a captured status up
|
|
* against this responder's log timestamps. Stamped here with the other
|
|
* field loads to mark the snapshot instant. */
|
|
struct timespec ctrlstatus_ts;
|
|
clock_gettime(CLOCK_REALTIME, &ctrlstatus_ts);
|
|
st->responder_time_ms = (uint64_t)ctrlstatus_ts.tv_sec * 1000u
|
|
+ (uint64_t)ctrlstatus_ts.tv_nsec / 1000000u;
|
|
|
|
/* loaded config version */
|
|
st->configVersion = mc ? mc->config_version_tag : 0;
|
|
|
|
/* configChecksum: 64-char lowercase-hex of the loaded config_sha256[32]
|
|
* (Req 5.2 content checksum, 5.3 divergence detection — see NOTE above).
|
|
* Heap-allocated; freed in ctrlstatus_free_fields(). */
|
|
st->configChecksum = NULL;
|
|
if (mc) {
|
|
static const char hexdig[] = "0123456789abcdef";
|
|
char* hex = (char*)malloc(2u * sizeof(mc->config_sha256) + 1u);
|
|
if (hex) {
|
|
for (size_t i = 0; i < sizeof(mc->config_sha256); i++) {
|
|
hex[2 * i] = hexdig[(mc->config_sha256[i] >> 4) & 0x0F];
|
|
hex[2 * i + 1] = hexdig[mc->config_sha256[i] & 0x0F];
|
|
}
|
|
hex[2 * sizeof(mc->config_sha256)] = '\0';
|
|
st->configChecksum = hex;
|
|
}
|
|
}
|
|
|
|
/* commit / apply progress (Req 6.2) — committed sourcing is role-branched
|
|
* (Req 10.1/10.2): a PRIMARY's own live markers are authoritative; any
|
|
* other role (STANDBY/ELECTING/INIT) has never served as primary, so its
|
|
* node-local committed marker is unset (0) and the only honest "committed"
|
|
* knowledge is what it has LEARNED from the primary's heartbeats. Reporting
|
|
* the learned pair avoids a misleading committed=0 while the node has
|
|
* clearly applied replicated entries, and keeps committed/applied sourced
|
|
* consistently (no spurious applied>committed inversion, Req 10.3). Pre-
|
|
* learn the learned values are 0 → well-defined "unknown" (Req 10.2). */
|
|
if (st->responderRole == (uint8_t)CTRL_PRIMARY) {
|
|
st->committedLsn = atomic_load_explicit(&c->persist.committed_lsn,
|
|
memory_order_acquire);
|
|
st->committedConfigVersion =
|
|
atomic_load_explicit(&ha->committed_config_tag,
|
|
memory_order_acquire);
|
|
} else {
|
|
st->committedLsn = atomic_load_explicit(&ha->last_known_committed_lsn,
|
|
memory_order_acquire);
|
|
/* last_known_committed_config_tag is an _Atomic uint64_t; an acquire
|
|
* load mirrors last_known_committed_lsn above and must NOT take the
|
|
* lock on this read path. */
|
|
st->committedConfigVersion =
|
|
atomic_load_explicit(&ha->last_known_committed_config_tag,
|
|
memory_order_acquire);
|
|
}
|
|
/* bug-1878: ha->applied_lsn is a standby-only counter (how far
|
|
* replicated WAL has been received and persisted); a primary never
|
|
* advances it, so every primary reported 0. A primary reports its apply
|
|
* cursor instead -- the last entry applied, never above committedLsn. */
|
|
if (st->responderRole == (uint8_t)CTRL_PRIMARY) {
|
|
st->appliedLsn = atomic_load_explicit(&c->persist.last_applied_lsn,
|
|
memory_order_acquire);
|
|
} else {
|
|
st->appliedLsn = atomic_load_explicit(&ha->applied_lsn,
|
|
memory_order_acquire);
|
|
}
|
|
|
|
/* fenced: catch-up OR recovery-unrecoverable (Req 6.3) */
|
|
int in_catch_up = atomic_load_explicit(&ha->in_catch_up,
|
|
memory_order_acquire);
|
|
st->fenced = (in_catch_up || persist_recovery_unrecoverable(&c->persist))
|
|
? 1u
|
|
: 0u;
|
|
|
|
/* configured controllers (Req 4.1, 4.2) */
|
|
int n = 0;
|
|
if (mc) {
|
|
n = mc->peers_cfg.candidates_count;
|
|
if (n < 0)
|
|
n = 0;
|
|
}
|
|
st->peerCount = (uint32_t)n;
|
|
st->peers.count = n;
|
|
st->peers.list = NULL;
|
|
if (n > 0) {
|
|
st->peers.list = (xce_ctrlpeer_t**)calloc((size_t)n,
|
|
sizeof(xce_ctrlpeer_t*));
|
|
|
|
/* per-controller connection status (Req 13.1-13.5; design.md Req 13
|
|
* delta, Locked Decisions 3-6): the responder's own entry always
|
|
* reports ok; when the responder is currently primary, every OTHER
|
|
* peer is sourced by host+port match against the live replication
|
|
* connection registry (replicator_t.standbys[]) -- no live
|
|
* connection -> dead, live but not yet caught up -> catching-up,
|
|
* live and caught up -> ok; when the responder is not primary,
|
|
* every other peer is unknown, since only the primary has direct
|
|
* connection visibility into every other node. */
|
|
int is_primary = (st->responderRole == (uint8_t)CTRL_PRIMARY);
|
|
/* ha->repl is a back-pointer wired once commit-engine setup runs
|
|
* (cluster.c) and may be NULL before that (or in a minimal test
|
|
* fixture); &c->persist.repl is always a valid fallback (mirrors
|
|
* the ha->repl ? ha->repl : &p->repl idiom at ha.c:2673). */
|
|
replicator_t* repl = ha->repl ? ha->repl : &c->persist.repl;
|
|
|
|
/* Self-match fallback (design-review finding): if this responder's
|
|
* own identity matches none of its configured peers -- a pre-
|
|
* existing misconfiguration, not new to this delta -- WARN once per
|
|
* call and let every peer fall through to the normal non-self
|
|
* branches below. No entry is force-corrected to ok in this case;
|
|
* this is an explicit, logged degradation rather than a silent or
|
|
* fabricated result. */
|
|
int self_found = 0;
|
|
for (int i = 0; i < n; i++) {
|
|
if (strcmp(mc->peers_cfg.candidates[i].name,
|
|
st->responderName) == 0) {
|
|
self_found = 1;
|
|
break;
|
|
}
|
|
}
|
|
if (!self_found) {
|
|
WARN("ctrlstatus: responder identity '%s' matches none of the "
|
|
"%d configured peers -- per-peer connection status falls "
|
|
"through to the non-self sourcing for every configured "
|
|
"peer (check this node's --peers/name configuration)",
|
|
st->responderName, n);
|
|
}
|
|
|
|
if (is_primary)
|
|
mylock_lock(&repl->lock);
|
|
|
|
for (int i = 0; i < n; i++) {
|
|
xce_ctrlpeer_t* p = (xce_ctrlpeer_t*)calloc(1, sizeof(*p));
|
|
p->name = mc->peers_cfg.candidates[i].name;
|
|
p->host = mc->peers_cfg.candidates[i].hostname;
|
|
p->port = (uint32_t)mc->peers_cfg.candidates[i].port;
|
|
|
|
if (strcmp(mc->peers_cfg.candidates[i].name,
|
|
st->responderName) == 0) {
|
|
p->status = 1u; /* self: always ok (Req 13.2) */
|
|
} else if (is_primary) {
|
|
/* Reuses repl_active_standby_count's exact ep != NULL &&
|
|
* !in_catch_up predicate (repl.c) rather than inventing a
|
|
* new one (Locked Decision 3).
|
|
*
|
|
* Scans ALL MAX_STANDBYS slots (does NOT stop at the first
|
|
* host/port match) and prefers any slot with ep != NULL
|
|
* over a slot with ep == NULL when multiple slots match the
|
|
* same configured peer's host:port (task 8.6 fix). A killed
|
|
* standby's slot is freed (ep cleared) but keeps its
|
|
* host/port populated; when it reconnects, repl_add_standby's
|
|
* ep-identity idempotency check cannot reuse that now-ep-NULL
|
|
* slot, so the reconnect lands in a NEW slot -- leaving two
|
|
* slots that share the same peer's host:port, one stale
|
|
* (ep == NULL) and one live (ep != NULL). A first-match-wins
|
|
* scan reported the stale, lower-index slot's dead status
|
|
* forever, even after the peer genuinely reconnected and
|
|
* caught up. If more than one live slot somehow matches, any
|
|
* one of them is an acceptable source -- they would all be
|
|
* observing the same live peer. */
|
|
uint8_t status = 3u; /* default: no matching slot at all */
|
|
int live_match = 0; /* found a slot with ep != NULL */
|
|
for (int j = 0; j < MAX_STANDBYS; j++) {
|
|
if (strcmp(repl->standbys[j].host,
|
|
mc->peers_cfg.candidates[i].hostname) != 0 ||
|
|
repl->standbys[j].port != p->port) {
|
|
continue; /* not a match for this peer */
|
|
}
|
|
if (repl->standbys[j].ep == NULL) {
|
|
if (!live_match)
|
|
status = 3u; /* stale match; keep scanning for a live one */
|
|
continue;
|
|
}
|
|
if (!live_match) {
|
|
live_match = 1;
|
|
status = atomic_load_explicit(
|
|
&repl->standbys[j].in_catch_up,
|
|
memory_order_acquire)
|
|
? 2u /* live, catching-up */
|
|
: 1u; /* live, caught up: ok */
|
|
}
|
|
}
|
|
p->status = status;
|
|
} else {
|
|
p->status = 0u; /* unknown: only the primary can see peers */
|
|
}
|
|
|
|
st->peers.list[i] = p;
|
|
}
|
|
|
|
if (is_primary)
|
|
mylock_unlock(&repl->lock);
|
|
}
|
|
}
|
|
|
|
/*
|
|
* ctrlstatus_free_fields — release the heap that ctrlstatus_fill allocated.
|
|
* Frees the per-peer structs + the pointer array and the primaryName copy.
|
|
* The peer name/host strings are NOT owned (they point into myconf) and are
|
|
* not freed. Safe to call once after the reply has been serialised.
|
|
*/
|
|
void ctrlstatus_free_fields(xce_ctrlstatus_t* st)
|
|
{
|
|
if (st->peers.list) {
|
|
for (int i = 0; i < st->peers.count; i++)
|
|
free(st->peers.list[i]);
|
|
free(st->peers.list);
|
|
st->peers.list = NULL;
|
|
st->peers.count = 0;
|
|
}
|
|
free(st->primaryName);
|
|
st->primaryName = NULL;
|
|
free(st->configChecksum);
|
|
st->configChecksum = NULL;
|
|
}
|
|
|
|
/*
|
|
* handle_ctrlstatus — dispatch entry point for OPCode_ctrlstatus.
|
|
* Builds a CtrlStatus from local state, replies, then frees. Mirrors
|
|
* handle_clusterconf (reply.c): stack reply, init_reply_by_req, point the union
|
|
* at the payload, xce_reply, then free what was allocated.
|
|
*/
|
|
int handle_ctrlstatus(cluster_t* c, endpoint_t* ep, xce_request_t* request)
|
|
{
|
|
xce_reply_t reply;
|
|
xce_ctrlstatus_t status;
|
|
|
|
ctrlstatus_fill(c, &status);
|
|
|
|
init_reply_by_req(&reply, request);
|
|
reply.status = 0;
|
|
reply.data_which = RequestReply_data_ctrlstatus;
|
|
reply.data.ctrlstatus = &status;
|
|
|
|
xce_reply(ep, c->myhost, &reply);
|
|
|
|
ctrlstatus_free_fields(&status);
|
|
return 0;
|
|
}
|