lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(ha-term-safety): out-of-band config bytes, hash-verify-before-activate, serve-fence (task 4.3)

Browse source 
Wires config_committed_marker to a production consumer (config_activate_committed):
verifies pushed bytes' sha256 against the committed marker before activation, fences
(loud, non-serving, stay-alive) on mismatch, and activates the latest committed marker
on recovery. Enforced via ha_dispatch_gate + promotion gate.

Fixes bug-981: a 2nd+ config upgrade could permanently fence a standby on a
push-before-commit ordering. Adds an off-iomgr delaymgr retry scheduled from the
commit-advance hook (atomic read + non-blocking enqueue only on the ACK thread) that
re-activates and clears the stale fence once the new marker commits and bytes verify.
This commit is contained in:
Rongsong Shen 2026-06-22 11:47:15 +00:00
parent c3201a2b1e
commit be4c1f0721
7 changed files with 1074 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

105
src/controller/ha.c
View file

@ -87,6 +87,17 @@ extern int push_config_transport_only(cluster_t *c, const uint8_t *bytes,
uint64_t new_tag, int *n_out);
#endif
/* config_repl.c is linked in BOTH production and XCE_TESTING builds.
* ha-term-safety task 4.3 (remediation r1): the commit-advance hook activates a
* newly-committed config marker (Finding B); ha_dispatch_gate / the pre-promotion
* gate read the serve-fence (Finding C). */
extern int config_activate_committed(cluster_t *c);
extern int config_serve_fenced(const cluster_t *c);
/* task 4.3 (remediation r2, Finding D): the commit-advance path schedules an
* off-iomgr retry that re-activates a newly-committed config marker and clears a
* stale serve-fence (non-blocking ordq_put only). */
extern int config_schedule_activate_retry(cluster_t *c);
static void on_vote_connected(endpoint_t *ep);
static void on_pre_vote_connected(endpoint_t *ep);
static int vote_ep_err_handler(void *ep_raw);
@ -1028,6 +1039,23 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
}
}
/* ── ha-term-safety task 4.3 (remediation r1, Finding C / R7.9): config
* serve-fence promotion gate ──────────────────────────────────────────────
* A member whose latest committed config marker references bytes it does not
* hold (config_serve_fenced) MUST NOT become authoritative CTRL_PRIMARY — it
* cannot serve under the committed configuration. Refuse promotion and STAY
* ALIVE (controller-never-self-terminate): the member remains non-promotable
* and keeps retrying; once the correct bytes arrive out-of-band and re-verify
* clears the fence, a later promotion attempt passes this gate. The check is a
* pure atomic read (no I/O) — safe on the election path. */
if (config_serve_fenced(c)) {
WARN("HA: promotion REFUSED — config serve-FENCED (committed config "
"marker references bytes not present/matching locally); member "
"stays ALIVE, non-promotable, awaiting out-of-band config bytes "
"(ha-term-safety R7.9)");
return; /* do NOT become CTRL_PRIMARY; never self-exit */
}
int is_first_boot = !ha->phase2_done;
/* First boot: call cluster_init_phase2 */
@ -1636,6 +1664,37 @@ static void ha_cb_commit_advanced_stub(void *ctx, uint64_t commit_index)
if (!ha || !ha->raft_c)
return; /* test mode without a cluster: no barrier to trigger */
commit_set_raft_commit_index(&ha->raft_c->commit, commit_index);
/* ha-term-safety task 4.3 (remediation r1, Finding B): a config marker may
* have just entered the committed prefix. Activation is NOT performed HERE:
* this callback runs on the iomgr ACK path, where the async-I/O invariant
* forbids the disk reads config_activate_committed performs
* (scan_latest_committed_marker pread + load_primary_config). The committed
* marker is instead consumed off the iomgr thread by the call sites that make
* §7.10 true in production:
* - boot/recovery (myapp.c step 7): activates the latest committed marker,
* hash-verified, after persist recovery.
* - handle_repl_config_push: after the out-of-band bytes are saved, gates
* activation on the committed marker (verify-before-activate).
*
* ha-term-safety task 4.3 (remediation r2, Finding D / R7.9): the two sites
* above are NOT sufficient when the BYTES arrive (and are saved) BEFORE the
* NEW marker commits — the config MARKER (WAL) and the config BYTES (out-of-
* band push) travel on INDEPENDENT transports. In that ordering the push path
* hash-checks the freshly-saved NEW bytes against the still-OLDER committed
* marker, MISMATCHES, and sets the serve-fence; when the NEW marker later
* commits HERE, nothing re-runs activation (this callback can't do disk I/O,
* and the primary's config-behind re-push net does not fire because the push
* already advanced this member's advertised config_version_tag) — so the
* standby would stay PERMANENTLY fenced. Close the gap: while serve-fenced,
* schedule an OFF-IOMGR delaymgr task (config_schedule_activate_retry, NON-
* BLOCKING ordq_put only) that re-runs config_activate_committed on a thread
* allowed to block on disk; once the NEW marker is in the committed prefix and
* the saved bytes verify, the retry clears the fence and activates. No-op once
* unfenced (no busy-spin); a still-genuine mismatch reschedules on the next
* commit advance. */
if (config_serve_fenced(ha->raft_c))
config_schedule_activate_retry(ha->raft_c);
}
static const raft_callbacks_t s_real_cbs = {
@ -2630,6 +2689,41 @@ int ha_dispatch_gate(ha_state_t *ha, int opcode)
switch (role) {
case CTRL_PRIMARY:
/* ha-term-safety task 4.3 (remediation r1, Finding C / R7.9): the config
* serve-fence is the PRODUCTION READER that makes the fence real. When
* set, the latest committed WAL_CONFIG_MARKER references config bytes that
* do not hash-match the locally-held bytes (or the bytes are absent), so
* this member MUST NOT serve under the new configuration. Refuse
* config-dependent state-changing ops (loud non-serving), but keep
* replication / read-only / status ops flowing so the correct bytes can
* arrive out-of-band and re-verify clears the fence. The member STAYS
* ALIVE (returns an error code, never exits — controller-never-self-
* terminate). Checked before the quorum gate so a fenced primary never
* serves divergent config even with quorum. */
if (config_serve_fenced(ha->raft_c)) {
switch (opcode) {
/* Replication ops — let the correct config bytes arrive + re-verify */
case OPCode_heartbeat:
case OPCode_replConfig:
case OPCode_replConfigPush:
case OPCode_upgradeConfigAck:
case OPCode_hello:
case OPCode_walAck:
case OPCode_replEntry:
case OPCode_replAck:
case OPCode_replCatchUpDone:
case OPCode_walCatchupEntry:
case OPCode_snapshotTransfer:
/* Read-only queries / status — answerable while fenced */
case OPCode_get:
case OPCode_probeConfigTag:
case OPCode_ctrlstatus:
break; /* fall through to the quorum check below */
default:
/* Config-dependent state-changing op while fenced: refuse. */
return -XCE_ERR_NOT_READY;
}
}
/* State-changing operations require majority so committed entries
* are durable across failover. Read-only queries and replication
* ops are always allowed. */
@ -2755,6 +2849,17 @@ int ha_dispatch_gate(ha_state_t *ha, int opcode)
}
}
/* ha_config_promotion_fenced (ha-term-safety task 4.3, R7.9): a member is
* non-promotable while config serve-fenced — its latest committed config marker
* references bytes it does not hold/match, so it must not become authoritative.
* Pure atomic read via config_serve_fenced (no I/O) — safe on the election path. */
int ha_config_promotion_fenced(const ha_state_t *ha)
{
if (!ha)
return 0;
return config_serve_fenced(ha->raft_c) ? 1 : 0;
}
/* ── Redirect helpers used by join.c ───────────────────────── */
/*