wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

feat(config-upgrade-snapshot): gate primary serving and promotion on the activation fence

Browse source 
Wires task 6.1's upgrade_materialization_verify/upgrade_materialization_fenced
into ha_dispatch_gate's CTRL_PRIMARY branch (sequential block alongside,
not merged into, the existing config_serve_fenced check -- same
replication/read-only opcode allowlist, refuses state-changing ops
with -XCE_ERR_NOT_READY while fenced) and the promotion gate
(raft_cb_become_leader refuses to become CTRL_PRIMARY while fenced,
introduces no new WAL/log-completeness eligibility mechanism, never
self-terminates). Adds a 4-function self-triggering retry trampoline
(upgrade_materialization_retry_fn/_vtask_entry/_delaymgr_trampoline/
schedule_upgrade_materialization_retry) mirroring
config_activate_retry_fn's exact shape, hooked into
ha_cb_commit_advanced_stub alongside the existing config-byte retry.

Fixes a real bug caught on review: the fence had no production path
that could ever engage it from its zero-initialized default -- every
setter call site was gated on the fence already being set (closed
loop), making the new gate checks permanently dead code. Fixed with
two unconditional upgrade_materialization_verify() call sites:
persist.c's apply_committed_entry WAL_CONFIG_MARKER case (right after
persist_checkpoint_marker_triggered returns, regardless of its rc) and
myapp.c's boot sequence (right after config_activate_committed, for
Req 5.2 crash/restart resume). New test drives the fence through the
real production trigger (a forced -EBUSY checkpoint miss on a real
marker commit+apply), not a direct verify() call, proving the
end-to-end path actually works. Task 6.2 of 19.

Requirements: 3.1, 3.2, 3.3, 3.4, 4.3, 5.5
This commit is contained in:
Rongsong Shen 2026-07-25 04:58:37 +00:00
parent 7b507156a0
commit 89b85e1680
6 changed files with 805 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

203
src/controller/ha.c
View file

@ -288,6 +288,124 @@ int upgrade_materialization_fenced(const cluster_t *c)
memory_order_acquire) ? 1 : 0;
}
/* ── config-upgrade-snapshot task 6.2: Activation Fence self-triggering retry ──
* Mirrors config_schedule_activate_retry's exact four-function shape
* (config_repl.c:409-507) for the upgrade_materialization_fence instead of
* config_serve_fence: a one-shot NON-BLOCKING delaymgr task offloads the
* actual blocking re-verify work (upgrade_materialization_verify's disk
* reads) onto a VTASK_LIVE_BLOCK vtask, so the delaymgr thread itself never
* blocks (ha-async-io.md thread-role contract). */
/* upgrade_materialization_retry_fn: re-runs upgrade_materialization_verify so
* a member whose materialization was incomplete or hash-mismatched re-checks
* and clears the fence once conditions allow. Does the actual blocking disk
* I/O; callers must invoke it off the delaymgr thread (see
* upgrade_materialization_retry_vtask_entry /
* schedule_upgrade_materialization_retry below). Public so tests can drive it
* directly without a vtask pool — mirrors config_activate_retry_fn. */
int upgrade_materialization_retry_fn(void *arg)
{
cluster_t *c = (cluster_t *)arg;
if (!c)
return -EINVAL;
if (!upgrade_materialization_fenced(c))
return 0; /* already cleared by another path — nothing to retry */
int rc = upgrade_materialization_verify(c);
if (rc == 0)
INFO("upgrade materialization retry: fence CLEARED — this member's "
"own materialization for the committed config marker now "
"verified");
else if (rc == -EAGAIN)
TRACE("upgrade materialization retry: still fenced (materialization "
"incomplete or hash-mismatched) — will retry on the next "
"commit advance");
else
WARN("upgrade materialization retry: upgrade_materialization_verify "
"rc=%d", rc);
return rc;
}
/* upgrade_materialization_retry_vtask_entry: vtask entry wrapping
* upgrade_materialization_retry_fn so its blocking disk I/O runs on a vtask
* worker, never inline on the single delaymgr thread. Mirrors
* config_activate_retry_vtask_entry. */
static int upgrade_materialization_retry_vtask_entry(vtask_t *task)
{
cluster_t *c = (cluster_t *)task->data->u.generic;
upgrade_materialization_retry_fn(c);
return 0;
}
/* upgrade_materialization_retry_delaymgr_trampoline: the delaymgr task body.
* Delaymgr must only trigger + enqueue, never block — dispatch the actual
* retry work to a vtask (VTASK_LIVE_BLOCK, the same class
* config_activate_retry_delaymgr_trampoline uses) and return immediately.
* Fails loud and runs inline (stay alive, never silently drop a scheduled
* retry) if the pool is unavailable. Mirrors
* config_activate_retry_delaymgr_trampoline. */
static int upgrade_materialization_retry_delaymgr_trampoline(void *arg)
{
cluster_t *c = (cluster_t *)arg;
if (!c)
return -EINVAL;
vtaskmgr_t *pool = &c->vtaskmgr;
vtask_data_t *data = calloc(1, sizeof(*data));
vtask_t *task = vtask_alloc();
if (data == NULL || task == NULL) {
WARN("upgrade materialization retry: vtask alloc failed — running "
"inline (stay alive)");
free(data);
free(task);
return upgrade_materialization_retry_fn(c);
}
data->vtype = VTASK_GENERIC;
data->u.generic = c;
vtask_init(task, (vtask_func_t)upgrade_materialization_retry_vtask_entry,
data);
if (vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK) < 0) {
WARN("upgrade materialization retry: vtask enqueue failed — running "
"inline (stay alive)");
free(data);
free(task);
return upgrade_materialization_retry_fn(c);
}
return 0;
}
/* schedule_upgrade_materialization_retry: enqueue the retry trampoline on the
* delaymgr (NON-BLOCKING — delay_task_ms only does ordq_put; the trampoline
* itself offloads the blocking work to a vtask, see above). Called from the
* commit-advance path (iomgr ACK thread) so the retry's disk I/O runs OFF
* that thread (ha-async-io invariant). No-op when the member is NOT
* upgrade-materialization fenced (steady state schedules nothing) or when
* the delaymgr is not running (the unit-test fixture path, which never inits
* delaymgr — avoids leaking a task into an un-drained queue). Mirrors
* config_schedule_activate_retry exactly. Returns 0 on enqueue/no-op,
* negative on enqueue failure. */
int schedule_upgrade_materialization_retry(cluster_t *c)
{
if (!c)
return -EINVAL;
if (!upgrade_materialization_fenced(c))
return 0; /* steady state: nothing to retry, schedule nothing */
if (!c->delaymgr.running)
return 0; /* delaymgr not running (test fixture) — caller drives directly */
/* One-shot, minimal delay: run promptly but off the iomgr ACK thread. */
int rc = delay_task_ms(&c->delaymgr, 1, 1,
(delay_task_entry_fn_t)upgrade_materialization_retry_delaymgr_trampoline,
c);
if (rc < 0)
WARN("upgrade materialization retry: delay_task_ms failed: %d — "
"fence will clear on a later commit advance", rc);
return rc;
}
static void on_vote_connected(endpoint_t *ep);
static void on_pre_vote_connected(endpoint_t *ep);
static int vote_ep_err_handler(void *ep_raw);
@ -1622,6 +1740,35 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
return; /* do NOT become CTRL_PRIMARY; never self-exit */
}
/* ── config-upgrade-snapshot task 6.2 (Req 1.8, 3.2, 3.3, 4.3, 5.5):
* Activation Fence promotion gate ──────────────────────────────────────
* A member whose OWN upgrade-snapshot materialization for the current
* committed config marker has not completed and hash-verified must not
* become authoritative CTRL_PRIMARY — doing so would let it win an
* election (Raft's log-freshness gate only proves it has the committed
* MARKER, not that it has finished MATERIALIZING under it) and then
* immediately violate the serving gate above the moment it starts
* accepting requests. This is a SEPARATE, SEQUENTIAL check from the
* config serve-fence directly above (same rationale as the dispatch-gate
* composition: byte-mismatch and materialization-incomplete are distinct
* conditions) — never a new promotion-eligibility mechanism for WAL/log
* completeness (Req 3.1 is explicit that Raft's own freshness gate
* already owns that; this fence is layered on top, purely about
* materialization completeness). Refuse promotion and STAY ALIVE
* (controller-never-self-terminate): the member remains non-promotable
* and keeps retrying (schedule_upgrade_materialization_retry, driven off
* the commit-advance path); once its own materialization completes and
* re-verify clears the fence, a later promotion attempt passes this
* gate. Pure atomic read (no I/O) — safe on the election path. */
if (upgrade_materialization_fenced(c)) {
WARN("HA: promotion REFUSED — upgrade materialization FENCED (this "
"member's own upgrade-snapshot materialization for the "
"committed config marker is incomplete or hash-mismatched); "
"member stays ALIVE, non-promotable, retrying materialization "
"(config-upgrade-snapshot Req 1.8/3.2/3.3/4.3/5.5)");
return; /* do NOT become CTRL_PRIMARY; never self-exit */
}
int is_first_boot = !ha->phase2_done;
/* First boot: call cluster_init_phase2 */
@ -2379,6 +2526,20 @@ static void ha_cb_commit_advanced_stub(void *ctx, uint64_t commit_index)
* commit advance. */
if (config_serve_fenced(ha->raft_c))
config_schedule_activate_retry(ha->raft_c);
/* config-upgrade-snapshot task 6.2 (Req 3.4, 5.5): symmetric commit-
* advance trigger for the Activation Fence. Materialization technically
* completes on APPLY (persist.c's WAL_CONFIG_MARKER case), which lags
* commit under raft-apply-decoupling — this hook does not need to fire
* at the PRECISE instant materialization becomes possible, only to
* eventually re-check and clear the fence once conditions allow (Req
* 3.4/5.5's "keep retrying" framing, not a precise event-driven
* guarantee). Commit advancing is a reasonable, precedented "something
* changed, worth re-checking" signal, and schedule_upgrade_materialization_
* retry itself is a no-op when unfenced, so this adds no busy-spin in the
* steady state. */
if (upgrade_materialization_fenced(ha->raft_c))
schedule_upgrade_materialization_retry(ha->raft_c);
}
static const raft_callbacks_t s_real_cbs = {
@ -4031,6 +4192,48 @@ int ha_dispatch_gate(ha_state_t *ha, int opcode)
return -XCE_ERR_NOT_READY;
}
}
/* config-upgrade-snapshot task 6.2 (Req 3.2, 3.3, 5.5): the Activation
* Fence is a SEPARATE, SEQUENTIAL check composed alongside the
* config-byte serve-fence above, not merged into it — the two
* conditions are logically distinct (byte-mismatch vs. this member's
* own materialization-incomplete) per design.md's Activation Fence
* component. Checked AFTER config_serve_fenced deliberately: byte
* delivery is the earlier step (the bytes must exist locally before
* this member's own checkpoint can even attempt to materialize
* against them), so checking byte-fence first surfaces the more
* fundamental blocker first in log/error ordering — but either fence
* alone is sufficient to refuse a state-changing op, so the final
* admit/refuse outcome does not depend on this ordering. Same
* allowlist as the byte-fence: replication + read-only/status ops
* stay admitted (they let materialization proceed and re-verify);
* everything else is refused with a loud non-serving error while the
* member STAYS ALIVE and keeps retrying (controller-never-self-
* terminate). */
if (upgrade_materialization_fenced(ha->raft_c)) {
switch (opcode) {
/* Replication ops — let commit/apply advance + re-verify */
case OPCode_heartbeat:
case OPCode_replConfig:
case OPCode_replConfigPush:
case OPCode_upgradeConfigAck:
case OPCode_hello:
case OPCode_walAck:
case OPCode_replEntry:
case OPCode_replAck:
case OPCode_replCatchUpDone:
case OPCode_walCatchupEntry:
case OPCode_snapshotTransfer:
/* Read-only queries / status — answerable while fenced */
case OPCode_get:
case OPCode_probeConfigTag:
case OPCode_ctrlstatus:
break; /* fall through to the quorum check below */
default:
/* State-changing op while this member's own upgrade-snapshot
* materialization is incomplete/unverified: refuse. */
return -XCE_ERR_NOT_READY;
}
}
/* State-changing operations require majority so committed entries
* are durable across failover. Read-only queries and replication
* ops are always allowed. */