lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): bug-1450/1451 offload delaymgr-inline blocking work to the slow lane

Browse source 
/xce-audit-invariants controller found two live violations of the async
I/O thread-role contract: ha_demotion_convergence_fn (ha_convergence.c)
ran WAL truncation + tlog_exit()+execv() directly on the delaymgr thread,
and ha_catchup_continuation_fn (config_repl.c) ran unbounded WAL-streaming
to a standby directly on delaymgr -- the same single thread that drives
heartbeats and election timeouts cluster-wide, so either blocks liveness
for the whole cluster and risks a spurious election.

A prior vtask-pool offload attempt (2026-07-10, bug-1429) deadlocked
controller_unit_test because c->ha.pool is never initialized by the
lightweight unit-test fixtures these functions run under, and was fully
reverted. This fix instead uses the codebase's established slow-lane
pattern (the same mechanism already used for snapshot-send, config-push,
and peer connect) -- a 4th and 5th instance, not a new mechanism.

New g_drive_async_demotion_offload / g_drive_async_catchup_offload flags
ship default OFF, matching the original rollout discipline of the three
reference offloads (shipped OFF, flipped ON later after a dedicated
compatibility audit). Flipping ON was tried during validation and
reproduced two distinct hazards against pre-existing fixtures that never
call slowlane_init: the already-known hang, and a new ASan SEGV in
commit_advance_locked (logged as bug-1452, not root-caused, deferred to
the future flip-to-ON audit). Both flags verified OFF and the full
976-test suite clean, then the committed state re-verified clean against
a live 10-round chaotic and 10-round sequential HA soak (zero job loss,
zero divergence, both PASS).
This commit is contained in:
Rongsong Shen 2026-07-19 08:14:00 +00:00
parent aeb6e37ce7
commit f88e4ef431
5 changed files with 877 additions and 55 deletions
Download patch file Download diff file Expand all files Collapse all files

197
src/controller/ha_convergence.c
View file

@ -184,36 +184,82 @@ int ha_self_restart(cluster_t *c)
}
/*
* ha_demotion_convergence_ctx_t — heap context for the one-shot delaymgr task.
* Holds only the cluster pointer; committed_lsn is read at task time (not
* schedule time) so a late-advancing commit still truncates to the right
* boundary.
* ha_demotion_convergence_ctx_t — heap context for the one-shot delaymgr task
* (and, with the offload flag ON, the slow-lane task it hands off to). Holds
* only the cluster pointer; committed_lsn is read at task time (not schedule
* time) so a late-advancing commit still truncates to the right boundary.
*/
typedef struct {
cluster_t *c;
} ha_demotion_convergence_ctx_t;
/*
* ha_demotion_convergence_fn — delaymgr task body (runs OFF the become-follower
* callback thread, D1). Truncate above committed_lsn (Req 1.2), WARN with the
* truncated range (Req 1.5), then self-restart (Req 1.3). On truncation failure
* (Req error strategy) ERROR then restart anyway — the rejoin reconciliation
* loop converges the untruncated case at the next join.
* g_drive_async_demotion_offload / ha_set_demotion_offload_for_test /
* ha_demotion_offload_enabled (xce-audit-invariants controller pass,
* thread-role violation 1 fix, 2026-07-18):
*
* In XCE_TESTING the test fixture drives this entry directly (un-started
* delaymgr); ha_self_restart routes to the restart seam there. The ctx is
* freed before restart in production restart never returns, so only the test
* path reaches the free — kept on every non-restarting exit for safety.
* Gates whether ha_demotion_convergence_fn hands the truncate + self-restart
* body off to the slow lane (ON — closes the thread-role violation: the
* delaymgr thread that also drives heartbeats/election timeouts cluster-wide
* never blocks on the truncation I/O or the tlog_exit()+execv() inside
* ha_self_restart) or runs it INLINE on the calling (delaymgr) thread (OFF,
* the PRODUCTION DEFAULT for now — see below). This is the SAME established
* slow-lane pattern already used for snapshot send
* (ha_snapshot_send_dispatch/snapshot_lane_fn), config-push
* (ha_config_push_offloadable/configpush_lane_fn), and peer connect
* (ha_connect_to_peer/reconnect_lane_fn) in ha.c — not a new mechanism. A
* prior attempt to fix this via a VTASK_LIVE_BLOCK vtask-pool dispatch
* (2026-07-10) deadlocked controller_unit_test and was reverted (bug-1429):
* the vtask pool used there (c->ha.pool) is never initialized/started by
* this file's lightweight unit-test fixtures (dc_setup), so a vtask
* enqueued VTASK_LIVE_BLOCK blocked forever waiting for a worker that never
* existed. The slow lane's own submit contract cannot deadlock a caller the
* way the vtask pool did — but a second, distinct hazard surfaces the SAME
* class of problem: raft_cb_become_follower (the demotion callback this task
* is scheduled from) is exercised by a very broad set of PRE-EXISTING
* controller_unit_test fixtures (HA lifecycle, connection-reuse,
* vep-teardown, startup, …) that were never written with a live slow lane in
* mind. Flipping this flag ON by default was tried and reproduced a REAL
* hang: such a fixture demotes a PRIMARY, the truncate+restart work is
* silently submitted to an uninitialized c->slowlane (queued, but never run
* — no live workers), and downstream code that waits for the now-never-
* happening completion hangs. OFF (0, PRODUCTION DEFAULT) matches the
* ORIGINAL rollout discipline of all three reference offloads — every one of
* snapshot/config-push/reconnect shipped OFF-by-default and was flipped ON
* only in a LATER, dedicated pass (raft-drive-async task 3.4) once the full
* test suite was audited for compatibility. This change lands the
* mechanism, correct and covered by its own targeted tests below; flipping
* the cluster-wide default to ON is the same kind of follow-up audit, out of
* scope for this fix (bug-1450, buglog.json). Compiled in ALL builds; the
* setter is the test seam to flip RED<->GREEN.
*/
int ha_demotion_convergence_fn(void *arg)
{
ha_demotion_convergence_ctx_t *ctx = (ha_demotion_convergence_ctx_t *)arg;
if (ctx == NULL || ctx->c == NULL) {
free(ctx);
return -EINVAL;
}
cluster_t *c = ctx->c;
static _Atomic int g_drive_async_demotion_offload = 0;
void ha_set_demotion_offload_for_test(int on)
{
atomic_store_explicit(&g_drive_async_demotion_offload, on ? 1 : 0,
memory_order_release);
}
int ha_demotion_offload_enabled(void)
{
return atomic_load_explicit(&g_drive_async_demotion_offload,
memory_order_acquire);
}
/*
* ha_demotion_convergence_do_run — the actual (blocking) convergence work:
* truncate above committed_lsn (Req 1.2), WARN with the truncated range
* (Req 1.5), then self-restart (Req 1.3). On truncation failure (Req error
* strategy) ERROR then restart anyway — the raft library's own (term,lsn) AE
* reconcile converges the untruncated case once the leader's AppendEntries
* arrives. Shared verbatim by the inline path (offload flag OFF) and the
* slow-lane worker (demotion_lane_fn, offload flag ON) so both behave
* identically — mirrors ha_snapshot_do_send / ha_config_push_do_send in
* ha.c.
*/
static int ha_demotion_convergence_do_run(cluster_t *c)
{
uint64_t committed = atomic_load_explicit(&c->persist.committed_lsn,
memory_order_acquire);
@ -237,11 +283,114 @@ int ha_demotion_convergence_fn(void *arg)
committed, prior_tip);
}
/* Free the task ctx before restart (production exec never returns). */
/* Req 1.3: rebuild in-memory state through boot recovery. Production:
* execv() replaces the process image and never returns. XCE_TESTING:
* the restart seam returns so the caller can observe the outcome. */
return ha_self_restart(c);
}
/*
* demotion_lane_fn — slow-lane worker entry. Runs ha_demotion_convergence_do_run
* OFF the delaymgr thread; frees ctx and clears the cluster-wide single-flight
* guard on the return path. In production ha_self_restart's execv() never
* returns, so this point is reached only under XCE_TESTING (or the rare
* production restart-seam-failure fence-and-retry path, which also returns).
*/
static void demotion_lane_fn(void *arg)
{
ha_demotion_convergence_ctx_t *ctx = (ha_demotion_convergence_ctx_t *)arg;
if (ctx == NULL)
return;
cluster_t *c = ctx->c;
free(ctx);
/* Req 1.3: rebuild in-memory state through boot recovery. */
return ha_self_restart(c);
(void)ha_demotion_convergence_do_run(c);
atomic_store_explicit(&c->ha.demotion_convergence_inflight, 0,
memory_order_release);
}
/*
* ha_demotion_convergence_fn — delaymgr task body (runs OFF the become-follower
* callback thread, D1 — the FIRST hop). With the offload flag ON (production
* default), this function does only fast, non-blocking admin work (ctx
* unpack, single-flight CAS, slowlane_submit) and returns immediately — the
* SECOND hop, off the delaymgr thread itself, onto a slow-lane worker
* (demotion_lane_fn), which is where the truncate + self-restart actually
* run. This closes the thread-role violation: delaymgr also drives
* heartbeats and election timeouts cluster-wide, so blocking it here would
* stall liveness for the whole cluster (ha-async-io.md thread-role
* contract). With the flag OFF, runs the legacy inline blocking path
* (retained solely so a TDD RED test can show the pre-fix stall).
*
* In XCE_TESTING the test fixture drives this entry directly (un-started
* delaymgr) via its stored entry/args, exactly as before this offload.
*/
int ha_demotion_convergence_fn(void *arg)
{
ha_demotion_convergence_ctx_t *ctx = (ha_demotion_convergence_ctx_t *)arg;
if (ctx == NULL || ctx->c == NULL) {
free(ctx);
return -EINVAL;
}
cluster_t *c = ctx->c;
if (!ha_demotion_offload_enabled()) {
/* Legacy inline path (flag OFF) — blocks the calling (delaymgr)
* thread. Retained solely so the RED test can show the pre-change
* stall; production runs with the flag ON. */
int rc = ha_demotion_convergence_do_run(c);
free(ctx);
return rc;
}
/* Cluster-wide single-flight (demotion convergence is a single one-shot
* event, like configpush — not per-peer): CAS 0->1. Already 1 should be
* structurally unreachable — raft_cb_become_follower only schedules on
* the was-PRIMARY edge, and the role flips to STANDBY synchronously
* BEFORE scheduling — but stay defensive rather than assume. */
int expected = 0;
if (!atomic_compare_exchange_strong_explicit(
&c->ha.demotion_convergence_inflight, &expected, 1,
memory_order_acq_rel, memory_order_acquire)) {
WARN("HA demotion convergence: a convergence task is already in "
"flight — skipping duplicate schedule");
free(ctx);
return 0;
}
int srv = slowlane_submit(&c->slowlane, /*key=*/1, demotion_lane_fn, ctx);
if (srv == -EEXIST) {
/* Lane still holds key=1 from a prior (already-cleared) convergence
* task whose key removal hasn't caught up yet. Release the guard we
* just set (mirrors the reconnect/configpush -EEXIST handling in
* ha.c) and fall back to running inline: unlike reconnect/configpush
* — which retry on the next heartbeat cycle — a demotion-convergence
* event is one-shot with no later trigger, and this is a
* safety-critical action (an ex-primary discarding provisional state
* before it can be trusted again), so it must never be silently
* dropped. */
atomic_store_explicit(&c->ha.demotion_convergence_inflight, 0,
memory_order_release);
int rc = ha_demotion_convergence_do_run(c);
free(ctx);
return rc;
}
if (srv < 0 && srv != -EAGAIN) {
/* Hard submit failure (alloc / lane stopping). -EAGAIN is NOT a
* failure (still queued, will run). Same "never drop" reasoning as
* -EEXIST above. */
WARN("HA demotion convergence: slowlane_submit failed: %d — running "
"inline (a one-shot safety action, never dropped)", srv);
atomic_store_explicit(&c->ha.demotion_convergence_inflight, 0,
memory_order_release);
int rc = ha_demotion_convergence_do_run(c);
free(ctx);
return rc;
}
/* 0 or -EAGAIN: enqueued exactly once; demotion_lane_fn frees ctx and
* clears the flag after the (production: non-returning) restart. */
return 0;
}
/*