lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): bug-1450/1451 offload delaymgr-inline blocking work to the slow lane

Browse source 
/xce-audit-invariants controller found two live violations of the async
I/O thread-role contract: ha_demotion_convergence_fn (ha_convergence.c)
ran WAL truncation + tlog_exit()+execv() directly on the delaymgr thread,
and ha_catchup_continuation_fn (config_repl.c) ran unbounded WAL-streaming
to a standby directly on delaymgr -- the same single thread that drives
heartbeats and election timeouts cluster-wide, so either blocks liveness
for the whole cluster and risks a spurious election.

A prior vtask-pool offload attempt (2026-07-10, bug-1429) deadlocked
controller_unit_test because c->ha.pool is never initialized by the
lightweight unit-test fixtures these functions run under, and was fully
reverted. This fix instead uses the codebase's established slow-lane
pattern (the same mechanism already used for snapshot-send, config-push,
and peer connect) -- a 4th and 5th instance, not a new mechanism.

New g_drive_async_demotion_offload / g_drive_async_catchup_offload flags
ship default OFF, matching the original rollout discipline of the three
reference offloads (shipped OFF, flipped ON later after a dedicated
compatibility audit). Flipping ON was tried during validation and
reproduced two distinct hazards against pre-existing fixtures that never
call slowlane_init: the already-known hang, and a new ASan SEGV in
commit_advance_locked (logged as bug-1452, not root-caused, deferred to
the future flip-to-ON audit). Both flags verified OFF and the full
976-test suite clean, then the committed state re-verified clean against
a live 10-round chaotic and 10-round sequential HA soak (zero job loss,
zero divergence, both PASS).
This commit is contained in:
Rongsong Shen 2026-07-19 08:14:00 +00:00
parent aeb6e37ce7
commit f88e4ef431
5 changed files with 877 additions and 55 deletions
Download patch file Download diff file Expand all files Collapse all files

100
src/controller/ha.h
View file

@ -359,6 +359,26 @@ typedef struct {
* set, the heartbeat decision loop /
* promotion enqueues no duplicate config
* push. 0 at init (ha_init memset). */
/* ── New field added by xce-audit-invariants controller pass (thread-role
* violation 1 fix, 2026-07-18) ──────────── */
_Atomic int demotion_convergence_inflight; /* CLUSTER-WIDE single-
* flight guard for the offloaded
* demotion-convergence task (truncate
* above committed_lsn + self-restart).
* Like configpush_inflight, this is a
* single scalar, not a per-peer array —
* demotion convergence is a one-shot,
* whole-node event, never per-peer. Set
* (CAS 0->1) when the task is submitted
* to the slow lane, cleared by the lane
* worker on completion. Defensive only:
* raft_cb_become_follower's was-PRIMARY
* check already makes scheduling this
* task structurally single-shot (role
* flips to STANDBY synchronously before
* scheduling), so this guard should never
* actually observe a collision in
* practice. 0 at init (ha_init memset). */
/* ── New field added by ha-entity-snapshots spec (Task 4.1) ──────── */
_Atomic uint64_t snapshot_inflight_lsn[MAX_HA_CANDIDATES]; /* Per-peer
* checkpoint LSN referenced by an
@ -560,6 +580,12 @@ int ha_repl_ack_abs_peer_idx(ha_state_t *ha, int ep_is_vote, int peer_idx);
* so all file I/O and the exec run off the become-follower callback thread (D1).
* Returns 0 on success, negative on alloc/schedule failure (suffix is then
* reconciled by rejoin reconciliation at the next join).
*
* The scheduled task body (ha_demotion_convergence_fn) CAN run off the
* delaymgr thread entirely — see ha_set_demotion_offload_for_test: with the
* offload flag ON, it hands the truncate + self-restart off to the slow
* lane and returns immediately (flag currently defaults OFF pending a
* cluster-wide test-suite compatibility audit — see the flag's doc comment).
*/
int ha_schedule_demotion_convergence(cluster_t *c);
@ -568,6 +594,11 @@ int ha_schedule_demotion_convergence(cluster_t *c);
* ha_demotion_convergence_fn: the delaymgr task body — exposed under XCE_TESTING
* only so the demotion-convergence suite can reference its contract; the test
* fixture drives the queued task through its stored entry pointer.
*
* With the offload flag ON, this function itself only does fast,
* non-blocking admin work (ctx unpack, single-flight CAS, slowlane_submit)
* and returns immediately; the truncate + ha_self_restart body runs on a
* slow-lane worker (see ha_set_demotion_offload_for_test).
*/
int ha_demotion_convergence_fn(void *arg);
#endif
@ -623,6 +654,13 @@ void cluster_apply_ctrlhello_reply(cluster_t *c, uint64_t committed_lsn,
* [acked_lsn+1, target] on delaymgr (never inline on
* iomgr), UNLESS c->ha.catchup_inflight[peer_idx] is
* already set (per-standby single-flight, Req 5.2).
* The scheduled task body (ha_catchup_continuation_fn)
* CAN run off the delaymgr thread entirely — see
* ha_set_catchup_offload_for_test: with the
* offload flag ON, the actual WAL streaming runs
* on the slow lane (flag currently defaults OFF
* pending a cluster-wide test-suite compatibility
* audit — see the flag's doc comment).
* if acked_lsn >= target → repl_complete_catchup(&c->persist.repl, ep) to
* re-include the standby in live replication (Req 7.1).
*
@ -1425,6 +1463,68 @@ int ha_snapshot_offload_enabled(void);
void ha_set_configpush_offload_for_test(int on);
int ha_configpush_offload_enabled(void);
/*
* ha_set_demotion_offload_for_test / ha_demotion_offload_enabled:
* TDD kill-switch (xce-audit-invariants controller pass, thread-role
* violation 1 fix, 2026-07-18) for the demotion-convergence slow-lane
* offload. ha_demotion_convergence_fn is scheduled directly as a delaymgr
* task body; it must not block that thread (it also drives heartbeats and
* election timeouts cluster-wide — ha-async-io.md thread-role contract).
* ON: the truncate-above-committed_lsn + ha_self_restart (tlog_exit +
* execv) body runs on a slow-lane worker; ha_demotion_convergence_fn
* itself only does the fast admin work (CAS + slowlane_submit) and
* returns immediately — this closes the thread-role violation.
* OFF (0, PRODUCTION DEFAULT FOR NOW): legacy inline path — the blocking
* body runs directly on the caller (delaymgr) thread.
* Mirrors ha_set_reconnect_offload_for_test / ha_set_configpush_offload_for_test
* (raft-drive-async); demotion convergence is a 4th instance of the SAME
* slow-lane pattern, not a new mechanism — a prior VTASK_LIVE_BLOCK vtask-
* pool attempt here (2026-07-10) deadlocked controller_unit_test and was
* reverted (bug-1429). The slow lane's own submit contract cannot
* deadlock a caller that way, but flipping this flag ON by default
* reproduced a DIFFERENT real hang: raft_cb_become_follower (the caller)
* is exercised by a very broad set of pre-existing controller_unit_test
* fixtures that never call slowlane_init, so the offloaded work is
* silently queued and never runs, and downstream code waiting on that
* now-never-happening completion hangs. OFF-by-default matches the
* ORIGINAL rollout discipline of all three reference offloads (every one
* shipped OFF, flipped ON only in a later dedicated compatibility-audit
* pass — raft-drive-async task 3.4). Flipping this default to ON is the
* same kind of follow-up, out of scope here (bug-1450, buglog.json).
*/
void ha_set_demotion_offload_for_test(int on);
int ha_demotion_offload_enabled(void);
/*
* ha_set_catchup_offload_for_test / ha_catchup_offload_enabled:
* TDD kill-switch (xce-audit-invariants controller pass, thread-role
* violation 2 fix, 2026-07-18) for the catch-up continuation slow-lane
* offload. ha_catchup_continuation_fn is scheduled directly as a delaymgr
* task body; catchup_stream_range -> ha_stream_wal_to_standby does blocking
* WAL file reads + socket writes to a standby, which can stream a variable,
* potentially large amount of WAL data — an unbounded block on the SAME
* single thread that drives heartbeats and election timeouts cluster-wide
* is a real spurious-election risk (ha-async-io.md thread-role contract).
* ON: the streaming body runs on a slow-lane worker; ha_catchup_
* continuation_fn itself only does the fast admin work (slowlane_submit)
* and returns immediately — this closes the thread-role violation.
* c->ha.catchup_inflight[peer_idx] (set by ha_walack_on_ack before
* scheduling) remains the authoritative per-standby single-flight guard;
* the lane's own per-key dedup (key = peer_idx+1) is a redundant backstop,
* same relationship as the reconnect offload.
* OFF (0, PRODUCTION DEFAULT FOR NOW): legacy inline path — the blocking
* body runs directly on the caller (delaymgr) thread.
* 5th instance of the SAME established slow-lane pattern (see
* ha_set_demotion_offload_for_test above for why NOT the vtask pool, and
* for why OFF-by-default for now — the identical hazard applies here:
* test_convergence_verification.c's pre-existing catch-up fixtures never
* call slowlane_init, and flipping this flag ON by default reproduced a
* real hang against them). Flipping the default to ON is a follow-up
* compatibility audit, out of scope here (bug-1451, buglog.json).
*/
void ha_set_catchup_offload_for_test(int on);
int ha_catchup_offload_enabled(void);
/*
* ha_set_peerctrl_dispatch_for_test / ha_peerctrl_dispatch_enabled:
* Feature-flag accessors (raft-drive-async Task 3.1). The dispatch flag gates