lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race

Browse source 
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).

bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.

Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
This commit is contained in:
Rongsong Shen 2026-07-11 12:16:28 +00:00
parent ea0ac81c92
commit 40f510fb14
25 changed files with 1617 additions and 226 deletions
Download patch file Download diff file Expand all files Collapse all files

123
src/controller/config_repl.c
View file

@ -391,20 +391,27 @@ int config_activate_committed(cluster_t *c)
* non-promotable) for that upgrade until reboot — a §7.9 liveness violation.
*
* The fix: when commit advances WHILE this member is serve-fenced, schedule a
* one-shot OFF-IOMGR delaymgr task that re-runs config_activate_committed. The
* delaymgr worker runs on a thread allowed to block on the disk reads
* (scan_latest_committed_marker pread + load_primary_config). Once the NEW marker
* is in the committed prefix AND the locally-held bytes hash-match it, the retry
* activates and CLEARS the fence; a genuine still-MISMATCH (or no-bytes-yet) keeps
* the fence and the next commit-advance reschedules. -ENOENT still falls back
* without fencing (the no-marker path is benign). Idempotent + self-limiting: the
* scheduler is a no-op once unfenced, so there is no busy-spin. */
* one-shot task that re-runs config_activate_committed off the iomgr ACK thread.
* The delaymgr thread itself must never block (ha-async-io.md thread-role
* contract — it also drives heartbeats and every election timeout), so the
* one-shot delaymgr task is only a trigger: it hands the actual blocking disk
* reads (scan_latest_committed_marker pread + load_primary_config) to a
* VTASK_LIVE_BLOCK vtask and returns immediately (2026-07-09 correction — this
* comment previously and incorrectly claimed the delaymgr worker itself was
* "allowed to block on the disk reads"; that predates the corrected contract
* ha-async-io.md documents). Once the NEW marker is in the committed prefix AND
* the locally-held bytes hash-match it, the retry activates and CLEARS the
* fence; a genuine still-MISMATCH (or no-bytes-yet) keeps the fence and the next
* commit-advance reschedules. -ENOENT still falls back without fencing (the
* no-marker path is benign). Idempotent + self-limiting: the scheduler is a
* no-op once unfenced, so there is no busy-spin. */
/* config_activate_retry_fn: the delaymgr task body (runs OFF the iomgr ACK
* thread). Re-runs config_activate_committed so a freshly-committed config marker
* is activated — and a stale serve-fence cleared — once its out-of-band bytes
* verify. `arg` is the cluster_t* (passed directly; no heap ctx to free, so the
* delaymgr node-killer that frees only the delay_task_t never leaks). */
/* config_activate_retry_fn: re-runs config_activate_committed so a freshly-
* committed config marker is activated — and a stale serve-fence cleared —
* once its out-of-band bytes verify. Does the actual blocking disk I/O; callers
* must invoke it off the delaymgr thread (see config_activate_retry_vtask_entry
* / config_schedule_activate_retry below). Public so tests can drive it
* directly without a vtask pool. */
int config_activate_retry_fn(void *arg)
{
cluster_t *c = (cluster_t *)arg;
@ -425,8 +432,56 @@ int config_activate_retry_fn(void *arg)
return rc;
}
/* config_schedule_activate_retry: enqueue config_activate_retry_fn on the
* delaymgr (NON-BLOCKING — delay_task_ms only does ordq_put). Called from the
/* config_activate_retry_vtask_entry: vtask entry wrapping config_activate_
* retry_fn so its blocking disk I/O runs on a vtask worker, never inline on
* the single delaymgr thread. */
static int config_activate_retry_vtask_entry(vtask_t *task)
{
cluster_t *c = (cluster_t *)task->data->u.generic;
config_activate_retry_fn(c);
return 0;
}
/* config_activate_retry_delaymgr_trampoline: the delaymgr task body. Delaymgr
* must only trigger + enqueue, never block — dispatch the actual retry work to
* a vtask (VTASK_LIVE_BLOCK, the same class the vote-grant term fsync uses,
* R4.6) and return immediately. Fails loud and runs inline (stay alive, never
* silently drop a scheduled retry) if the pool is unavailable. */
static int config_activate_retry_delaymgr_trampoline(void *arg)
{
cluster_t *c = (cluster_t *)arg;
if (!c)
return -EINVAL;
vtaskmgr_t *pool = &c->vtaskmgr;
vtask_data_t *data = calloc(1, sizeof(*data));
vtask_t *task = vtask_alloc();
if (data == NULL || task == NULL) {
WARN("config commit-advance retry: vtask alloc failed — running "
"inline (stay alive)");
free(data);
free(task);
return config_activate_retry_fn(c);
}
data->vtype = VTASK_GENERIC;
data->u.generic = c;
vtask_init(task, (vtask_func_t)config_activate_retry_vtask_entry, data);
if (vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK) < 0) {
WARN("config commit-advance retry: vtask enqueue failed — running "
"inline (stay alive)");
free(data);
free(task);
return config_activate_retry_fn(c);
}
return 0;
}
/* config_schedule_activate_retry: enqueue the retry trampoline on the delaymgr
* (NON-BLOCKING — delay_task_ms only does ordq_put; the trampoline itself
* offloads the blocking work to a vtask, see above). Called from the
* commit-advance path (iomgr ACK thread) so activation's disk I/O runs OFF that
* thread (ha-async-io invariant). No-op when the member is NOT serve-fenced (so
* the steady state schedules nothing) or when the delaymgr is not running (the
@ -443,7 +498,8 @@ int config_schedule_activate_retry(cluster_t *c)
/* One-shot, minimal delay: run promptly but off the iomgr ACK thread. */
int rc = delay_task_ms(&c->delaymgr, 1, 1,
(delay_task_entry_fn_t)config_activate_retry_fn, c);
(delay_task_entry_fn_t)config_activate_retry_delaymgr_trampoline,
c);
if (rc < 0)
WARN("config commit-advance retry: delay_task_ms failed: %d — fence will "
"clear on a later commit advance or config-push", rc);
@ -2540,12 +2596,21 @@ typedef struct {
} ha_catchup_continuation_ctx_t;
/*
* ha_catchup_continuation_fn — delaymgr task body (runs OFF the iomgr walAck
* branch, D6/D1). Re-validate the endpoint against the registry, re-derive
* target, stream [acked_lsn+1, target], and clear catchup_inflight on EVERY exit
* path (success, ep-gone, nothing-to-stream) so a stuck flag never blocks future
* continuations (Req 5.2). Frees its own ctx (body-frees-own-ctx pattern,
* mirroring ha_demotion_convergence_fn).
* ha_catchup_continuation_fn — the real catch-up work. Re-validate the
* endpoint against the registry, re-derive target, stream [acked_lsn+1,
* target], and clear catchup_inflight on EVERY exit path (success, ep-gone,
* nothing-to-stream) so a stuck flag never blocks future continuations (Req
* 5.2). Frees its own ctx (body-frees-own-ctx pattern, mirroring
* ha_demotion_convergence_run).
*
* Runs as a delaymgr task body (see ha_schedule_catchup_continuation below) —
* must run off the iomgr walAck thread, since catchup_stream_range ->
* ha_stream_wal_to_standby does blocking WAL file reads + socket writes.
* (2026-07-10: a VTASK_LIVE_BLOCK dispatch was attempted here to also get it
* off the delaymgr thread per ha-async-io.md's thread-role contract, but it
* deadlocked the controller_unit_test harness — reverted pending a proper
* fix; tracked as a known, still-open thread-role violation alongside the
* other raft-drive-async P2 items.)
*/
int ha_catchup_continuation_fn(void *arg)
{
@ -2587,10 +2652,11 @@ clear_and_done:
}
/*
* ha_schedule_catchup_continuation — enqueue one one-shot delaymgr task running
* ha_catchup_continuation_fn. The single-flight flag is set by the caller
* (ha_walack_on_ack) before this call; on a scheduling failure this clears it so
* the slot does not stay stuck. Returns 0 on success, negative on failure.
* ha_schedule_catchup_continuation — enqueue one one-shot delaymgr task
* running ha_catchup_continuation_fn directly. The single-flight flag is set
* by the caller (ha_walack_on_ack) before this call; on a scheduling failure
* this clears it so the slot does not stay stuck. Returns 0 on success,
* negative on failure.
*/
static int ha_schedule_catchup_continuation(cluster_t *c, repl_ep_t ep,
int peer_idx, uint64_t acked_lsn)
@ -2610,8 +2676,9 @@ static int ha_schedule_catchup_continuation(cluster_t *c, repl_ep_t ep,
/* One-shot (runcount=1), minimal delay: streaming reads WAL files and must
* run off the iomgr walAck thread. delay_task_ms only enqueues (ordq_put);
* the delaymgr worker runs it. In XCE_TESTING the fixture drives the queued
* task synchronously via its stored entry/args. */
* the delaymgr worker runs ha_catchup_continuation_fn itself. In
* XCE_TESTING the fixture drives the queued task synchronously via its
* stored entry/args. */
int rc = delay_task_ms(&c->delaymgr, 1, 1,
(delay_task_entry_fn_t)ha_catchup_continuation_fn,
ctx);