wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(raft-drive-async): endpoint-quiescence for bug-1197 vote-ep teardown (task 3.4a)

Browse source 
When the (still-default-OFF) peer-control dispatch flag is ON,
raft_cb_become_follower runs as a vtask pool worker and frees every peer
vote-ep while other pool workers may still hold those eps' data — a UAF
(bug-1197). This makes that path quiescence-safe so the flag can later be
flipped ON (deferred to the 4.x prod-flip gate; production stays OFF here).

Mechanism (no libxce-wide endpoint refcount; reuses the existing per-ep FIFO +
busy serialization):

- Deferred teardown-as-vtask: the loser loop, when dispatch is ON and the pool
  is wired, does per vote-ep endpoint_mark_tearing_down -> iomgr_unmonitor ->
  enqueue a VTASK_IO/VTASK_LIVENESS teardown vtask bound to that ep. Per-ep busy
  FIFO makes it run strictly after any in-flight/queued same-ep peer-control
  vtask; the body NULLs iod->ep before endpoint_fini+free(ep) to disarm
  run_task's post-body endpoint_unblock_tasks (which would touch the freed
  ep->lock). OFF / pool-unwired path is byte-for-byte the legacy endpoint_fini.

- RCU-like drain closes the dispatch/teardown TOCTOU window: new _Atomic int
  endpoint_t.dispatch_inflight (+ tearing_down:1 bit). The ON dispatch brackets
  its flag-check+enqueue with fetch_add/fetch_sub (seq_cst);
  endpoint_mark_tearing_down sets the flag under ep->lock, releases, then spins
  until dispatch_inflight drains to 0. So every dispatch that read the flag
  false finishes its enqueue before mark returns (FIFO-before the teardown
  vtask); any later dispatch reads the flag set and drops the packet. ep->lock
  is deliberately NOT held across the enqueue (it inverts with the taskq lock in
  vtask_eligible); the counter is the lock-free handshake. Drain is bounded: the
  held window is just a calloc + non-blocking enqueue on the single io_worker.

Still open before the production flag flip (4.x): the borrowed-ep race in
snapshot_lane_fn/configpush_lane_fn is distinct (slowlane thread vs io_worker
RDHUP teardown, not the vtask-pool FIFO) and needs its own per-standby-ep
discipline.

Tests: test_ha_vep_teardown.c — 5 ASan tests (ordering, late-event drop, run_task
unblock disarm, OFF-path unchanged, deterministic mark-drains-inflight-dispatch).
xcelib_test 282/0; controller_unit_test EXIT=0; ASan-clean. Adversarial
cpp-review: no CRITICAL/HIGH; handshake formally sound, flag-ON path race-free.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
This commit is contained in:
Rongsong Shen 2026-06-26 17:37:26 +00:00
parent bd02064ebc
commit 6d510d8cf4
10 changed files with 1037 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

190
src/controller/ha.c
View file

@ -108,6 +108,9 @@ extern int config_schedule_activate_retry(cluster_t *c);
static void on_vote_connected(endpoint_t *ep);
static void on_pre_vote_connected(endpoint_t *ep);
static int vote_ep_err_handler(void *ep_raw);
/* bug-1197: defer a vote-ep teardown onto the per-ep FIFO of the vtask pool so
* it runs strictly after the last live use of the ep (flag-ON loser path). */
static void vote_ep_teardown_async(ha_state_t *ha, endpoint_t *ep);
/* ha-single-leader-role task 2.2: shared promotion seam — every road to
* PRIMARY registers the persist hooks and arms the own-term gate. */
static void ha_register_persist_hooks(cluster_t *c);
@ -1285,11 +1288,24 @@ static void raft_cb_become_follower(void *ctx, uint64_t term, int leader_idx)
&ha->elect_ctx, memory_order_acquire);
if (ectx) {
for (int i = 0; i < ectx->n_peers; i++) {
if (ectx->votes[i].ep != NULL) {
INFO("HA: closing vote ep[%d] (loser path)", i);
endpoint_fini(ectx->votes[i].ep);
ectx->votes[i].ep = NULL;
endpoint_t *ep = ectx->votes[i].ep;
if (ep == NULL)
continue;
INFO("HA: closing vote ep[%d] (loser path)", i);
if (ha_peerctrl_dispatch_enabled() && ha->pool != NULL) {
/* bug-1197: this callback runs on a pool worker; another
* worker may be mid-vtask (or have a queued vtask) on this
* same ep reading ep->data. Defer teardown onto the ep's
* per-ep FIFO so endpoint_fini+free runs strictly after the
* last live use, and gate late dispatch (tearing_down). */
vote_ep_teardown_async(ha, ep);
} else {
/* Flag OFF / pool unwired: UNCHANGED legacy behavior —
* endpoint_fini only, no free(ep) (the struct is leaked per
* lost election, exactly as before this commit). */
endpoint_fini(ep);
}
ectx->votes[i].ep = NULL;
}
}
}
@ -5365,6 +5381,127 @@ static vtask_class_t peer_ctrl_classify(ha_state_t *ha, io_packet_t *pkt)
return VTASK_LIVENESS;
}
/* ── bug-1197: deferred vote-ep teardown ──────────────────────────────────────
*
* When g_drive_async_peerctrl_dispatch_enabled is ON, raft_cb_become_follower
* runs on a pool worker and must endpoint_fini() every loser vote-ep. But
* another pool worker can be mid-vtask (or have a queued vtask) on one of those
* same eps reading ep->data (a vote_ep_ctx_t) — freeing ep->data underneath it
* is a UAF. Instead of freeing inline we enqueue a teardown vtask BOUND to the
* ep (VTASK_IO): the pool's per-ep busy serialization guarantees it runs after
* any in-flight or earlier-queued same-ep vtask completes. We also mark the ep
* tearing_down + iomgr_unmonitor it BEFORE enqueue so the io_worker drops any
* late same-batch event instead of dispatching a new vtask onto a doomed ep. */
#ifdef XCE_TESTING
/* test-only ordering probe: fired (with the real ep) at the top of the teardown
* vtask body, before endpoint_fini, so a test can assert teardown ran strictly
* after the last live same-ep use. NULL in production. */
void (*ha_vep_teardown_probe)(endpoint_t *) = NULL;
/* test-only drain probe: fired in peer_ctrl_msg_handler AFTER the tearing_down
* check passes but BEFORE the pool enqueue (inside the dispatch_inflight
* bracket), so a test can park a dispatch in the drained window and prove
* endpoint_mark_tearing_down blocks until the enqueue completes. NULL in
* production. */
void (*ha_peerctrl_predispatch_probe)(endpoint_t *) = NULL;
#endif
/* Runs on a pool worker, bound VTASK_IO to ep, strictly AFTER any in-flight or
* queued same-ep peer-control vtask (per-ep busy FIFO). On the loser path
* ectx->votes[i].ep is the sole reference to ep, so this owns the free. */
static int vote_ep_teardown_vtask_entry(vtask_t *task)
{
io_task_data_t *iod = &task->data->u.iodata;
endpoint_t *ep = iod->ep;
/* Disarm run_task's post-body endpoint_unblock_tasks: it would touch
* ep->lock AFTER we free ep below. NULLing the task's ep makes run_task
* skip the unblock (run_task: `if (ep != NULL) endpoint_unblock_tasks`). */
iod->ep = NULL;
if (ep != NULL) {
#ifdef XCE_TESTING
if (ha_vep_teardown_probe)
ha_vep_teardown_probe(ep);
#endif
endpoint_fini(ep);
free(ep);
}
return 0;
}
/* Build + enqueue the teardown vtask. Caller has ALREADY
* endpoint_mark_tearing_down(ep) + iomgr_unmonitor(ep). Controllers never
* self-terminate: on any alloc/enqueue failure we fail loud (WARN) and tear the
* ep down inline — with the pool unwired or the task rejected there is no other
* worker that could be racing this ep, so an inline endpoint_fini+free is safe. */
static void vote_ep_teardown_enqueue(ha_state_t *ha, endpoint_t *ep)
{
vtaskmgr_t *pool = (vtaskmgr_t *)ha->pool;
vtask_data_t *data = calloc(1, sizeof(*data));
vtask_t *task = calloc(1, sizeof(*task));
if (pool == NULL || data == NULL || task == NULL) {
WARN("bug-1197: vote-ep teardown enqueue unavailable (pool=%p data=%p "
"task=%p) — tearing down inline (stay alive)",
(void *)pool, (void *)data, (void *)task);
free(data);
free(task);
endpoint_fini(ep);
free(ep);
return;
}
data->vtype = VTASK_IO;
data->u.iodata.ep = ep; /* drives per-ep serialization in bind_vtask */
data->u.iodata.pkt = NULL;
vtask_init(task, (vtask_func_t)vote_ep_teardown_vtask_entry, data);
/* VTASK_LIVENESS is never admission-gated, so the teardown can always be
* enqueued even when the pool is saturated by blocking tasks. */
if (vtask_enq_classified(pool, task, VTASK_LIVENESS) < 0) {
WARN("bug-1197: vote-ep teardown enqueue failed — tearing down inline "
"(stay alive)");
free(data);
free(task);
endpoint_fini(ep);
free(ep);
}
}
/* The flag-ON loser-path teardown for one vote-ep: mark tearing_down, stop epoll
* wakeups, then defer the free onto the ep's per-ep FIFO. ORDERING (bug-1197):
* - endpoint_mark_tearing_down sets the flag under ep->lock; the io_worker's
* late-event gate (peer_ctrl_msg_handler, ON branch) reads it under ep->lock
* via endpoint_is_tearing_down, so any dispatch DECISION taken after the
* mark observes it and drops — nothing new is enqueued for this ep.
* - iomgr_unmonitor stops FUTURE epoll events (an event already in the current
* batch is handled by the tearing_down gate above).
* - the teardown vtask is itself VTASK_IO bound to ep, so the pool's per-ep
* busy serialization runs it strictly after any in-flight / earlier-queued
* same-ep vtask completes.
* NOTE: we cannot hold ep->lock ACROSS the dispatch enqueue (vtask_enq_classified
* → dataq_enq takes the taskq lock, while the pool dispatcher takes ep->lock
* UNDER the taskq lock in vtask_eligible → endpoint_is_task_blocked — an AB/BA
* inversion). The check is therefore done under ep->lock and the enqueue
* outside it; see the io_worker gate for the residual-window note. */
static void vote_ep_teardown_async(ha_state_t *ha, endpoint_t *ep)
{
endpoint_mark_tearing_down(ep);
if (ep->iomgr != NULL)
iomgr_unmonitor(ep->iomgr, ep);
vote_ep_teardown_enqueue(ha, ep);
}
#ifdef XCE_TESTING
/* ha_vote_ep_teardown_for_test: drive the flag-ON per-ep async teardown branch
* for a single ep without staging a full election/become_follower. */
void ha_vote_ep_teardown_for_test(ha_state_t *ha, endpoint_t *ep)
{
vote_ep_teardown_async(ha, ep);
}
#endif
/*
* peer_ctrl_dispatch_to_pool (raft-drive-async task 3.1): hand the decoded
* peer-control packet to the vtask pool and return promptly. Per-endpoint
@ -5466,7 +5603,50 @@ static int peer_ctrl_msg_handler(void *ep_raw)
if (!ha_peerctrl_dispatch_enabled())
return peer_ctrl_msg_handler_inline(ep, pkt);
return peer_ctrl_dispatch_to_pool(ep, pkt);
/* bug-1197 late-event gate (ON path only): if this ep is being torn down,
* a same-batch event may still be delivered by the io_worker after the
* teardown thread called iomgr_unmonitor. Dropping it here keeps the doomed
* ep off the vtask pool.
*
* The window between the tearing_down check and the pool enqueue is closed
* by ep->dispatch_inflight: we increment it (seq_cst) BEFORE reading the
* flag and decrement it AFTER the enqueue completes. endpoint_mark_tearing_
* down sets the flag (under ep->lock) and then DRAINS this counter to 0
* before returning. So once the teardown caller's mark returns, every
* dispatch that read the flag as false has already finished its enqueue;
* because the teardown vtask is enqueued only AFTER mark returns, that
* peer-control vtask is strictly FIFO-before the teardown vtask (and gates
* on the same ep busy bit). Any dispatch starting after the flag is set
* reads it true here and drops. No same-ep vtask is ever enqueued after the
* teardown vtask — the window is fully closed. We must NOT hold ep->lock
* across the enqueue (it would invert with the taskq lock, see
* vote_ep_teardown_async); the counter is the lock-free handshake instead. */
atomic_fetch_add_explicit(&ep->dispatch_inflight, 1, memory_order_seq_cst);
if (endpoint_is_tearing_down(ep)) {
if (pkt) { free_packet(pkt); free(pkt); }
atomic_fetch_sub_explicit(&ep->dispatch_inflight, 1,
memory_order_seq_cst);
return 0;
}
#ifdef XCE_TESTING
/* test-only seam: fires AFTER the flag-check passed but BEFORE the enqueue,
* so a test can park a dispatch in exactly the drained window and prove
* endpoint_mark_tearing_down blocks until the enqueue completes. */
if (ha_peerctrl_predispatch_probe)
ha_peerctrl_predispatch_probe(ep);
#endif
/* NOTE (drain liveness): this call runs while dispatch_inflight is held, so
* a concurrent endpoint_mark_tearing_down on this ep spins until it returns.
* peer_ctrl_dispatch_to_pool only ever enqueues (non-blocking) or, on OOM,
* runs peer_ctrl_msg_handler_inline — which must also stay non-blocking
* (the vtask-handler no-block invariant). Keep it bounded or the drain is. */
int rc = peer_ctrl_dispatch_to_pool(ep, pkt);
atomic_fetch_sub_explicit(&ep->dispatch_inflight, 1, memory_order_seq_cst);
return rc;
}
#ifdef XCE_TESTING