myxce/.kiro/specs/raft-drive-async/tasks.md

Implementation Plan

Tasks mirror the design's 4-phase migration. The slow lane is foundational (every offload submits to it); the peer_ctrl→vtask move is coupled with its long-op offload in Phase 3 so no phase runs long I/O inline in a blocked vtask.

• 1. Phase 1 — SlowLane executor and bounded connect (foundation)

• 1.1 Add drive-async configuration constants

  • Define connect-timeout, slow-lane worker count, and max-in-flight constants with documented defaults (sized to peer count).
  • Define the liveness reservation as a peer-scaled derivation, not a flat literal: VTASK_LIVENESS_RESERVED = clamp(n_peers − 1, 1, n_vcpus − 1) (floor 1 = deadlock-freedom; ceiling n_vcpus − 1 keeps ≥1 commit slot); WARN at init when clamped by n_vcpus.
  • Wire the constants into the controller config so they are overridable.
  • Observable: a build with the new constants compiles; defaults are readable at controller startup; the resolved reservation logged at init.
  • Requirements: 2.3, 3.1

• 1.2 Implement the SlowLane executor

  • Bounded pool of dedicated worker threads, isolated from the vtask request pool; non-blocking submit that enqueues and returns immediately.
  • Admission cap on concurrent in-flight tasks; per-key in-flight dedup (key 0 disables dedup); init/fini wired into the controller lifecycle.
  • Observable: a submitted task runs exactly once on a lane worker; submit never blocks the caller; concurrent running tasks never exceed the cap; a submit with an in-flight key is a no-op.
  • Requirements: 3.1, 3.2, 3.3, 3.4, 1.5
  • Boundary: SlowLane

• 1.3 Bound the peer connect path

  • Reorder socket_connect to set non-blocking before connect and bound the attempt with a poll timeout (reuse the existing probe pattern); return success/timeout/error.
  • Update connect callers to pass the configured timeout.
  • Observable: a connect to an unreachable host returns within the configured timeout instead of the OS connect timeout; a connect to a reachable host still succeeds.
  • Requirements: 2.3
  • Boundary: BoundedConnect
  • Depends: 1.1

• 1.4 (P) SlowLane unit tests

  • Assert non-blocking submit, admission-cap bounds on concurrency, per-key dedup returns the deduped result, and run-exactly-once.
  • Observable: tests pass and fail if the cap or dedup is removed.
  • Requirements: 3.1, 3.2, 3.3, 3.4, 1.5
  • Boundary: SlowLane
  • Depends: 1.2

• 1.5 (P) Bounded-connect unit test

  • Assert socket_connect returns within the timeout against an unreachable host and succeeds against a reachable one.
  • Observable: test passes; reverting the reorder makes it hang/fail.
  • Requirements: 2.3
  • Boundary: BoundedConnect
  • Depends: 1.3

• 1.6 LivenessAdmission — pool-level task-class reservation in vtaskmgr

  • Add vtask_class_t (VTASK_LIVENESS / VTASK_LIVE_BLOCK / VTASK_REQ_BLOCK) + a classified submit; the dispatcher reserves the VTASK_LIVENESS_RESERVED value resolved in 1.1 (single source of truth — consume it, do not recompute the clamp/WARN here) and fast-rejects a VTASK_REQ_BLOCK with -XCE_ERR_NOT_READY when admitting it would breach the reservation (never parks a worker, never silently queues).
  • Fold commit.c's inflight_waits/COMMIT_WAIT_RESERVED into this — commit_submit submits a VTASK_REQ_BLOCK; the pool owns the single reservation.
  • Dormant infra in Phase 1: no behavior change until peer_ctrl dispatches through it (3.1). Hard prerequisite of 3.1 — without it, moving ack/vote handling onto the pool is not deadlock-safe (vote-fsync/config-ack waits, invisible to the old commit-only counter, could starve the ack that wakes commit-waiters).
  • Observable: a VTASK_LIVENESS task always finds a worker even when *_BLOCK tasks saturate n_vcpus − VTASK_LIVENESS_RESERVED; commit-wait still fast-rejects -XCE_ERR_NOT_READY at the cap (backpressure unchanged).
  • Requirements: 3.1, 3.2, 3.3, 4.5
  • Boundary: LivenessAdmission
  • Depends: 1.1

• 1.7 (P) LivenessAdmission unit test

  • Assert: a VTASK_LIVENESS task is dispatched while *_BLOCK tasks hold n_vcpus − VTASK_LIVENESS_RESERVED vcpus; VTASK_REQ_BLOCK over the cap returns -XCE_ERR_NOT_READY without parking; folding commit-wait through the new path preserves the existing reject-at-cap behavior. Reverting the reservation makes a liveness task starve behind blockers.
  • Requirements: 3.1, 3.2, 3.3, 4.5
  • Boundary: LivenessAdmission
  • Depends: 1.6

• 2. Phase 2 — Reconnect offload (sender-side liveness)

• 2.1 Offload the delaymgr peer reconnect to the slow lane

  • The heartbeat timer detects a disconnected peer and submits the reconnect to the lane (per-peer dedup key), then returns immediately; the timer never performs the connect inline.
  • Refcount the endpoint across the in-flight window.
  • Observable: while one peer is unreachable, the delaymgr timer keeps firing and heartbeats to reachable peers continue within the interval; only one reconnect attempt per peer is in flight at a time.
  • Requirements: 2.1, 2.2, 2.5
  • Boundary: ReconnectOffload, SlowLane
  • Depends: 1.2, 1.3

• 2.2 Complete the reconnect on the lane worker

  • The lane worker performs the bounded connect, fully prepares the endpoint, and registers it via the lock-guarded monitor interface; on failure it clears the in-flight flag and lets the next tick retry without terminating; resolve the inbound-vs-outbound connection race per the connection-ownership rules.
  • Observable: a successful reconnect registers the new connection with no lost or reordered queued messages; a failed attempt leaves the controller alive and retrying.
  • Requirements: 2.4, 2.6, 4.4
  • Boundary: ReconnectOffload
  • Depends: 2.1

• 2.3 Reconnect-offload integration test

  • With one peer made unreachable, assert heartbeats to the surviving peers continue within the interval, the delaymgr is not stalled, and reconnect is deduped per peer.
  • Observable: test passes; reverting the offload (inline connect) makes heartbeats to survivors stall.
  • Requirements: 2.1, 2.2, 2.5
  • Boundary: ReconnectOffload
  • Depends: 2.2

• 3. Phase 3 — Peer-control dispatch to vtask, coupled with long-op offload (receiver-side)

• 3.1 Dispatch peer-control message handling to the vtask pool

  • Register peer endpoints so peer-control messages dispatch through the vtask pool instead of running inline on iomgr; convert the peer-control handler into a vtask entry that does only bounded protocol work inline.
  • Preserve per-connection processing order via the per-endpoint block primitive and preserve vote/ack/term semantics and persist-before-vote ordering; keep the steady-state heartbeat/vote send paths as non-blocking enqueues.
  • Observable: peer-control no longer runs on the iomgr loop; existing election/heartbeat/repl behavior is unchanged; messages on one connection are processed in arrival order.
  • Deadlock-freedom gate (bug-invariant reversal): moving ack decode to the pool co-locates the commit-barrier wakers (acks) and waiters (commit_submit(COMMIT_MODE_WAIT)) on one pool — today acks run on iomgr precisely to avoid this (commit.c:16). The old commit-only COMMIT_WAIT_RESERVED is insufficient (it can't see vote-fsync/config-ack blockers this move also puts on the pool), so safety rests on LivenessAdmission (task 1.6): ack/vote-receipt dispatched as VTASK_LIVENESS (never gated, never offloaded), commit_submit as VTASK_REQ_BLOCK under the pool reservation. Task 3.4 is the admission gate for this task — 3.1 is not complete until 3.4 is green.
  • Requirements: 1.4, 4.1, 4.2, 4.5, 4.6, 5.1, 5.4, 8.1, 8.2
  • Boundary: PeerCtrlDispatch
  • Depends: 1.2, 1.6

• [~] 3.2 Offload snapshot send to the slow lane

  • Trigger snapshot send + checkpoint to the lane from the vtask entry without holding the per-endpoint block across the transfer; dedup one snapshot per standby.
  • Enforce the snapshot/checkpoint concurrency invariant: snapshot reads a frozen term/lsn boundary; segment trim never deletes at/above any in-flight snapshot boundary or a standby catch-up cursor; at most one checkpoint in flight; bound the in-lane snapshot socket write so an unreachable-mid-transfer peer cannot pin a worker.
  • Add the trim cursor-bound guard (bug-1147, stopgap): trim_wal_segments (persist.c:1374) currently bounds deletion ONLY by the open segment and consults no standby cursor; before this task moves checkpoint/trim onto a concurrent lane worker, change it to compute floor = min(all standby acked_lsn, in-flight snapshot boundaries) under p->lock and refuse to delete at/above floor. This is strategy-A retention discipline; it does NOT fix empty/far-behind join (cursor=0) — that needs entity-bearing snapshots (separate spec).
  • Observable: a catch-up that triggers a snapshot does not block iomgr; the triggering peer's later messages are not queued behind the transfer; concurrent appends plus checkpoint/trim never delete a segment at/above any live standby acked_lsn or in-flight snapshot boundary (reverting the guard makes the test re-create a bug-795/828 gap).
  • Requirements: 1.1, 1.3, 1.5, 4.3
  • Boundary: PeerCtrlDispatch, SlowLane
  • Depends: 3.1

• [~] 3.3 (P) Offload the transport-only config-push to the slow lane

  • Move the requester-less iomgr/promotion config-push transport (and its wait) to the lane; replace the drive-thread join with a lane-internal wait and non-blocking return; leave the admin upgrade path (with a waiting requester) on its existing vtask-blocking path unchanged.
  • Observable: a promotion/heartbeat-driven config push no longer blocks iomgr; the admin upgrade reply path is unchanged.
  • Requirements: 1.2
  • Boundary: PeerCtrlDispatch, SlowLane
  • Depends: 3.1

• 3.4 Receiver-side integration tests

  • 3.4a (DONE, committed 6d510d8): bug-1197 endpoint-quiescence fix landed so flag-ON peer-control dispatch is UAF-free (deferred teardown-vtask + RCU dispatch_inflight drain).
  • 3.4b (DONE): receiver-side liveness integration tests (test_ha_dispatch_integration.c) with the dispatch flags flipped ON in-test — P1 a parked snapshot/config-push lane transfer does not block another endpoint's peer-control on the pool; P2 a VTASK_LIVENESS ack runs under REQ_BLOCK pool saturation (reservation never starves it; one-over fast-rejects to prove the boundary); P3 cross-endpoint peer-control runs concurrently (per-ep FIFO does not serialize across eps). Fixed a flaky teardown-race hang (bug-1224: poll end-of-body completion counters before the cancel-based vtaskmgr_fini). 5/5 deterministic, full controller suite EXIT=0, ASan-clean.
  • Production flags STILL DEFAULT OFF (peerctrl/snapshot/configpush). Per the "full fix, defer prod flip" decision, the production flip happens only after 4.1 partition soak + 4.2 regression sweep pass; the snapshot/configpush lane borrowed-ep refcount (distinct from 3.4a) is also required before flipping those two ON.
  • Assert that during a snapshot/config-push transfer the controller still processes other peers' heartbeats and votes; that a quorum of WAL-acks advances the commit position promptly even with the vtask pool near saturation; and that cross-endpoint acks are not serialized behind one another.
  • Observable: tests pass; commit advances after a quorum of acks under load; iomgr stays responsive during transfers.
  • Requirements: 1.1, 1.3, 1.4, 4.3, 4.5
  • Boundary: PeerCtrlDispatch
  • Depends: 3.2, 3.3

• 4. Phase 4 — Verification, partition test, and regression

• 4.1 Network-partition (unreachable-peer) soak scenario

  • Add a partition primitive that renders one node unreachable (distinct from process kill). Primary = in-process socket-drop hook in xce_test (XCE_TESTING) — blackholes a node's peer traffic at the endpoint layer with NO privilege, so R7 actually runs in the unprivileged/overlayfs sandbox (avoids the perpetual-skip fate of ha-term-safety 5.2/5.4). Secondary (optional) = firewall iptables/nft DROP for real-network fidelity, capability-gated with a loud skip when unprivileged.
  • Assert: heartbeats to survivors continue, no spurious election while the leader holds the majority, the peer rejoins on heal, and no committed job is lost.
  • Observable: the partition soak runs and passes; the same scenario fails against the pre-change build (heartbeat stall / spurious election).
  • Requirements: 7.1, 7.2, 7.3, 7.4
  • Boundary: PartitionTest
  • Depends: 2.3, 3.4

• 4.2 Existing-feature regression sweep

  • DONE — verdict: raft-drive-async introduces NO new regression. Unit-level: controller_unit_test EXIT=0, xcelib_test 282/0 (all HA suites — election/prevote/liveness/failover/rdhup/repl/config-repl/persist-recovery/commit-barrier/durability — green, dormant flags OFF). Live kill-based failover soak (real fs ~/tmp, release+ASAN builds): zero committed-JOB loss across every completed round (jobs 3→12 monotonic). The 6-round CHAOTIC soak does not run clean to completion — it fails around round 4 — but a baseline comparison against pre-Phase-3 binaries (commit 21e26a1) fails round 4 IDENTICALLY (election stall / reconverge), and the round-4 committed-prefix divergence at LSN 9 is the documented pre-existing bug-851/795 family (chaotic-failover silent divergence), present before Phase 3 and out of scope for this spec (needs a separate HA-core effort). See bug-1225.
  • Run the existing HA suites unchanged — election, prevote, liveness, failover, rdhup-integration, repl majority/apply/catchup, config-repl/repl-push, persist-recovery, commit-barrier/commit-durability — and the kill-based failover soak.
  • Observable: all listed suites pass with no change to their asserted behavior.
  • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7
  • Depends: 3.4

• 4.3 Thread-role compliance and heartbeat performance check

  • Verify no blocking operation runs inline on the iomgr loop or delaymgr timer for the in-scope paths; bounded blocking runs in the vtask pool only under the admission reservation and long blocking I/O only on the lane.
  • Measure steady-state heartbeat cadence at 3–5 controllers and confirm no added latency attributable to the new dispatch path.
  • Observable: a compliance assertion/inspection shows the drive threads non-blocking; heartbeat cadence matches the pre-change baseline.
  • Requirements: 5.1, 5.2, 5.3, 5.4, 8.1, 8.2
  • Depends: 2.3, 3.4

Implementation Notes

• Build/test env: builds MUST run in the nix dev shell (nix develop --command meson compile -C src/xce.build <target>). The IDE clang reports false errors (missing openssl/unity headers, unknown -fvar-tracking) because it isn't in the nix shell — ignore those. Controller tests: rm -rf /tmp/xce_* first (bug-924 overlayfs), run ./src/xce.build/controller/controller_unit_test, set ASAN_OPTIONS=detect_leaks=0.

• Pre-existing leak (not ours): the default-options full controller suite exits non-zero due to a LeakSanitizer leak in test_commit_barrier_reply.c / test_worker_join.c / test_egress_filter.c (~174KB). Unrelated to this spec. Run with detect_leaks=0 (or a -t filter) so it doesn't mask raft-drive-async regressions.

• 1.1: drive_async_liveness_reserved(n_peers, n_vcpus) = max(1, min(n_peers−1, n_vcpus−1)) in ha.c is the single source of truth for VTASK_LIVENESS_RESERVED (floor applied LAST). Consumed by 1.6. myconf_t CLI override fields for the new DEFAULT_* constants were deferred to the consumer tasks (1.2 connect-timeout, 1.3/1.6) per YAGNI — add each field with its reader.

• 2.1: reconnect offload is gated by a default-ON kill-switch g_drive_async_reconnect_offload (legacy inline path retained for the TDD-RED proof + operational fallback). Per-peer single-flight via ha_state_t.reconnect_inflight[] CAS + slowlane key; flag cleared on all 4 terminal paths (prep-fail, lane-fn success/-EAGAIN, -EEXIST, hard-error). Lane fn = reconnect_lane_fn → ha_connect_finish_peer (happy path: bounded connect + lock-guarded iomgr_monitor; 2.2 adds inbound/outbound race + failure hardening via the documented seam). FOLLOW-UP (test hygiene, LOW): the GREEN reconnect tests in test_ha_liveness.c leak the prepped endpoint on the simulated-success hook path (production-clean; masked by detect_leaks=0) — free the un-adopted ep in teardown when convenient.

• 1.6: pool reservation lives in vtaskmgr (vtask_class_t + blocking_inflight/liveness_reserved + vtaskmgr_blocking_acquire/release); commit folds in via opt-in commit_set_pool (kept commit_engine_init(n_vcpus) signature — the n_vcpus local cap is the genuine pool==NULL fallback for unit fixtures). Production wiring is in cluster.c phase2 (once-guarded, right after commit_engine_init): vtaskmgr_set_liveness_reserved(R) + commit_set_pool — DO NOT remove; without it the reservation is dead. Lock order p->lock → ce->lock → blk_lock (cycle-free). New-cap = n_vcpus−R (tighter than old n_vcpus−2 for 5/7-node — intended). LOW deferred: vtaskmgr_blocking_cap() reads liveness_reserved without blk_lock (init-time set-once, benign). When 3.1 dispatches peer_ctrl, REQ_BLOCK slot is released in run_task — ensure no double-count vs the commit path.

• 3.1 (landed DORMANT — checkbox stays unchecked until 3.4 gate is green): peer_ctrl_msg_handler is now a shim gated by g_drive_async_peerctrl_dispatch_enabled (DEFAULT 0 / OFF — production behavior UNCHANGED; toggle via ha_set_peerctrl_dispatch_for_test). Flag OFF → peer_ctrl_msg_handler_inline runs on io_worker, byte-for-byte prior path. Flag ON → peer_ctrl_dispatch_to_pool builds a VTASK_IO task and vtask_enq_classifieds it (per-endpoint order auto-enforced by bind_vtask→endpoint_block_tasks). The 3 routers (vote_reply/pre_vote_reply/repl_ack_decode_and_dispatch) take pkt as an explicit param now (no cross-thread ep->fsm_in.pkt re-read). Packet ownership: shim clears ep->fsm_in.pkt immediately (sole-owner transfer); peer_ctrl_vtask_entry NULLs iod->pkt before the body frees it once; new vtask_t.owns_io_pkt (default 0; controller sets 1) lets taskq_item_killer free an un-executed queued task's packet without double-freeing the libxce endpoint-owned path (bug-1199). ha_state_t.pool wired in cluster.c to &c->vtaskmgr.

  • TWO FLAG-ON GATE ITEMS — MUST be resolved before 3.4 flips the flag ON (documented with loud code comments; harmless while flag OFF):
    1. bug-1197 (CRITICAL UAF): raft_cb_become_follower endpoint_fini()s peer vote-eps from a pool worker → frees ep->data while the single io_worker can concurrently read ep_j->data in peer_ctrl_msg_handler_inline for a same-epoll-batch queued event. FIX (part of 3.4 enablement): iomgr_unmonitor(ep) BEFORE endpoint_fini(ep) in every pool-reachable teardown.
    2. bug-1198 (HIGH classification): peer_ctrl_classify returns VTASK_LIVENESS for all, but the CTRL_PRIMARY repl-ack path can run a blocking HA_REPL_SEND_SNAPSHOT→persist_checkpoint inline under the never-gated liveness reservation → breaks deadlock-freedom. The 3.2 SlowLane offload mechanism now EXISTS but is DORMANT (flag OFF, below), so bug-1198 stays OPEN until 3.4 flips it ON together with the peerctrl flag.

• 3.2 — Part B (trim guard, bug-1147) is LIVE; Part A (snapshot offload) landed DORMANT (checkbox [~] until 3.4 gate):

  • Part B (production-live, default-ON, no flag): trim_wal_segments (persist.c) now bounds deletion by floor = repl_trim_lsn(&p->repl, active_seg_lsn, snap_interval_s) = min(active_seg_lsn, every active standby acked_lsn, every valid disconnected-slot last_disconnected_lsn), after a repl_expire_disconnect_slots call to age out stale slot guards. floor ≤ active_seg_lsn structurally (repl_trim_lsn caps at its checkpoint_lsn arg), so the new seg_start_lsn < floor check subsumes the old < active_seg_lsn guard and can only PRESERVE more segments — never delete one a live standby still needs (closes the bug-795/828 trim-vs-cursor race for connected/recently-disconnected standbys; does NOT fix empty/cursor=0 join — needs entity-bearing snapshots, separate spec). Lock-safe: trim_wal_segments runs with p->lock released; repl_trim_lsn/repl_expire_disconnect_slots take only r->lock — no inversion. Tests: test_trim_guard_preserves_standby_segments, test_trim_without_standbys_proceeds_normally (test_e2e_checkpoint.c).
  • Part A (DORMANT, g_drive_async_snapshot_offload DEFAULT 0/OFF): ha_snapshot_send_dispatch routes HA_REPL_SEND_SNAPSHOT inline (flag OFF — byte-for-byte legacy, send on io_worker) or to slowlane_submit+snapshot_lane_fn (flag ON). Send body factored into ha_snapshot_do_send (shared by both paths, holds the XCE_TESTING hook). Per-peer single-flight via ha_state_t.snapshot_inflight[MAX_HA_CANDIDATES] CAS + slowlane key peer_idx+1; flag cleared on all 4 terminal paths (alloc-fail, -EEXIST, hard-error, lane-fn completion) exactly like reconnect (2.1). Tests: test_snapshot_offload_runs_on_lane, test_snapshot_inline_when_flag_off, test_snapshot_offload_dedups_per_peer (test_ha_liveness.c).
  • WHY DORMANT (flag-ON gate, owned by 3.4): snapshot_lane_fn captures a BORROWED standby endpoint (unlike reconnect, which owns its ep). With the flag ON the send runs on a lane thread while the io_worker may RDHUP-endpoint_fini that ep → cross-thread UAF (bug-1197 family). Flag OFF keeps send+teardown serialized on the io_worker. 3.4 is the single gate that flips BOTH g_drive_async_snapshot_offload AND g_drive_async_peerctrl_dispatch ON, after establishing the endpoint-lifetime discipline (refcount or lane-serialized teardown) and proving liveness under pool saturation. Caught + fixed pre-commit: the offload flag was initially authored DEFAULT-ON (would have shipped the UAF) → corrected to OFF in both ha.c and the ha.h accessor doc (bug-1210).

• 3.3 (landed DORMANT — checkbox [~] until 3.4 gate): requester-less config-push transport offload, flag g_drive_async_configpush_offload DEFAULT 0/OFF (toggle ha_set_configpush_offload_for_test). ha_config_push_offloadable(c, bytes, blen, sha256, new_tag) routes inline (flag OFF — verbatim push_config_transport_only, caller still owns+frees bytes) or to slowlane_submit+configpush_lane_fn (flag ON — DEEP-COPIES bytes+sha into a ctx the lane frees). Single cluster-wide single-flight via scalar ha_state_t.configpush_inflight CAS + lane key=1 (config push is a cluster-wide fan-out, not per-peer); flag cleared on all terminal paths (alloc-fail×2, -EEXIST, hard-error, lane completion). Send body factored into ha_config_push_do_send (production → push_config_transport_only; XCE_TESTING → ha_configpush_probe stub since cluster.c/transport isn't linked in test). THREE requester-less drive-thread call sites converted: raft_cb_become_leader (promotion, both bytes/NULL branches), the HA_REPL_PUSH_CONFIG heartbeat case, and the direct-CTRL_PRIMARY FSM promotion branch (~ha.c:4476). UNCHANGED: push_config_to_standbys admin-upgrade barrier (has a waiting requester) and join.c. Tests: test_configpush_offload_runs_on_lane, test_configpush_inline_when_flag_off, test_configpush_offload_dedups (test_ha_liveness.c). Same BORROWED-ENDPOINT/bug-1197 dormancy rationale as 3.2 (lane reads c->ha.standby_endpoints); 3.4 flips this ON together with snapshot + peerctrl flags. cpp-review: no CRITICAL/HIGH/MEDIUM; controller suite EXIT=0.

• 3.4a (bug-1197 endpoint-quiescence fix — committed; flags STILL default OFF): makes the flag-ON peer-control dispatch path UAF-free so raft_cb_become_follower (running as a pool vtask) can free peer vote-eps while other pool workers process peer-control on the same eps. Mechanism (no libxce-wide endpoint refcount; reuses the existing per-ep FIFO + busy serialization):

  • Deferred teardown-as-vtask: the loser loop, when ha_peerctrl_dispatch_enabled() && ha->pool, does per vote-ep endpoint_mark_tearing_down(ep) → iomgr_unmonitor(ep) → enqueue a VTASK_IO/VTASK_LIVENESS teardown vtask bound to that ep (vote_ep_teardown_vtask_entry). Per-ep busy FIFO makes it run strictly AFTER any in-flight/queued same-ep peer-control vtask; it NULLs iod->ep before endpoint_fini+free(ep) to disarm run_task's post-body endpoint_unblock_tasks (which would otherwise touch the freed ep->lock). OFF / pool-unwired path is byte-for-byte the legacy endpoint_fini (no free).
  • RCU-like drain closes the TOCTOU window: a new _Atomic int endpoint_t.dispatch_inflight (+ tearing_down:1 bit). The ON dispatch brackets its flag-check+enqueue: fetch_add(seq_cst) → if tearing_down drop pkt + fetch_sub → else enqueue → fetch_sub. endpoint_mark_tearing_down sets the flag under ep->lock, releases, then spins while(load(dispatch_inflight,seq_cst)>0) sched_yield(). become_follower marks (drains) BEFORE enqueuing the teardown vtask, so any dispatch that read the flag false is FIFO-before teardown; any later dispatch drops. Could NOT hold ep->lock across the enqueue — it inverts with the taskq lock (dispatcher takes taskq→ep in vtask_eligible); the counter is the lock-free handshake. Drain is bounded (the held window is just calloc+non-blocking enqueue on the single io_worker).
  • Files: src/include/netio.h (tearing_down:1, dispatch_inflight, mark/is helpers), src/lib/netio.c (helpers + drain + do_endpoint_init zero-init), src/controller/ha.c (vote_ep_teardown_*, loser-loop async branch, dispatch bracket + late-drop, test seams), src/controller/ha.h. Tests: test_ha_vep_teardown.c (5 ASan tests: ordering, late-drop, disarm, OFF-path, deterministic drain-blocks). xcelib_test 282/0; controller EXIT=0; ASan-clean. Adversarial cpp-review: no CRITICAL/HIGH; handshake formally sound (SC total order on the counter + monotone tearing_down under ep->lock); flag-ON path proven race-free.
  • STILL OPEN for the prod flip (4.x): the borrowed-ep hazard in snapshot_lane_fn/configpush_lane_fn is a DISTINCT race (slowlane thread vs io_worker RDHUP teardown, not the vtask-pool FIFO) — needs its own per-standby-ep refcount/lane-serialized teardown before g_drive_async_snapshot_offload/configpush_offload go ON in production. 3.4b integration tests may enable the flags in-test (no mid-transfer ep teardown, so the lane race isn't exercised).