162 lines
27 KiB
Markdown
162 lines
27 KiB
Markdown
# Implementation Plan
|
||
|
||
Tasks mirror the design's 4-phase migration. The slow lane is foundational (every offload submits to it); the peer_ctrl→vtask move is coupled with its long-op offload in Phase 3 so no phase runs long I/O inline in a blocked vtask.
|
||
|
||
- [ ] 1. Phase 1 — SlowLane executor and bounded connect (foundation)
|
||
- [x] 1.1 Add drive-async configuration constants
|
||
- Define connect-timeout, slow-lane worker count, and max-in-flight constants with documented defaults (sized to peer count).
|
||
- Define the liveness reservation as a peer-scaled derivation, not a flat literal: `VTASK_LIVENESS_RESERVED = clamp(n_peers − 1, 1, n_vcpus − 1)` (floor 1 = deadlock-freedom; ceiling `n_vcpus − 1` keeps ≥1 commit slot); WARN at init when clamped by `n_vcpus`.
|
||
- Wire the constants into the controller config so they are overridable.
|
||
- Observable: a build with the new constants compiles; defaults are readable at controller startup; the resolved reservation logged at init.
|
||
- _Requirements: 2.3, 3.1_
|
||
|
||
- [x] 1.2 Implement the SlowLane executor
|
||
- Bounded pool of dedicated worker threads, isolated from the vtask request pool; non-blocking submit that enqueues and returns immediately.
|
||
- Admission cap on concurrent in-flight tasks; per-key in-flight dedup (key 0 disables dedup); init/fini wired into the controller lifecycle.
|
||
- Observable: a submitted task runs exactly once on a lane worker; submit never blocks the caller; concurrent running tasks never exceed the cap; a submit with an in-flight key is a no-op.
|
||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 1.5_
|
||
- _Boundary: SlowLane_
|
||
|
||
- [x] 1.3 Bound the peer connect path
|
||
- Reorder `socket_connect` to set non-blocking before connect and bound the attempt with a poll timeout (reuse the existing probe pattern); return success/timeout/error.
|
||
- Update connect callers to pass the configured timeout.
|
||
- Observable: a connect to an unreachable host returns within the configured timeout instead of the OS connect timeout; a connect to a reachable host still succeeds.
|
||
- _Requirements: 2.3_
|
||
- _Boundary: BoundedConnect_
|
||
- _Depends: 1.1_
|
||
|
||
- [x] 1.4 (P) SlowLane unit tests
|
||
- Assert non-blocking submit, admission-cap bounds on concurrency, per-key dedup returns the deduped result, and run-exactly-once.
|
||
- Observable: tests pass and fail if the cap or dedup is removed.
|
||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 1.5_
|
||
- _Boundary: SlowLane_
|
||
- _Depends: 1.2_
|
||
|
||
- [x] 1.5 (P) Bounded-connect unit test
|
||
- Assert `socket_connect` returns within the timeout against an unreachable host and succeeds against a reachable one.
|
||
- Observable: test passes; reverting the reorder makes it hang/fail.
|
||
- _Requirements: 2.3_
|
||
- _Boundary: BoundedConnect_
|
||
- _Depends: 1.3_
|
||
|
||
- [x] 1.6 LivenessAdmission — pool-level task-class reservation in vtaskmgr
|
||
- Add `vtask_class_t` (`VTASK_LIVENESS` / `VTASK_LIVE_BLOCK` / `VTASK_REQ_BLOCK`) + a classified submit; the dispatcher reserves the `VTASK_LIVENESS_RESERVED` value **resolved in 1.1** (single source of truth — consume it, do not recompute the clamp/WARN here) and fast-rejects a `VTASK_REQ_BLOCK` with `-XCE_ERR_NOT_READY` when admitting it would breach the reservation (never parks a worker, never silently queues).
|
||
- Fold `commit.c`'s `inflight_waits`/`COMMIT_WAIT_RESERVED` into this — `commit_submit` submits a `VTASK_REQ_BLOCK`; the pool owns the single reservation.
|
||
- Dormant infra in Phase 1: no behavior change until peer_ctrl dispatches through it (3.1). **Hard prerequisite of 3.1** — without it, moving ack/vote handling onto the pool is not deadlock-safe (vote-fsync/config-ack waits, invisible to the old commit-only counter, could starve the ack that wakes commit-waiters).
|
||
- Observable: a `VTASK_LIVENESS` task always finds a worker even when `*_BLOCK` tasks saturate `n_vcpus − VTASK_LIVENESS_RESERVED`; commit-wait still fast-rejects `-XCE_ERR_NOT_READY` at the cap (backpressure unchanged).
|
||
- _Requirements: 3.1, 3.2, 3.3, 4.5_
|
||
- _Boundary: LivenessAdmission_
|
||
- _Depends: 1.1_
|
||
|
||
- [x] 1.7 (P) LivenessAdmission unit test
|
||
- Assert: a `VTASK_LIVENESS` task is dispatched while `*_BLOCK` tasks hold `n_vcpus − VTASK_LIVENESS_RESERVED` vcpus; `VTASK_REQ_BLOCK` over the cap returns `-XCE_ERR_NOT_READY` without parking; folding commit-wait through the new path preserves the existing reject-at-cap behavior. Reverting the reservation makes a liveness task starve behind blockers.
|
||
- _Requirements: 3.1, 3.2, 3.3, 4.5_
|
||
- _Boundary: LivenessAdmission_
|
||
- _Depends: 1.6_
|
||
|
||
- [ ] 2. Phase 2 — Reconnect offload (sender-side liveness)
|
||
- [x] 2.1 Offload the delaymgr peer reconnect to the slow lane
|
||
- The heartbeat timer detects a disconnected peer and submits the reconnect to the lane (per-peer dedup key), then returns immediately; the timer never performs the connect inline.
|
||
- Refcount the endpoint across the in-flight window.
|
||
- Observable: while one peer is unreachable, the delaymgr timer keeps firing and heartbeats to reachable peers continue within the interval; only one reconnect attempt per peer is in flight at a time.
|
||
- _Requirements: 2.1, 2.2, 2.5_
|
||
- _Boundary: ReconnectOffload, SlowLane_
|
||
- _Depends: 1.2, 1.3_
|
||
|
||
- [x] 2.2 Complete the reconnect on the lane worker
|
||
- The lane worker performs the bounded connect, fully prepares the endpoint, and registers it via the lock-guarded monitor interface; on failure it clears the in-flight flag and lets the next tick retry without terminating; resolve the inbound-vs-outbound connection race per the connection-ownership rules.
|
||
- Observable: a successful reconnect registers the new connection with no lost or reordered queued messages; a failed attempt leaves the controller alive and retrying.
|
||
- _Requirements: 2.4, 2.6, 4.4_
|
||
- _Boundary: ReconnectOffload_
|
||
- _Depends: 2.1_
|
||
|
||
- [x] 2.3 Reconnect-offload integration test
|
||
- With one peer made unreachable, assert heartbeats to the surviving peers continue within the interval, the delaymgr is not stalled, and reconnect is deduped per peer.
|
||
- Observable: test passes; reverting the offload (inline connect) makes heartbeats to survivors stall.
|
||
- _Requirements: 2.1, 2.2, 2.5_
|
||
- _Boundary: ReconnectOffload_
|
||
- _Depends: 2.2_
|
||
|
||
- [ ] 3. Phase 3 — Peer-control dispatch to vtask, coupled with long-op offload (receiver-side)
|
||
- [ ] 3.1 Dispatch peer-control message handling to the vtask pool
|
||
- Register peer endpoints so peer-control messages dispatch through the vtask pool instead of running inline on iomgr; convert the peer-control handler into a vtask entry that does only bounded protocol work inline.
|
||
- Preserve per-connection processing order via the per-endpoint block primitive and preserve vote/ack/term semantics and persist-before-vote ordering; keep the steady-state heartbeat/vote send paths as non-blocking enqueues.
|
||
- Observable: peer-control no longer runs on the iomgr loop; existing election/heartbeat/repl behavior is unchanged; messages on one connection are processed in arrival order.
|
||
- **Deadlock-freedom gate (bug-invariant reversal):** moving ack decode to the pool co-locates the commit-barrier *wakers* (acks) and *waiters* (`commit_submit(COMMIT_MODE_WAIT)`) on one pool — today acks run on iomgr precisely to avoid this (`commit.c:16`). The old commit-only `COMMIT_WAIT_RESERVED` is insufficient (it can't see vote-fsync/config-ack blockers this move also puts on the pool), so safety rests on **LivenessAdmission (task 1.6)**: ack/vote-receipt dispatched as `VTASK_LIVENESS` (never gated, never offloaded), `commit_submit` as `VTASK_REQ_BLOCK` under the pool reservation. **Task 3.4 is the admission gate for this task — 3.1 is not complete until 3.4 is green.**
|
||
- _Requirements: 1.4, 4.1, 4.2, 4.5, 4.6, 5.1, 5.4, 8.1, 8.2_
|
||
- _Boundary: PeerCtrlDispatch_
|
||
- _Depends: 1.2, 1.6_
|
||
|
||
- [~] 3.2 Offload snapshot send to the slow lane <!-- Part B (trim guard) LIVE; Part A (offload) DORMANT pending 3.4 flip — see Implementation Notes -->
|
||
- Trigger snapshot send + checkpoint to the lane from the vtask entry without holding the per-endpoint block across the transfer; dedup one snapshot per standby.
|
||
- Enforce the snapshot/checkpoint concurrency invariant: snapshot reads a frozen term/lsn boundary; segment trim never deletes at/above any in-flight snapshot boundary or a standby catch-up cursor; at most one checkpoint in flight; bound the in-lane snapshot socket write so an unreachable-mid-transfer peer cannot pin a worker.
|
||
- **Add the trim cursor-bound guard (bug-1147, stopgap):** `trim_wal_segments` (`persist.c:1374`) currently bounds deletion ONLY by the open segment and consults no standby cursor; before this task moves checkpoint/trim onto a concurrent lane worker, change it to compute `floor = min(all standby acked_lsn, in-flight snapshot boundaries)` under `p->lock` and refuse to delete at/above `floor`. This is strategy-A retention discipline; it does NOT fix empty/far-behind join (cursor=0) — that needs entity-bearing snapshots (separate spec).
|
||
- Observable: a catch-up that triggers a snapshot does not block iomgr; the triggering peer's later messages are not queued behind the transfer; concurrent appends plus checkpoint/trim never delete a segment at/above any live standby `acked_lsn` or in-flight snapshot boundary (reverting the guard makes the test re-create a bug-795/828 gap).
|
||
- _Requirements: 1.1, 1.3, 1.5, 4.3_
|
||
- _Boundary: PeerCtrlDispatch, SlowLane_
|
||
- _Depends: 3.1_
|
||
|
||
- [~] 3.3 (P) Offload the transport-only config-push to the slow lane <!-- landed DORMANT (flag g_drive_async_configpush_offload default OFF) pending 3.4 flip — see Implementation Notes -->
|
||
- Move the requester-less iomgr/promotion config-push transport (and its wait) to the lane; replace the drive-thread join with a lane-internal wait and non-blocking return; leave the admin upgrade path (with a waiting requester) on its existing vtask-blocking path unchanged.
|
||
- Observable: a promotion/heartbeat-driven config push no longer blocks iomgr; the admin upgrade reply path is unchanged.
|
||
- _Requirements: 1.2_
|
||
- _Boundary: PeerCtrlDispatch, SlowLane_
|
||
- _Depends: 3.1_
|
||
|
||
- [x] 3.4 Receiver-side integration tests <!-- 3.4a UAF fix + 3.4b integration tests DONE (committed); production flag-flip deferred to after 4.1/4.2 per the "full fix, defer prod flip" decision -->
|
||
- **3.4a (DONE, committed 6d510d8)**: bug-1197 endpoint-quiescence fix landed so flag-ON peer-control dispatch is UAF-free (deferred teardown-vtask + RCU dispatch_inflight drain).
|
||
- **3.4b (DONE)**: receiver-side liveness integration tests (`test_ha_dispatch_integration.c`) with the dispatch flags flipped ON in-test — P1 a parked snapshot/config-push lane transfer does not block another endpoint's peer-control on the pool; P2 a VTASK_LIVENESS ack runs under REQ_BLOCK pool saturation (reservation never starves it; one-over fast-rejects to prove the boundary); P3 cross-endpoint peer-control runs concurrently (per-ep FIFO does not serialize across eps). Fixed a flaky teardown-race hang (bug-1224: poll end-of-body completion counters before the cancel-based vtaskmgr_fini). 5/5 deterministic, full controller suite EXIT=0, ASan-clean.
|
||
- **Production flags STILL DEFAULT OFF** (peerctrl/snapshot/configpush). Per the "full fix, defer prod flip" decision, the production flip happens only after 4.1 partition soak + 4.2 regression sweep pass; the snapshot/configpush lane borrowed-ep refcount (distinct from 3.4a) is also required before flipping those two ON.
|
||
- Assert that during a snapshot/config-push transfer the controller still processes other peers' heartbeats and votes; that a quorum of WAL-acks advances the commit position promptly even with the vtask pool near saturation; and that cross-endpoint acks are not serialized behind one another.
|
||
- Observable: tests pass; commit advances after a quorum of acks under load; iomgr stays responsive during transfers.
|
||
- _Requirements: 1.1, 1.3, 1.4, 4.3, 4.5_
|
||
- _Boundary: PeerCtrlDispatch_
|
||
- _Depends: 3.2, 3.3_
|
||
|
||
- [ ] 4. Phase 4 — Verification, partition test, and regression
|
||
- [x] 4.1 Network-partition (unreachable-peer) soak scenario
|
||
- Add a partition primitive that renders one node unreachable (distinct from process kill). **Primary = in-process socket-drop hook in `xce_test` (`XCE_TESTING`)** — blackholes a node's peer traffic at the endpoint layer with NO privilege, so R7 actually runs in the unprivileged/overlayfs sandbox (avoids the perpetual-skip fate of `ha-term-safety` 5.2/5.4). **Secondary (optional) = firewall `iptables`/`nft` DROP** for real-network fidelity, capability-gated with a loud skip when unprivileged.
|
||
- Assert: heartbeats to survivors continue, no spurious election while the leader holds the majority, the peer rejoins on heal, and no committed job is lost.
|
||
- Observable: the partition soak runs and passes; the same scenario fails against the pre-change build (heartbeat stall / spurious election).
|
||
- _Requirements: 7.1, 7.2, 7.3, 7.4_
|
||
- _Boundary: PartitionTest_
|
||
- _Depends: 2.3, 3.4_
|
||
|
||
- [x] 4.2 Existing-feature regression sweep <!-- NO new regression: unit suites green + live failover soak baseline-compared (bug-1225) -->
|
||
- **DONE — verdict: raft-drive-async introduces NO new regression.** Unit-level: controller_unit_test EXIT=0, xcelib_test 282/0 (all HA suites — election/prevote/liveness/failover/rdhup/repl/config-repl/persist-recovery/commit-barrier/durability — green, dormant flags OFF). Live kill-based failover soak (real fs ~/tmp, release+ASAN builds): **zero committed-JOB loss** across every completed round (jobs 3→12 monotonic). The 6-round CHAOTIC soak does not run clean to completion — it fails around round 4 — but a **baseline comparison against pre-Phase-3 binaries (commit 21e26a1) fails round 4 IDENTICALLY** (election stall / reconverge), and the round-4 committed-prefix divergence at LSN 9 is the documented pre-existing **bug-851/795 family** (chaotic-failover silent divergence), present before Phase 3 and out of scope for this spec (needs a separate HA-core effort). See bug-1225.
|
||
- Run the existing HA suites unchanged — election, prevote, liveness, failover, rdhup-integration, repl majority/apply/catchup, config-repl/repl-push, persist-recovery, commit-barrier/commit-durability — and the kill-based failover soak.
|
||
- Observable: all listed suites pass with no change to their asserted behavior.
|
||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7_
|
||
- _Depends: 3.4_
|
||
|
||
- [x] 4.3 Thread-role compliance and heartbeat performance check
|
||
- Verify no blocking operation runs inline on the iomgr loop or delaymgr timer for the in-scope paths; bounded blocking runs in the vtask pool only under the admission reservation and long blocking I/O only on the lane.
|
||
- Measure steady-state heartbeat cadence at 3–5 controllers and confirm no added latency attributable to the new dispatch path.
|
||
- Observable: a compliance assertion/inspection shows the drive threads non-blocking; heartbeat cadence matches the pre-change baseline.
|
||
- _Requirements: 5.1, 5.2, 5.3, 5.4, 8.1, 8.2_
|
||
- _Depends: 2.3, 3.4_
|
||
|
||
## Implementation Notes
|
||
|
||
- **Build/test env**: builds MUST run in the nix dev shell (`nix develop --command meson compile -C src/xce.build <target>`). The IDE clang reports false errors (missing openssl/unity headers, unknown `-fvar-tracking`) because it isn't in the nix shell — ignore those. Controller tests: `rm -rf /tmp/xce_*` first (bug-924 overlayfs), run `./src/xce.build/controller/controller_unit_test`, set `ASAN_OPTIONS=detect_leaks=0`.
|
||
- **Pre-existing leak (not ours)**: the default-options full controller suite exits non-zero due to a LeakSanitizer leak in `test_commit_barrier_reply.c` / `test_worker_join.c` / `test_egress_filter.c` (~174KB). Unrelated to this spec. Run with `detect_leaks=0` (or a `-t` filter) so it doesn't mask raft-drive-async regressions.
|
||
- **1.1**: `drive_async_liveness_reserved(n_peers, n_vcpus) = max(1, min(n_peers−1, n_vcpus−1))` in `ha.c` is the single source of truth for `VTASK_LIVENESS_RESERVED` (floor applied LAST). Consumed by 1.6. `myconf_t` CLI override fields for the new `DEFAULT_*` constants were deferred to the consumer tasks (1.2 connect-timeout, 1.3/1.6) per YAGNI — add each field with its reader.
|
||
- **2.1**: reconnect offload is gated by a default-ON kill-switch `g_drive_async_reconnect_offload` (legacy inline path retained for the TDD-RED proof + operational fallback). Per-peer single-flight via `ha_state_t.reconnect_inflight[]` CAS + slowlane key; flag cleared on all 4 terminal paths (prep-fail, lane-fn success/-EAGAIN, -EEXIST, hard-error). Lane fn = `reconnect_lane_fn` → `ha_connect_finish_peer` (happy path: bounded connect + lock-guarded iomgr_monitor; 2.2 adds inbound/outbound race + failure hardening via the documented seam). **FOLLOW-UP (test hygiene, LOW)**: the GREEN reconnect tests in test_ha_liveness.c leak the prepped endpoint on the simulated-success hook path (production-clean; masked by detect_leaks=0) — free the un-adopted ep in teardown when convenient.
|
||
- **1.6**: pool reservation lives in vtaskmgr (`vtask_class_t` + `blocking_inflight`/`liveness_reserved` + `vtaskmgr_blocking_acquire/release`); commit folds in via opt-in `commit_set_pool` (kept `commit_engine_init(n_vcpus)` signature — the `n_vcpus` local cap is the genuine pool==NULL fallback for unit fixtures). **Production wiring is in `cluster.c` phase2** (once-guarded, right after `commit_engine_init`): `vtaskmgr_set_liveness_reserved(R)` + `commit_set_pool` — DO NOT remove; without it the reservation is dead. Lock order `p->lock → ce->lock → blk_lock` (cycle-free). New-cap = `n_vcpus−R` (tighter than old `n_vcpus−2` for 5/7-node — intended). LOW deferred: `vtaskmgr_blocking_cap()` reads `liveness_reserved` without `blk_lock` (init-time set-once, benign). When 3.1 dispatches peer_ctrl, REQ_BLOCK slot is released in `run_task` — ensure no double-count vs the commit path.
|
||
- **3.1 (landed DORMANT — checkbox stays unchecked until 3.4 gate is green)**: `peer_ctrl_msg_handler` is now a shim gated by `g_drive_async_peerctrl_dispatch_enabled` (**DEFAULT 0 / OFF** — production behavior UNCHANGED; toggle via `ha_set_peerctrl_dispatch_for_test`). Flag OFF → `peer_ctrl_msg_handler_inline` runs on io_worker, byte-for-byte prior path. Flag ON → `peer_ctrl_dispatch_to_pool` builds a `VTASK_IO` task and `vtask_enq_classified`s it (per-endpoint order auto-enforced by `bind_vtask`→`endpoint_block_tasks`). The 3 routers (`vote_reply`/`pre_vote_reply`/`repl_ack`_decode_and_dispatch) take `pkt` as an explicit param now (no cross-thread `ep->fsm_in.pkt` re-read). **Packet ownership**: shim clears `ep->fsm_in.pkt` immediately (sole-owner transfer); `peer_ctrl_vtask_entry` NULLs `iod->pkt` before the body frees it once; new `vtask_t.owns_io_pkt` (default 0; controller sets 1) lets `taskq_item_killer` free an un-executed queued task's packet without double-freeing the libxce endpoint-owned path (bug-1199). `ha_state_t.pool` wired in `cluster.c` to `&c->vtaskmgr`.
|
||
- **TWO FLAG-ON GATE ITEMS — MUST be resolved before 3.4 flips the flag ON** (documented with loud code comments; harmless while flag OFF):
|
||
1. **bug-1197 (CRITICAL UAF)**: `raft_cb_become_follower` `endpoint_fini()`s peer vote-eps from a pool worker → frees `ep->data` while the single io_worker can concurrently read `ep_j->data` in `peer_ctrl_msg_handler_inline` for a same-epoll-batch queued event. FIX (part of 3.4 enablement): `iomgr_unmonitor(ep)` BEFORE `endpoint_fini(ep)` in every pool-reachable teardown.
|
||
2. **bug-1198 (HIGH classification)**: `peer_ctrl_classify` returns `VTASK_LIVENESS` for all, but the CTRL_PRIMARY repl-ack path can run a blocking `HA_REPL_SEND_SNAPSHOT`→`persist_checkpoint` inline under the never-gated liveness reservation → breaks deadlock-freedom. The 3.2 SlowLane offload mechanism now EXISTS but is DORMANT (flag OFF, below), so bug-1198 stays OPEN until 3.4 flips it ON together with the peerctrl flag.
|
||
|
||
- **3.2 — Part B (trim guard, bug-1147) is LIVE; Part A (snapshot offload) landed DORMANT (checkbox `[~]` until 3.4 gate)**:
|
||
- **Part B (production-live, default-ON, no flag)**: `trim_wal_segments` (`persist.c`) now bounds deletion by `floor = repl_trim_lsn(&p->repl, active_seg_lsn, snap_interval_s)` = `min(active_seg_lsn, every active standby acked_lsn, every valid disconnected-slot last_disconnected_lsn)`, after a `repl_expire_disconnect_slots` call to age out stale slot guards. `floor ≤ active_seg_lsn` structurally (repl_trim_lsn caps at its `checkpoint_lsn` arg), so the new `seg_start_lsn < floor` check subsumes the old `< active_seg_lsn` guard and can only PRESERVE more segments — never delete one a live standby still needs (closes the bug-795/828 trim-vs-cursor race for connected/recently-disconnected standbys; does NOT fix empty/cursor=0 join — needs entity-bearing snapshots, separate spec). Lock-safe: `trim_wal_segments` runs with `p->lock` released; `repl_trim_lsn`/`repl_expire_disconnect_slots` take only `r->lock` — no inversion. Tests: `test_trim_guard_preserves_standby_segments`, `test_trim_without_standbys_proceeds_normally` (test_e2e_checkpoint.c).
|
||
- **Part A (DORMANT, `g_drive_async_snapshot_offload` DEFAULT 0/OFF)**: `ha_snapshot_send_dispatch` routes `HA_REPL_SEND_SNAPSHOT` inline (flag OFF — byte-for-byte legacy, send on io_worker) or to `slowlane_submit`+`snapshot_lane_fn` (flag ON). Send body factored into `ha_snapshot_do_send` (shared by both paths, holds the XCE_TESTING hook). Per-peer single-flight via `ha_state_t.snapshot_inflight[MAX_HA_CANDIDATES]` CAS + slowlane key `peer_idx+1`; flag cleared on all 4 terminal paths (alloc-fail, -EEXIST, hard-error, lane-fn completion) exactly like reconnect (2.1). Tests: `test_snapshot_offload_runs_on_lane`, `test_snapshot_inline_when_flag_off`, `test_snapshot_offload_dedups_per_peer` (test_ha_liveness.c).
|
||
- **WHY DORMANT (flag-ON gate, owned by 3.4)**: `snapshot_lane_fn` captures a BORROWED standby endpoint (unlike reconnect, which owns its ep). With the flag ON the send runs on a lane thread while the io_worker may RDHUP-`endpoint_fini` that ep → cross-thread UAF (bug-1197 family). Flag OFF keeps send+teardown serialized on the io_worker. **3.4 is the single gate** that flips BOTH `g_drive_async_snapshot_offload` AND `g_drive_async_peerctrl_dispatch` ON, after establishing the endpoint-lifetime discipline (refcount or lane-serialized teardown) and proving liveness under pool saturation. Caught + fixed pre-commit: the offload flag was initially authored DEFAULT-ON (would have shipped the UAF) → corrected to OFF in both ha.c and the ha.h accessor doc (bug-1210).
|
||
|
||
- **3.3 (landed DORMANT — checkbox `[~]` until 3.4 gate)**: requester-less config-push transport offload, flag `g_drive_async_configpush_offload` **DEFAULT 0/OFF** (toggle `ha_set_configpush_offload_for_test`). `ha_config_push_offloadable(c, bytes, blen, sha256, new_tag)` routes inline (flag OFF — verbatim `push_config_transport_only`, caller still owns+frees `bytes`) or to `slowlane_submit`+`configpush_lane_fn` (flag ON — DEEP-COPIES `bytes`+sha into a ctx the lane frees). Single cluster-wide single-flight via scalar `ha_state_t.configpush_inflight` CAS + lane key=1 (config push is a cluster-wide fan-out, not per-peer); flag cleared on all terminal paths (alloc-fail×2, -EEXIST, hard-error, lane completion). Send body factored into `ha_config_push_do_send` (production → `push_config_transport_only`; XCE_TESTING → `ha_configpush_probe` stub since cluster.c/transport isn't linked in test). THREE requester-less drive-thread call sites converted: `raft_cb_become_leader` (promotion, both bytes/NULL branches), the `HA_REPL_PUSH_CONFIG` heartbeat case, and the direct-CTRL_PRIMARY FSM promotion branch (~ha.c:4476). UNCHANGED: `push_config_to_standbys` admin-upgrade barrier (has a waiting requester) and `join.c`. Tests: `test_configpush_offload_runs_on_lane`, `test_configpush_inline_when_flag_off`, `test_configpush_offload_dedups` (test_ha_liveness.c). Same BORROWED-ENDPOINT/bug-1197 dormancy rationale as 3.2 (lane reads `c->ha.standby_endpoints`); **3.4 flips this ON together with snapshot + peerctrl flags**. cpp-review: no CRITICAL/HIGH/MEDIUM; controller suite EXIT=0.
|
||
|
||
- **3.4a (bug-1197 endpoint-quiescence fix — committed; flags STILL default OFF)**: makes the flag-ON peer-control dispatch path UAF-free so `raft_cb_become_follower` (running as a pool vtask) can free peer vote-eps while other pool workers process peer-control on the same eps. Mechanism (no libxce-wide endpoint refcount; reuses the existing per-ep FIFO + busy serialization):
|
||
- **Deferred teardown-as-vtask**: the loser loop, when `ha_peerctrl_dispatch_enabled() && ha->pool`, does per vote-ep `endpoint_mark_tearing_down(ep)` → `iomgr_unmonitor(ep)` → enqueue a `VTASK_IO`/`VTASK_LIVENESS` teardown vtask bound to that ep (`vote_ep_teardown_vtask_entry`). Per-ep busy FIFO makes it run strictly AFTER any in-flight/queued same-ep peer-control vtask; it NULLs `iod->ep` before `endpoint_fini`+`free(ep)` to disarm `run_task`'s post-body `endpoint_unblock_tasks` (which would otherwise touch the freed `ep->lock`). OFF / pool-unwired path is byte-for-byte the legacy `endpoint_fini` (no free).
|
||
- **RCU-like drain closes the TOCTOU window**: a new `_Atomic int endpoint_t.dispatch_inflight` (+ `tearing_down:1` bit). The ON dispatch brackets its flag-check+enqueue: `fetch_add(seq_cst)` → if `tearing_down` drop pkt + `fetch_sub` → else enqueue → `fetch_sub`. `endpoint_mark_tearing_down` sets the flag under `ep->lock`, releases, then spins `while(load(dispatch_inflight,seq_cst)>0) sched_yield()`. become_follower marks (drains) BEFORE enqueuing the teardown vtask, so any dispatch that read the flag false is FIFO-before teardown; any later dispatch drops. Could NOT hold `ep->lock` across the enqueue — it inverts with the taskq lock (dispatcher takes taskq→ep in `vtask_eligible`); the counter is the lock-free handshake. Drain is bounded (the held window is just calloc+non-blocking enqueue on the single io_worker).
|
||
- Files: `src/include/netio.h` (`tearing_down:1`, `dispatch_inflight`, mark/is helpers), `src/lib/netio.c` (helpers + drain + `do_endpoint_init` zero-init), `src/controller/ha.c` (`vote_ep_teardown_*`, loser-loop async branch, dispatch bracket + late-drop, test seams), `src/controller/ha.h`. Tests: `test_ha_vep_teardown.c` (5 ASan tests: ordering, late-drop, disarm, OFF-path, deterministic drain-blocks). xcelib_test 282/0; controller EXIT=0; ASan-clean. Adversarial cpp-review: no CRITICAL/HIGH; handshake formally sound (SC total order on the counter + monotone `tearing_down` under `ep->lock`); flag-ON path proven race-free.
|
||
- **STILL OPEN for the prod flip (4.x)**: the borrowed-ep hazard in `snapshot_lane_fn`/`configpush_lane_fn` is a DISTINCT race (slowlane thread vs io_worker RDHUP teardown, not the vtask-pool FIFO) — needs its own per-standby-ep refcount/lane-serialized teardown before `g_drive_async_snapshot_offload`/`configpush_offload` go ON in production. 3.4b integration tests may enable the flags in-test (no mid-transfer ep teardown, so the lane race isn't exercised).
|