myxce/.kiro/specs/raft-drive-async/tasks.md

# Implementation Plan

Tasks mirror the design's 4-phase migration. The slow lane is foundational (every offload submits to it); the peer_ctrl→vtask move is coupled with its long-op offload in Phase 3 so no phase runs long I/O inline in a blocked vtask.

- [x] 1. Phase 1 — SlowLane executor and bounded connect (foundation)
- [x] 1.1 Add drive-async configuration constants
  - Define connect-timeout, slow-lane worker count, and max-in-flight constants with documented defaults (sized to peer count).
  - Define the liveness reservation as a peer-scaled derivation, not a flat literal: `VTASK_LIVENESS_RESERVED = clamp(n_peers − 1, 1, n_vcpus − 1)` (floor 1 = deadlock-freedom; ceiling `n_vcpus − 1` keeps ≥1 commit slot); WARN at init when clamped by `n_vcpus`.
  - Wire the constants into the controller config so they are overridable.
  - Observable: a build with the new constants compiles; defaults are readable at controller startup; the resolved reservation logged at init.
  - _Requirements: 2.3, 3.1_

- [x] 1.2 Implement the SlowLane executor
  - Bounded pool of dedicated worker threads, isolated from the vtask request pool; non-blocking submit that enqueues and returns immediately.
  - Admission cap on concurrent in-flight tasks; per-key in-flight dedup (key 0 disables dedup); init/fini wired into the controller lifecycle.
  - Observable: a submitted task runs exactly once on a lane worker; submit never blocks the caller; concurrent running tasks never exceed the cap; a submit with an in-flight key is a no-op.
  - _Requirements: 3.1, 3.2, 3.3, 3.4, 1.5_
  - _Boundary: SlowLane_

- [x] 1.3 Bound the peer connect path
  - Reorder `socket_connect` to set non-blocking before connect and bound the attempt with a poll timeout (reuse the existing probe pattern); return success/timeout/error.
  - Update connect callers to pass the configured timeout.
  - Observable: a connect to an unreachable host returns within the configured timeout instead of the OS connect timeout; a connect to a reachable host still succeeds.
  - _Requirements: 2.3_
  - _Boundary: BoundedConnect_
  - _Depends: 1.1_

- [x] 1.4 (P) SlowLane unit tests
  - Assert non-blocking submit, admission-cap bounds on concurrency, per-key dedup returns the deduped result, and run-exactly-once.
  - Observable: tests pass and fail if the cap or dedup is removed.
  - _Requirements: 3.1, 3.2, 3.3, 3.4, 1.5_
  - _Boundary: SlowLane_
  - _Depends: 1.2_

- [x] 1.5 (P) Bounded-connect unit test
  - Assert `socket_connect` returns within the timeout against an unreachable host and succeeds against a reachable one.
  - Observable: test passes; reverting the reorder makes it hang/fail.
  - _Requirements: 2.3_
  - _Boundary: BoundedConnect_
  - _Depends: 1.3_

- [x] 1.6 LivenessAdmission — pool-level task-class reservation in vtaskmgr
  - Add `vtask_class_t` (`VTASK_LIVENESS` / `VTASK_LIVE_BLOCK` / `VTASK_REQ_BLOCK`) + a classified submit; the dispatcher reserves the `VTASK_LIVENESS_RESERVED` value **resolved in 1.1** (single source of truth — consume it, do not recompute the clamp/WARN here) and fast-rejects a `VTASK_REQ_BLOCK` with `-XCE_ERR_NOT_READY` when admitting it would breach the reservation (never parks a worker, never silently queues).
  - Fold `commit.c`'s `inflight_waits`/`COMMIT_WAIT_RESERVED` into this — `commit_submit` submits a `VTASK_REQ_BLOCK`; the pool owns the single reservation.
  - Dormant infra in Phase 1: no behavior change until peer_ctrl dispatches through it (3.1). **Hard prerequisite of 3.1** — without it, moving ack/vote handling onto the pool is not deadlock-safe (vote-fsync/config-ack waits, invisible to the old commit-only counter, could starve the ack that wakes commit-waiters).
  - Observable: a `VTASK_LIVENESS` task always finds a worker even when `*_BLOCK` tasks saturate `n_vcpus − VTASK_LIVENESS_RESERVED`; commit-wait still fast-rejects `-XCE_ERR_NOT_READY` at the cap (backpressure unchanged).
  - _Requirements: 3.1, 3.2, 3.3, 4.5_
  - _Boundary: LivenessAdmission_
  - _Depends: 1.1_

- [x] 1.7 (P) LivenessAdmission unit test
  - Assert: a `VTASK_LIVENESS` task is dispatched while `*_BLOCK` tasks hold `n_vcpus − VTASK_LIVENESS_RESERVED` vcpus; `VTASK_REQ_BLOCK` over the cap returns `-XCE_ERR_NOT_READY` without parking; folding commit-wait through the new path preserves the existing reject-at-cap behavior. Reverting the reservation makes a liveness task starve behind blockers.
  - _Requirements: 3.1, 3.2, 3.3, 4.5_
  - _Boundary: LivenessAdmission_
  - _Depends: 1.6_

- [x] 2. Phase 2 — Reconnect offload (sender-side liveness)
- [x] 2.1 Offload the delaymgr peer reconnect to the slow lane
  - The heartbeat timer detects a disconnected peer and submits the reconnect to the lane (per-peer dedup key), then returns immediately; the timer never performs the connect inline.
  - Refcount the endpoint across the in-flight window.
  - Observable: while one peer is unreachable, the delaymgr timer keeps firing and heartbeats to reachable peers continue within the interval; only one reconnect attempt per peer is in flight at a time.
  - _Requirements: 2.1, 2.2, 2.5_
  - _Boundary: ReconnectOffload, SlowLane_
  - _Depends: 1.2, 1.3_

- [x] 2.2 Complete the reconnect on the lane worker
  - The lane worker performs the bounded connect, fully prepares the endpoint, and registers it via the lock-guarded monitor interface; on failure it clears the in-flight flag and lets the next tick retry without terminating; resolve the inbound-vs-outbound connection race per the connection-ownership rules.
  - Observable: a successful reconnect registers the new connection with no lost or reordered queued messages; a failed attempt leaves the controller alive and retrying.
  - _Requirements: 2.4, 2.6, 4.4_
  - _Boundary: ReconnectOffload_
  - _Depends: 2.1_

- [x] 2.3 Reconnect-offload integration test
  - With one peer made unreachable, assert heartbeats to the surviving peers continue within the interval, the delaymgr is not stalled, and reconnect is deduped per peer.
  - Observable: test passes; reverting the offload (inline connect) makes heartbeats to survivors stall.
  - _Requirements: 2.1, 2.2, 2.5_
  - _Boundary: ReconnectOffload_
  - _Depends: 2.2_

- [x] 3. Phase 3 — Peer-control dispatch to vtask, coupled with long-op offload (receiver-side)
- [x] 3.1 Dispatch peer-control message handling to the vtask pool
  - Register peer endpoints so peer-control messages dispatch through the vtask pool instead of running inline on iomgr; convert the peer-control handler into a vtask entry that does only bounded protocol work inline.
  - Preserve per-connection processing order via the per-endpoint block primitive and preserve vote/ack/term semantics and persist-before-vote ordering; keep the steady-state heartbeat/vote send paths as non-blocking enqueues.
  - Observable: peer-control no longer runs on the iomgr loop; existing election/heartbeat/repl behavior is unchanged; messages on one connection are processed in arrival order.
  - **Deadlock-freedom gate (bug-invariant reversal):** moving ack decode to the pool co-locates the commit-barrier *wakers* (acks) and *waiters* (`commit_submit(COMMIT_MODE_WAIT)`) on one pool — today acks run on iomgr precisely to avoid this (`commit.c:16`). The old commit-only `COMMIT_WAIT_RESERVED` is insufficient (it can't see vote-fsync/config-ack blockers this move also puts on the pool), so safety rests on **LivenessAdmission (task 1.6)**: ack/vote-receipt dispatched as `VTASK_LIVENESS` (never gated, never offloaded), `commit_submit` as `VTASK_REQ_BLOCK` under the pool reservation. **Task 3.4 is the admission gate for this task — 3.1 is not complete until 3.4 is green.**
  - _Requirements: 1.4, 4.1, 4.2, 4.5, 4.6, 5.1, 5.4, 8.1, 8.2_
  - _Boundary: PeerCtrlDispatch_
  - _Depends: 1.2, 1.6_
  - _2026-07-01/07-02 session (production flip — DONE, both flags now default ON): with 4.1+4.2 green, executed the "full fix, defer prod flip" decision's flip. (1) Closed the snapshot-lane borrowed-endpoint hazard (task 3.2's Part A) properly: `ha_snapshot_send_dispatch` takes `endpoint_ref(ep)` before `slowlane_submit`, released on every non-lane terminal path and by `snapshot_lane_fn` on completion — reuses the general bug-1298 refcount system (standby-ep teardown already routes through `endpoint_unref` via `cluster.c`'s `endpoint_killer`, confirmed by reading it — this infra landed in raft-ha-core-safety task 5.8, AFTER this spec's 3.2/3.4 notes were written, so the hazard they describe is now closed for real). New unit test `test_snapshot_offload_holds_endpoint_ref_across_transfer` (RED→GREEN). (2) Flipped `g_drive_async_snapshot_offload` and `g_drive_async_peerctrl_dispatch_enabled` to default ON; full controller suite green; a 6-round `--chaotic` soak passed clean (zero divergence, zero job loss). (3) A `--force-snapshot` soak (genuinely empty node cold-start rejoin — the ONE scenario chaotic mode never exercises, since it restarts nodes with WAL intact) then surfaced a SEPARATE pre-existing bug, unmasked (not caused) by the flip's timing shift: `ha_stream_wal_to_standby` had no filter for `WAL_CHECKPOINT` markers (local-only, never a logical Raft entry — consumes a real LSN slot via `persist_write_checkpoint_marker`'s own `p->current_lsn++`) and shipped one to a catching-up standby, which correctly refused it and got stuck retrying the same LSN forever. FIXED (`config_repl.c`, skip `hdr.type == WAL_CHECKPOINT` entirely, mirroring the existing `replay_wal`/term-index precedent of treating checkpoint-marker LSNs as invisible gaps) with a RED→GREEN unit test (`test_wal_stream_skips_checkpoint_marker`, differential-verified). (4) Re-ran `--force-snapshot` with BOTH flags ON and the checkpoint-skip fix in place — it STILL failed, this time with a THIRD, different issue logged as **bug-1376**: the freshly-wiped empty node's own self-election races `cluster_init_phase2`'s vtask-pool wiring, then the process hangs silently. Live-repro'd deterministically via a `GDB_WRAP` debug seam added to `run-ha-soak.sh` (launches a named node directly under `gdb --batch`, so it becomes ptrace-legal as gdb's direct child under yama scope 1 — bypasses a live-attach restriction) and a bounded `SIGINT`-to-gdb interrupt for a full ASan-free thread backtrace. **Root-caused and FIXED**: `peer_ctrl_msg_handler` held `ep->dispatch_inflight` incremented across the ENTIRE `peer_ctrl_dispatch_to_pool` call, including its inline fallback (the common cold-start shape when `ha->pool` isn't wired yet). That inline body can reach `raft_cb_become_follower`, which — now that `cluster_init_phase2` ran synchronously earlier in the SAME call chain and wired the pool — tears down its OWN loser vote-ep (the very ep whose reply triggered this call) via the deferred path (`vote_ep_teardown_async`), whose `endpoint_mark_tearing_down` spins waiting for `dispatch_inflight==0` on that SAME ep — a count only this exact call, on this exact thread, holds — a permanent self-deadlock. Fixed by moving the `dispatch_inflight` release into `peer_ctrl_dispatch_to_pool` itself, at the exact right point per exit path: released BEFORE calling the inline body on every fallback path (pool==NULL, OOM, enqueue failure) — safe because `io_worker` is single-threaded and the inline path never enqueues anything, so there is no pool-worker race left to protect against once inline execution is committed to — and only AFTER the (non-blocking) enqueue completes on the success path, preserving the original protection for the genuinely concurrent pool-worker case. New unit test `test_vep_self_teardown_does_not_deadlock` (`test_ha_vep_teardown.c`, bounded-timeout `pthread_timedjoin_np` join so a regression fails loud instead of hanging the suite; RED confirmed via a temporary manual revert — `Expected 0 Was 110` (ETIMEDOUT) — then GREEN restored). **Live-verified with both flags ON**: `--force-snapshot` soak now completes without hanging (2/3 runs confirmed a real snapshot install + `committed_divergence=0 committed_job_loss=0`; 1/3 hit the separate known-flaky "checkpoint didn't fire before the wipe this round" — a timing non-issue the harness itself already calls out, NOT a hang); a 6-round `--chaotic` soak with both flags ON also passed clean; full controller suite green throughout. **Both `g_drive_async_snapshot_offload` and `g_drive_async_peerctrl_dispatch_enabled` now default ON in production.** `g_drive_async_configpush_offload` (task 3.3) stays OFF — it has its own DISTINCT, still-open hazard (see task 3.3's note) unrelated to bug-1376._

- [x] 3.2 Offload snapshot send to the slow lane  <!-- Part B (trim guard) LIVE; Part A (offload) now ON by default (bug-1197 endpoint_ref/unref fix, bug-1376 fix) — see Implementation Notes -->
  - Trigger snapshot send + checkpoint to the lane from the vtask entry without holding the per-endpoint block across the transfer; dedup one snapshot per standby.
  - Enforce the snapshot/checkpoint concurrency invariant: snapshot reads a frozen term/lsn boundary; segment trim never deletes at/above any in-flight snapshot boundary or a standby catch-up cursor; at most one checkpoint in flight; bound the in-lane snapshot socket write so an unreachable-mid-transfer peer cannot pin a worker.
  - **Add the trim cursor-bound guard (bug-1147, stopgap):** `trim_wal_segments` (`persist.c:1374`) currently bounds deletion ONLY by the open segment and consults no standby cursor; before this task moves checkpoint/trim onto a concurrent lane worker, change it to compute `floor = min(all standby acked_lsn, in-flight snapshot boundaries)` under `p->lock` and refuse to delete at/above `floor`. This is strategy-A retention discipline; it does NOT fix empty/far-behind join (cursor=0) — that needs entity-bearing snapshots (separate spec).
  - Observable: a catch-up that triggers a snapshot does not block iomgr; the triggering peer's later messages are not queued behind the transfer; concurrent appends plus checkpoint/trim never delete a segment at/above any live standby `acked_lsn` or in-flight snapshot boundary (reverting the guard makes the test re-create a bug-795/828 gap).
  - _Requirements: 1.1, 1.3, 1.5, 4.3_
  - _Boundary: PeerCtrlDispatch, SlowLane_
  - _Depends: 3.1_

- [x] 3.3 (P) Offload the transport-only config-push to the slow lane  <!-- LANDED, flag g_drive_async_configpush_offload DEFAULT ON as of 2026-07-02. bug-1381 (endpoint UAF) and bug-1386 (silent chunk-drop) and bug-1389 (vtask-pool chunk reordering) all CLOSED; flip soak-validated: 6-round --chaotic PASS (zero divergence/loss) + 3/3 --force-snapshot PASS (genuine snapshot installs confirmed each time), see Implementation Notes -->
  - Move the requester-less iomgr/promotion config-push transport (and its wait) to the lane; replace the drive-thread join with a lane-internal wait and non-blocking return; leave the admin upgrade path (with a waiting requester) on its existing vtask-blocking path unchanged.
  - Observable: a promotion/heartbeat-driven config push no longer blocks iomgr; the admin upgrade reply path is unchanged.
  - _Requirements: 1.2_
  - _Boundary: PeerCtrlDispatch, SlowLane_
  - _Depends: 3.1_

- [x] 3.4 Receiver-side integration tests  <!-- 3.4a UAF fix + 3.4b integration tests DONE (committed); production flag-flip deferred to after 4.1/4.2 per the "full fix, defer prod flip" decision -->
  - **3.4a (DONE, committed 6d510d8)**: bug-1197 endpoint-quiescence fix landed so flag-ON peer-control dispatch is UAF-free (deferred teardown-vtask + RCU dispatch_inflight drain).
  - **3.4b (DONE)**: receiver-side liveness integration tests (`test_ha_dispatch_integration.c`) with the dispatch flags flipped ON in-test — P1 a parked snapshot/config-push lane transfer does not block another endpoint's peer-control on the pool; P2 a VTASK_LIVENESS ack runs under REQ_BLOCK pool saturation (reservation never starves it; one-over fast-rejects to prove the boundary); P3 cross-endpoint peer-control runs concurrently (per-ep FIFO does not serialize across eps). Fixed a flaky teardown-race hang (bug-1224: poll end-of-body completion counters before the cancel-based vtaskmgr_fini). 5/5 deterministic, full controller suite EXIT=0, ASan-clean.
  - **Production flags STILL DEFAULT OFF** (peerctrl/snapshot/configpush). Per the "full fix, defer prod flip" decision, the production flip happens only after 4.1 partition soak + 4.2 regression sweep pass; the snapshot/configpush lane borrowed-ep refcount (distinct from 3.4a) is also required before flipping those two ON.
  - Assert that during a snapshot/config-push transfer the controller still processes other peers' heartbeats and votes; that a quorum of WAL-acks advances the commit position promptly even with the vtask pool near saturation; and that cross-endpoint acks are not serialized behind one another.
  - Observable: tests pass; commit advances after a quorum of acks under load; iomgr stays responsive during transfers.
  - _Requirements: 1.1, 1.3, 1.4, 4.3, 4.5_
  - _Boundary: PeerCtrlDispatch_
  - _Depends: 3.2, 3.3_

- [x] 4. Phase 4 — Verification, partition test, and regression
- [x] 4.1 Network-partition (unreachable-peer) soak scenario
  - Add a partition primitive that renders one node unreachable (distinct from process kill). **Primary = in-process socket-drop hook in `xce_test` (`XCE_TESTING`)** — blackholes a node's peer traffic at the endpoint layer with NO privilege, so R7 actually runs in the unprivileged/overlayfs sandbox (avoids the perpetual-skip fate of `ha-term-safety` 5.2/5.4). **Secondary (optional) = firewall `iptables`/`nft` DROP** for real-network fidelity, capability-gated with a loud skip when unprivileged.
  - Assert: heartbeats to survivors continue, no spurious election while the leader holds the majority, the peer rejoins on heal, and no committed job is lost.
  - Observable: the partition soak runs and passes; the same scenario fails against the pre-change build (heartbeat stall / spurious election).
  - _Requirements: 7.1, 7.2, 7.3, 7.4_
  - _Boundary: PartitionTest_
  - _Depends: 2.3, 3.4_

- [x] 4.2 Existing-feature regression sweep  <!-- NO new regression: unit suites green + live failover soak baseline-compared (bug-1225) -->
  - **DONE — verdict: raft-drive-async introduces NO new regression.** Unit-level: controller_unit_test EXIT=0, xcelib_test 282/0 (all HA suites — election/prevote/liveness/failover/rdhup/repl/config-repl/persist-recovery/commit-barrier/durability — green, dormant flags OFF). Live kill-based failover soak (real fs ~/tmp, release+ASAN builds): **zero committed-JOB loss** across every completed round (jobs 3→12 monotonic). The 6-round CHAOTIC soak does not run clean to completion — it fails around round 4 — but a **baseline comparison against pre-Phase-3 binaries (commit 21e26a1) fails round 4 IDENTICALLY** (election stall / reconverge), and the round-4 committed-prefix divergence at LSN 9 is the documented pre-existing **bug-851/795 family** (chaotic-failover silent divergence), present before Phase 3 and out of scope for this spec (needs a separate HA-core effort). See bug-1225.
  - Run the existing HA suites unchanged — election, prevote, liveness, failover, rdhup-integration, repl majority/apply/catchup, config-repl/repl-push, persist-recovery, commit-barrier/commit-durability — and the kill-based failover soak.
  - Observable: all listed suites pass with no change to their asserted behavior.
  - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7_
  - _Depends: 3.4_

- [x] 4.3 Thread-role compliance and heartbeat performance check
  - Verify no blocking operation runs inline on the iomgr loop or delaymgr timer for the in-scope paths; bounded blocking runs in the vtask pool only under the admission reservation and long blocking I/O only on the lane.
  - Measure steady-state heartbeat cadence at 3–5 controllers and confirm no added latency attributable to the new dispatch path.
  - Observable: a compliance assertion/inspection shows the drive threads non-blocking; heartbeat cadence matches the pre-change baseline.
  - _Requirements: 5.1, 5.2, 5.3, 5.4, 8.1, 8.2_
  - _Depends: 2.3, 3.4_

## Implementation Notes

- **Build/test env**: builds MUST run in the nix dev shell (`nix develop --command meson compile -C src/xce.build <target>`). The IDE clang reports false errors (missing openssl/unity headers, unknown `-fvar-tracking`) because it isn't in the nix shell — ignore those. Controller tests: `rm -rf /tmp/xce_*` first (bug-924 overlayfs), run `./src/xce.build/controller/controller_unit_test`, set `ASAN_OPTIONS=detect_leaks=0`.
- **Pre-existing leak (not ours)**: the default-options full controller suite exits non-zero due to a LeakSanitizer leak in `test_commit_barrier_reply.c` / `test_worker_join.c` / `test_egress_filter.c` (~174KB). Unrelated to this spec. Run with `detect_leaks=0` (or a `-t` filter) so it doesn't mask raft-drive-async regressions.
- **1.1**: `drive_async_liveness_reserved(n_peers, n_vcpus) = max(1, min(n_peers−1, n_vcpus−1))` in `ha.c` is the single source of truth for `VTASK_LIVENESS_RESERVED` (floor applied LAST). Consumed by 1.6. `myconf_t` CLI override fields for the new `DEFAULT_*` constants were deferred to the consumer tasks (1.2 connect-timeout, 1.3/1.6) per YAGNI — add each field with its reader.
- **2.1**: reconnect offload is gated by a default-ON kill-switch `g_drive_async_reconnect_offload` (legacy inline path retained for the TDD-RED proof + operational fallback). Per-peer single-flight via `ha_state_t.reconnect_inflight[]` CAS + slowlane key; flag cleared on all 4 terminal paths (prep-fail, lane-fn success/-EAGAIN, -EEXIST, hard-error). Lane fn = `reconnect_lane_fn` → `ha_connect_finish_peer` (happy path: bounded connect + lock-guarded iomgr_monitor; 2.2 adds inbound/outbound race + failure hardening via the documented seam). **FOLLOW-UP (test hygiene, LOW)**: the GREEN reconnect tests in test_ha_liveness.c leak the prepped endpoint on the simulated-success hook path (production-clean; masked by detect_leaks=0) — free the un-adopted ep in teardown when convenient.
- **1.6**: pool reservation lives in vtaskmgr (`vtask_class_t` + `blocking_inflight`/`liveness_reserved` + `vtaskmgr_blocking_acquire/release`); commit folds in via opt-in `commit_set_pool` (kept `commit_engine_init(n_vcpus)` signature — the `n_vcpus` local cap is the genuine pool==NULL fallback for unit fixtures). **Production wiring is in `cluster.c` phase2** (once-guarded, right after `commit_engine_init`): `vtaskmgr_set_liveness_reserved(R)` + `commit_set_pool` — DO NOT remove; without it the reservation is dead. Lock order `p->lock → ce->lock → blk_lock` (cycle-free). New-cap = `n_vcpus−R` (tighter than old `n_vcpus−2` for 5/7-node — intended). LOW deferred: `vtaskmgr_blocking_cap()` reads `liveness_reserved` without `blk_lock` (init-time set-once, benign). When 3.1 dispatches peer_ctrl, REQ_BLOCK slot is released in `run_task` — ensure no double-count vs the commit path.
- **3.1 (DONE — flag now DEFAULT ON, bug-1376 fixed 2026-07-02)**: `peer_ctrl_msg_handler` is now a shim gated by `g_drive_async_peerctrl_dispatch_enabled` (**DEFAULT 1 / ON**; toggle via `ha_set_peerctrl_dispatch_for_test` for a regression comparison). Flag OFF → `peer_ctrl_msg_handler_inline` runs on io_worker, byte-for-byte prior path. Flag ON → `peer_ctrl_dispatch_to_pool` builds a `VTASK_IO` task and `vtask_enq_classified`s it (per-endpoint order auto-enforced by `bind_vtask`→`endpoint_block_tasks`). The 3 routers (`vote_reply`/`pre_vote_reply`/`repl_ack`_decode_and_dispatch) take `pkt` as an explicit param now (no cross-thread `ep->fsm_in.pkt` re-read). **Packet ownership**: shim clears `ep->fsm_in.pkt` immediately (sole-owner transfer); `peer_ctrl_vtask_entry` NULLs `iod->pkt` before the body frees it once; new `vtask_t.owns_io_pkt` (default 0; controller sets 1) lets `taskq_item_killer` free an un-executed queued task's packet without double-freeing the libxce endpoint-owned path (bug-1199). `ha_state_t.pool` wired in `cluster.c` to `&c->vtaskmgr`.
  - **TWO FLAG-ON GATE ITEMS — MUST be resolved before 3.4 flips the flag ON** (documented with loud code comments; harmless while flag OFF):
    1. **bug-1197 (CRITICAL UAF)**: `raft_cb_become_follower` `endpoint_fini()`s peer vote-eps from a pool worker → frees `ep->data` while the single io_worker can concurrently read `ep_j->data` in `peer_ctrl_msg_handler_inline` for a same-epoll-batch queued event. FIX (part of 3.4 enablement): `iomgr_unmonitor(ep)` BEFORE `endpoint_fini(ep)` in every pool-reachable teardown.
    2. **bug-1198 (HIGH classification)**: `peer_ctrl_classify` returns `VTASK_LIVENESS` for all, but the CTRL_PRIMARY repl-ack path can run a blocking `HA_REPL_SEND_SNAPSHOT`→`persist_checkpoint` inline under the never-gated liveness reservation → breaks deadlock-freedom. The 3.2 SlowLane offload mechanism now EXISTS but is DORMANT (flag OFF, below), so bug-1198 stays OPEN until 3.4 flips it ON together with the peerctrl flag.

- **3.2 — Part B (trim guard, bug-1147) is LIVE; Part A (snapshot offload) is now DEFAULT ON (bug-1197 + bug-1376 fixed 2026-07-02)**:
  - **Part B (production-live, default-ON, no flag)**: `trim_wal_segments` (`persist.c`) now bounds deletion by `floor = repl_trim_lsn(&p->repl, active_seg_lsn, snap_interval_s)` = `min(active_seg_lsn, every active standby acked_lsn, every valid disconnected-slot last_disconnected_lsn)`, after a `repl_expire_disconnect_slots` call to age out stale slot guards. `floor ≤ active_seg_lsn` structurally (repl_trim_lsn caps at its `checkpoint_lsn` arg), so the new `seg_start_lsn < floor` check subsumes the old `< active_seg_lsn` guard and can only PRESERVE more segments — never delete one a live standby still needs (closes the bug-795/828 trim-vs-cursor race for connected/recently-disconnected standbys; does NOT fix empty/cursor=0 join — needs entity-bearing snapshots, separate spec). Lock-safe: `trim_wal_segments` runs with `p->lock` released; `repl_trim_lsn`/`repl_expire_disconnect_slots` take only `r->lock` — no inversion. Tests: `test_trim_guard_preserves_standby_segments`, `test_trim_without_standbys_proceeds_normally` (test_e2e_checkpoint.c).
  - **Part A (DORMANT, `g_drive_async_snapshot_offload` DEFAULT 0/OFF)**: `ha_snapshot_send_dispatch` routes `HA_REPL_SEND_SNAPSHOT` inline (flag OFF — byte-for-byte legacy, send on io_worker) or to `slowlane_submit`+`snapshot_lane_fn` (flag ON). Send body factored into `ha_snapshot_do_send` (shared by both paths, holds the XCE_TESTING hook). Per-peer single-flight via `ha_state_t.snapshot_inflight[MAX_HA_CANDIDATES]` CAS + slowlane key `peer_idx+1`; flag cleared on all 4 terminal paths (alloc-fail, -EEXIST, hard-error, lane-fn completion) exactly like reconnect (2.1). Tests: `test_snapshot_offload_runs_on_lane`, `test_snapshot_inline_when_flag_off`, `test_snapshot_offload_dedups_per_peer` (test_ha_liveness.c).
  - **WHY IT WAS DORMANT, AND HOW IT CLOSED**: `snapshot_lane_fn` captures a BORROWED standby endpoint (unlike reconnect, which owns its ep). With the flag ON the send runs on a lane thread while the io_worker may RDHUP-`endpoint_fini` that ep → cross-thread UAF (bug-1197 family). Flag OFF kept send+teardown serialized on the io_worker. Caught + fixed pre-commit early on: the offload flag was initially authored DEFAULT-ON (would have shipped the UAF) → corrected to OFF in both ha.c and the ha.h accessor doc (bug-1210). **Closed 2026-07-02**: `ha_snapshot_send_dispatch`/`snapshot_lane_fn` now bracket the lane dispatch with `endpoint_ref`/`endpoint_unref` (reusing the general bug-1298 refcount system), so the borrowed-endpoint hazard cannot recur regardless of thread timing — no lane-serialized-teardown redesign needed. `g_drive_async_snapshot_offload` and `g_drive_async_peerctrl_dispatch` both now default ON together (bug-1376, a distinct cold-start self-deadlock unmasked by flipping both, is also fixed).

- **3.3 (LANDED — checkbox `[x]`, flag DEFAULT ON)**: requester-less config-push transport offload, flag `g_drive_async_configpush_offload` **DEFAULT 0/OFF** (toggle `ha_set_configpush_offload_for_test`). `ha_config_push_offloadable(c, bytes, blen, sha256, new_tag)` routes inline (flag OFF — verbatim `push_config_transport_only`, caller still owns+frees `bytes`) or to `slowlane_submit`+`configpush_lane_fn` (flag ON — DEEP-COPIES `bytes`+sha into a ctx the lane frees). Single cluster-wide single-flight via scalar `ha_state_t.configpush_inflight` CAS + lane key=1 (config push is a cluster-wide fan-out, not per-peer); flag cleared on all terminal paths (alloc-fail×2, -EEXIST, hard-error, lane completion). Send body factored into `ha_config_push_do_send` (production → `push_config_transport_only`; XCE_TESTING → `ha_configpush_probe` stub since cluster.c/transport isn't linked in test). THREE requester-less drive-thread call sites converted: `raft_cb_become_leader` (promotion, both bytes/NULL branches), the `HA_REPL_PUSH_CONFIG` heartbeat case, and the direct-CTRL_PRIMARY FSM promotion branch (~ha.c:4476). UNCHANGED: `push_config_to_standbys` admin-upgrade barrier (has a waiting requester) and `join.c`. Tests: `test_configpush_offload_runs_on_lane`, `test_configpush_inline_when_flag_off`, `test_configpush_offload_dedups` (test_ha_liveness.c). **bug-1381 CLOSED 2026-07-02** (see task 3.4a's note below for the fix) — `push_config_transport_only` itself (not the lane fn) no longer races raw standby endpoint pointers. cpp-review: no CRITICAL/HIGH/MEDIUM; controller suite EXIT=0. **Flip attempted same day, REVERTED**: with bug-1381 closed, flipped `g_drive_async_configpush_offload` to default ON, rebuilt `xce.build` (confirmed binary mtime > source mtime — see the Do-Not-Repeat cerebrum entry this session added after almost repeating the earlier stale-binary mistake), then ran `--chaotic 6` (PASS, 21 jobs, zero divergence/loss) and `--force-snapshot` x3 (0/3 PASS — "no snapshot install" / "did not reconverge to 3-of-3"). A flag-OFF control on the identical rebuilt binary went 1/2 on the same two failure signatures, proving the defect is PRE-EXISTING and NOT caused by this flag: `ha_send_snapshot_to_standby` (config_repl.c) never checks `xce_snapshot_chunk_req_raw`'s return in its chunk-send loop, so a chunk that fails to enqueue (e.g. `-ECONNRESET` during the empty-rejoin reconnect window) is silently dropped while the function still logs "transfer complete" and returns 0 — logged as **bug-1386** (open). Flag-ON's 0/3 vs flag-OFF's 1/2 is too small a sample to rule out a secondary timing interaction (configpush lane traffic narrowing the snapshot lane's completion window before reconnect), so the flag was reverted to OFF rather than accepted on a "probably fine" basis. Full controller suite re-confirmed green (230/230) at OFF. **bug-1386 FIXED 2026-07-02**: `ha_send_snapshot_to_standby` (config_repl.c) now captures the return value of `xce_snapshot_chunk_req_raw`/`xce_snapshot_chunk_hook` in its chunk loop, breaks on a non-zero `send_rc`, and propagates it as `rc` instead of silently continuing and reporting "complete"; the success log is gated on `rc==0`, a new WARN logs the failure (checkpoint_lsn/size/sent-offset/chunk_index/rc) otherwise. No receive-side timeout/re-request was added — unnecessary: `snapshot_inflight[peer_idx]` already clears on every send-thread exit (`snapshot_lane_fn`), so the next heartbeat's SEND_SNAPSHOT decision naturally retries a peer whose `applied_lsn` never advanced. New RED->GREEN test `test_snapshot_transfer_stops_and_propagates_send_failure` (test_config_repl.c) injects a mid-loop `-ECONNRESET` via the existing `xce_snapshot_chunk_hook` seam; RED confirmed via temporary revert ("Expected -104 Was 0"), GREEN restored, full controller suite clean (0 failures). **Re-soak to verify bug-1386's fix (2026-07-02, same session) surfaced a THIRD distinct pre-existing bug, STILL BLOCKING**: ran `--force-snapshot` (flag still OFF, just verifying bug-1386 in isolation before re-attempting the flip) — FAIL again, but a NEW signature: ctrl-1 (sender) now correctly logs "transfer complete" for the full 735546-byte/12-chunk transfer both times (confirming bug-1386's fix works — no more false-complete), but ctrl-2 (receiver) fails with `ha_handle_snapshot_chunk: persist_load_snapshot failed rc=-5` after receiving all 12 chunks. Byte-comparing the sender's source checkpoint against the receiver's assembled file (two independent repro runs, checkpoint LSNs 0x75 and 0x76) showed IDENTICAL total size but DIFFERENT CONTENT — the received file diverges from the source at an exact `SNAPSHOT_CHUNK_SIZE` (65536-byte) chunk boundary, then contains the correct data byte-SHIFTED (same job-frame strings, same occurrence count, wrong offset), re-synchronizing later. Shift direction/magnitude differed between the two attempts (-912 bytes / +4684 bytes) — nondeterministic, not fixed-offset. This is a chunk-content-CORRUPTION bug, not a drop (bug-1386) — logged as **bug-1389** (open, NOT fixed). Prime suspect: `xce_snapshot_chunk_encode_raw`'s capnp `Data` encode (`request.c:630-671`, `capn_new_list8`/`capn_setv8` pattern) — same family as bug-847 (WAL catch-up streaming payload corruption at a capnp segment boundary). Receive-side `persist_load_snapshot` is confirmed NOT the bug — it correctly rejects the corrupted file. **bug-1389 ROOT-CAUSED AND FIXED 2026-07-02**: added live per-chunk crc32 TRACE instrumentation at both `ha_send_snapshot_to_standby` (send) and `ha_handle_snapshot_chunk` (receive), correlated by `(snapshot_lsn, chunk_index)`, then re-ran `--force-snapshot` — the crc32 log proved every one of 12 chunks' CONTENT crossed the wire byte-for-byte intact (all 12 crc32s matched exactly), so capnp/transport was NEVER the bug (the earlier "capnp encode suspected" hypothesis, and an exhaustive prior trace of the wire framing and far-pointer resolution, are both correctly refuted). The real defect: the RECEIVE log showed chunks arriving out of `chunk_index` order — `0,1,3,4,2,5,6,7,8,9,10,11` — because the vtask pool dispatches one transfer's chunks across multiple workers (`cpu#0-3` in the log) with no ordering guarantee, but `ha_handle_snapshot_chunk` (ha.c) appended every chunk's bytes to `snap.part` in ARRIVAL order (`fopen("ab")`/`fwrite`) instead of at the chunk's own `chunk_offset` (a field already carried on the wire but never used). Reordered chunks landed at the wrong file position — same total byte count, wrong content from the first reordered chunk onward, exactly the "shifted, resyncing later, nondeterministic magnitude" signature originally observed. **Fix**: (1) `ha_handle_snapshot_chunk` now writes each chunk via `pwrite(fd, payload, len, chunk->chunk_offset)` — order-independent and safe for concurrent writers (positioned writes to disjoint byte ranges of the same fd don't race). (2) New `ha_state_t.snapshot_recv_count`/`snapshot_recv_total` fields (ha.h, guarded by the existing `ha->lock`) track how many distinct chunks have actually been written; finalize (verify/rename/`persist_load_snapshot`) now fires only once `count >= total` (known once `is_last`'s `chunk_index+1` is seen), not merely because THIS call happens to carry `is_last` — under reordering, `is_last` can process before an earlier chunk. `snapshot_recv_total` is claimed (reset to 0) by whichever thread crosses the completion threshold, preventing a late straggler from double-finalizing. `chunk_index==0` still resets both the stale-file unlink and the new counters (Req 4.4 unchanged). New RED->GREEN test `test_snapshot_recv_out_of_order_chunks_assemble_correctly` (test_config_repl.c) replays the exact non-sequential delivery order and asserts byte-identical assembly; RED confirmed via temporary revert (`Expected 0 Was -22`, `job_count` field corrupted by the misordered chunk boundary), GREEN restored, full controller suite clean. Also added 2 new full-size (65536-byte) capnp encode/decode round-trip tests (`test_wal_wire.c`) — permanent regression coverage that definitively rules out the capnp layer (the prior test only covered 300 bytes). **Live-validated**: 3/3 `--force-snapshot` rounds PASS (genuine `HA: snapshot loaded ... full job/queue entity restoration` installs confirmed each time, zero divergence/loss) + 6-round `--chaotic` PASS. **Flag re-flipped to ON 2026-07-02, LANDED**: with bug-1389 closed, `g_drive_async_configpush_offload` flipped to default 1/ON, full build clean (all components, mtime-verified), full controller suite 0 failures. Soak-validated on the flag-ON binary: `--chaotic 6` PASS (21 jobs, zero divergence/loss); `--force-snapshot` 3/3 genuine PASS (each confirmed a real `HA: snapshot loaded` install, zero divergence/loss) — one intervening run hit the unrelated pre-existing job-submission flakiness ("could not submit 110 jobs to trigger a checkpoint", never reached the snapshot-transfer phase at all) and was retried clean. Task 3.3 is now fully landed — all three requester-less config-push call sites route through the slow lane by default.

- **3.4a (bug-1197 endpoint-quiescence fix — committed; flags STILL default OFF)**: makes the flag-ON peer-control dispatch path UAF-free so `raft_cb_become_follower` (running as a pool vtask) can free peer vote-eps while other pool workers process peer-control on the same eps. Mechanism (no libxce-wide endpoint refcount; reuses the existing per-ep FIFO + busy serialization):
  - **Deferred teardown-as-vtask**: the loser loop, when `ha_peerctrl_dispatch_enabled() && ha->pool`, does per vote-ep `endpoint_mark_tearing_down(ep)` → `iomgr_unmonitor(ep)` → enqueue a `VTASK_IO`/`VTASK_LIVENESS` teardown vtask bound to that ep (`vote_ep_teardown_vtask_entry`). Per-ep busy FIFO makes it run strictly AFTER any in-flight/queued same-ep peer-control vtask; it NULLs `iod->ep` before `endpoint_fini`+`free(ep)` to disarm `run_task`'s post-body `endpoint_unblock_tasks` (which would otherwise touch the freed `ep->lock`). OFF / pool-unwired path is byte-for-byte the legacy `endpoint_fini` (no free).
  - **RCU-like drain closes the TOCTOU window**: a new `_Atomic int endpoint_t.dispatch_inflight` (+ `tearing_down:1` bit). The ON dispatch brackets its flag-check+enqueue: `fetch_add(seq_cst)` → if `tearing_down` drop pkt + `fetch_sub` → else enqueue → `fetch_sub`. `endpoint_mark_tearing_down` sets the flag under `ep->lock`, releases, then spins `while(load(dispatch_inflight,seq_cst)>0) sched_yield()`. become_follower marks (drains) BEFORE enqueuing the teardown vtask, so any dispatch that read the flag false is FIFO-before teardown; any later dispatch drops. Could NOT hold `ep->lock` across the enqueue — it inverts with the taskq lock (dispatcher takes taskq→ep in `vtask_eligible`); the counter is the lock-free handshake. Drain is bounded (the held window is just calloc+non-blocking enqueue on the single io_worker).
  - Files: `src/include/netio.h` (`tearing_down:1`, `dispatch_inflight`, mark/is helpers), `src/lib/netio.c` (helpers + drain + `do_endpoint_init` zero-init), `src/controller/ha.c` (`vote_ep_teardown_*`, loser-loop async branch, dispatch bracket + late-drop, test seams), `src/controller/ha.h`. Tests: `test_ha_vep_teardown.c` (5 ASan tests: ordering, late-drop, disarm, OFF-path, deterministic drain-blocks). xcelib_test 282/0; controller EXIT=0; ASan-clean. Adversarial cpp-review: no CRITICAL/HIGH; handshake formally sound (SC total order on the counter + monotone `tearing_down` under `ep->lock`); flag-ON path proven race-free.
  - **CLOSED for `configpush_offload`'s prod flip**: `snapshot_lane_fn`'s borrowed-ep hazard closed 2026-07-02 (endpoint_ref/unref, see task 3.2's note). `configpush_lane_fn`'s hazard was a DIFFERENT race — logged as **bug-1381**: `push_config_transport_only` (not `configpush_lane_fn` itself) grabbed raw standby endpoint pointers and spawned unprotected `pthread_create` workers for the fan-out, a race that existed independent of the offload flag. **Fixed 2026-07-02**: each standby ep is now `endpoint_ref()`'d under `c->ha.lock` at the same point its pointer is snapshotted into `push_ctx_t`, with exactly one matching `endpoint_unref()` per ref — from `push_thread` after it finishes using `ctx->ep`, or from `push_config_transport_only`'s own cleanup if a thread never started (pthread_create failure or a total threads[] alloc failure). A concurrent RDHUP teardown can now only drop the OWNER's reference while a push is in flight, never free `ep` out from under `push_thread`. New seam `push_thread_send_hook` (config_repl.c, same pattern as `xce_wal_stream_request_hook`) lets `test_configpush_transport_holds_endpoint_ref_across_send` (test_config_repl.c) park a send mid-flight and assert the refcount bracket; RED→GREEN verified via temporary revert (`Expected 2 Was 1`), full controller suite green (230/230) after restore. `g_drive_async_configpush_offload` remains default OFF — this closes the correctness blocker for the flip, but the flip itself (and its own soak validation) is a separate follow-up, not part of this fix.