myxce/.kiro/specs/raft-drive-async/tasks.md

Implementation Plan

Tasks mirror the design's 4-phase migration. The slow lane is foundational (every offload submits to it); the peer_ctrl→vtask move is coupled with its long-op offload in Phase 3 so no phase runs long I/O inline in a blocked vtask.

• 1. Phase 1 — SlowLane executor and bounded connect (foundation)

• 1.1 Add drive-async configuration constants

  • Define connect-timeout, slow-lane worker count, and max-in-flight constants with documented defaults (sized to peer count).
  • Define the liveness reservation as a peer-scaled derivation, not a flat literal: VTASK_LIVENESS_RESERVED = clamp(n_peers − 1, 1, n_vcpus − 1) (floor 1 = deadlock-freedom; ceiling n_vcpus − 1 keeps ≥1 commit slot); WARN at init when clamped by n_vcpus.
  • Wire the constants into the controller config so they are overridable.
  • Observable: a build with the new constants compiles; defaults are readable at controller startup; the resolved reservation logged at init.
  • Requirements: 2.3, 3.1

• 1.2 Implement the SlowLane executor

  • Bounded pool of dedicated worker threads, isolated from the vtask request pool; non-blocking submit that enqueues and returns immediately.
  • Admission cap on concurrent in-flight tasks; per-key in-flight dedup (key 0 disables dedup); init/fini wired into the controller lifecycle.
  • Observable: a submitted task runs exactly once on a lane worker; submit never blocks the caller; concurrent running tasks never exceed the cap; a submit with an in-flight key is a no-op.
  • Requirements: 3.1, 3.2, 3.3, 3.4, 1.5
  • Boundary: SlowLane

• 1.3 Bound the peer connect path

  • Reorder socket_connect to set non-blocking before connect and bound the attempt with a poll timeout (reuse the existing probe pattern); return success/timeout/error.
  • Update connect callers to pass the configured timeout.
  • Observable: a connect to an unreachable host returns within the configured timeout instead of the OS connect timeout; a connect to a reachable host still succeeds.
  • Requirements: 2.3
  • Boundary: BoundedConnect
  • Depends: 1.1

• 1.4 (P) SlowLane unit tests

  • Assert non-blocking submit, admission-cap bounds on concurrency, per-key dedup returns the deduped result, and run-exactly-once.
  • Observable: tests pass and fail if the cap or dedup is removed.
  • Requirements: 3.1, 3.2, 3.3, 3.4, 1.5
  • Boundary: SlowLane
  • Depends: 1.2

• 1.5 (P) Bounded-connect unit test

  • Assert socket_connect returns within the timeout against an unreachable host and succeeds against a reachable one.
  • Observable: test passes; reverting the reorder makes it hang/fail.
  • Requirements: 2.3
  • Boundary: BoundedConnect
  • Depends: 1.3

• 1.6 LivenessAdmission — pool-level task-class reservation in vtaskmgr

  • Add vtask_class_t (VTASK_LIVENESS / VTASK_LIVE_BLOCK / VTASK_REQ_BLOCK) + a classified submit; the dispatcher reserves the VTASK_LIVENESS_RESERVED value resolved in 1.1 (single source of truth — consume it, do not recompute the clamp/WARN here) and fast-rejects a VTASK_REQ_BLOCK with -XCE_ERR_NOT_READY when admitting it would breach the reservation (never parks a worker, never silently queues).
  • Fold commit.c's inflight_waits/COMMIT_WAIT_RESERVED into this — commit_submit submits a VTASK_REQ_BLOCK; the pool owns the single reservation.
  • Dormant infra in Phase 1: no behavior change until peer_ctrl dispatches through it (3.1). Hard prerequisite of 3.1 — without it, moving ack/vote handling onto the pool is not deadlock-safe (vote-fsync/config-ack waits, invisible to the old commit-only counter, could starve the ack that wakes commit-waiters).
  • Observable: a VTASK_LIVENESS task always finds a worker even when *_BLOCK tasks saturate n_vcpus − VTASK_LIVENESS_RESERVED; commit-wait still fast-rejects -XCE_ERR_NOT_READY at the cap (backpressure unchanged).
  • Requirements: 3.1, 3.2, 3.3, 4.5
  • Boundary: LivenessAdmission
  • Depends: 1.1

• 1.7 (P) LivenessAdmission unit test

  • Assert: a VTASK_LIVENESS task is dispatched while *_BLOCK tasks hold n_vcpus − VTASK_LIVENESS_RESERVED vcpus; VTASK_REQ_BLOCK over the cap returns -XCE_ERR_NOT_READY without parking; folding commit-wait through the new path preserves the existing reject-at-cap behavior. Reverting the reservation makes a liveness task starve behind blockers.
  • Requirements: 3.1, 3.2, 3.3, 4.5
  • Boundary: LivenessAdmission
  • Depends: 1.6

• 2. Phase 2 — Reconnect offload (sender-side liveness)

• 2.1 Offload the delaymgr peer reconnect to the slow lane

  • The heartbeat timer detects a disconnected peer and submits the reconnect to the lane (per-peer dedup key), then returns immediately; the timer never performs the connect inline.
  • Refcount the endpoint across the in-flight window.
  • Observable: while one peer is unreachable, the delaymgr timer keeps firing and heartbeats to reachable peers continue within the interval; only one reconnect attempt per peer is in flight at a time.
  • Requirements: 2.1, 2.2, 2.5
  • Boundary: ReconnectOffload, SlowLane
  • Depends: 1.2, 1.3

• 2.2 Complete the reconnect on the lane worker

  • The lane worker performs the bounded connect, fully prepares the endpoint, and registers it via the lock-guarded monitor interface; on failure it clears the in-flight flag and lets the next tick retry without terminating; resolve the inbound-vs-outbound connection race per the connection-ownership rules.
  • Observable: a successful reconnect registers the new connection with no lost or reordered queued messages; a failed attempt leaves the controller alive and retrying.
  • Requirements: 2.4, 2.6, 4.4
  • Boundary: ReconnectOffload
  • Depends: 2.1

• 2.3 Reconnect-offload integration test

  • With one peer made unreachable, assert heartbeats to the surviving peers continue within the interval, the delaymgr is not stalled, and reconnect is deduped per peer.
  • Observable: test passes; reverting the offload (inline connect) makes heartbeats to survivors stall.
  • Requirements: 2.1, 2.2, 2.5
  • Boundary: ReconnectOffload
  • Depends: 2.2

• 3. Phase 3 — Peer-control dispatch to vtask, coupled with long-op offload (receiver-side)

• 3.1 Dispatch peer-control message handling to the vtask pool

  • Register peer endpoints so peer-control messages dispatch through the vtask pool instead of running inline on iomgr; convert the peer-control handler into a vtask entry that does only bounded protocol work inline.
  • Preserve per-connection processing order via the per-endpoint block primitive and preserve vote/ack/term semantics and persist-before-vote ordering; keep the steady-state heartbeat/vote send paths as non-blocking enqueues.
  • Observable: peer-control no longer runs on the iomgr loop; existing election/heartbeat/repl behavior is unchanged; messages on one connection are processed in arrival order.
  • Deadlock-freedom gate (bug-invariant reversal): moving ack decode to the pool co-locates the commit-barrier wakers (acks) and waiters (commit_submit(COMMIT_MODE_WAIT)) on one pool — today acks run on iomgr precisely to avoid this (commit.c:16). The old commit-only COMMIT_WAIT_RESERVED is insufficient (it can't see vote-fsync/config-ack blockers this move also puts on the pool), so safety rests on LivenessAdmission (task 1.6): ack/vote-receipt dispatched as VTASK_LIVENESS (never gated, never offloaded), commit_submit as VTASK_REQ_BLOCK under the pool reservation. Task 3.4 is the admission gate for this task — 3.1 is not complete until 3.4 is green.
  • Requirements: 1.4, 4.1, 4.2, 4.5, 4.6, 5.1, 5.4, 8.1, 8.2
  • Boundary: PeerCtrlDispatch
  • Depends: 1.2, 1.6
  • 2026-07-01/07-02 session (production flip — DONE, both flags now default ON): with 4.1+4.2 green, executed the "full fix, defer prod flip" decision's flip. (1) Closed the snapshot-lane borrowed-endpoint hazard (task 3.2's Part A) properly: ha_snapshot_send_dispatch takes endpoint_ref(ep) before slowlane_submit, released on every non-lane terminal path and by snapshot_lane_fn on completion — reuses the general bug-1298 refcount system (standby-ep teardown already routes through endpoint_unref via cluster.c's endpoint_killer, confirmed by reading it — this infra landed in raft-ha-core-safety task 5.8, AFTER this spec's 3.2/3.4 notes were written, so the hazard they describe is now closed for real). New unit test test_snapshot_offload_holds_endpoint_ref_across_transfer (RED→GREEN). (2) Flipped g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch_enabled to default ON; full controller suite green; a 6-round --chaotic soak passed clean (zero divergence, zero job loss). (3) A --force-snapshot soak (genuinely empty node cold-start rejoin — the ONE scenario chaotic mode never exercises, since it restarts nodes with WAL intact) then surfaced a SEPARATE pre-existing bug, unmasked (not caused) by the flip's timing shift: ha_stream_wal_to_standby had no filter for WAL_CHECKPOINT markers (local-only, never a logical Raft entry — consumes a real LSN slot via persist_write_checkpoint_marker's own p->current_lsn++) and shipped one to a catching-up standby, which correctly refused it and got stuck retrying the same LSN forever. FIXED (config_repl.c, skip hdr.type == WAL_CHECKPOINT entirely, mirroring the existing replay_wal/term-index precedent of treating checkpoint-marker LSNs as invisible gaps) with a RED→GREEN unit test (test_wal_stream_skips_checkpoint_marker, differential-verified). (4) Re-ran --force-snapshot with BOTH flags ON and the checkpoint-skip fix in place — it STILL failed, this time with a THIRD, different issue logged as bug-1376: the freshly-wiped empty node's own self-election races cluster_init_phase2's vtask-pool wiring, then the process hangs silently. Live-repro'd deterministically via a GDB_WRAP debug seam added to run-ha-soak.sh (launches a named node directly under gdb --batch, so it becomes ptrace-legal as gdb's direct child under yama scope 1 — bypasses a live-attach restriction) and a bounded SIGINT-to-gdb interrupt for a full ASan-free thread backtrace. Root-caused and FIXED: peer_ctrl_msg_handler held ep->dispatch_inflight incremented across the ENTIRE peer_ctrl_dispatch_to_pool call, including its inline fallback (the common cold-start shape when ha->pool isn't wired yet). That inline body can reach raft_cb_become_follower, which — now that cluster_init_phase2 ran synchronously earlier in the SAME call chain and wired the pool — tears down its OWN loser vote-ep (the very ep whose reply triggered this call) via the deferred path (vote_ep_teardown_async), whose endpoint_mark_tearing_down spins waiting for dispatch_inflight==0 on that SAME ep — a count only this exact call, on this exact thread, holds — a permanent self-deadlock. Fixed by moving the dispatch_inflight release into peer_ctrl_dispatch_to_pool itself, at the exact right point per exit path: released BEFORE calling the inline body on every fallback path (pool==NULL, OOM, enqueue failure) — safe because io_worker is single-threaded and the inline path never enqueues anything, so there is no pool-worker race left to protect against once inline execution is committed to — and only AFTER the (non-blocking) enqueue completes on the success path, preserving the original protection for the genuinely concurrent pool-worker case. New unit test test_vep_self_teardown_does_not_deadlock (test_ha_vep_teardown.c, bounded-timeout pthread_timedjoin_np join so a regression fails loud instead of hanging the suite; RED confirmed via a temporary manual revert — Expected 0 Was 110 (ETIMEDOUT) — then GREEN restored). Live-verified with both flags ON: --force-snapshot soak now completes without hanging (2/3 runs confirmed a real snapshot install + committed_divergence=0 committed_job_loss=0; 1/3 hit the separate known-flaky "checkpoint didn't fire before the wipe this round" — a timing non-issue the harness itself already calls out, NOT a hang); a 6-round --chaotic soak with both flags ON also passed clean; full controller suite green throughout. Both g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch_enabled now default ON in production. g_drive_async_configpush_offload (task 3.3) stays OFF — it has its own DISTINCT, still-open hazard (see task 3.3's note) unrelated to bug-1376.

• 3.2 Offload snapshot send to the slow lane

  • Trigger snapshot send + checkpoint to the lane from the vtask entry without holding the per-endpoint block across the transfer; dedup one snapshot per standby.
  • Enforce the snapshot/checkpoint concurrency invariant: snapshot reads a frozen term/lsn boundary; segment trim never deletes at/above any in-flight snapshot boundary or a standby catch-up cursor; at most one checkpoint in flight; bound the in-lane snapshot socket write so an unreachable-mid-transfer peer cannot pin a worker.
  • Add the trim cursor-bound guard (bug-1147, stopgap): trim_wal_segments (persist.c:1374) currently bounds deletion ONLY by the open segment and consults no standby cursor; before this task moves checkpoint/trim onto a concurrent lane worker, change it to compute floor = min(all standby acked_lsn, in-flight snapshot boundaries) under p->lock and refuse to delete at/above floor. This is strategy-A retention discipline; it does NOT fix empty/far-behind join (cursor=0) — that needs entity-bearing snapshots (separate spec).
  • Observable: a catch-up that triggers a snapshot does not block iomgr; the triggering peer's later messages are not queued behind the transfer; concurrent appends plus checkpoint/trim never delete a segment at/above any live standby acked_lsn or in-flight snapshot boundary (reverting the guard makes the test re-create a bug-795/828 gap).
  • Requirements: 1.1, 1.3, 1.5, 4.3
  • Boundary: PeerCtrlDispatch, SlowLane
  • Depends: 3.1

• 3.3 (P) Offload the transport-only config-push to the slow lane

  • Move the requester-less iomgr/promotion config-push transport (and its wait) to the lane; replace the drive-thread join with a lane-internal wait and non-blocking return; leave the admin upgrade path (with a waiting requester) on its existing vtask-blocking path unchanged.
  • Observable: a promotion/heartbeat-driven config push no longer blocks iomgr; the admin upgrade reply path is unchanged.
  • Requirements: 1.2
  • Boundary: PeerCtrlDispatch, SlowLane
  • Depends: 3.1

• 3.4 Receiver-side integration tests

  • 3.4a (DONE, committed 6d510d8): bug-1197 endpoint-quiescence fix landed so flag-ON peer-control dispatch is UAF-free (deferred teardown-vtask + RCU dispatch_inflight drain).
  • 3.4b (DONE): receiver-side liveness integration tests (test_ha_dispatch_integration.c) with the dispatch flags flipped ON in-test — P1 a parked snapshot/config-push lane transfer does not block another endpoint's peer-control on the pool; P2 a VTASK_LIVENESS ack runs under REQ_BLOCK pool saturation (reservation never starves it; one-over fast-rejects to prove the boundary); P3 cross-endpoint peer-control runs concurrently (per-ep FIFO does not serialize across eps). Fixed a flaky teardown-race hang (bug-1224: poll end-of-body completion counters before the cancel-based vtaskmgr_fini). 5/5 deterministic, full controller suite EXIT=0, ASan-clean.
  • Production flags STILL DEFAULT OFF (peerctrl/snapshot/configpush). Per the "full fix, defer prod flip" decision, the production flip happens only after 4.1 partition soak + 4.2 regression sweep pass; the snapshot/configpush lane borrowed-ep refcount (distinct from 3.4a) is also required before flipping those two ON.
  • Assert that during a snapshot/config-push transfer the controller still processes other peers' heartbeats and votes; that a quorum of WAL-acks advances the commit position promptly even with the vtask pool near saturation; and that cross-endpoint acks are not serialized behind one another.
  • Observable: tests pass; commit advances after a quorum of acks under load; iomgr stays responsive during transfers.
  • Requirements: 1.1, 1.3, 1.4, 4.3, 4.5
  • Boundary: PeerCtrlDispatch
  • Depends: 3.2, 3.3

• 4. Phase 4 — Verification, partition test, and regression

• 4.1 Network-partition (unreachable-peer) soak scenario

  • Add a partition primitive that renders one node unreachable (distinct from process kill). Primary = in-process socket-drop hook in xce_test (XCE_TESTING) — blackholes a node's peer traffic at the endpoint layer with NO privilege, so R7 actually runs in the unprivileged/overlayfs sandbox (avoids the perpetual-skip fate of ha-term-safety 5.2/5.4). Secondary (optional) = firewall iptables/nft DROP for real-network fidelity, capability-gated with a loud skip when unprivileged.
  • Assert: heartbeats to survivors continue, no spurious election while the leader holds the majority, the peer rejoins on heal, and no committed job is lost.
  • Observable: the partition soak runs and passes; the same scenario fails against the pre-change build (heartbeat stall / spurious election).
  • Requirements: 7.1, 7.2, 7.3, 7.4
  • Boundary: PartitionTest
  • Depends: 2.3, 3.4

• 4.2 Existing-feature regression sweep

  • DONE — verdict: raft-drive-async introduces NO new regression. Unit-level: controller_unit_test EXIT=0, xcelib_test 282/0 (all HA suites — election/prevote/liveness/failover/rdhup/repl/config-repl/persist-recovery/commit-barrier/durability — green, dormant flags OFF). Live kill-based failover soak (real fs ~/tmp, release+ASAN builds): zero committed-JOB loss across every completed round (jobs 3→12 monotonic). The 6-round CHAOTIC soak does not run clean to completion — it fails around round 4 — but a baseline comparison against pre-Phase-3 binaries (commit 21e26a1) fails round 4 IDENTICALLY (election stall / reconverge), and the round-4 committed-prefix divergence at LSN 9 is the documented pre-existing bug-851/795 family (chaotic-failover silent divergence), present before Phase 3 and out of scope for this spec (needs a separate HA-core effort). See bug-1225.
  • Run the existing HA suites unchanged — election, prevote, liveness, failover, rdhup-integration, repl majority/apply/catchup, config-repl/repl-push, persist-recovery, commit-barrier/commit-durability — and the kill-based failover soak.
  • Observable: all listed suites pass with no change to their asserted behavior.
  • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7
  • Depends: 3.4

• 4.3 Thread-role compliance and heartbeat performance check

  • Verify no blocking operation runs inline on the iomgr loop or delaymgr timer for the in-scope paths; bounded blocking runs in the vtask pool only under the admission reservation and long blocking I/O only on the lane.
  • Measure steady-state heartbeat cadence at 3–5 controllers and confirm no added latency attributable to the new dispatch path.
  • Observable: a compliance assertion/inspection shows the drive threads non-blocking; heartbeat cadence matches the pre-change baseline.
  • Requirements: 5.1, 5.2, 5.3, 5.4, 8.1, 8.2
  • Depends: 2.3, 3.4

Implementation Notes

• Build/test env: builds MUST run in the nix dev shell (nix develop --command meson compile -C src/xce.build <target>). The IDE clang reports false errors (missing openssl/unity headers, unknown -fvar-tracking) because it isn't in the nix shell — ignore those. Controller tests: rm -rf /tmp/xce_* first (bug-924 overlayfs), run ./src/xce.build/controller/controller_unit_test, set ASAN_OPTIONS=detect_leaks=0.

• Pre-existing leak (not ours): the default-options full controller suite exits non-zero due to a LeakSanitizer leak in test_commit_barrier_reply.c / test_worker_join.c / test_egress_filter.c (~174KB). Unrelated to this spec. Run with detect_leaks=0 (or a -t filter) so it doesn't mask raft-drive-async regressions.

• 1.1: drive_async_liveness_reserved(n_peers, n_vcpus) = max(1, min(n_peers−1, n_vcpus−1)) in ha.c is the single source of truth for VTASK_LIVENESS_RESERVED (floor applied LAST). Consumed by 1.6. myconf_t CLI override fields for the new DEFAULT_* constants were deferred to the consumer tasks (1.2 connect-timeout, 1.3/1.6) per YAGNI — add each field with its reader.

• 2.1: reconnect offload is gated by a default-ON kill-switch g_drive_async_reconnect_offload (legacy inline path retained for the TDD-RED proof + operational fallback). Per-peer single-flight via ha_state_t.reconnect_inflight[] CAS + slowlane key; flag cleared on all 4 terminal paths (prep-fail, lane-fn success/-EAGAIN, -EEXIST, hard-error). Lane fn = reconnect_lane_fn → ha_connect_finish_peer (happy path: bounded connect + lock-guarded iomgr_monitor; 2.2 adds inbound/outbound race + failure hardening via the documented seam). FOLLOW-UP (test hygiene, LOW): the GREEN reconnect tests in test_ha_liveness.c leak the prepped endpoint on the simulated-success hook path (production-clean; masked by detect_leaks=0) — free the un-adopted ep in teardown when convenient.

• 1.6: pool reservation lives in vtaskmgr (vtask_class_t + blocking_inflight/liveness_reserved + vtaskmgr_blocking_acquire/release); commit folds in via opt-in commit_set_pool (kept commit_engine_init(n_vcpus) signature — the n_vcpus local cap is the genuine pool==NULL fallback for unit fixtures). Production wiring is in cluster.c phase2 (once-guarded, right after commit_engine_init): vtaskmgr_set_liveness_reserved(R) + commit_set_pool — DO NOT remove; without it the reservation is dead. Lock order p->lock → ce->lock → blk_lock (cycle-free). New-cap = n_vcpus−R (tighter than old n_vcpus−2 for 5/7-node — intended). LOW deferred: vtaskmgr_blocking_cap() reads liveness_reserved without blk_lock (init-time set-once, benign). When 3.1 dispatches peer_ctrl, REQ_BLOCK slot is released in run_task — ensure no double-count vs the commit path.

• 3.1 (DONE — flag now DEFAULT ON, bug-1376 fixed 2026-07-02): peer_ctrl_msg_handler is now a shim gated by g_drive_async_peerctrl_dispatch_enabled (DEFAULT 1 / ON; toggle via ha_set_peerctrl_dispatch_for_test for a regression comparison). Flag OFF → peer_ctrl_msg_handler_inline runs on io_worker, byte-for-byte prior path. Flag ON → peer_ctrl_dispatch_to_pool builds a VTASK_IO task and vtask_enq_classifieds it (per-endpoint order auto-enforced by bind_vtask→endpoint_block_tasks). The 3 routers (vote_reply/pre_vote_reply/repl_ack_decode_and_dispatch) take pkt as an explicit param now (no cross-thread ep->fsm_in.pkt re-read). Packet ownership: shim clears ep->fsm_in.pkt immediately (sole-owner transfer); peer_ctrl_vtask_entry NULLs iod->pkt before the body frees it once; new vtask_t.owns_io_pkt (default 0; controller sets 1) lets taskq_item_killer free an un-executed queued task's packet without double-freeing the libxce endpoint-owned path (bug-1199). ha_state_t.pool wired in cluster.c to &c->vtaskmgr.

  • TWO FLAG-ON GATE ITEMS — MUST be resolved before 3.4 flips the flag ON (documented with loud code comments; harmless while flag OFF):
    1. bug-1197 (CRITICAL UAF): raft_cb_become_follower endpoint_fini()s peer vote-eps from a pool worker → frees ep->data while the single io_worker can concurrently read ep_j->data in peer_ctrl_msg_handler_inline for a same-epoll-batch queued event. FIX (part of 3.4 enablement): iomgr_unmonitor(ep) BEFORE endpoint_fini(ep) in every pool-reachable teardown.
    2. bug-1198 (HIGH classification): peer_ctrl_classify returns VTASK_LIVENESS for all, but the CTRL_PRIMARY repl-ack path can run a blocking HA_REPL_SEND_SNAPSHOT→persist_checkpoint inline under the never-gated liveness reservation → breaks deadlock-freedom. The 3.2 SlowLane offload mechanism now EXISTS but is DORMANT (flag OFF, below), so bug-1198 stays OPEN until 3.4 flips it ON together with the peerctrl flag.

• 3.2 — Part B (trim guard, bug-1147) is LIVE; Part A (snapshot offload) is now DEFAULT ON (bug-1197 + bug-1376 fixed 2026-07-02):

  • Part B (production-live, default-ON, no flag): trim_wal_segments (persist.c) now bounds deletion by floor = repl_trim_lsn(&p->repl, active_seg_lsn, snap_interval_s) = min(active_seg_lsn, every active standby acked_lsn, every valid disconnected-slot last_disconnected_lsn), after a repl_expire_disconnect_slots call to age out stale slot guards. floor ≤ active_seg_lsn structurally (repl_trim_lsn caps at its checkpoint_lsn arg), so the new seg_start_lsn < floor check subsumes the old < active_seg_lsn guard and can only PRESERVE more segments — never delete one a live standby still needs (closes the bug-795/828 trim-vs-cursor race for connected/recently-disconnected standbys; does NOT fix empty/cursor=0 join — needs entity-bearing snapshots, separate spec). Lock-safe: trim_wal_segments runs with p->lock released; repl_trim_lsn/repl_expire_disconnect_slots take only r->lock — no inversion. Tests: test_trim_guard_preserves_standby_segments, test_trim_without_standbys_proceeds_normally (test_e2e_checkpoint.c).
  • Part A (DORMANT, g_drive_async_snapshot_offload DEFAULT 0/OFF): ha_snapshot_send_dispatch routes HA_REPL_SEND_SNAPSHOT inline (flag OFF — byte-for-byte legacy, send on io_worker) or to slowlane_submit+snapshot_lane_fn (flag ON). Send body factored into ha_snapshot_do_send (shared by both paths, holds the XCE_TESTING hook). Per-peer single-flight via ha_state_t.snapshot_inflight[MAX_HA_CANDIDATES] CAS + slowlane key peer_idx+1; flag cleared on all 4 terminal paths (alloc-fail, -EEXIST, hard-error, lane-fn completion) exactly like reconnect (2.1). Tests: test_snapshot_offload_runs_on_lane, test_snapshot_inline_when_flag_off, test_snapshot_offload_dedups_per_peer (test_ha_liveness.c).
  • WHY IT WAS DORMANT, AND HOW IT CLOSED: snapshot_lane_fn captures a BORROWED standby endpoint (unlike reconnect, which owns its ep). With the flag ON the send runs on a lane thread while the io_worker may RDHUP-endpoint_fini that ep → cross-thread UAF (bug-1197 family). Flag OFF kept send+teardown serialized on the io_worker. Caught + fixed pre-commit early on: the offload flag was initially authored DEFAULT-ON (would have shipped the UAF) → corrected to OFF in both ha.c and the ha.h accessor doc (bug-1210). Closed 2026-07-02: ha_snapshot_send_dispatch/snapshot_lane_fn now bracket the lane dispatch with endpoint_ref/endpoint_unref (reusing the general bug-1298 refcount system), so the borrowed-endpoint hazard cannot recur regardless of thread timing — no lane-serialized-teardown redesign needed. g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch both now default ON together (bug-1376, a distinct cold-start self-deadlock unmasked by flipping both, is also fixed).

• 3.3 (LANDED — checkbox [x], flag DEFAULT ON): requester-less config-push transport offload, flag g_drive_async_configpush_offload DEFAULT 0/OFF (toggle ha_set_configpush_offload_for_test). ha_config_push_offloadable(c, bytes, blen, sha256, new_tag) routes inline (flag OFF — verbatim push_config_transport_only, caller still owns+frees bytes) or to slowlane_submit+configpush_lane_fn (flag ON — DEEP-COPIES bytes+sha into a ctx the lane frees). Single cluster-wide single-flight via scalar ha_state_t.configpush_inflight CAS + lane key=1 (config push is a cluster-wide fan-out, not per-peer); flag cleared on all terminal paths (alloc-fail×2, -EEXIST, hard-error, lane completion). Send body factored into ha_config_push_do_send (production → push_config_transport_only; XCE_TESTING → ha_configpush_probe stub since cluster.c/transport isn't linked in test). THREE requester-less drive-thread call sites converted: raft_cb_become_leader (promotion, both bytes/NULL branches), the HA_REPL_PUSH_CONFIG heartbeat case, and the direct-CTRL_PRIMARY FSM promotion branch (~ha.c:4476). UNCHANGED: push_config_to_standbys admin-upgrade barrier (has a waiting requester) and join.c. Tests: test_configpush_offload_runs_on_lane, test_configpush_inline_when_flag_off, test_configpush_offload_dedups (test_ha_liveness.c). bug-1381 CLOSED 2026-07-02 (see task 3.4a's note below for the fix) — push_config_transport_only itself (not the lane fn) no longer races raw standby endpoint pointers. cpp-review: no CRITICAL/HIGH/MEDIUM; controller suite EXIT=0. Flip attempted same day, REVERTED: with bug-1381 closed, flipped g_drive_async_configpush_offload to default ON, rebuilt xce.build (confirmed binary mtime > source mtime — see the Do-Not-Repeat cerebrum entry this session added after almost repeating the earlier stale-binary mistake), then ran --chaotic 6 (PASS, 21 jobs, zero divergence/loss) and --force-snapshot x3 (0/3 PASS — "no snapshot install" / "did not reconverge to 3-of-3"). A flag-OFF control on the identical rebuilt binary went 1/2 on the same two failure signatures, proving the defect is PRE-EXISTING and NOT caused by this flag: ha_send_snapshot_to_standby (config_repl.c) never checks xce_snapshot_chunk_req_raw's return in its chunk-send loop, so a chunk that fails to enqueue (e.g. -ECONNRESET during the empty-rejoin reconnect window) is silently dropped while the function still logs "transfer complete" and returns 0 — logged as bug-1386 (open). Flag-ON's 0/3 vs flag-OFF's 1/2 is too small a sample to rule out a secondary timing interaction (configpush lane traffic narrowing the snapshot lane's completion window before reconnect), so the flag was reverted to OFF rather than accepted on a "probably fine" basis. Full controller suite re-confirmed green (230/230) at OFF. bug-1386 FIXED 2026-07-02: ha_send_snapshot_to_standby (config_repl.c) now captures the return value of xce_snapshot_chunk_req_raw/xce_snapshot_chunk_hook in its chunk loop, breaks on a non-zero send_rc, and propagates it as rc instead of silently continuing and reporting "complete"; the success log is gated on rc==0, a new WARN logs the failure (checkpoint_lsn/size/sent-offset/chunk_index/rc) otherwise. No receive-side timeout/re-request was added — unnecessary: snapshot_inflight[peer_idx] already clears on every send-thread exit (snapshot_lane_fn), so the next heartbeat's SEND_SNAPSHOT decision naturally retries a peer whose applied_lsn never advanced. New RED->GREEN test test_snapshot_transfer_stops_and_propagates_send_failure (test_config_repl.c) injects a mid-loop -ECONNRESET via the existing xce_snapshot_chunk_hook seam; RED confirmed via temporary revert ("Expected -104 Was 0"), GREEN restored, full controller suite clean (0 failures). Re-soak to verify bug-1386's fix (2026-07-02, same session) surfaced a THIRD distinct pre-existing bug, STILL BLOCKING: ran --force-snapshot (flag still OFF, just verifying bug-1386 in isolation before re-attempting the flip) — FAIL again, but a NEW signature: ctrl-1 (sender) now correctly logs "transfer complete" for the full 735546-byte/12-chunk transfer both times (confirming bug-1386's fix works — no more false-complete), but ctrl-2 (receiver) fails with ha_handle_snapshot_chunk: persist_load_snapshot failed rc=-5 after receiving all 12 chunks. Byte-comparing the sender's source checkpoint against the receiver's assembled file (two independent repro runs, checkpoint LSNs 0x75 and 0x76) showed IDENTICAL total size but DIFFERENT CONTENT — the received file diverges from the source at an exact SNAPSHOT_CHUNK_SIZE (65536-byte) chunk boundary, then contains the correct data byte-SHIFTED (same job-frame strings, same occurrence count, wrong offset), re-synchronizing later. Shift direction/magnitude differed between the two attempts (-912 bytes / +4684 bytes) — nondeterministic, not fixed-offset. This is a chunk-content-CORRUPTION bug, not a drop (bug-1386) — logged as bug-1389 (open, NOT fixed). Prime suspect: xce_snapshot_chunk_encode_raw's capnp Data encode (request.c:630-671, capn_new_list8/capn_setv8 pattern) — same family as bug-847 (WAL catch-up streaming payload corruption at a capnp segment boundary). Receive-side persist_load_snapshot is confirmed NOT the bug — it correctly rejects the corrupted file. bug-1389 ROOT-CAUSED AND FIXED 2026-07-02: added live per-chunk crc32 TRACE instrumentation at both ha_send_snapshot_to_standby (send) and ha_handle_snapshot_chunk (receive), correlated by (snapshot_lsn, chunk_index), then re-ran --force-snapshot — the crc32 log proved every one of 12 chunks' CONTENT crossed the wire byte-for-byte intact (all 12 crc32s matched exactly), so capnp/transport was NEVER the bug (the earlier "capnp encode suspected" hypothesis, and an exhaustive prior trace of the wire framing and far-pointer resolution, are both correctly refuted). The real defect: the RECEIVE log showed chunks arriving out of chunk_index order — 0,1,3,4,2,5,6,7,8,9,10,11 — because the vtask pool dispatches one transfer's chunks across multiple workers (cpu#0-3 in the log) with no ordering guarantee, but ha_handle_snapshot_chunk (ha.c) appended every chunk's bytes to snap.part in ARRIVAL order (fopen("ab")/fwrite) instead of at the chunk's own chunk_offset (a field already carried on the wire but never used). Reordered chunks landed at the wrong file position — same total byte count, wrong content from the first reordered chunk onward, exactly the "shifted, resyncing later, nondeterministic magnitude" signature originally observed. Fix: (1) ha_handle_snapshot_chunk now writes each chunk via pwrite(fd, payload, len, chunk->chunk_offset) — order-independent and safe for concurrent writers (positioned writes to disjoint byte ranges of the same fd don't race). (2) New ha_state_t.snapshot_recv_count/snapshot_recv_total fields (ha.h, guarded by the existing ha->lock) track how many distinct chunks have actually been written; finalize (verify/rename/persist_load_snapshot) now fires only once count >= total (known once is_last's chunk_index+1 is seen), not merely because THIS call happens to carry is_last — under reordering, is_last can process before an earlier chunk. snapshot_recv_total is claimed (reset to 0) by whichever thread crosses the completion threshold, preventing a late straggler from double-finalizing. chunk_index==0 still resets both the stale-file unlink and the new counters (Req 4.4 unchanged). New RED->GREEN test test_snapshot_recv_out_of_order_chunks_assemble_correctly (test_config_repl.c) replays the exact non-sequential delivery order and asserts byte-identical assembly; RED confirmed via temporary revert (Expected 0 Was -22, job_count field corrupted by the misordered chunk boundary), GREEN restored, full controller suite clean. Also added 2 new full-size (65536-byte) capnp encode/decode round-trip tests (test_wal_wire.c) — permanent regression coverage that definitively rules out the capnp layer (the prior test only covered 300 bytes). Live-validated: 3/3 --force-snapshot rounds PASS (genuine HA: snapshot loaded ... full job/queue entity restoration installs confirmed each time, zero divergence/loss) + 6-round --chaotic PASS. Flag re-flipped to ON 2026-07-02, LANDED: with bug-1389 closed, g_drive_async_configpush_offload flipped to default 1/ON, full build clean (all components, mtime-verified), full controller suite 0 failures. Soak-validated on the flag-ON binary: --chaotic 6 PASS (21 jobs, zero divergence/loss); --force-snapshot 3/3 genuine PASS (each confirmed a real HA: snapshot loaded install, zero divergence/loss) — one intervening run hit the unrelated pre-existing job-submission flakiness ("could not submit 110 jobs to trigger a checkpoint", never reached the snapshot-transfer phase at all) and was retried clean. Task 3.3 is now fully landed — all three requester-less config-push call sites route through the slow lane by default.

• 3.4a (bug-1197 endpoint-quiescence fix — committed; flags STILL default OFF): makes the flag-ON peer-control dispatch path UAF-free so raft_cb_become_follower (running as a pool vtask) can free peer vote-eps while other pool workers process peer-control on the same eps. Mechanism (no libxce-wide endpoint refcount; reuses the existing per-ep FIFO + busy serialization):

  • Deferred teardown-as-vtask: the loser loop, when ha_peerctrl_dispatch_enabled() && ha->pool, does per vote-ep endpoint_mark_tearing_down(ep) → iomgr_unmonitor(ep) → enqueue a VTASK_IO/VTASK_LIVENESS teardown vtask bound to that ep (vote_ep_teardown_vtask_entry). Per-ep busy FIFO makes it run strictly AFTER any in-flight/queued same-ep peer-control vtask; it NULLs iod->ep before endpoint_fini+free(ep) to disarm run_task's post-body endpoint_unblock_tasks (which would otherwise touch the freed ep->lock). OFF / pool-unwired path is byte-for-byte the legacy endpoint_fini (no free).
  • RCU-like drain closes the TOCTOU window: a new _Atomic int endpoint_t.dispatch_inflight (+ tearing_down:1 bit). The ON dispatch brackets its flag-check+enqueue: fetch_add(seq_cst) → if tearing_down drop pkt + fetch_sub → else enqueue → fetch_sub. endpoint_mark_tearing_down sets the flag under ep->lock, releases, then spins while(load(dispatch_inflight,seq_cst)>0) sched_yield(). become_follower marks (drains) BEFORE enqueuing the teardown vtask, so any dispatch that read the flag false is FIFO-before teardown; any later dispatch drops. Could NOT hold ep->lock across the enqueue — it inverts with the taskq lock (dispatcher takes taskq→ep in vtask_eligible); the counter is the lock-free handshake. Drain is bounded (the held window is just calloc+non-blocking enqueue on the single io_worker).
  • Files: src/include/netio.h (tearing_down:1, dispatch_inflight, mark/is helpers), src/lib/netio.c (helpers + drain + do_endpoint_init zero-init), src/controller/ha.c (vote_ep_teardown_*, loser-loop async branch, dispatch bracket + late-drop, test seams), src/controller/ha.h. Tests: test_ha_vep_teardown.c (5 ASan tests: ordering, late-drop, disarm, OFF-path, deterministic drain-blocks). xcelib_test 282/0; controller EXIT=0; ASan-clean. Adversarial cpp-review: no CRITICAL/HIGH; handshake formally sound (SC total order on the counter + monotone tearing_down under ep->lock); flag-ON path proven race-free.
  • CLOSED for configpush_offload's prod flip: snapshot_lane_fn's borrowed-ep hazard closed 2026-07-02 (endpoint_ref/unref, see task 3.2's note). configpush_lane_fn's hazard was a DIFFERENT race — logged as bug-1381: push_config_transport_only (not configpush_lane_fn itself) grabbed raw standby endpoint pointers and spawned unprotected pthread_create workers for the fan-out, a race that existed independent of the offload flag. Fixed 2026-07-02: each standby ep is now endpoint_ref()'d under c->ha.lock at the same point its pointer is snapshotted into push_ctx_t, with exactly one matching endpoint_unref() per ref — from push_thread after it finishes using ctx->ep, or from push_config_transport_only's own cleanup if a thread never started (pthread_create failure or a total threads[] alloc failure). A concurrent RDHUP teardown can now only drop the OWNER's reference while a push is in flight, never free ep out from under push_thread. New seam push_thread_send_hook (config_repl.c, same pattern as xce_wal_stream_request_hook) lets test_configpush_transport_holds_endpoint_ref_across_send (test_config_repl.c) park a send mid-flight and assert the refcount bracket; RED→GREEN verified via temporary revert (Expected 2 Was 1), full controller suite green (230/230) after restore. g_drive_async_configpush_offload remains default OFF — this closes the correctness blocker for the flip, but the flip itself (and its own soak validation) is a separate follow-up, not part of this fix.