myxce/.kiro/specs/raft-drive-async/tasks.md

Implementation Plan

Tasks mirror the design's 4-phase migration. The slow lane is foundational (every offload submits to it); the peer_ctrl→vtask move is coupled with its long-op offload in Phase 3 so no phase runs long I/O inline in a blocked vtask.

• 1. Phase 1 — SlowLane executor and bounded connect (foundation)

• 1.1 Add drive-async configuration constants

  • Define connect-timeout, slow-lane worker count, and max-in-flight constants with documented defaults (sized to peer count).
  • Define the liveness reservation as a peer-scaled derivation, not a flat literal: VTASK_LIVENESS_RESERVED = clamp(n_peers − 1, 1, n_vcpus − 1) (floor 1 = deadlock-freedom; ceiling n_vcpus − 1 keeps ≥1 commit slot); WARN at init when clamped by n_vcpus.
  • Wire the constants into the controller config so they are overridable.
  • Observable: a build with the new constants compiles; defaults are readable at controller startup; the resolved reservation logged at init.
  • Requirements: 2.3, 3.1

• 1.2 Implement the SlowLane executor

  • Bounded pool of dedicated worker threads, isolated from the vtask request pool; non-blocking submit that enqueues and returns immediately.
  • Admission cap on concurrent in-flight tasks; per-key in-flight dedup (key 0 disables dedup); init/fini wired into the controller lifecycle.
  • Observable: a submitted task runs exactly once on a lane worker; submit never blocks the caller; concurrent running tasks never exceed the cap; a submit with an in-flight key is a no-op.
  • Requirements: 3.1, 3.2, 3.3, 3.4, 1.5
  • Boundary: SlowLane

• 1.3 Bound the peer connect path

  • Reorder socket_connect to set non-blocking before connect and bound the attempt with a poll timeout (reuse the existing probe pattern); return success/timeout/error.
  • Update connect callers to pass the configured timeout.
  • Observable: a connect to an unreachable host returns within the configured timeout instead of the OS connect timeout; a connect to a reachable host still succeeds.
  • Requirements: 2.3
  • Boundary: BoundedConnect
  • Depends: 1.1

• 1.4 (P) SlowLane unit tests

  • Assert non-blocking submit, admission-cap bounds on concurrency, per-key dedup returns the deduped result, and run-exactly-once.
  • Observable: tests pass and fail if the cap or dedup is removed.
  • Requirements: 3.1, 3.2, 3.3, 3.4, 1.5
  • Boundary: SlowLane
  • Depends: 1.2

• 1.5 (P) Bounded-connect unit test

  • Assert socket_connect returns within the timeout against an unreachable host and succeeds against a reachable one.
  • Observable: test passes; reverting the reorder makes it hang/fail.
  • Requirements: 2.3
  • Boundary: BoundedConnect
  • Depends: 1.3

• 1.6 LivenessAdmission — pool-level task-class reservation in vtaskmgr

  • Add vtask_class_t (VTASK_LIVENESS / VTASK_LIVE_BLOCK / VTASK_REQ_BLOCK) + a classified submit; the dispatcher reserves the VTASK_LIVENESS_RESERVED value resolved in 1.1 (single source of truth — consume it, do not recompute the clamp/WARN here) and fast-rejects a VTASK_REQ_BLOCK with -XCE_ERR_NOT_READY when admitting it would breach the reservation (never parks a worker, never silently queues).
  • Fold commit.c's inflight_waits/COMMIT_WAIT_RESERVED into this — commit_submit submits a VTASK_REQ_BLOCK; the pool owns the single reservation.
  • Dormant infra in Phase 1: no behavior change until peer_ctrl dispatches through it (3.1). Hard prerequisite of 3.1 — without it, moving ack/vote handling onto the pool is not deadlock-safe (vote-fsync/config-ack waits, invisible to the old commit-only counter, could starve the ack that wakes commit-waiters).
  • Observable: a VTASK_LIVENESS task always finds a worker even when *_BLOCK tasks saturate n_vcpus − VTASK_LIVENESS_RESERVED; commit-wait still fast-rejects -XCE_ERR_NOT_READY at the cap (backpressure unchanged).
  • Requirements: 3.1, 3.2, 3.3, 4.5
  • Boundary: LivenessAdmission
  • Depends: 1.1

• 1.7 (P) LivenessAdmission unit test

  • Assert: a VTASK_LIVENESS task is dispatched while *_BLOCK tasks hold n_vcpus − VTASK_LIVENESS_RESERVED vcpus; VTASK_REQ_BLOCK over the cap returns -XCE_ERR_NOT_READY without parking; folding commit-wait through the new path preserves the existing reject-at-cap behavior. Reverting the reservation makes a liveness task starve behind blockers.
  • Requirements: 3.1, 3.2, 3.3, 4.5
  • Boundary: LivenessAdmission
  • Depends: 1.6

• 2. Phase 2 — Reconnect offload (sender-side liveness)

• 2.1 Offload the delaymgr peer reconnect to the slow lane

  • The heartbeat timer detects a disconnected peer and submits the reconnect to the lane (per-peer dedup key), then returns immediately; the timer never performs the connect inline.
  • Refcount the endpoint across the in-flight window.
  • Observable: while one peer is unreachable, the delaymgr timer keeps firing and heartbeats to reachable peers continue within the interval; only one reconnect attempt per peer is in flight at a time.
  • Requirements: 2.1, 2.2, 2.5
  • Boundary: ReconnectOffload, SlowLane
  • Depends: 1.2, 1.3

• 2.2 Complete the reconnect on the lane worker

  • The lane worker performs the bounded connect, fully prepares the endpoint, and registers it via the lock-guarded monitor interface; on failure it clears the in-flight flag and lets the next tick retry without terminating; resolve the inbound-vs-outbound connection race per the connection-ownership rules.
  • Observable: a successful reconnect registers the new connection with no lost or reordered queued messages; a failed attempt leaves the controller alive and retrying.
  • Requirements: 2.4, 2.6, 4.4
  • Boundary: ReconnectOffload
  • Depends: 2.1

• 2.3 Reconnect-offload integration test

  • With one peer made unreachable, assert heartbeats to the surviving peers continue within the interval, the delaymgr is not stalled, and reconnect is deduped per peer.
  • Observable: test passes; reverting the offload (inline connect) makes heartbeats to survivors stall.
  • Requirements: 2.1, 2.2, 2.5
  • Boundary: ReconnectOffload
  • Depends: 2.2

• 3. Phase 3 — Peer-control dispatch to vtask, coupled with long-op offload (receiver-side)

• 3.1 Dispatch peer-control message handling to the vtask pool

  • Register peer endpoints so peer-control messages dispatch through the vtask pool instead of running inline on iomgr; convert the peer-control handler into a vtask entry that does only bounded protocol work inline.
  • Preserve per-connection processing order via the per-endpoint block primitive and preserve vote/ack/term semantics and persist-before-vote ordering; keep the steady-state heartbeat/vote send paths as non-blocking enqueues.
  • Observable: peer-control no longer runs on the iomgr loop; existing election/heartbeat/repl behavior is unchanged; messages on one connection are processed in arrival order.
  • Deadlock-freedom gate (bug-invariant reversal): moving ack decode to the pool co-locates the commit-barrier wakers (acks) and waiters (commit_submit(COMMIT_MODE_WAIT)) on one pool — today acks run on iomgr precisely to avoid this (commit.c:16). The old commit-only COMMIT_WAIT_RESERVED is insufficient (it can't see vote-fsync/config-ack blockers this move also puts on the pool), so safety rests on LivenessAdmission (task 1.6): ack/vote-receipt dispatched as VTASK_LIVENESS (never gated, never offloaded), commit_submit as VTASK_REQ_BLOCK under the pool reservation. Task 3.4 is the admission gate for this task — 3.1 is not complete until 3.4 is green.
  • Requirements: 1.4, 4.1, 4.2, 4.5, 4.6, 5.1, 5.4, 8.1, 8.2
  • Boundary: PeerCtrlDispatch
  • Depends: 1.2, 1.6

• 3.2 Offload snapshot send to the slow lane

  • Trigger snapshot send + checkpoint to the lane from the vtask entry without holding the per-endpoint block across the transfer; dedup one snapshot per standby.
  • Enforce the snapshot/checkpoint concurrency invariant: snapshot reads a frozen term/lsn boundary; segment trim never deletes at/above any in-flight snapshot boundary or a standby catch-up cursor; at most one checkpoint in flight; bound the in-lane snapshot socket write so an unreachable-mid-transfer peer cannot pin a worker.
  • Add the trim cursor-bound guard (bug-1147, stopgap): trim_wal_segments (persist.c:1374) currently bounds deletion ONLY by the open segment and consults no standby cursor; before this task moves checkpoint/trim onto a concurrent lane worker, change it to compute floor = min(all standby acked_lsn, in-flight snapshot boundaries) under p->lock and refuse to delete at/above floor. This is strategy-A retention discipline; it does NOT fix empty/far-behind join (cursor=0) — that needs entity-bearing snapshots (separate spec).
  • Observable: a catch-up that triggers a snapshot does not block iomgr; the triggering peer's later messages are not queued behind the transfer; concurrent appends plus checkpoint/trim never delete a segment at/above any live standby acked_lsn or in-flight snapshot boundary (reverting the guard makes the test re-create a bug-795/828 gap).
  • Requirements: 1.1, 1.3, 1.5, 4.3
  • Boundary: PeerCtrlDispatch, SlowLane
  • Depends: 3.1

• 3.3 (P) Offload the transport-only config-push to the slow lane

  • Move the requester-less iomgr/promotion config-push transport (and its wait) to the lane; replace the drive-thread join with a lane-internal wait and non-blocking return; leave the admin upgrade path (with a waiting requester) on its existing vtask-blocking path unchanged.
  • Observable: a promotion/heartbeat-driven config push no longer blocks iomgr; the admin upgrade reply path is unchanged.
  • Requirements: 1.2
  • Boundary: PeerCtrlDispatch, SlowLane
  • Depends: 3.1

• 3.4 Receiver-side integration tests

  • Assert that during a snapshot/config-push transfer the controller still processes other peers' heartbeats and votes; that a quorum of WAL-acks advances the commit position promptly even with the vtask pool near saturation; and that cross-endpoint acks are not serialized behind one another.
  • Observable: tests pass; commit advances after a quorum of acks under load; iomgr stays responsive during transfers.
  • Requirements: 1.1, 1.3, 1.4, 4.3, 4.5
  • Boundary: PeerCtrlDispatch
  • Depends: 3.2, 3.3

• 4. Phase 4 — Verification, partition test, and regression

• 4.1 Network-partition (unreachable-peer) soak scenario

  • Add a partition primitive that renders one node unreachable (distinct from process kill). Primary = in-process socket-drop hook in xce_test (XCE_TESTING) — blackholes a node's peer traffic at the endpoint layer with NO privilege, so R7 actually runs in the unprivileged/overlayfs sandbox (avoids the perpetual-skip fate of ha-term-safety 5.2/5.4). Secondary (optional) = firewall iptables/nft DROP for real-network fidelity, capability-gated with a loud skip when unprivileged.
  • Assert: heartbeats to survivors continue, no spurious election while the leader holds the majority, the peer rejoins on heal, and no committed job is lost.
  • Observable: the partition soak runs and passes; the same scenario fails against the pre-change build (heartbeat stall / spurious election).
  • Requirements: 7.1, 7.2, 7.3, 7.4
  • Boundary: PartitionTest
  • Depends: 2.3, 3.4

• 4.2 Existing-feature regression sweep

  • Run the existing HA suites unchanged — election, prevote, liveness, failover, rdhup-integration, repl majority/apply/catchup, config-repl/repl-push, persist-recovery, commit-barrier/commit-durability — and the kill-based failover soak.
  • Observable: all listed suites pass with no change to their asserted behavior.
  • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7
  • Depends: 3.4

• 4.3 Thread-role compliance and heartbeat performance check

  • Verify no blocking operation runs inline on the iomgr loop or delaymgr timer for the in-scope paths; bounded blocking runs in the vtask pool only under the admission reservation and long blocking I/O only on the lane.
  • Measure steady-state heartbeat cadence at 3–5 controllers and confirm no added latency attributable to the new dispatch path.
  • Observable: a compliance assertion/inspection shows the drive threads non-blocking; heartbeat cadence matches the pre-change baseline.
  • Requirements: 5.1, 5.2, 5.3, 5.4, 8.1, 8.2
  • Depends: 2.3, 3.4

Implementation Notes

• Build/test env: builds MUST run in the nix dev shell (nix develop --command meson compile -C src/xce.build <target>). The IDE clang reports false errors (missing openssl/unity headers, unknown -fvar-tracking) because it isn't in the nix shell — ignore those. Controller tests: rm -rf /tmp/xce_* first (bug-924 overlayfs), run ./src/xce.build/controller/controller_unit_test, set ASAN_OPTIONS=detect_leaks=0.
• Pre-existing leak (not ours): the default-options full controller suite exits non-zero due to a LeakSanitizer leak in test_commit_barrier_reply.c / test_worker_join.c / test_egress_filter.c (~174KB). Unrelated to this spec. Run with detect_leaks=0 (or a -t filter) so it doesn't mask raft-drive-async regressions.
• 1.1: drive_async_liveness_reserved(n_peers, n_vcpus) = max(1, min(n_peers−1, n_vcpus−1)) in ha.c is the single source of truth for VTASK_LIVENESS_RESERVED (floor applied LAST). Consumed by 1.6. myconf_t CLI override fields for the new DEFAULT_* constants were deferred to the consumer tasks (1.2 connect-timeout, 1.3/1.6) per YAGNI — add each field with its reader.