fix(raft-drive-async): close snapshot-lane borrowed-endpoint hazard + catch-up checkpoint-marker bug

Both fixes are real and independently verified; the drive-async flags
(g_drive_async_snapshot_offload, g_drive_async_peerctrl_dispatch_enabled)
stay at their production-default OFF -- see the tasks.md session note for
why the flip itself doesn't land in this commit.

1. Snapshot-lane borrowed-endpoint hazard (task 3.2, bug-1197 family):
   ha_snapshot_send_dispatch now takes endpoint_ref(ep) before submitting to
   the slow lane, released on every non-lane terminal path and by
   snapshot_lane_fn on completion. Reuses the general bug-1298 endpoint
   refcount system (standby-ep teardown already routes through
   endpoint_unref via cluster.c's endpoint_killer) -- a concurrent RDHUP
   teardown can now only drop the owner's reference while a transfer is in
   flight, never free the ep out from under it. New test:
   test_snapshot_offload_holds_endpoint_ref_across_transfer (RED->GREEN).

2. Catch-up streamer WAL_CHECKPOINT bug (found while verifying #1 via a live
   soak, pre-existing, unrelated to the drive-async flags): ha_stream_wal_
   to_standby had no filter for WAL_CHECKPOINT markers -- local-only
   bookkeeping that still consumes a real LSN slot via persist_write_
   checkpoint_marker's own current_lsn++. It shipped one to a catching-up
   standby, which correctly refused it (persist_wal_write_replicated
   rejects type==WAL_CHECKPOINT) and got stuck retrying the same LSN
   forever. Fixed by skipping WAL_CHECKPOINT entirely in the streamer loop,
   mirroring the existing replay_wal/term-index precedent of treating
   checkpoint-marker LSNs as invisible gaps. New test:
   test_wal_stream_skips_checkpoint_marker (RED->GREEN).

Attempted flipping both drive-async flags ON after landing these fixes (the
"full fix, defer prod flip" decision's prerequisite -- 4.1+4.2 are green).
A 6-round --chaotic soak passed clean, but a --force-snapshot soak (a
genuinely empty node cold-start rejoin, the one scenario chaotic mode never
exercises) surfaced a THIRD, not-yet-root-caused issue: the freshly-wiped
node's own self-election appears to race cluster_init_phase2's vtask-pool
wiring, then the process hangs silently (no crash, no ASAN report) until
killed. Reverted both flags to OFF rather than ship an unproven state --
production behavior is unchanged by this commit. Full controller suite
green and both soaks (--force-snapshot, 6-round --chaotic) pass clean with
flags OFF, confirming no regression from the two logic fixes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014dZVSx3CdBjys9BB6hqpxP

.kiro/specs/raft-drive-async/tasks.md

@@ -86,6 +86,7 @@ Tasks mirror the design's 4-phase migration. The slow lane is foundational (ever
   - _Requirements: 1.4, 4.1, 4.2, 4.5, 4.6, 5.1, 5.4, 8.1, 8.2_
   - _Boundary: PeerCtrlDispatch_
   - _Depends: 1.2, 1.6_
+  - _2026-07-01 session (production flip attempt, both gate items closed but a NEW blocker found — flags stay OFF, task NOT complete): with 4.1+4.2 green, attempted the "full fix, defer prod flip" decision's flip. (1) Closed the snapshot-lane borrowed-endpoint hazard (task 3.2's Part A) properly: `ha_snapshot_send_dispatch` takes `endpoint_ref(ep)` before `slowlane_submit`, released on every non-lane terminal path and by `snapshot_lane_fn` on completion — reuses the general bug-1298 refcount system (standby-ep teardown already routes through `endpoint_unref` via `cluster.c`'s `endpoint_killer`, confirmed by reading it — this infra landed in raft-ha-core-safety task 5.8, AFTER this spec's 3.2/3.4 notes were written, so the hazard they describe is now closed for real). New unit test `test_snapshot_offload_holds_endpoint_ref_across_transfer` (RED→GREEN). (2) Flipped `g_drive_async_snapshot_offload` and `g_drive_async_peerctrl_dispatch_enabled` to default ON; full controller suite green; a 6-round `--chaotic` soak passed clean (zero divergence, zero job loss). (3) A `--force-snapshot` soak (genuinely empty node cold-start rejoin — the ONE scenario chaotic mode never exercises, since it restarts nodes with WAL intact) then surfaced a SEPARATE pre-existing bug, unmasked (not caused) by the flip's timing shift: `ha_stream_wal_to_standby` had no filter for `WAL_CHECKPOINT` markers (local-only, never a logical Raft entry — consumes a real LSN slot via `persist_write_checkpoint_marker`'s own `p->current_lsn++`) and shipped one to a catching-up standby, which correctly refused it and got stuck retrying the same LSN forever. FIXED (`config_repl.c`, skip `hdr.type == WAL_CHECKPOINT` entirely, mirroring the existing `replay_wal`/term-index precedent of treating checkpoint-marker LSNs as invisible gaps) with a RED→GREEN unit test (`test_wal_stream_skips_checkpoint_marker`, differential-verified: this specific soak scenario failed before the fix, passed after, with flags OFF as a control). (4) Re-ran `--force-snapshot` with BOTH flags ON and the checkpoint-skip fix in place — it STILL failed, this time with a THIRD, different, NOT-YET-ROOT-CAUSED issue: the freshly-wiped empty node's own self-election appears to race `cluster_init_phase2`'s vtask-pool wiring (log shows `peer_ctrl_dispatch_to_pool: vtask pool not wired — running inline` immediately followed by `cluster_init_phase2` wiring the pool while `raft_cb_become_follower`'s deferred-teardown vtasks were being enqueued), after which the process went completely silent (no crash, no ASAN report) until the harness killed it — a genuine hang, not yet diagnosed. **Given a second real bug found from this flag flip in a row, reverted both flags to OFF (production behavior unchanged) rather than ship an unproven state.** The two logic fixes (endpoint-ref bracket, checkpoint-marker filter) are real, independently-verified, and shipped regardless of flag state — they are correct and safe on their own. Verified final state (flags OFF): full controller suite green, `--force-snapshot` PASS, 6-round `--chaotic` PASS — no regression from the two logic fixes. **Task 3.1/3.2/3.3 remain open.** Next step for a future session: reproduce the cold-start pool-wiring race deterministically (likely a unit/integration test driving `cluster_init_phase2` concurrently with an inbound vote reply on a freshly-constructed `cluster_t`, both flags ON) before attempting the flip again._
 
 - [~] 3.2 Offload snapshot send to the slow lane  <!-- Part B (trim guard) LIVE; Part A (offload) DORMANT pending 3.4 flip — see Implementation Notes -->
   - Trigger snapshot send + checkpoint to the lane from the vtask entry without holding the per-endpoint block across the transfer; dedup one snapshot per standby.