fix(controller): offload checkpoint_task_fn off the delaymgr thread (bug-1694)

Last of the 9 confirmed findings from the /xce-audit-invariants controller
pass. checkpoint_task_fn was registered directly as a delaymgr entry and
ran its full blocking body inline: checkpoint marker pwrite+fdatasync,
write_snapshot()'s genuinely unbounded per-entity walk (scales with
cluster state size) held under apply_pump_lock, WAL trim, and snapshot
rotation/unlink -- all on the same thread that drives heartbeats/election
timeouts. Unlike every other delaymgr operation (snapshot send,
config-push, reconnect, catch-up), checkpoint had no offload mechanism
at all -- the one gap left from the raft-drive-async hardening pass.

Design: full offload via the SlowLane (slowlane_submit), default ON --
NOT the vtask pool. persist_apply_pump_run() already holds apply_pump_lock
across its own per-entry walk while running ON the vtask pool
(VTASK_LIVE_BLOCK); putting checkpoint's unbounded, lock-held
write_snapshot() on that same pool risked starving it, exactly what
ha-async-io.md's own thread-role table warns against. The SlowLane is
the actual established pattern for this I/O class (push_config_transport_only,
the snapshot-send path, catchup_lane_fn, demotion_lane_fn all dispatch
there) -- and persist_checkpoint() already runs on the slow lane in
production TODAY via ha_snapshot_do_send() -> snapshot_lane_fn, so
apply_pump_lock's acquisition from a slow-lane thread is proven-safe,
exercised daily, not new surface. Unlike catchup/demotion (which stay
flag-gated OFF for an unrelated, still-not-root-caused hang, bug-1429),
checkpoint's new flag defaults ON given this reuses an already-proven
code path. Inline-fallback + WARN only on a genuine slowlane_submit
failure (not -EAGAIN/-EEXIST, both non-failures) -- never silently drops
a scheduled checkpoint.

New test_persist_checkpoint_offload.c (3 tests). RED-phase forced the
pre-fix always-inline path: the suite never printed a single result and
had to be killed, proving the test genuinely detects the violation, not
a tautology. GREEN: all 3 pass. Full suite: 1184 tests, 2 failures --
both the pre-existing, unrelated bug-1561 overlayfs sandbox flake
(exact signature match). No live HA soak was run for this specific fix
in isolation (explicitly scoped out, not hidden) -- the mandatory
live-soak proof for this whole audit campaign covers it next.

This closes the last of the 9 confirmed findings from the controller
audit (bug-1684, bug-1687/1689/1690/1691/1692, bug-1693, bug-1694).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg

src/include/persist.h

@@ -1058,11 +1058,41 @@ int persist_test_force_new_segment(persist_t* p);
  * comment above). Pass a negative index to disable. Lock-free atomic
  * write; compiled out of production. */
 void persist_test_set_force_job_encode_fail_at(persist_t* p, int index);
+
+/* Test seam: directly drive the delaymgr-triggered checkpoint task
+ * (checkpoint_task_fn, persist.c) without waiting out delay_task's real 1s
+ * delay. Returns the task fn's own return value (always 0). Compiled out of
+ * production — the only real call site is via delay_task. */
+int persist_checkpoint_task_fn_for_test(persist_t* p);
+
+/* Test-injectable substitute for the real (blocking) checkpoint work; when
+ * non-NULL, checkpoint_lane_fn (persist.c) calls this INSTEAD OF
+ * persist_checkpoint() — lets a test park/gate the slow-lane worker in
+ * flight. NULL in every fixture that never installs it. Symbol always
+ * compiled (like ha.c's ha_repl_send_snapshot_hook) so test TUs can extern
+ * it; unused in a non-XCE_TESTING build. */
+extern void (*persist_checkpoint_test_hook)(persist_t*);
 #endif /* XCE_TESTING */
 
 /* Checkpoint */
 int persist_checkpoint(persist_t* p);
 
+/*
+ * persist_set_checkpoint_offload_for_test / persist_checkpoint_offload_enabled:
+ * Feature-flag accessors (xce-audit-invariants controller pass, dimension 1
+ * async-I/O thread-role fix, bug-1694). The offload flag gates whether
+ * checkpoint_task_fn (persist.c) submits the real checkpoint work to the
+ * cluster's slow lane (ON, the production default — the delaymgr thread
+ * never blocks on checkpoint I/O) or runs the legacy inline blocking
+ * checkpoint on the delaymgr thread (OFF, retained only so the TDD RED test
+ * can demonstrate the pre-fix stall). See checkpoint_task_fn's own doc
+ * comment (persist.c) for the full design rationale, including why the
+ * destination is the slow lane and not the vtaskmgr pool. These accessors
+ * are compiled in ALL builds; the setter is used by tests to flip the flag.
+ */
+void persist_set_checkpoint_offload_for_test(int on);
+int  persist_checkpoint_offload_enabled(void);
+
 /* WAL truncation — remove all entries with LSN > cutoff_lsn.
  * Idempotent; returns 0 on success, negative errno on I/O error. */
 int persist_truncate_wal_above(persist_t* p, uint64_t cutoff_lsn);