fix(controller): offload checkpoint_task_fn off the delaymgr thread (bug-1694)
Last of the 9 confirmed findings from the /xce-audit-invariants controller pass. checkpoint_task_fn was registered directly as a delaymgr entry and ran its full blocking body inline: checkpoint marker pwrite+fdatasync, write_snapshot()'s genuinely unbounded per-entity walk (scales with cluster state size) held under apply_pump_lock, WAL trim, and snapshot rotation/unlink -- all on the same thread that drives heartbeats/election timeouts. Unlike every other delaymgr operation (snapshot send, config-push, reconnect, catch-up), checkpoint had no offload mechanism at all -- the one gap left from the raft-drive-async hardening pass. Design: full offload via the SlowLane (slowlane_submit), default ON -- NOT the vtask pool. persist_apply_pump_run() already holds apply_pump_lock across its own per-entry walk while running ON the vtask pool (VTASK_LIVE_BLOCK); putting checkpoint's unbounded, lock-held write_snapshot() on that same pool risked starving it, exactly what ha-async-io.md's own thread-role table warns against. The SlowLane is the actual established pattern for this I/O class (push_config_transport_only, the snapshot-send path, catchup_lane_fn, demotion_lane_fn all dispatch there) -- and persist_checkpoint() already runs on the slow lane in production TODAY via ha_snapshot_do_send() -> snapshot_lane_fn, so apply_pump_lock's acquisition from a slow-lane thread is proven-safe, exercised daily, not new surface. Unlike catchup/demotion (which stay flag-gated OFF for an unrelated, still-not-root-caused hang, bug-1429), checkpoint's new flag defaults ON given this reuses an already-proven code path. Inline-fallback + WARN only on a genuine slowlane_submit failure (not -EAGAIN/-EEXIST, both non-failures) -- never silently drops a scheduled checkpoint. New test_persist_checkpoint_offload.c (3 tests). RED-phase forced the pre-fix always-inline path: the suite never printed a single result and had to be killed, proving the test genuinely detects the violation, not a tautology. GREEN: all 3 pass. Full suite: 1184 tests, 2 failures -- both the pre-existing, unrelated bug-1561 overlayfs sandbox flake (exact signature match). No live HA soak was run for this specific fix in isolation (explicitly scoped out, not hidden) -- the mandatory live-soak proof for this whole audit campaign covers it next. This closes the last of the 9 confirmed findings from the controller audit (bug-1684, bug-1687/1689/1690/1691/1692, bug-1693, bug-1694). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
parent
d1803376d8
commit
b4eb9ec821
6 changed files with 561 additions and 8 deletions
|
|
@ -180,6 +180,7 @@ controller_unit_test_src = [
|
|||
'testsuite/test_apply_correctness_verification.c',
|
||||
'testsuite/test_apply_pump.c',
|
||||
'testsuite/test_apply_pump_integration.c',
|
||||
'testsuite/test_persist_checkpoint_offload.c',
|
||||
'testsuite/test_job_array_apply.c',
|
||||
'testsuite/test_job_array_fence.c',
|
||||
'testsuite/test_job_array_replay_fence.c',
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue