Rongsong Shen b4eb9ec821 fix(controller): offload checkpoint_task_fn off the delaymgr thread (bug-1694) ... Last of the 9 confirmed findings from the /xce-audit-invariants controller pass. checkpoint_task_fn was registered directly as a delaymgr entry and ran its full blocking body inline: checkpoint marker pwrite+fdatasync, write_snapshot()'s genuinely unbounded per-entity walk (scales with cluster state size) held under apply_pump_lock, WAL trim, and snapshot rotation/unlink -- all on the same thread that drives heartbeats/election timeouts. Unlike every other delaymgr operation (snapshot send, config-push, reconnect, catch-up), checkpoint had no offload mechanism at all -- the one gap left from the raft-drive-async hardening pass. Design: full offload via the SlowLane (slowlane_submit), default ON -- NOT the vtask pool. persist_apply_pump_run() already holds apply_pump_lock across its own per-entry walk while running ON the vtask pool (VTASK_LIVE_BLOCK); putting checkpoint's unbounded, lock-held write_snapshot() on that same pool risked starving it, exactly what ha-async-io.md's own thread-role table warns against. The SlowLane is the actual established pattern for this I/O class (push_config_transport_only, the snapshot-send path, catchup_lane_fn, demotion_lane_fn all dispatch there) -- and persist_checkpoint() already runs on the slow lane in production TODAY via ha_snapshot_do_send() -> snapshot_lane_fn, so apply_pump_lock's acquisition from a slow-lane thread is proven-safe, exercised daily, not new surface. Unlike catchup/demotion (which stay flag-gated OFF for an unrelated, still-not-root-caused hang, bug-1429), checkpoint's new flag defaults ON given this reuses an already-proven code path. Inline-fallback + WARN only on a genuine slowlane_submit failure (not -EAGAIN/-EEXIST, both non-failures) -- never silently drops a scheduled checkpoint. New test_persist_checkpoint_offload.c (3 tests). RED-phase forced the pre-fix always-inline path: the suite never printed a single result and had to be killed, proving the test genuinely detects the violation, not a tautology. GREEN: all 3 pass. Full suite: 1184 tests, 2 failures -- both the pre-existing, unrelated bug-1561 overlayfs sandbox flake (exact signature match). No live HA soak was run for this specific fix in isolation (explicitly scoped out, not hidden) -- the mandatory live-soak proof for this whole audit campaign covers it next. This closes the last of the 9 confirmed findings from the controller audit (bug-1684, bug-1687/1689/1690/1691/1692, bug-1693, bug-1694). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg		2026-08-25 14:54:53 +00:00
..
clean_failover_preservation.sh	test(ha-term-safety): harness post-failover client probes target a live node (bug-854 workaround)	2026-06-23 04:12:32 +00:00
configcompare.c	feat(config-upgrade-snapshot): add configcompare tool, extend snapverify (task 8.4)	2026-07-25 09:06:24 +00:00
configcompare.h	feat(config-upgrade-snapshot): add configcompare tool, extend snapverify (task 8.4)	2026-07-25 09:06:24 +00:00
controller_test.c	fix(controller): offload checkpoint_task_fn off the delaymgr thread (bug-1694)	2026-08-25 14:54:53 +00:00
ctrlstatus_fmt.h	feat(controller-status): render responderTime in cmd_ctrlstatus, log-comparable format (task 6.3)	2026-06-20 09:28:49 +00:00
resval_typeresolve.c	feat(resource-value-verification): wire resolver into resupdate, add -t flag (task 2.1)	2026-08-17 23:56:18 +00:00
resval_typeresolve.h	feat(resource-value-verification): wire resolver into resupdate, add -t flag (task 2.1)	2026-08-17 23:56:18 +00:00
snapdiff.c	fix(raft-drive-async): resolve snapshot chunk-drop (bug-1386) and chunk misordering (bug-1389)	2026-07-03 02:40:12 +00:00
snapdiff.h	fix(raft-drive-async): resolve snapshot chunk-drop (bug-1386) and chunk misordering (bug-1389)	2026-07-03 02:40:12 +00:00
snapverify.c	feat(config-upgrade-snapshot): add configcompare tool, extend snapverify (task 8.4)	2026-07-25 09:06:24 +00:00
snapverify.h	feat(config-upgrade-snapshot): add configcompare tool, extend snapverify (task 8.4)	2026-07-25 09:06:24 +00:00
test.c	fix(worker): job ids print human-readable format via fmt_jobid() (bug-1588)	2026-08-22 02:42:01 +00:00
test_apply_committed_entry.c	fix(controller): job start_time/finish_time/run_time never set by apply-pump	2026-08-20 04:24:44 +00:00
test_apply_correctness_verification.c	fix(controller): close 6 data-safety races + apply-pump O(N^2) regression + boot TOCTOU	2026-08-16 10:20:02 +00:00
test_apply_pump.c	fix(controller): close 6 data-safety races + apply-pump O(N^2) regression + boot TOCTOU	2026-08-16 10:20:02 +00:00
test_apply_pump_integration.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_apply_reapply_idempotency.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_apply_seam.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_bug857_regression.c	feat(config-upgrade-snapshot): add bug-857 regression test, correct design.md (task 8.5)	2026-07-25 10:44:14 +00:00
test_cluster_persist.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_coldstart_lsn0.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_commit_admission_pool.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_commit_barrier_reply.c	fix(controller): WAL_JOB_START loses pidinfo/start_time on restart replay (bug-1687)	2026-08-25 09:23:00 +00:00
test_commit_barrier_rollout.c	feat(config-upgrade-snapshot): remove restart/replay fallback, marker-trigger materializes alone	2026-07-25 05:37:27 +00:00
test_commit_durability.c	fix(controller): persist_flush now fires durable-advance callback (bug-1564)	2026-08-17 10:20:24 +00:00
test_commit_engine.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_commit_resolve.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_config_repl.c	test(raft-adapter-conformance): full config-push reply integrity coverage	2026-07-29 13:13:25 +00:00
test_configcompare.c	feat(config-upgrade-snapshot): add configcompare tool, extend snapverify (task 8.4)	2026-07-25 09:06:24 +00:00
test_convergence_verification.c	fix(ha): close permanent catch-up wedge from duplicated replication-target formula (bug-1478)	2026-07-28 09:53:20 +00:00
test_ctrl_metrics.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ctrlstatus.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_demotion_convergence.c	fix(controller): bug-1450/1451 offload delaymgr-inline blocking work to the slow lane	2026-07-19 08:14:00 +00:00
test_dispatch_commit.c	test(controller): cover live dispatch's apiserver notification (bug-1679)	2026-08-25 03:44:24 +00:00
test_dispatch_fail_retry.c	test(scheduler-dispatch-commit): retryable and bounded-retry dispatch failure e2e coverage (task 5.2)	2026-08-23 20:14:18 +00:00
test_dispatch_fail_terminal.c	feat(scheduler-dispatch-commit): controller resolves terminal dispatch failures to EXIT (task 3.2)	2026-08-23 16:36:39 +00:00
test_drive_async_config.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_e2e_capnp.c	fix(controller): stop SIGSEGV in WalCatchupEntry capnp encode (bug-1505)	2026-08-07 11:09:27 +00:00
test_e2e_checkpoint.c	fix(controller): close 6 data-safety races + apply-pump O(N^2) regression + boot TOCTOU	2026-08-16 10:20:02 +00:00
test_e2e_gc.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_e2e_hooks.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_e2e_wal.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_egress_filter.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_fetchfile_disconnect.c	fix(controller): fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693)	2026-08-25 14:13:26 +00:00
test_gc_retention.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_bootstrap.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_bootstrap_support.c	feat(election-iomgr-refactor): remove old blocking election code (am_i_primary_quorum, vote_request_thread, bootstrap_config_if_needed) — task 5.3	2026-05-30 13:04:19 +00:00
test_ha_cluster.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_connection_reuse.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_core_safety.c	test(raft-adapter-conformance): WAL catch-up reject reply-construction regression lock	2026-07-29 16:11:47 +00:00
test_ha_ctrlhello.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_dispatch_integration.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_durability.c	fix(controller): raft_cb_persist must deny vote and roll back state on persist failure (bug-1684)	2026-08-25 07:49:49 +00:00
test_ha_elect_ctx_lifecycle.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_election.c	fix(controller): clamp committed_config_tag to own loaded config on promotion	2026-07-23 04:23:50 +00:00
test_ha_failover.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_failover_prevote.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_heartbeat_committed.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_liveness.c	fix(controller): close 6 data-safety races + apply-pump O(N^2) regression + boot TOCTOU	2026-08-16 10:20:02 +00:00
test_ha_partition.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_prevote.c	fix(ha): free liveness poll ctx on failed/skipped delaymgr schedule (bug-1483)	2026-08-02 04:36:37 +00:00
test_ha_rdhup_integration.c	fix(ha): close standby election-freeze (bug-1562) + flusher retry storm (bug-1563)	2026-08-17 06:35:06 +00:00
test_ha_redirect.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_repl_push.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_role.c	fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)	2026-07-29 04:11:32 +00:00
test_ha_role_lifecycle.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_start_election_async.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_startup.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_vep_teardown.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_vote_floor_heartbeat.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_vote_gate.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_ha_vote_send.c	test(raft-adapter-conformance): PreVote post-connect failure routing + fast-quorum coverage	2026-07-29 14:38:36 +00:00
test_hooks.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_job_array_apiserver_push.c	fix(schema): rename struct Data -> XceData to unshadow capnp builtin blob type	2026-07-22 09:22:04 +00:00
test_job_array_apply.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_job_array_fence.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_job_array_replay_fence.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_job_array_sched_push.c	fix(controller): fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693)	2026-08-25 14:13:26 +00:00
test_job_dispatch.c	feat(scheduler-dispatch-commit): add controller dispatch-failure classification (task 1.1)	2026-08-23 06:36:51 +00:00
test_job_dispatch_fail_count.c	feat(scheduler-dispatch-commit): add dispatch-fail retry counter to job_t (task 1.3)	2026-08-23 07:44:22 +00:00
test_jobdispatch_no_double_notify.c	test(scheduler-dispatch-commit): terminal dispatch failure e2e coverage (task 5.3)	2026-08-23 20:47:15 +00:00
test_join_redirect.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_p1_integration.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_p2_integration.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_p3_hardening.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_p3_manual_failover.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_persist_checkpoint.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_persist_checkpoint_offload.c	fix(controller): offload checkpoint_task_fn off the delaymgr thread (bug-1694)	2026-08-25 14:54:53 +00:00
test_persist_hooks.c	fix(controller): close 6 data-safety races + apply-pump O(N^2) regression + boot TOCTOU	2026-08-16 10:20:02 +00:00
test_persist_recovery.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_persist_repl.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_persist_schema.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_persist_snapshot_entities.c	feat(config-upgrade-snapshot): decode host/queue admin-state frames on checkpoint load	2026-07-24 14:08:12 +00:00
test_persist_truncate.c	fix(controller): bug-1452 fix commit_engine_fini/persist_stop teardown race	2026-07-19 09:27:50 +00:00
test_persist_types.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_persist_wal.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_repl_apply.c	fix(controller): persist dispatch-time resource allocation into WAL_JOB_DISPATCH (bug-1678, gitea #43)	2026-08-25 03:12:49 +00:00
test_repl_catchup.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_repl_majority.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_repl_unit.c	fix(repl): harden standby slot allocation against WAL-retention guard clobber (bug-1474)	2026-07-26 04:11:27 +00:00
test_replay_apply_seam.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_resval_typeresolve.c	feat(resource-value-verification): extract resupdate type-resolution logic (task 1.1)	2026-08-17 23:03:57 +00:00
test_sched_admin_dispatch.c	fix(controller): fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693)	2026-08-25 14:13:26 +00:00
test_sched_worker_change.c	fix(controller): fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693)	2026-08-25 14:13:26 +00:00
test_slowlane.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_snapdiff.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_snapverify.c	feat(config-upgrade-snapshot): add configcompare tool, extend snapverify (task 8.4)	2026-07-25 09:06:24 +00:00
test_term_adoption_vectors.c	test(ha): add term-adoption regression tests for heartbeat/WAL-NACK vectors (bug-1484)	2026-08-02 04:36:46 +00:00
test_term_safety_adapter.c	fix(ha): close standby election-freeze (bug-1562) + flusher retry storm (bug-1563)	2026-08-17 06:35:06 +00:00
test_upgrade_snapshot_apply.c	test(config-upgrade-snapshot): fault-inject crash/restart + leadership change (task 8.7)	2026-07-25 18:20:43 +00:00
test_upgrade_snapshot_restore.c	feat(config-upgrade-snapshot): restore host admin-state frame safely (create-if-missing)	2026-07-24 12:48:28 +00:00
test_wal_job_ckpt_fact.c	fix(controller): WAL_JOB_CKPT loses job.ckptinfo on replay (bug-1691)	2026-08-25 12:29:10 +00:00
test_wal_job_dispatch_fact.c	test(controller): cover WAL_JOB_DISPATCH allocation persistence (bug-1678)	2026-08-25 03:12:58 +00:00
test_wal_job_finish_fact.c	fix(controller): WAL_JOB_FINISH loses exit_info/finish_time on replay -- and exit_info was never set at all in production (bug-1689)	2026-08-25 09:58:55 +00:00
test_wal_job_preason_fact.c	fix(controller): WAL_JOB_PREASON loses job.pend_reasons on replay (bug-1692)	2026-08-25 13:14:28 +00:00
test_wal_job_rusage_fact.c	fix(controller): WAL_JOB_RUSAGE loses job.run_usage on replay (bug-1690)	2026-08-25 10:53:32 +00:00
test_wal_job_start_fact.c	fix(controller): WAL_JOB_START loses pidinfo/start_time on restart replay (bug-1687)	2026-08-25 09:23:00 +00:00
test_walcompare.c	fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide	2026-07-17 14:49:12 +00:00
test_waldump.c	fix(waldump): decode WAL_CONFIG_MARKER, distinguish trimmed gaps from real loss	2026-07-31 09:05:22 +00:00
test_worker_join.c	fix(controller,xceadmin): fix zero-slot scheduling block, misleading resval logs, byte-unit display	2026-08-13 08:12:09 +00:00
walcompare.c	feat(walcompare): add per-member checkpoint-floor awareness (raft-ha-core-safety 5.10a / ha-entity-snapshots 7.1)	2026-07-02 02:59:08 +00:00
walcompare.h	feat(walcompare): add per-member checkpoint-floor awareness (raft-ha-core-safety 5.10a / ha-entity-snapshots 7.1)	2026-07-02 02:59:08 +00:00
waldump.c	fix(waldump): decode WAL_CONFIG_MARKER, distinguish trimmed gaps from real loss	2026-07-31 09:05:22 +00:00
waldump.h	fix(waldump): decode WAL_CONFIG_MARKER, distinguish trimmed gaps from real loss	2026-07-31 09:05:22 +00:00