/xce-audit-invariants controller found two live violations of the async
I/O thread-role contract: ha_demotion_convergence_fn (ha_convergence.c)
ran WAL truncation + tlog_exit()+execv() directly on the delaymgr thread,
and ha_catchup_continuation_fn (config_repl.c) ran unbounded WAL-streaming
to a standby directly on delaymgr -- the same single thread that drives
heartbeats and election timeouts cluster-wide, so either blocks liveness
for the whole cluster and risks a spurious election.
A prior vtask-pool offload attempt (2026-07-10, bug-1429) deadlocked
controller_unit_test because c->ha.pool is never initialized by the
lightweight unit-test fixtures these functions run under, and was fully
reverted. This fix instead uses the codebase's established slow-lane
pattern (the same mechanism already used for snapshot-send, config-push,
and peer connect) -- a 4th and 5th instance, not a new mechanism.
New g_drive_async_demotion_offload / g_drive_async_catchup_offload flags
ship default OFF, matching the original rollout discipline of the three
reference offloads (shipped OFF, flipped ON later after a dedicated
compatibility audit). Flipping ON was tried during validation and
reproduced two distinct hazards against pre-existing fixtures that never
call slowlane_init: the already-known hang, and a new ASan SEGV in
commit_advance_locked (logged as bug-1452, not root-caused, deferred to
the future flip-to-ON audit). Both flags verified OFF and the full
976-test suite clean, then the committed state re-verified clean against
a live 10-round chaotic and 10-round sequential HA soak (zero job loss,
zero divergence, both PASS).
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).
bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.
Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
Mirror the sibling last_known_committed_lsn: all writes release, reads acquire.
Prereq for a heartbeat-receive writer + the ctrlstatus reader to interact
safely (Req 2.3). Pure access migration, no behavior change; controller suite
0 failures.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
