wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race

Browse source 
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).

bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.

Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
This commit is contained in:
Rongsong Shen 2026-07-11 12:16:28 +00:00
parent ea0ac81c92
commit 40f510fb14
25 changed files with 1617 additions and 226 deletions
Download patch file Download diff file Expand all files Collapse all files

527
src/controller/ha.c
View file

@ -108,9 +108,26 @@ extern int config_schedule_activate_retry(cluster_t *c);
static void on_vote_connected(endpoint_t *ep);
static void on_pre_vote_connected(endpoint_t *ep);
static int vote_ep_err_handler(void *ep_raw);
/* bug-1423: forward-declared so raft_cb_send_vote_request's connect/monitor
* failure branches can route a synthetic "no vote" reply through the SAME
* production path a real reply uses (see vote_reply_dispatch's own comment,
* below, for why this must never be the legacy handle_request_vote_reply). */
static void vote_reply_dispatch(ha_state_t *ha, ha_elect_ctx_t *ctx, int idx,
uint64_t term, int vote_granted,
uint64_t peer_tag, uint64_t peer_lsn);
/* bug-1423 (PreVote twin): same rationale as vote_reply_dispatch above, for
* raft_cb_send_pre_vote_request's connect/monitor failure branches. */
static void pre_vote_reply_dispatch(ha_state_t *ha, cluster_t *c,
int peer_idx, int granted);
/* bug-1197: defer a vote-ep teardown onto the per-ep FIFO of the vtask pool so
* it runs strictly after the last live use of the ep (flag-ON loser path). */
static void vote_ep_teardown_async(ha_state_t *ha, endpoint_t *ep);
/* 2026-07-10 controller-invariant audit: forward-declared so the catch-up
* ingest path (ha_apply_wal_catchup_entry, earlier in this file) can kick the
* standby apply pump against the already-known commit position without
* waiting for the next heartbeat. Defined near ha_observe_heartbeat_committed. */
static void ha_standby_advance_committed_and_apply(cluster_t *c,
uint64_t learned_committed_lsn);
/* ha-single-leader-role task 2.2: shared promotion seam — every road to
* PRIMARY registers the persist hooks and arms the own-term gate. */
static void ha_register_persist_hooks(cluster_t *c);
@ -222,15 +239,36 @@ static int raft_cb_send_vote_request(void *ctx, int peer_idx, uint64_t term)
DEFAULT_CONNECT_TIMEOUT_MS);
if (rc < 0) {
endpoint_unref(ep); /* bug-1298: refcnt 1->0 → fini+free (not monitored) */
/* bug-1423: a connect failure (e.g. peer dead/unreachable) must still
* register as a "no vote" reply through the SAME path a real reply
* uses. Without this, raft_state_t.replies_received never reaches
* n_peers-1 whenever any peer is unreachable, so the library's
* "everyone's replied, decide now" fast path (raft.c "Step 6:
* all-replied-no-quorum") is structurally unreachable — recovery then
* depends solely on the raw election timeout, observed to leave a
* candidate stuck in CTRL_ELECTING far longer than expected in a
* 3-node cluster with one node down. Test mode (c==NULL) keeps the
* legacy handle_request_vote_reply call that existing unit tests
* depend on; production routes through vote_reply_dispatch, the same
* call a genuine reply makes (ha.c vote_reply_dispatch). */
if (!c)
handle_request_vote_reply(ha, ectx, votes_idx, term, 0, 0, 0);
else
vote_reply_dispatch(ha, ectx, votes_idx, term, /*vote_granted=*/0, 0, 0);
return 1;
}
rc = iomgr_monitor(ep);
if (rc < 0) {
endpoint_unref(ep); /* bug-1298: fini frees ep->data (vctx), then frees ep */
handle_request_vote_reply(ha, ectx, votes_idx, term, 0, 0, 0);
/* bug-1423: same fix as the connect-failure branch above — this was
* unconditionally calling the legacy handle_request_vote_reply even in
* production, bypassing raft_on_vote_reply (the sole production
* promotion driver per vote_reply_dispatch's own comment). */
if (!c)
handle_request_vote_reply(ha, ectx, votes_idx, term, 0, 0, 0);
else
vote_reply_dispatch(ha, ectx, votes_idx, term, /*vote_granted=*/0, 0, 0);
return 1;
}
@ -332,10 +370,30 @@ static int raft_cb_send_pre_vote_request(void *ctx, int peer_idx,
int rc = endpoint_connect(ep, peer->hostname, peer->port,
DEFAULT_CONNECT_TIMEOUT_MS);
if (rc < 0) { endpoint_unref(ep); return 1; } /* bug-1298: fini+free */
if (rc < 0) {
endpoint_unref(ep); /* bug-1298: fini+free */
/* bug-1423 (PreVote twin): a connect failure (dead/unreachable peer)
* must still register as a "not granted" reply with the raft library's
* PreVote tally, same as the real-vote send path. PreVote's own round
* timeout (ha_prevote_round_timeout_fn) is an unconditional fallback
* that always resolves the round regardless, so this gap is not a
* permanent-deadlock class bug like the real-vote one was — but
* leaving it means every round against a down peer wastes the full
* round timeout instead of concluding as soon as the live peer
* replies. */
if (c)
pre_vote_reply_dispatch(ha, c, peer_idx, /*granted=*/0);
return 1;
}
rc = iomgr_monitor(ep);
if (rc < 0) { endpoint_unref(ep); return 1; } /* bug-1298: fini+free */
if (rc < 0) {
endpoint_unref(ep); /* bug-1298: fini+free */
/* bug-1423 (PreVote twin): same fix as the connect-failure branch above. */
if (c)
pre_vote_reply_dispatch(ha, c, peer_idx, /*granted=*/0);
return 1;
}
return 0;
}
@ -482,15 +540,64 @@ static void raft_cb_cancel_timeout(void *ctx)
ha->raft_election_timeout_ctx = NULL;
}
/* raft_election_start_vtask_entry: vtask entry wrapping raft_check_and_start_
* election. raft_start_election's on_persist callback (raft_cb_persist ->
* ha_persist_term) does a synchronous fsync -- must never run inline on the
* delaymgr thread (ha-async-io.md thread-role contract: delaymgr only fires
* timers and triggers work, then returns; it never blocks, since the same
* thread also drives heartbeats and every other election timeout). */
static int raft_election_start_vtask_entry(vtask_t *task)
{
ha_state_t *ha = (ha_state_t *)task->data->u.generic;
raft_check_and_start_election(&ha->raft);
return 0;
}
/* ha_dispatch_election_start: offload raft_check_and_start_election onto a
* vtask (VTASK_LIVE_BLOCK -- the same class already used for the vote-GRANT
* term fsync, R4.6) so its persist-before-campaign fsync never blocks the
* delaymgr thread. On pool unavailability/enqueue failure, fail loud and run
* inline anyway (stay alive, never silently drop a scheduled election --
* mirrors persist_apply_pump_kick's and vote_ep_teardown_enqueue's fallback). */
static void ha_dispatch_election_start(ha_state_t *ha)
{
vtaskmgr_t *pool = (vtaskmgr_t *)ha->pool;
vtask_data_t *data = calloc(1, sizeof(*data));
vtask_t *task = vtask_alloc();
if (pool == NULL || data == NULL || task == NULL) {
WARN("ha_dispatch_election_start: vtask unavailable (pool=%p data=%p "
"task=%p) — starting election inline (stay alive)",
(void *)pool, (void *)data, (void *)task);
free(data);
free(task);
raft_check_and_start_election(&ha->raft);
return;
}
data->vtype = VTASK_GENERIC;
data->u.generic = ha;
vtask_init(task, (vtask_func_t)raft_election_start_vtask_entry, data);
if (vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK) < 0) {
WARN("ha_dispatch_election_start: vtask enqueue failed — starting "
"election inline (stay alive)");
free(data);
free(task);
raft_check_and_start_election(&ha->raft);
}
}
/* raft_rdhup_delay_fn: delaymgr callback for RDHUP-triggered delayed election.
* Owns its ctx (election_timeout_ctx_t) — frees it on exit.
* Calls raft_check_and_start_election to start election only if not already voted. */
* Dispatches the actual election start off the delaymgr thread (see
* ha_dispatch_election_start) only if this node hasn't already voted. */
static int raft_rdhup_delay_fn(void *arg)
{
election_timeout_ctx_t *t = (election_timeout_ctx_t *)arg;
ha_state_t *ha = t->ha;
if (!atomic_load(&t->cancelled))
raft_check_and_start_election(&ha->raft);
ha_dispatch_election_start(ha);
free(t);
return 0;
}
@ -506,9 +613,11 @@ static void raft_cb_start_election_after_delay(void *ctx, uint32_t delay_ms)
cluster_t *c = ha->raft_c;
if (!c) return; /* test mode */
/* Priority 0 → delay_ms == 0: highest-priority node starts immediately. */
/* Priority 0 → delay_ms == 0: highest-priority node starts immediately.
* Dispatched off the calling thread (see ha_dispatch_election_start) since
* this callback's caller is not guaranteed to be vtask-safe. */
if (delay_ms == 0) {
raft_check_and_start_election(&ha->raft);
ha_dispatch_election_start(ha);
return;
}
@ -870,10 +979,19 @@ static int promotion_fence_timeout_fn(void *arg)
* unconditionally for the same reason — see persist.c). */
bool apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT;
/* self_restart_pending: same unconditional, never-live-cleared refusal as
* apply_divergent — see the field's doc comment in ha.h. This purely
* structural re-check (durable floor vs HWM) must not release a fence set
* because a demoted ex-primary's execv() failed; that node's WAL state can
* look fully caught-up while its in-memory state is still stale. */
bool self_restart_pending =
atomic_load_explicit(&ha->self_restart_pending, memory_order_acquire);
/* Caught up: not truncated AND durable position reached the boundary AND
* the current apply verdict is not DIVERGENT → release the fence + clear
* the sticky flag; the member is eligible on its next election (Req 2.6). */
if (!truncated && !below_hwm && !apply_divergent) {
* the current apply verdict is not DIVERGENT AND no self-restart is
* pending → release the fence + clear the sticky flag; the member is
* eligible on its next election (Req 2.6). */
if (!truncated && !below_hwm && !apply_divergent && !self_restart_pending) {
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
persist_clear_recovery_unrecoverable(&c->persist);
INFO("HA: promotion fence RELEASED after catch-up — durable_floor=%"
@ -1046,10 +1164,20 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
uint64_t trunc_floor =
ha_promotion_truncate_floor(safe_lsn, durable_tip);
persist_truncate_wal_above(&c->persist, trunc_floor);
/* 2026-07-10 (chaotic-soak finding): persist_truncate_wal_above
* only advances persist->committed_lsn on the branches that
* perform a REAL truncation; its no-op path (this leader's own
* log already covers safe_lsn -- the common case) leaves
* committed_lsn untouched at its pre-election value, stalling
* commit-gated apply until the normal ack-driven commit path
* organically catches up seconds later. See
* persist_promotion_advance_committed's doc comment. */
persist_promotion_advance_committed(&c->persist, safe_lsn);
}
/* Task 4.2 (ha-repl-hardening): set committed_lsn = safe_lsn.
* persist_truncate_wal_above already sets c->persist.committed_lsn
* in the production path (c != NULL). Also update
* persist_promotion_advance_committed (above) advances
* c->persist.committed_lsn in the production path (c != NULL),
* durable-capped and monotonic. Also update
* ha->last_known_committed_lsn so the test-mode decision loop
* (XCE_TESTING path in standby heartbeat reply handler) and any
* code reading ha->last_known_committed_lsn see the safe commit
@ -1098,6 +1226,17 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
* recomputed by a fresh boot-time persist_recover() replay. */
bool apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT;
/* self_restart_pending (2026-07-10, ha-convergence-recovery Req 1.4
* supersession): a demoted ex-primary whose execv() re-exec failed
* must never become primary again while carrying its stale pre-
* demotion in-memory state — same unconditional-refuse treatment as
* apply_divergent, and for the same reason: this node's WAL/durability
* state can look perfectly ELIGIBLE while the in-memory state the
* restart exists to discard is still live. No live-clear path exists;
* only a successful restart (a fresh process) clears it. */
bool self_restart_pending =
atomic_load_explicit(&ha->self_restart_pending, memory_order_acquire);
/* A healthy peer is reachable for catch-up iff some voter granted with a
* live endpoint survives this election (these become the standby
* channels promoted below). */
@ -1127,30 +1266,35 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
truncated, durable_floor, promotion_safe_lsn,
peer_reachable, /*catch_up_succeeded=*/false);
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable && !apply_divergent) {
/* ELIGIBLE, not sticky-unrecoverable, AND not apply-divergent:
* release the fence (a caught-up member regains eligibility,
* Req 2.6), clear the sticky flag if set, and cancel any pending
* re-alert timer. Proceed to authoritative promotion below. */
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable && !apply_divergent
&& !self_restart_pending) {
/* ELIGIBLE, not sticky-unrecoverable, not apply-divergent, AND no
* pending self-restart: release the fence (a caught-up member
* regains eligibility, Req 2.6), clear the sticky flag if set, and
* cancel any pending re-alert timer. Proceed to authoritative
* promotion below. */
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
persist_clear_recovery_unrecoverable(&c->persist);
ha_cancel_promotion_fence_timer(ha);
} else {
/* FENCE, UNRECOVERABLE, sticky-unrecoverable, or apply-divergent:
* refuse promotion — do NOT become CTRL_PRIMARY. Fence the member
* and arm the catch-up re-alert timer. The member STAYS ALIVE,
* non-promotable, retrying catch-up (Req 2.2/2.4). NEVER
* self-exit, never fence silently. An apply-divergent member is
* refused unconditionally (Req 8.4) — no peer/HWM condition can
* clear that trigger; only a fresh, clean restart replay can. */
/* FENCE, UNRECOVERABLE, sticky-unrecoverable, apply-divergent, or
* self-restart-pending: refuse promotion — do NOT become
* CTRL_PRIMARY. Fence the member and arm the catch-up re-alert
* timer. The member STAYS ALIVE, non-promotable, retrying catch-up
* (Req 2.2/2.4). NEVER self-exit, never fence silently. An
* apply-divergent or self-restart-pending member is refused
* unconditionally (Req 8.4; ha-convergence-recovery Req 1.4) — no
* peer/HWM condition can clear either trigger; only a fresh boot
* (clean restart replay, or a successful re-exec) can. */
WARN("HA: promotion REFUSED (decision=%d unrecoverable=%d "
"apply_divergent=%d) — recovery_truncated=%d durable_floor=%"
PRIu64 " cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + "
"apply_divergent=%d self_restart_pending=%d) — "
"recovery_truncated=%d durable_floor=%" PRIu64
" cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + "
"arming catch-up re-alert window, staying ALIVE "
"(ha-commit-durability Req 2.2/2.4; raft-apply-decoupling "
"Req 8.4)",
(int)gate, unrecoverable, apply_divergent, truncated,
durable_floor, promotion_safe_lsn, peer_reachable);
"Req 8.4; ha-convergence-recovery Req 1.4)",
(int)gate, unrecoverable, apply_divergent, self_restart_pending,
truncated, durable_floor, promotion_safe_lsn, peer_reachable);
ha_arm_promotion_fence_timer(ha, c, truncated, durable_floor,
promotion_safe_lsn);
return;
@ -1201,6 +1345,52 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
INFO("HA FSM (raft): → CTRL_PRIMARY (term=%" PRIu64
", first_boot=%d)", term, is_first_boot);
/* 2026-07-10 (chaotic-soak regression finding, bug-1043 family recurrence):
* a node promoted from standby can carry an apply-lag backlog -- entries
* already committed per c->persist.committed_lsn (just finalized above by
* the truncate-clamp / safe_lsn assignment) but never materialized into
* the job/queue table, because standby-side materialization is now
* commit-gated and only kicked by heartbeat consumption or catch-up
* ingest (ha_standby_advance_committed_and_apply) -- BOTH of which stop
* firing the instant this node becomes CTRL_PRIMARY (ha_consume_heartbeat_
* committed explicitly skips primaries). Without draining here, that
* backlog sits unapplied: expect-job/joblist on the new primary would
* report a genuinely committed (WAL-consistent, walcompare-clean) job as
* missing.
*
* SYNCHRONOUS on purpose (persist_apply_pump_run_sync, not the async
* persist_apply_pump_kick): the async kick only ENQUEUES the drain onto a
* vtask and returns immediately, so this function (and thus promotion)
* could complete -- and the node could start answering expect-job/joblist
* queries as primary -- before the vtask worker actually finished
* applying the backlog. A live chaotic HA soak caught exactly that race
* (2026-07-10): the async version still lost committed jobs on a freshly
* promoted primary.
*
* THREAD-CONTEXT CORRECTION (2026-07-10, controller-invariant-audit): this
* comment previously claimed promotion "already runs on a VTASK_LIVE_BLOCK
* vtask" -- that is TRUE only for the single-node self-grant and RDHUP-
* failover election-start paths (ha_dispatch_election_start). It is FALSE
* for the common real multi-node case: a winning RequestVoteReply is
* dispatched via peer_ctrl_classify, which hardcodes VTASK_LIVENESS (never
* admission-gated, documented as never expected to block -- see the
* bug-1198 hazard box above peer_ctrl_classify's definition, which closed
* this EXACT class of bug for HA_REPL_SEND_SNAPSHOT's persist_checkpoint).
* persist_apply_pump_run_sync's pread()s of recently-written WAL segments
* are lower practical risk than a bare fdatasync (almost always page-cache
* hot, since the entries were just appended) but are still, strictly,
* blocking I/O on a task class the design says never blocks. This residual
* gap is tracked (see buglog.json) rather than fixed inline here: closing
* it properly means either reclassifying the winning-vote-reply case in
* peer_ctrl_classify or dispatching promotion's tail onto a fresh
* VTASK_LIVE_BLOCK vtask, both larger changes needing their own dedicated
* soak-validation pass. Do NOT remove this call to "fix" the classification
* gap -- it is what closes the bug-1043-family apply-lag/job-loss bug;
* removing it reopens that regression. The backlog itself stays inherently
* small (only entries committed since the last heartbeat-driven kick). */
if (c)
persist_apply_pump_run_sync(&c->persist);
/* Req 2.1 (ha-repl-hardening): re-initialize per-standby convergence
* tracking for THIS primary term. Stale converged flags from a
* previous term (demote → re-elect cycle) must not satisfy the
@ -2132,9 +2322,11 @@ void ha_fini(ha_state_t *ha)
* Two modes (Req 5.3, 5.5):
*
* Batch mode (entry->n_entries > 0):
* Iterate entry->entries[0..n_entries-1] in order. Apply each sub-entry
* individually via apply_wal_entry(). NULL sub-entries are treated as
* corrupt and skipped (Req 3.3). After ALL sub-entries are processed,
* Iterate entry->entries[0..n_entries-1] in order. Persist each
* sub-entry individually (entity materialization happens later, once
* committed — see ha_standby_advance_committed_and_apply). NULL
* sub-entries are treated as corrupt and skipped (Req 3.3). After ALL
* sub-entries are processed,
* advance applied_lsn to the last-successfully-applied lsn so the next
* heartbeat reply reflects post-batch state (Req 5.3).
*
@ -2166,11 +2358,10 @@ static void ha_advance_applied_lsn(ha_state_t *ha, uint64_t new_lsn)
}
}
/* ha_apply_single_entry: apply one decoded wal_catchup_entry_t sub-entry.
* Payload bytes (raw_payload/raw_payload_len, extracted by xce_decode)
* are passed through to apply_wal_entry so entity state is restored —
* not just the LSN chain (task 7.3).
* Returns 0 on success, non-zero on error (caller logs and skips).
/* ha_apply_single_entry: durably persist one decoded wal_catchup_entry_t
* sub-entry (name retained for history; despite the name it no longer
* materializes entity state here — see ha_standby_advance_committed_and_apply,
* 2026-07-10). Returns 0 on success, non-zero on error (caller logs and skips).
*
* raft-ha-core-safety task 6.2 — TERM-BLIND DIRECT-PERSIST BYPASS, SAFE-BY-
* GUARD (documented, not structurally retired): this function does NOT go
@ -2229,32 +2420,24 @@ static int ha_apply_single_entry(persist_t *p, wal_catchup_entry_t *e)
return prc;
}
/* bug-1416 (raft-apply-decoupling task 7.2 round 2): defer the in-memory
* entity apply while this node's OWN boot-time persist_recover() replay+
* verification pass is still running. The entry is already durably
* persisted above (persist_wal_write_replicated, fsync'd) — only the
* ACK and applied-lsn advancement depend on that, per the persist-
* before-apply-before-ACK contract this function's own header comment
* documents, and bug-1043's precedent of decoupling durability/ACK from
* entity materialization. Applying here too, concurrently with
* replay_wal()'s single-threaded, unlocked bookkeeping (last_applied_lsn)
* and the live job table it reads, can produce a false APPLY_DIVERGENT
* verdict on a perfectly healthy boot (confirmed live). No entry is
* lost: this node's own in-progress replay_wal() pass will pick it up
* directly off disk if it lands within that pass's scan, and
* persist_recover()'s own deferred catch-up kick (persist_apply_pump_
* kick, issued right before it returns) covers it unconditionally
* either way, since apply_pump_read_entry() does a fresh on-disk lookup
* every call rather than trusting any earlier directory listing. */
if (!persist_recovery_complete(p))
return 0;
/* Logical entity restore only. persist_wal_write_replicated (above) already
* wrote the entry and owns the on-disk CRC chain + current_lsn — so we pass
* the entry fields directly rather than fabricating a header. (bug-778:
* a synthetic crc=0 header here used to clobber p->last_crc.) */
return apply_wal_entry(p, (uint32_t)e->entry_type, e->lsn,
e->raw_payload, (uint32_t)e->raw_payload_len);
/* commit-then-apply on the standby (2026-07-10): entity materialization
* (restore_job/restore_job_state) does NOT happen here on receipt. This
* function's job ends at durable persist + ACK — the entry becomes
* in-memory state only once this node's OWN p->committed_lsn reaches it,
* driven by ha_standby_advance_committed_and_apply (heartbeat-learned
* commit position, clamped to this node's local durable tip) and applied
* through apply_committed_entry() by the same pump the primary's own
* online-apply and restart replay use. Applying eagerly here — before
* this node (or the sender) knows the entry reached cluster majority —
* could materialize a job the sender never actually committed: if a
* later (term,lsn) conflict truncate removes the WAL bytes (the sender
* lost leadership before reaching majority), nothing rolls back the
* already-applied entity mutation, leaving a permanent phantom job with
* no backing WAL entry (2026-07-10 controller-invariant audit finding).
* This mirrors bug-1043's own precedent of decoupling durability/ACK from
* entity materialization, extended one step further: apply now waits for
* COMMIT, not just local durability. */
return 0;
}
/* ha_send_wal_ack_reply: send WalAckReply {term, lsn, success=1} back to
@ -2329,32 +2512,18 @@ static int ha_apply_term_entry_via_library(ha_state_t *ha, persist_t *p,
* logOk and stalling commit. */
(void)before_term;
/* bug-1043: the library route (on_persist_entry → persist_wal_write_
* replicated) writes the entry to the WAL but does NOT apply it to the
* in-memory entity state (job/queue/worker tables). The legacy direct-
* apply path (ha_apply_single_entry) called apply_wal_entry after the
* persist; the term-safe library route dropped it. So a standby that
* caught up entries 8..N via replication held them on disk but NOT in the
* queryable job table — on promotion `joblist`/`expect-job` could not
* find a committed job (it IS on disk; walcompare passes), looking like a
* committed-job loss. Re-apply here, mirroring the legacy path. Idempotent
* on a re-stream (restore_job upserts by jobid). WAL_NOOP / WAL_CONFIG_
* MARKER carry no entity payload — apply_wal_entry no-ops on them.
*
* bug-1416 (raft-apply-decoupling task 7.2 round 2): gated on
* persist_recovery_complete() for the identical reason documented on
* ha_apply_single_entry's own matching guard above — defer the
* in-memory apply while this node's own boot-time persist_recover()
* pass is in progress, never the ACK/applied_lsn advance below (the
* entry is already durably persisted by on_persist_entry's
* persist_wal_write_replicated, invoked via ha_consume_append_entries
* above). This node's own in-progress replay, or the deferred
* catch-up kick persist_recover() issues before returning, applies
* it instead. */
if (persist_recovery_complete(p))
apply_wal_entry(p, (uint32_t)e->entry_type, e->lsn,
e->raw_payload, (uint32_t)e->raw_payload_len);
/* commit-then-apply on the standby (2026-07-10, supersedes the prior
* bug-1043 re-apply-here fix): entity materialization does NOT happen
* on receipt/persist. bug-1043's original problem — a caught-up
* standby held entries on disk but not in the queryable job table —
* is now solved by ha_standby_advance_committed_and_apply, driven by
* this node's heartbeat-learned commit position, applying through the
* SAME apply_committed_entry() seam the primary's online-apply and
* restart replay use. Applying here instead (right on receipt, before
* this entry is known to have reached cluster majority) risks a
* permanent phantom job: a later (term,lsn) conflict truncate removes
* the WAL bytes with no entity rollback (2026-07-10 controller-
* invariant audit finding). */
ha_advance_applied_lsn(ha, e->lsn);
ha_send_wal_ack_reply(ha, ep, c, e->lsn);
return 1;
@ -2386,6 +2555,13 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
if (c && c->ha.raft_c == c && entry->n_entries == 1 &&
entry->entries && entry->entries[0] && entry->entries[0]->term != 0) {
ha_apply_term_entry_via_library(ha, p, c, ep, entry->entries[0]);
/* 2026-07-10: a fresh entry just landed on disk -- kick the apply
* pump against the already-known commit position so a standby mid
* catch-up doesn't wait for the next heartbeat tick to materialize
* entries it's already known to have committed. */
ha_standby_advance_committed_and_apply(
c, atomic_load_explicit(&ha->last_known_committed_lsn,
memory_order_acquire));
return 0;
}
@ -2461,6 +2637,15 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
if (had_good)
ha_advance_applied_lsn(ha, last_good_lsn);
/* 2026-07-10: kick the apply pump against the already-known commit
* position so a batch catch-up materializes promptly instead of
* waiting for the next heartbeat tick (see the single-entry-term-safe
* branch above for the full rationale). */
if (had_good)
ha_standby_advance_committed_and_apply(
c, atomic_load_explicit(&ha->last_known_committed_lsn,
memory_order_acquire));
return 0;
}
@ -2499,6 +2684,12 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
/* Task 3.1: ACK the successfully applied single entry */
ha_send_wal_ack_reply(ha, ep, c, entry->lsn);
/* 2026-07-10: kick the apply pump against the already-known commit
* position (see the batch-mode branch above for the full rationale). */
ha_standby_advance_committed_and_apply(
c, atomic_load_explicit(&ha->last_known_committed_lsn,
memory_order_acquire));
return 0;
}
@ -3560,6 +3751,7 @@ int ha_dispatch_gate(ha_state_t *ha, int opcode)
if (opcode == OPCode_requestVote ||
opcode == OPCode_preVoteRequest ||
opcode == OPCode_probeConfigTag ||
opcode == OPCode_heartbeat ||
opcode == OPCode_hello ||
opcode == OPCode_nop ||
opcode == OPCode_replConfigPush ||
@ -6806,6 +6998,28 @@ int ha_prevote_round_timeout_fn(void *arg)
{
prevote_round_ctx_t *ctx = (prevote_round_ctx_t *)arg;
/* bug (2026-07-10 chaotic-soak ASAN finding): clear s_prevote_round_ctx
* for THIS ctx BEFORE freeing it, unconditionally -- regardless of which
* branch below runs. The prior code only cleared the static pointer
* inside the "round still in progress" branch; when pre_vote_in_progress
* was ALREADY false here (the round was decided via a reply arriving on
* a different path, one that doesn't itself clear this pointer) this
* function fell straight through to free(ctx) below with
* s_prevote_round_ctx still pointing at the about-to-be-freed memory.
* The NEXT PreVote round's ha_start_prevote_async then calls
* ha_prevote_cancel_round_timeout() defensively before arming its own
* timer, reads that dangling pointer, and writes through it --
* heap-use-after-free (confirmed via ASAN on a live chaotic HA soak,
* both crashing nodes wedged with an identical ha.c:6712 UAF). CAS so a
* newer round's context (if one has already replaced this one by the
* time this delaymgr task runs) is never incorrectly cleared. */
{
prevote_round_ctx_t *expected = ctx;
atomic_compare_exchange_strong_explicit(
&s_prevote_round_ctx, &expected, NULL,
memory_order_acq_rel, memory_order_relaxed);
}
if (atomic_load_explicit(&ctx->cancelled, memory_order_acquire)) {
#ifndef XCE_TESTING
free(ctx);
@ -6819,7 +7033,6 @@ int ha_prevote_round_timeout_fn(void *arg)
/* Only act if the round is still in progress (not already decided). */
if (ha->raft.pre_vote_in_progress) {
ha->raft.pre_vote_in_progress = 0;
atomic_store_explicit(&s_prevote_round_ctx, NULL, memory_order_release);
WARN("HA: PreVote round timed out before a decision — treating as LOSS, "
"staying CTRL_STANDBY; re-arming liveness watchdog (Req 5.4)");
ha_prevote_rearm_liveness(ha, c);
@ -7335,6 +7548,61 @@ typedef struct {
*/
static liveness_poll_ctx_t * _Atomic s_liveness_ctx = NULL;
/* rdhup_failover_vtask_entry / ha_dispatch_rdhup_failover: offload
* ha_on_primary_ctrl_fd_rdhup (which chains into ha_start_election_async ->
* raft_start_election -> on_persist -> ha_persist_term's synchronous fsync)
* off the delaymgr thread onto a VTASK_LIVE_BLOCK vtask, mirroring
* ha_dispatch_election_start. This is the real, common production trigger for
* a standby's failover election (heartbeat-timeout watchdog) -- delaymgr must
* only detect the timeout and hand off, never run the election itself. */
typedef struct {
ha_state_t *ha;
cluster_t *c;
} rdhup_failover_dispatch_t;
static int rdhup_failover_vtask_entry(vtask_t *task)
{
rdhup_failover_dispatch_t *d =
(rdhup_failover_dispatch_t *)task->data->u.generic;
ha_on_primary_ctrl_fd_rdhup(d->ha, d->c);
free(d);
return 0;
}
static void ha_dispatch_rdhup_failover(ha_state_t *ha, cluster_t *c)
{
vtaskmgr_t *pool = (vtaskmgr_t *)ha->pool;
vtask_data_t *data = calloc(1, sizeof(*data));
vtask_t *task = vtask_alloc();
rdhup_failover_dispatch_t *d = calloc(1, sizeof(*d));
if (pool == NULL || data == NULL || task == NULL || d == NULL) {
WARN("ha_dispatch_rdhup_failover: vtask unavailable (pool=%p data=%p "
"task=%p d=%p) — failing over inline (stay alive)",
(void *)pool, (void *)data, (void *)task, (void *)d);
free(data);
free(task);
free(d);
ha_on_primary_ctrl_fd_rdhup(ha, c);
return;
}
d->ha = ha;
d->c = c;
data->vtype = VTASK_GENERIC;
data->u.generic = d;
vtask_init(task, (vtask_func_t)rdhup_failover_vtask_entry, data);
if (vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK) < 0) {
WARN("ha_dispatch_rdhup_failover: vtask enqueue failed — failing over "
"inline (stay alive)");
free(data);
free(task);
free(d);
ha_on_primary_ctrl_fd_rdhup(ha, c);
}
}
/*
* liveness_poll_fn: delaymgr task replacing hb_timeout_thread.
*
@ -7384,7 +7652,7 @@ static int liveness_poll_fn(void *arg)
/* Silence exceeded timeout — trigger failover */
WARN("HA: liveness poll: no primary activity for %lus (timeout=%us) → failover",
(unsigned long)elapsed, timeout_s);
ha_on_primary_ctrl_fd_rdhup(ha, c);
ha_dispatch_rdhup_failover(ha, c);
/* Do NOT re-arm — failover handles re-election */
return 0;
}
@ -7555,6 +7823,56 @@ void ha_observe_heartbeat_committed(cluster_t *c, uint64_t committed_lsn,
committed_lsn, committed_config_tag, cur);
}
/*
* ha_standby_advance_committed_and_apply (2026-07-10, controller-invariant
* audit finding — commit/apply decoupling extended to the standby): drive
* this node's OWN p->committed_lsn from a learned primary committed position,
* clamped to what this node has actually durably received, then kick the
* SAME apply pump (persist_apply_pump_kick -> apply_committed_entry()) the
* primary's own online-apply and restart replay use.
*
* This is the missing half of the standby ingest path's commit-then-apply
* story: ha_apply_single_entry / ha_apply_term_entry_via_library persist +
* ACK a replicated entry unconditionally (required for durability and the
* primary's commit majority) but no longer apply it — entity materialization
* now waits for THIS function to observe that the entry is actually
* committed, never merely durable-on-receipt. Safe against a lagging
* standby: the clamp to local_tip means committed_lsn here can never run
* ahead of what this node has itself received, so persist_apply_pump_run's
* apply_pump_read_entry can never be asked to read an LSN this node doesn't
* have on disk yet.
*
* Race-safe against a CONCURRENT writer (2026-07-10, controller-invariant-
* audit finding): this function is gated to non-primary roles by its
* callers, but raft_cb_become_leader's own persist_promotion_advance_committed
* call happens BEFORE role flips to CTRL_PRIMARY (ha.c, the role store sits
* ~170 lines after the advance call) -- so a still-in-flight WAL catch-up
* entry (this function is also called from ha_apply_wal_catchup_entry's exit
* points) can legitimately run on another thread during that window, racing
* this function against the promotion path's own advance. Both now go
* through persist_committed_lsn_advance_atomic's CAS loop, which is safe
* against that race by construction (never loses an update, never
* regresses) -- no shared lock needed across the two call sites.
*/
static void ha_standby_advance_committed_and_apply(cluster_t *c,
uint64_t learned_committed_lsn)
{
if (!c || learned_committed_lsn == 0)
return;
persist_t *p = &c->persist;
pthread_mutex_lock(&p->lock);
uint64_t local_tip = p->current_lsn > 0 ? p->current_lsn - 1 : 0;
pthread_mutex_unlock(&p->lock);
uint64_t target = learned_committed_lsn < local_tip
? learned_committed_lsn : local_tip;
persist_committed_lsn_advance_atomic(p, target);
persist_apply_pump_kick(p, c->ha.pool);
}
/*
* ha_consume_heartbeat_committed: standby OP_nop consume branch — the
* non-primary gate (Req 3.4) + the stale-term guard (Req 3.2), then the
@ -7612,6 +7930,11 @@ void ha_consume_heartbeat_committed(cluster_t *c, const heartbeat_msg_t *hb)
* non-advancing LSN no-ops inside the observe (Req 4.2). */
ha_observe_heartbeat_committed(c, hb->committed_lsn,
hb->committed_config_tag);
/* 2026-07-10: drive this node's own committed_lsn + apply pump from the
* position just learned (see ha_standby_advance_committed_and_apply) —
* the commit-then-apply half of the standby ingest path. */
ha_standby_advance_committed_and_apply(c, hb->committed_lsn);
}
void ha_disarm_heartbeat_timeout_vtask(ha_state_t *ha)
@ -8326,6 +8649,28 @@ void ha_start_primary_heartbeat(cluster_t *c)
return;
}
/* 2026-07-10 (chaotic-soak finding): fire the FIRST heartbeat synchronously,
* right now, instead of only arming the periodic delay_task_ms below.
* delay_task_ms's first invocation fires only AFTER interval_ms elapses
* (it is a periodic timer, not "run now then repeat") -- so despite this
* function's own long-standing comment/doc ("start primary heartbeat
* immediately so standbys receive NOP"), a freshly-promoted leader was
* actually waiting a full heartbeat_interval_ms (default 3000ms) before a
* reconnected/catching-up standby ever got a heartbeat to reply to. That
* reply (stateReport) is what lets the primary learn the standby's true
* applied_lsn (ha_process_standby_state) and start catch-up streaming --
* which gates the per-entry majority-ack that advances committed_lsn past
* this election's conservative safe_committed_lsn floor. A live chaotic
* soak showed jobs above safe_lsn (legitimately majority-committed before
* the old primary died, per Leader Completeness) invisible via
* expect-job/joblist for the full 3s wait even after the
* persist_promotion_advance_committed fix above -- because nothing
* corroborated them was ever ASKED for that long. primary_heartbeat_fn
* only performs non-blocking async sends (xce_ha_heartbeat_req), so
* calling it inline here carries the same non-blocking-I/O contract as
* the rest of promotion and needs no vtask/thread-role change. */
primary_heartbeat_fn(c);
/* runcount=-1 (infinite): delaymgr re-enqueues automatically.
* The role check in primary_heartbeat_fn no-ops on demotion. */
int rc = delay_task_ms(&c->delaymgr, interval_ms, -1,