lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race

Browse source 
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).

bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.

Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
This commit is contained in:
Rongsong Shen 2026-07-11 12:16:28 +00:00
parent ea0ac81c92
commit 40f510fb14
25 changed files with 1617 additions and 226 deletions
Download patch file Download diff file Expand all files Collapse all files

123
src/controller/config_repl.c
View file

@ -391,20 +391,27 @@ int config_activate_committed(cluster_t *c)
* non-promotable) for that upgrade until reboot — a §7.9 liveness violation.
*
* The fix: when commit advances WHILE this member is serve-fenced, schedule a
* one-shot OFF-IOMGR delaymgr task that re-runs config_activate_committed. The
* delaymgr worker runs on a thread allowed to block on the disk reads
* (scan_latest_committed_marker pread + load_primary_config). Once the NEW marker
* is in the committed prefix AND the locally-held bytes hash-match it, the retry
* activates and CLEARS the fence; a genuine still-MISMATCH (or no-bytes-yet) keeps
* the fence and the next commit-advance reschedules. -ENOENT still falls back
* without fencing (the no-marker path is benign). Idempotent + self-limiting: the
* scheduler is a no-op once unfenced, so there is no busy-spin. */
* one-shot task that re-runs config_activate_committed off the iomgr ACK thread.
* The delaymgr thread itself must never block (ha-async-io.md thread-role
* contract — it also drives heartbeats and every election timeout), so the
* one-shot delaymgr task is only a trigger: it hands the actual blocking disk
* reads (scan_latest_committed_marker pread + load_primary_config) to a
* VTASK_LIVE_BLOCK vtask and returns immediately (2026-07-09 correction — this
* comment previously and incorrectly claimed the delaymgr worker itself was
* "allowed to block on the disk reads"; that predates the corrected contract
* ha-async-io.md documents). Once the NEW marker is in the committed prefix AND
* the locally-held bytes hash-match it, the retry activates and CLEARS the
* fence; a genuine still-MISMATCH (or no-bytes-yet) keeps the fence and the next
* commit-advance reschedules. -ENOENT still falls back without fencing (the
* no-marker path is benign). Idempotent + self-limiting: the scheduler is a
* no-op once unfenced, so there is no busy-spin. */
/* config_activate_retry_fn: the delaymgr task body (runs OFF the iomgr ACK
* thread). Re-runs config_activate_committed so a freshly-committed config marker
* is activated — and a stale serve-fence cleared — once its out-of-band bytes
* verify. `arg` is the cluster_t* (passed directly; no heap ctx to free, so the
* delaymgr node-killer that frees only the delay_task_t never leaks). */
/* config_activate_retry_fn: re-runs config_activate_committed so a freshly-
* committed config marker is activated — and a stale serve-fence cleared —
* once its out-of-band bytes verify. Does the actual blocking disk I/O; callers
* must invoke it off the delaymgr thread (see config_activate_retry_vtask_entry
* / config_schedule_activate_retry below). Public so tests can drive it
* directly without a vtask pool. */
int config_activate_retry_fn(void *arg)
{
cluster_t *c = (cluster_t *)arg;
@ -425,8 +432,56 @@ int config_activate_retry_fn(void *arg)
return rc;
}
/* config_schedule_activate_retry: enqueue config_activate_retry_fn on the
* delaymgr (NON-BLOCKING — delay_task_ms only does ordq_put). Called from the
/* config_activate_retry_vtask_entry: vtask entry wrapping config_activate_
* retry_fn so its blocking disk I/O runs on a vtask worker, never inline on
* the single delaymgr thread. */
static int config_activate_retry_vtask_entry(vtask_t *task)
{
cluster_t *c = (cluster_t *)task->data->u.generic;
config_activate_retry_fn(c);
return 0;
}
/* config_activate_retry_delaymgr_trampoline: the delaymgr task body. Delaymgr
* must only trigger + enqueue, never block — dispatch the actual retry work to
* a vtask (VTASK_LIVE_BLOCK, the same class the vote-grant term fsync uses,
* R4.6) and return immediately. Fails loud and runs inline (stay alive, never
* silently drop a scheduled retry) if the pool is unavailable. */
static int config_activate_retry_delaymgr_trampoline(void *arg)
{
cluster_t *c = (cluster_t *)arg;
if (!c)
return -EINVAL;
vtaskmgr_t *pool = &c->vtaskmgr;
vtask_data_t *data = calloc(1, sizeof(*data));
vtask_t *task = vtask_alloc();
if (data == NULL || task == NULL) {
WARN("config commit-advance retry: vtask alloc failed — running "
"inline (stay alive)");
free(data);
free(task);
return config_activate_retry_fn(c);
}
data->vtype = VTASK_GENERIC;
data->u.generic = c;
vtask_init(task, (vtask_func_t)config_activate_retry_vtask_entry, data);
if (vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK) < 0) {
WARN("config commit-advance retry: vtask enqueue failed — running "
"inline (stay alive)");
free(data);
free(task);
return config_activate_retry_fn(c);
}
return 0;
}
/* config_schedule_activate_retry: enqueue the retry trampoline on the delaymgr
* (NON-BLOCKING — delay_task_ms only does ordq_put; the trampoline itself
* offloads the blocking work to a vtask, see above). Called from the
* commit-advance path (iomgr ACK thread) so activation's disk I/O runs OFF that
* thread (ha-async-io invariant). No-op when the member is NOT serve-fenced (so
* the steady state schedules nothing) or when the delaymgr is not running (the
@ -443,7 +498,8 @@ int config_schedule_activate_retry(cluster_t *c)
/* One-shot, minimal delay: run promptly but off the iomgr ACK thread. */
int rc = delay_task_ms(&c->delaymgr, 1, 1,
(delay_task_entry_fn_t)config_activate_retry_fn, c);
(delay_task_entry_fn_t)config_activate_retry_delaymgr_trampoline,
c);
if (rc < 0)
WARN("config commit-advance retry: delay_task_ms failed: %d — fence will "
"clear on a later commit advance or config-push", rc);
@ -2540,12 +2596,21 @@ typedef struct {
} ha_catchup_continuation_ctx_t;
/*
* ha_catchup_continuation_fn — delaymgr task body (runs OFF the iomgr walAck
* branch, D6/D1). Re-validate the endpoint against the registry, re-derive
* target, stream [acked_lsn+1, target], and clear catchup_inflight on EVERY exit
* path (success, ep-gone, nothing-to-stream) so a stuck flag never blocks future
* continuations (Req 5.2). Frees its own ctx (body-frees-own-ctx pattern,
* mirroring ha_demotion_convergence_fn).
* ha_catchup_continuation_fn — the real catch-up work. Re-validate the
* endpoint against the registry, re-derive target, stream [acked_lsn+1,
* target], and clear catchup_inflight on EVERY exit path (success, ep-gone,
* nothing-to-stream) so a stuck flag never blocks future continuations (Req
* 5.2). Frees its own ctx (body-frees-own-ctx pattern, mirroring
* ha_demotion_convergence_run).
*
* Runs as a delaymgr task body (see ha_schedule_catchup_continuation below) —
* must run off the iomgr walAck thread, since catchup_stream_range ->
* ha_stream_wal_to_standby does blocking WAL file reads + socket writes.
* (2026-07-10: a VTASK_LIVE_BLOCK dispatch was attempted here to also get it
* off the delaymgr thread per ha-async-io.md's thread-role contract, but it
* deadlocked the controller_unit_test harness — reverted pending a proper
* fix; tracked as a known, still-open thread-role violation alongside the
* other raft-drive-async P2 items.)
*/
int ha_catchup_continuation_fn(void *arg)
{
@ -2587,10 +2652,11 @@ clear_and_done:
}
/*
* ha_schedule_catchup_continuation — enqueue one one-shot delaymgr task running
* ha_catchup_continuation_fn. The single-flight flag is set by the caller
* (ha_walack_on_ack) before this call; on a scheduling failure this clears it so
* the slot does not stay stuck. Returns 0 on success, negative on failure.
* ha_schedule_catchup_continuation — enqueue one one-shot delaymgr task
* running ha_catchup_continuation_fn directly. The single-flight flag is set
* by the caller (ha_walack_on_ack) before this call; on a scheduling failure
* this clears it so the slot does not stay stuck. Returns 0 on success,
* negative on failure.
*/
static int ha_schedule_catchup_continuation(cluster_t *c, repl_ep_t ep,
int peer_idx, uint64_t acked_lsn)
@ -2610,8 +2676,9 @@ static int ha_schedule_catchup_continuation(cluster_t *c, repl_ep_t ep,
/* One-shot (runcount=1), minimal delay: streaming reads WAL files and must
* run off the iomgr walAck thread. delay_task_ms only enqueues (ordq_put);
* the delaymgr worker runs it. In XCE_TESTING the fixture drives the queued
* task synchronously via its stored entry/args. */
* the delaymgr worker runs ha_catchup_continuation_fn itself. In
* XCE_TESTING the fixture drives the queued task synchronously via its
* stored entry/args. */
int rc = delay_task_ms(&c->delaymgr, 1, 1,
(delay_task_entry_fn_t)ha_catchup_continuation_fn,
ctx);

527
src/controller/ha.c
View file

@ -108,9 +108,26 @@ extern int config_schedule_activate_retry(cluster_t *c);
static void on_vote_connected(endpoint_t *ep);
static void on_pre_vote_connected(endpoint_t *ep);
static int vote_ep_err_handler(void *ep_raw);
/* bug-1423: forward-declared so raft_cb_send_vote_request's connect/monitor
* failure branches can route a synthetic "no vote" reply through the SAME
* production path a real reply uses (see vote_reply_dispatch's own comment,
* below, for why this must never be the legacy handle_request_vote_reply). */
static void vote_reply_dispatch(ha_state_t *ha, ha_elect_ctx_t *ctx, int idx,
uint64_t term, int vote_granted,
uint64_t peer_tag, uint64_t peer_lsn);
/* bug-1423 (PreVote twin): same rationale as vote_reply_dispatch above, for
* raft_cb_send_pre_vote_request's connect/monitor failure branches. */
static void pre_vote_reply_dispatch(ha_state_t *ha, cluster_t *c,
int peer_idx, int granted);
/* bug-1197: defer a vote-ep teardown onto the per-ep FIFO of the vtask pool so
* it runs strictly after the last live use of the ep (flag-ON loser path). */
static void vote_ep_teardown_async(ha_state_t *ha, endpoint_t *ep);
/* 2026-07-10 controller-invariant audit: forward-declared so the catch-up
* ingest path (ha_apply_wal_catchup_entry, earlier in this file) can kick the
* standby apply pump against the already-known commit position without
* waiting for the next heartbeat. Defined near ha_observe_heartbeat_committed. */
static void ha_standby_advance_committed_and_apply(cluster_t *c,
uint64_t learned_committed_lsn);
/* ha-single-leader-role task 2.2: shared promotion seam — every road to
* PRIMARY registers the persist hooks and arms the own-term gate. */
static void ha_register_persist_hooks(cluster_t *c);
@ -222,15 +239,36 @@ static int raft_cb_send_vote_request(void *ctx, int peer_idx, uint64_t term)
DEFAULT_CONNECT_TIMEOUT_MS);
if (rc < 0) {
endpoint_unref(ep); /* bug-1298: refcnt 1->0 → fini+free (not monitored) */
/* bug-1423: a connect failure (e.g. peer dead/unreachable) must still
* register as a "no vote" reply through the SAME path a real reply
* uses. Without this, raft_state_t.replies_received never reaches
* n_peers-1 whenever any peer is unreachable, so the library's
* "everyone's replied, decide now" fast path (raft.c "Step 6:
* all-replied-no-quorum") is structurally unreachable — recovery then
* depends solely on the raw election timeout, observed to leave a
* candidate stuck in CTRL_ELECTING far longer than expected in a
* 3-node cluster with one node down. Test mode (c==NULL) keeps the
* legacy handle_request_vote_reply call that existing unit tests
* depend on; production routes through vote_reply_dispatch, the same
* call a genuine reply makes (ha.c vote_reply_dispatch). */
if (!c)
handle_request_vote_reply(ha, ectx, votes_idx, term, 0, 0, 0);
else
vote_reply_dispatch(ha, ectx, votes_idx, term, /*vote_granted=*/0, 0, 0);
return 1;
}
rc = iomgr_monitor(ep);
if (rc < 0) {
endpoint_unref(ep); /* bug-1298: fini frees ep->data (vctx), then frees ep */
handle_request_vote_reply(ha, ectx, votes_idx, term, 0, 0, 0);
/* bug-1423: same fix as the connect-failure branch above — this was
* unconditionally calling the legacy handle_request_vote_reply even in
* production, bypassing raft_on_vote_reply (the sole production
* promotion driver per vote_reply_dispatch's own comment). */
if (!c)
handle_request_vote_reply(ha, ectx, votes_idx, term, 0, 0, 0);
else
vote_reply_dispatch(ha, ectx, votes_idx, term, /*vote_granted=*/0, 0, 0);
return 1;
}
@ -332,10 +370,30 @@ static int raft_cb_send_pre_vote_request(void *ctx, int peer_idx,
int rc = endpoint_connect(ep, peer->hostname, peer->port,
DEFAULT_CONNECT_TIMEOUT_MS);
if (rc < 0) { endpoint_unref(ep); return 1; } /* bug-1298: fini+free */
if (rc < 0) {
endpoint_unref(ep); /* bug-1298: fini+free */
/* bug-1423 (PreVote twin): a connect failure (dead/unreachable peer)
* must still register as a "not granted" reply with the raft library's
* PreVote tally, same as the real-vote send path. PreVote's own round
* timeout (ha_prevote_round_timeout_fn) is an unconditional fallback
* that always resolves the round regardless, so this gap is not a
* permanent-deadlock class bug like the real-vote one was — but
* leaving it means every round against a down peer wastes the full
* round timeout instead of concluding as soon as the live peer
* replies. */
if (c)
pre_vote_reply_dispatch(ha, c, peer_idx, /*granted=*/0);
return 1;
}
rc = iomgr_monitor(ep);
if (rc < 0) { endpoint_unref(ep); return 1; } /* bug-1298: fini+free */
if (rc < 0) {
endpoint_unref(ep); /* bug-1298: fini+free */
/* bug-1423 (PreVote twin): same fix as the connect-failure branch above. */
if (c)
pre_vote_reply_dispatch(ha, c, peer_idx, /*granted=*/0);
return 1;
}
return 0;
}
@ -482,15 +540,64 @@ static void raft_cb_cancel_timeout(void *ctx)
ha->raft_election_timeout_ctx = NULL;
}
/* raft_election_start_vtask_entry: vtask entry wrapping raft_check_and_start_
* election. raft_start_election's on_persist callback (raft_cb_persist ->
* ha_persist_term) does a synchronous fsync -- must never run inline on the
* delaymgr thread (ha-async-io.md thread-role contract: delaymgr only fires
* timers and triggers work, then returns; it never blocks, since the same
* thread also drives heartbeats and every other election timeout). */
static int raft_election_start_vtask_entry(vtask_t *task)
{
ha_state_t *ha = (ha_state_t *)task->data->u.generic;
raft_check_and_start_election(&ha->raft);
return 0;
}
/* ha_dispatch_election_start: offload raft_check_and_start_election onto a
* vtask (VTASK_LIVE_BLOCK -- the same class already used for the vote-GRANT
* term fsync, R4.6) so its persist-before-campaign fsync never blocks the
* delaymgr thread. On pool unavailability/enqueue failure, fail loud and run
* inline anyway (stay alive, never silently drop a scheduled election --
* mirrors persist_apply_pump_kick's and vote_ep_teardown_enqueue's fallback). */
static void ha_dispatch_election_start(ha_state_t *ha)
{
vtaskmgr_t *pool = (vtaskmgr_t *)ha->pool;
vtask_data_t *data = calloc(1, sizeof(*data));
vtask_t *task = vtask_alloc();
if (pool == NULL || data == NULL || task == NULL) {
WARN("ha_dispatch_election_start: vtask unavailable (pool=%p data=%p "
"task=%p) — starting election inline (stay alive)",
(void *)pool, (void *)data, (void *)task);
free(data);
free(task);
raft_check_and_start_election(&ha->raft);
return;
}
data->vtype = VTASK_GENERIC;
data->u.generic = ha;
vtask_init(task, (vtask_func_t)raft_election_start_vtask_entry, data);
if (vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK) < 0) {
WARN("ha_dispatch_election_start: vtask enqueue failed — starting "
"election inline (stay alive)");
free(data);
free(task);
raft_check_and_start_election(&ha->raft);
}
}
/* raft_rdhup_delay_fn: delaymgr callback for RDHUP-triggered delayed election.
* Owns its ctx (election_timeout_ctx_t) — frees it on exit.
* Calls raft_check_and_start_election to start election only if not already voted. */
* Dispatches the actual election start off the delaymgr thread (see
* ha_dispatch_election_start) only if this node hasn't already voted. */
static int raft_rdhup_delay_fn(void *arg)
{
election_timeout_ctx_t *t = (election_timeout_ctx_t *)arg;
ha_state_t *ha = t->ha;
if (!atomic_load(&t->cancelled))
raft_check_and_start_election(&ha->raft);
ha_dispatch_election_start(ha);
free(t);
return 0;
}
@ -506,9 +613,11 @@ static void raft_cb_start_election_after_delay(void *ctx, uint32_t delay_ms)
cluster_t *c = ha->raft_c;
if (!c) return; /* test mode */
/* Priority 0 → delay_ms == 0: highest-priority node starts immediately. */
/* Priority 0 → delay_ms == 0: highest-priority node starts immediately.
* Dispatched off the calling thread (see ha_dispatch_election_start) since
* this callback's caller is not guaranteed to be vtask-safe. */
if (delay_ms == 0) {
raft_check_and_start_election(&ha->raft);
ha_dispatch_election_start(ha);
return;
}
@ -870,10 +979,19 @@ static int promotion_fence_timeout_fn(void *arg)
* unconditionally for the same reason — see persist.c). */
bool apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT;
/* self_restart_pending: same unconditional, never-live-cleared refusal as
* apply_divergent — see the field's doc comment in ha.h. This purely
* structural re-check (durable floor vs HWM) must not release a fence set
* because a demoted ex-primary's execv() failed; that node's WAL state can
* look fully caught-up while its in-memory state is still stale. */
bool self_restart_pending =
atomic_load_explicit(&ha->self_restart_pending, memory_order_acquire);
/* Caught up: not truncated AND durable position reached the boundary AND
* the current apply verdict is not DIVERGENT → release the fence + clear
* the sticky flag; the member is eligible on its next election (Req 2.6). */
if (!truncated && !below_hwm && !apply_divergent) {
* the current apply verdict is not DIVERGENT AND no self-restart is
* pending → release the fence + clear the sticky flag; the member is
* eligible on its next election (Req 2.6). */
if (!truncated && !below_hwm && !apply_divergent && !self_restart_pending) {
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
persist_clear_recovery_unrecoverable(&c->persist);
INFO("HA: promotion fence RELEASED after catch-up — durable_floor=%"
@ -1046,10 +1164,20 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
uint64_t trunc_floor =
ha_promotion_truncate_floor(safe_lsn, durable_tip);
persist_truncate_wal_above(&c->persist, trunc_floor);
/* 2026-07-10 (chaotic-soak finding): persist_truncate_wal_above
* only advances persist->committed_lsn on the branches that
* perform a REAL truncation; its no-op path (this leader's own
* log already covers safe_lsn -- the common case) leaves
* committed_lsn untouched at its pre-election value, stalling
* commit-gated apply until the normal ack-driven commit path
* organically catches up seconds later. See
* persist_promotion_advance_committed's doc comment. */
persist_promotion_advance_committed(&c->persist, safe_lsn);
}
/* Task 4.2 (ha-repl-hardening): set committed_lsn = safe_lsn.
* persist_truncate_wal_above already sets c->persist.committed_lsn
* in the production path (c != NULL). Also update
* persist_promotion_advance_committed (above) advances
* c->persist.committed_lsn in the production path (c != NULL),
* durable-capped and monotonic. Also update
* ha->last_known_committed_lsn so the test-mode decision loop
* (XCE_TESTING path in standby heartbeat reply handler) and any
* code reading ha->last_known_committed_lsn see the safe commit
@ -1098,6 +1226,17 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
* recomputed by a fresh boot-time persist_recover() replay. */
bool apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT;
/* self_restart_pending (2026-07-10, ha-convergence-recovery Req 1.4
* supersession): a demoted ex-primary whose execv() re-exec failed
* must never become primary again while carrying its stale pre-
* demotion in-memory state — same unconditional-refuse treatment as
* apply_divergent, and for the same reason: this node's WAL/durability
* state can look perfectly ELIGIBLE while the in-memory state the
* restart exists to discard is still live. No live-clear path exists;
* only a successful restart (a fresh process) clears it. */
bool self_restart_pending =
atomic_load_explicit(&ha->self_restart_pending, memory_order_acquire);
/* A healthy peer is reachable for catch-up iff some voter granted with a
* live endpoint survives this election (these become the standby
* channels promoted below). */
@ -1127,30 +1266,35 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
truncated, durable_floor, promotion_safe_lsn,
peer_reachable, /*catch_up_succeeded=*/false);
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable && !apply_divergent) {
/* ELIGIBLE, not sticky-unrecoverable, AND not apply-divergent:
* release the fence (a caught-up member regains eligibility,
* Req 2.6), clear the sticky flag if set, and cancel any pending
* re-alert timer. Proceed to authoritative promotion below. */
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable && !apply_divergent
&& !self_restart_pending) {
/* ELIGIBLE, not sticky-unrecoverable, not apply-divergent, AND no
* pending self-restart: release the fence (a caught-up member
* regains eligibility, Req 2.6), clear the sticky flag if set, and
* cancel any pending re-alert timer. Proceed to authoritative
* promotion below. */
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
persist_clear_recovery_unrecoverable(&c->persist);
ha_cancel_promotion_fence_timer(ha);
} else {
/* FENCE, UNRECOVERABLE, sticky-unrecoverable, or apply-divergent:
* refuse promotion — do NOT become CTRL_PRIMARY. Fence the member
* and arm the catch-up re-alert timer. The member STAYS ALIVE,
* non-promotable, retrying catch-up (Req 2.2/2.4). NEVER
* self-exit, never fence silently. An apply-divergent member is
* refused unconditionally (Req 8.4) — no peer/HWM condition can
* clear that trigger; only a fresh, clean restart replay can. */
/* FENCE, UNRECOVERABLE, sticky-unrecoverable, apply-divergent, or
* self-restart-pending: refuse promotion — do NOT become
* CTRL_PRIMARY. Fence the member and arm the catch-up re-alert
* timer. The member STAYS ALIVE, non-promotable, retrying catch-up
* (Req 2.2/2.4). NEVER self-exit, never fence silently. An
* apply-divergent or self-restart-pending member is refused
* unconditionally (Req 8.4; ha-convergence-recovery Req 1.4) — no
* peer/HWM condition can clear either trigger; only a fresh boot
* (clean restart replay, or a successful re-exec) can. */
WARN("HA: promotion REFUSED (decision=%d unrecoverable=%d "
"apply_divergent=%d) — recovery_truncated=%d durable_floor=%"
PRIu64 " cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + "
"apply_divergent=%d self_restart_pending=%d) — "
"recovery_truncated=%d durable_floor=%" PRIu64
" cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + "
"arming catch-up re-alert window, staying ALIVE "
"(ha-commit-durability Req 2.2/2.4; raft-apply-decoupling "
"Req 8.4)",
(int)gate, unrecoverable, apply_divergent, truncated,
durable_floor, promotion_safe_lsn, peer_reachable);
"Req 8.4; ha-convergence-recovery Req 1.4)",
(int)gate, unrecoverable, apply_divergent, self_restart_pending,
truncated, durable_floor, promotion_safe_lsn, peer_reachable);
ha_arm_promotion_fence_timer(ha, c, truncated, durable_floor,
promotion_safe_lsn);
return;
@ -1201,6 +1345,52 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
INFO("HA FSM (raft): → CTRL_PRIMARY (term=%" PRIu64
", first_boot=%d)", term, is_first_boot);
/* 2026-07-10 (chaotic-soak regression finding, bug-1043 family recurrence):
* a node promoted from standby can carry an apply-lag backlog -- entries
* already committed per c->persist.committed_lsn (just finalized above by
* the truncate-clamp / safe_lsn assignment) but never materialized into
* the job/queue table, because standby-side materialization is now
* commit-gated and only kicked by heartbeat consumption or catch-up
* ingest (ha_standby_advance_committed_and_apply) -- BOTH of which stop
* firing the instant this node becomes CTRL_PRIMARY (ha_consume_heartbeat_
* committed explicitly skips primaries). Without draining here, that
* backlog sits unapplied: expect-job/joblist on the new primary would
* report a genuinely committed (WAL-consistent, walcompare-clean) job as
* missing.
*
* SYNCHRONOUS on purpose (persist_apply_pump_run_sync, not the async
* persist_apply_pump_kick): the async kick only ENQUEUES the drain onto a
* vtask and returns immediately, so this function (and thus promotion)
* could complete -- and the node could start answering expect-job/joblist
* queries as primary -- before the vtask worker actually finished
* applying the backlog. A live chaotic HA soak caught exactly that race
* (2026-07-10): the async version still lost committed jobs on a freshly
* promoted primary.
*
* THREAD-CONTEXT CORRECTION (2026-07-10, controller-invariant-audit): this
* comment previously claimed promotion "already runs on a VTASK_LIVE_BLOCK
* vtask" -- that is TRUE only for the single-node self-grant and RDHUP-
* failover election-start paths (ha_dispatch_election_start). It is FALSE
* for the common real multi-node case: a winning RequestVoteReply is
* dispatched via peer_ctrl_classify, which hardcodes VTASK_LIVENESS (never
* admission-gated, documented as never expected to block -- see the
* bug-1198 hazard box above peer_ctrl_classify's definition, which closed
* this EXACT class of bug for HA_REPL_SEND_SNAPSHOT's persist_checkpoint).
* persist_apply_pump_run_sync's pread()s of recently-written WAL segments
* are lower practical risk than a bare fdatasync (almost always page-cache
* hot, since the entries were just appended) but are still, strictly,
* blocking I/O on a task class the design says never blocks. This residual
* gap is tracked (see buglog.json) rather than fixed inline here: closing
* it properly means either reclassifying the winning-vote-reply case in
* peer_ctrl_classify or dispatching promotion's tail onto a fresh
* VTASK_LIVE_BLOCK vtask, both larger changes needing their own dedicated
* soak-validation pass. Do NOT remove this call to "fix" the classification
* gap -- it is what closes the bug-1043-family apply-lag/job-loss bug;
* removing it reopens that regression. The backlog itself stays inherently
* small (only entries committed since the last heartbeat-driven kick). */
if (c)
persist_apply_pump_run_sync(&c->persist);
/* Req 2.1 (ha-repl-hardening): re-initialize per-standby convergence
* tracking for THIS primary term. Stale converged flags from a
* previous term (demote → re-elect cycle) must not satisfy the
@ -2132,9 +2322,11 @@ void ha_fini(ha_state_t *ha)
* Two modes (Req 5.3, 5.5):
*
* Batch mode (entry->n_entries > 0):
* Iterate entry->entries[0..n_entries-1] in order. Apply each sub-entry
* individually via apply_wal_entry(). NULL sub-entries are treated as
* corrupt and skipped (Req 3.3). After ALL sub-entries are processed,
* Iterate entry->entries[0..n_entries-1] in order. Persist each
* sub-entry individually (entity materialization happens later, once
* committed — see ha_standby_advance_committed_and_apply). NULL
* sub-entries are treated as corrupt and skipped (Req 3.3). After ALL
* sub-entries are processed,
* advance applied_lsn to the last-successfully-applied lsn so the next
* heartbeat reply reflects post-batch state (Req 5.3).
*
@ -2166,11 +2358,10 @@ static void ha_advance_applied_lsn(ha_state_t *ha, uint64_t new_lsn)
}
}
/* ha_apply_single_entry: apply one decoded wal_catchup_entry_t sub-entry.
* Payload bytes (raw_payload/raw_payload_len, extracted by xce_decode)
* are passed through to apply_wal_entry so entity state is restored —
* not just the LSN chain (task 7.3).
* Returns 0 on success, non-zero on error (caller logs and skips).
/* ha_apply_single_entry: durably persist one decoded wal_catchup_entry_t
* sub-entry (name retained for history; despite the name it no longer
* materializes entity state here — see ha_standby_advance_committed_and_apply,
* 2026-07-10). Returns 0 on success, non-zero on error (caller logs and skips).
*
* raft-ha-core-safety task 6.2 — TERM-BLIND DIRECT-PERSIST BYPASS, SAFE-BY-
* GUARD (documented, not structurally retired): this function does NOT go
@ -2229,32 +2420,24 @@ static int ha_apply_single_entry(persist_t *p, wal_catchup_entry_t *e)
return prc;
}
/* bug-1416 (raft-apply-decoupling task 7.2 round 2): defer the in-memory
* entity apply while this node's OWN boot-time persist_recover() replay+
* verification pass is still running. The entry is already durably
* persisted above (persist_wal_write_replicated, fsync'd) — only the
* ACK and applied-lsn advancement depend on that, per the persist-
* before-apply-before-ACK contract this function's own header comment
* documents, and bug-1043's precedent of decoupling durability/ACK from
* entity materialization. Applying here too, concurrently with
* replay_wal()'s single-threaded, unlocked bookkeeping (last_applied_lsn)
* and the live job table it reads, can produce a false APPLY_DIVERGENT
* verdict on a perfectly healthy boot (confirmed live). No entry is
* lost: this node's own in-progress replay_wal() pass will pick it up
* directly off disk if it lands within that pass's scan, and
* persist_recover()'s own deferred catch-up kick (persist_apply_pump_
* kick, issued right before it returns) covers it unconditionally
* either way, since apply_pump_read_entry() does a fresh on-disk lookup
* every call rather than trusting any earlier directory listing. */
if (!persist_recovery_complete(p))
return 0;
/* Logical entity restore only. persist_wal_write_replicated (above) already
* wrote the entry and owns the on-disk CRC chain + current_lsn — so we pass
* the entry fields directly rather than fabricating a header. (bug-778:
* a synthetic crc=0 header here used to clobber p->last_crc.) */
return apply_wal_entry(p, (uint32_t)e->entry_type, e->lsn,
e->raw_payload, (uint32_t)e->raw_payload_len);
/* commit-then-apply on the standby (2026-07-10): entity materialization
* (restore_job/restore_job_state) does NOT happen here on receipt. This
* function's job ends at durable persist + ACK — the entry becomes
* in-memory state only once this node's OWN p->committed_lsn reaches it,
* driven by ha_standby_advance_committed_and_apply (heartbeat-learned
* commit position, clamped to this node's local durable tip) and applied
* through apply_committed_entry() by the same pump the primary's own
* online-apply and restart replay use. Applying eagerly here — before
* this node (or the sender) knows the entry reached cluster majority —
* could materialize a job the sender never actually committed: if a
* later (term,lsn) conflict truncate removes the WAL bytes (the sender
* lost leadership before reaching majority), nothing rolls back the
* already-applied entity mutation, leaving a permanent phantom job with
* no backing WAL entry (2026-07-10 controller-invariant audit finding).
* This mirrors bug-1043's own precedent of decoupling durability/ACK from
* entity materialization, extended one step further: apply now waits for
* COMMIT, not just local durability. */
return 0;
}
/* ha_send_wal_ack_reply: send WalAckReply {term, lsn, success=1} back to
@ -2329,32 +2512,18 @@ static int ha_apply_term_entry_via_library(ha_state_t *ha, persist_t *p,
* logOk and stalling commit. */
(void)before_term;
/* bug-1043: the library route (on_persist_entry → persist_wal_write_
* replicated) writes the entry to the WAL but does NOT apply it to the
* in-memory entity state (job/queue/worker tables). The legacy direct-
* apply path (ha_apply_single_entry) called apply_wal_entry after the
* persist; the term-safe library route dropped it. So a standby that
* caught up entries 8..N via replication held them on disk but NOT in the
* queryable job table — on promotion `joblist`/`expect-job` could not
* find a committed job (it IS on disk; walcompare passes), looking like a
* committed-job loss. Re-apply here, mirroring the legacy path. Idempotent
* on a re-stream (restore_job upserts by jobid). WAL_NOOP / WAL_CONFIG_
* MARKER carry no entity payload — apply_wal_entry no-ops on them.
*
* bug-1416 (raft-apply-decoupling task 7.2 round 2): gated on
* persist_recovery_complete() for the identical reason documented on
* ha_apply_single_entry's own matching guard above — defer the
* in-memory apply while this node's own boot-time persist_recover()
* pass is in progress, never the ACK/applied_lsn advance below (the
* entry is already durably persisted by on_persist_entry's
* persist_wal_write_replicated, invoked via ha_consume_append_entries
* above). This node's own in-progress replay, or the deferred
* catch-up kick persist_recover() issues before returning, applies
* it instead. */
if (persist_recovery_complete(p))
apply_wal_entry(p, (uint32_t)e->entry_type, e->lsn,
e->raw_payload, (uint32_t)e->raw_payload_len);
/* commit-then-apply on the standby (2026-07-10, supersedes the prior
* bug-1043 re-apply-here fix): entity materialization does NOT happen
* on receipt/persist. bug-1043's original problem — a caught-up
* standby held entries on disk but not in the queryable job table —
* is now solved by ha_standby_advance_committed_and_apply, driven by
* this node's heartbeat-learned commit position, applying through the
* SAME apply_committed_entry() seam the primary's online-apply and
* restart replay use. Applying here instead (right on receipt, before
* this entry is known to have reached cluster majority) risks a
* permanent phantom job: a later (term,lsn) conflict truncate removes
* the WAL bytes with no entity rollback (2026-07-10 controller-
* invariant audit finding). */
ha_advance_applied_lsn(ha, e->lsn);
ha_send_wal_ack_reply(ha, ep, c, e->lsn);
return 1;
@ -2386,6 +2555,13 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
if (c && c->ha.raft_c == c && entry->n_entries == 1 &&
entry->entries && entry->entries[0] && entry->entries[0]->term != 0) {
ha_apply_term_entry_via_library(ha, p, c, ep, entry->entries[0]);
/* 2026-07-10: a fresh entry just landed on disk -- kick the apply
* pump against the already-known commit position so a standby mid
* catch-up doesn't wait for the next heartbeat tick to materialize
* entries it's already known to have committed. */
ha_standby_advance_committed_and_apply(
c, atomic_load_explicit(&ha->last_known_committed_lsn,
memory_order_acquire));
return 0;
}
@ -2461,6 +2637,15 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
if (had_good)
ha_advance_applied_lsn(ha, last_good_lsn);
/* 2026-07-10: kick the apply pump against the already-known commit
* position so a batch catch-up materializes promptly instead of
* waiting for the next heartbeat tick (see the single-entry-term-safe
* branch above for the full rationale). */
if (had_good)
ha_standby_advance_committed_and_apply(
c, atomic_load_explicit(&ha->last_known_committed_lsn,
memory_order_acquire));
return 0;
}
@ -2499,6 +2684,12 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
/* Task 3.1: ACK the successfully applied single entry */
ha_send_wal_ack_reply(ha, ep, c, entry->lsn);
/* 2026-07-10: kick the apply pump against the already-known commit
* position (see the batch-mode branch above for the full rationale). */
ha_standby_advance_committed_and_apply(
c, atomic_load_explicit(&ha->last_known_committed_lsn,
memory_order_acquire));
return 0;
}
@ -3560,6 +3751,7 @@ int ha_dispatch_gate(ha_state_t *ha, int opcode)
if (opcode == OPCode_requestVote ||
opcode == OPCode_preVoteRequest ||
opcode == OPCode_probeConfigTag ||
opcode == OPCode_heartbeat ||
opcode == OPCode_hello ||
opcode == OPCode_nop ||
opcode == OPCode_replConfigPush ||
@ -6806,6 +6998,28 @@ int ha_prevote_round_timeout_fn(void *arg)
{
prevote_round_ctx_t *ctx = (prevote_round_ctx_t *)arg;
/* bug (2026-07-10 chaotic-soak ASAN finding): clear s_prevote_round_ctx
* for THIS ctx BEFORE freeing it, unconditionally -- regardless of which
* branch below runs. The prior code only cleared the static pointer
* inside the "round still in progress" branch; when pre_vote_in_progress
* was ALREADY false here (the round was decided via a reply arriving on
* a different path, one that doesn't itself clear this pointer) this
* function fell straight through to free(ctx) below with
* s_prevote_round_ctx still pointing at the about-to-be-freed memory.
* The NEXT PreVote round's ha_start_prevote_async then calls
* ha_prevote_cancel_round_timeout() defensively before arming its own
* timer, reads that dangling pointer, and writes through it --
* heap-use-after-free (confirmed via ASAN on a live chaotic HA soak,
* both crashing nodes wedged with an identical ha.c:6712 UAF). CAS so a
* newer round's context (if one has already replaced this one by the
* time this delaymgr task runs) is never incorrectly cleared. */
{
prevote_round_ctx_t *expected = ctx;
atomic_compare_exchange_strong_explicit(
&s_prevote_round_ctx, &expected, NULL,
memory_order_acq_rel, memory_order_relaxed);
}
if (atomic_load_explicit(&ctx->cancelled, memory_order_acquire)) {
#ifndef XCE_TESTING
free(ctx);
@ -6819,7 +7033,6 @@ int ha_prevote_round_timeout_fn(void *arg)
/* Only act if the round is still in progress (not already decided). */
if (ha->raft.pre_vote_in_progress) {
ha->raft.pre_vote_in_progress = 0;
atomic_store_explicit(&s_prevote_round_ctx, NULL, memory_order_release);
WARN("HA: PreVote round timed out before a decision — treating as LOSS, "
"staying CTRL_STANDBY; re-arming liveness watchdog (Req 5.4)");
ha_prevote_rearm_liveness(ha, c);
@ -7335,6 +7548,61 @@ typedef struct {
*/
static liveness_poll_ctx_t * _Atomic s_liveness_ctx = NULL;
/* rdhup_failover_vtask_entry / ha_dispatch_rdhup_failover: offload
* ha_on_primary_ctrl_fd_rdhup (which chains into ha_start_election_async ->
* raft_start_election -> on_persist -> ha_persist_term's synchronous fsync)
* off the delaymgr thread onto a VTASK_LIVE_BLOCK vtask, mirroring
* ha_dispatch_election_start. This is the real, common production trigger for
* a standby's failover election (heartbeat-timeout watchdog) -- delaymgr must
* only detect the timeout and hand off, never run the election itself. */
typedef struct {
ha_state_t *ha;
cluster_t *c;
} rdhup_failover_dispatch_t;
static int rdhup_failover_vtask_entry(vtask_t *task)
{
rdhup_failover_dispatch_t *d =
(rdhup_failover_dispatch_t *)task->data->u.generic;
ha_on_primary_ctrl_fd_rdhup(d->ha, d->c);
free(d);
return 0;
}
static void ha_dispatch_rdhup_failover(ha_state_t *ha, cluster_t *c)
{
vtaskmgr_t *pool = (vtaskmgr_t *)ha->pool;
vtask_data_t *data = calloc(1, sizeof(*data));
vtask_t *task = vtask_alloc();
rdhup_failover_dispatch_t *d = calloc(1, sizeof(*d));
if (pool == NULL || data == NULL || task == NULL || d == NULL) {
WARN("ha_dispatch_rdhup_failover: vtask unavailable (pool=%p data=%p "
"task=%p d=%p) — failing over inline (stay alive)",
(void *)pool, (void *)data, (void *)task, (void *)d);
free(data);
free(task);
free(d);
ha_on_primary_ctrl_fd_rdhup(ha, c);
return;
}
d->ha = ha;
d->c = c;
data->vtype = VTASK_GENERIC;
data->u.generic = d;
vtask_init(task, (vtask_func_t)rdhup_failover_vtask_entry, data);
if (vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK) < 0) {
WARN("ha_dispatch_rdhup_failover: vtask enqueue failed — failing over "
"inline (stay alive)");
free(data);
free(task);
free(d);
ha_on_primary_ctrl_fd_rdhup(ha, c);
}
}
/*
* liveness_poll_fn: delaymgr task replacing hb_timeout_thread.
*
@ -7384,7 +7652,7 @@ static int liveness_poll_fn(void *arg)
/* Silence exceeded timeout — trigger failover */
WARN("HA: liveness poll: no primary activity for %lus (timeout=%us) → failover",
(unsigned long)elapsed, timeout_s);
ha_on_primary_ctrl_fd_rdhup(ha, c);
ha_dispatch_rdhup_failover(ha, c);
/* Do NOT re-arm — failover handles re-election */
return 0;
}
@ -7555,6 +7823,56 @@ void ha_observe_heartbeat_committed(cluster_t *c, uint64_t committed_lsn,
committed_lsn, committed_config_tag, cur);
}
/*
* ha_standby_advance_committed_and_apply (2026-07-10, controller-invariant
* audit finding — commit/apply decoupling extended to the standby): drive
* this node's OWN p->committed_lsn from a learned primary committed position,
* clamped to what this node has actually durably received, then kick the
* SAME apply pump (persist_apply_pump_kick -> apply_committed_entry()) the
* primary's own online-apply and restart replay use.
*
* This is the missing half of the standby ingest path's commit-then-apply
* story: ha_apply_single_entry / ha_apply_term_entry_via_library persist +
* ACK a replicated entry unconditionally (required for durability and the
* primary's commit majority) but no longer apply it — entity materialization
* now waits for THIS function to observe that the entry is actually
* committed, never merely durable-on-receipt. Safe against a lagging
* standby: the clamp to local_tip means committed_lsn here can never run
* ahead of what this node has itself received, so persist_apply_pump_run's
* apply_pump_read_entry can never be asked to read an LSN this node doesn't
* have on disk yet.
*
* Race-safe against a CONCURRENT writer (2026-07-10, controller-invariant-
* audit finding): this function is gated to non-primary roles by its
* callers, but raft_cb_become_leader's own persist_promotion_advance_committed
* call happens BEFORE role flips to CTRL_PRIMARY (ha.c, the role store sits
* ~170 lines after the advance call) -- so a still-in-flight WAL catch-up
* entry (this function is also called from ha_apply_wal_catchup_entry's exit
* points) can legitimately run on another thread during that window, racing
* this function against the promotion path's own advance. Both now go
* through persist_committed_lsn_advance_atomic's CAS loop, which is safe
* against that race by construction (never loses an update, never
* regresses) -- no shared lock needed across the two call sites.
*/
static void ha_standby_advance_committed_and_apply(cluster_t *c,
uint64_t learned_committed_lsn)
{
if (!c || learned_committed_lsn == 0)
return;
persist_t *p = &c->persist;
pthread_mutex_lock(&p->lock);
uint64_t local_tip = p->current_lsn > 0 ? p->current_lsn - 1 : 0;
pthread_mutex_unlock(&p->lock);
uint64_t target = learned_committed_lsn < local_tip
? learned_committed_lsn : local_tip;
persist_committed_lsn_advance_atomic(p, target);
persist_apply_pump_kick(p, c->ha.pool);
}
/*
* ha_consume_heartbeat_committed: standby OP_nop consume branch — the
* non-primary gate (Req 3.4) + the stale-term guard (Req 3.2), then the
@ -7612,6 +7930,11 @@ void ha_consume_heartbeat_committed(cluster_t *c, const heartbeat_msg_t *hb)
* non-advancing LSN no-ops inside the observe (Req 4.2). */
ha_observe_heartbeat_committed(c, hb->committed_lsn,
hb->committed_config_tag);
/* 2026-07-10: drive this node's own committed_lsn + apply pump from the
* position just learned (see ha_standby_advance_committed_and_apply) —
* the commit-then-apply half of the standby ingest path. */
ha_standby_advance_committed_and_apply(c, hb->committed_lsn);
}
void ha_disarm_heartbeat_timeout_vtask(ha_state_t *ha)
@ -8326,6 +8649,28 @@ void ha_start_primary_heartbeat(cluster_t *c)
return;
}
/* 2026-07-10 (chaotic-soak finding): fire the FIRST heartbeat synchronously,
* right now, instead of only arming the periodic delay_task_ms below.
* delay_task_ms's first invocation fires only AFTER interval_ms elapses
* (it is a periodic timer, not "run now then repeat") -- so despite this
* function's own long-standing comment/doc ("start primary heartbeat
* immediately so standbys receive NOP"), a freshly-promoted leader was
* actually waiting a full heartbeat_interval_ms (default 3000ms) before a
* reconnected/catching-up standby ever got a heartbeat to reply to. That
* reply (stateReport) is what lets the primary learn the standby's true
* applied_lsn (ha_process_standby_state) and start catch-up streaming --
* which gates the per-entry majority-ack that advances committed_lsn past
* this election's conservative safe_committed_lsn floor. A live chaotic
* soak showed jobs above safe_lsn (legitimately majority-committed before
* the old primary died, per Leader Completeness) invisible via
* expect-job/joblist for the full 3s wait even after the
* persist_promotion_advance_committed fix above -- because nothing
* corroborated them was ever ASKED for that long. primary_heartbeat_fn
* only performs non-blocking async sends (xce_ha_heartbeat_req), so
* calling it inline here carries the same non-blocking-I/O contract as
* the rest of promotion and needs no vtask/thread-role change. */
primary_heartbeat_fn(c);
/* runcount=-1 (infinite): delaymgr re-enqueues automatically.
* The role check in primary_heartbeat_fn no-ops on demotion. */
int rc = delay_task_ms(&c->delaymgr, interval_ms, -1,

19
src/controller/ha.h
View file

@ -227,6 +227,25 @@ typedef struct {
* once per edge (no log storms across
* repeated dispatch-gate rejections).
* 0 at init (cleared by ha_init memset). */
/* ── New field (2026-07-10, ha-convergence-recovery Req 1.4 supersession) ── */
_Atomic int self_restart_pending; /* Set when ha_self_restart's execv()
* fails (a demoted ex-primary could not
* re-exec to discard provisional state).
* Checked unconditionally, alongside
* apply_divergent, by raft_cb_become_
* leader and promotion_fence_timeout_fn
* -- NEVER live-cleared (no clear site
* exists anywhere): a periodic re-check
* like promotion_fence_timeout_fn's
* catch-up clear must NOT lift this, since
* this node's WAL/durability state can
* look perfectly healthy while its
* in-memory state is still the stale
* pre-demotion primary-era state the
* restart exists to discard. Clears only
* via a fresh process (a successful
* retry replaces the image entirely).
* 0 at init (cleared by ha_init memset). */
/* ── New field added by ha-commit-durability spec (Task 4.4) ──────── */
void * _Atomic promotion_fence_ctx; /* promotion_fence_ctx_t* for the
* bounded-window catch-up timer (Req

102
src/controller/ha_convergence.c
View file

@ -65,19 +65,73 @@ void (*ha_quorum_transition_log_hook)(int lost, int connected,
int required) = NULL;
#endif
#ifndef XCE_TESTING
#define HA_SELF_RESTART_RETRY_MS 5000
static int ha_self_restart_retry_fn(void *arg);
/* ha_self_restart_enter_fence_and_retry: the non-abort response to a failed
* re-exec (2026-07-10, supersedes ha-convergence-recovery Req 1.4's original
* "terminate fatally" text — see requirements.md's superseding note). Sets
* the sticky, never-live-cleared ha->self_restart_pending flag (checked
* unconditionally by raft_cb_become_leader / promotion_fence_timeout_fn
* alongside apply_divergent, ha.c) so this node can never become primary
* while it still carries its stale pre-demotion in-memory state, then
* schedules a bounded periodic retry of the re-exec. This satisfies the same
* safety property Req 1.4 protects — a divergent ex-primary never serves
* authoritatively — via the project-wide fail-loud invariant (a controller
* MUST NEVER self-terminate as a designed response to an operational
* condition; CLAUDE.md "Process Lifecycle") instead of abort(): staying
* alive and retrying self-heals automatically once the transient condition
* (ENOMEM, EMFILE, a permissions change mid-upgrade) clears, whereas a dead
* process needs manual admin recovery and degrades quorum meanwhile. */
static void ha_self_restart_enter_fence_and_retry(cluster_t *c)
{
atomic_store_explicit(&c->ha.self_restart_pending, 1, memory_order_release);
if (c->delaymgr.running) {
int rc = delay_task_ms(&c->delaymgr, HA_SELF_RESTART_RETRY_MS, 1,
ha_self_restart_retry_fn, c);
if (rc < 0)
WARN("HA self-restart: retry scheduling failed: %d — node stays "
"fenced (non-promotable) with no further automatic retry "
"attempt", rc);
} else {
WARN("HA self-restart: delaymgr not running — node stays fenced "
"(non-promotable) with no automatic retry");
}
}
/* ha_self_restart_retry_fn: delaymgr task body — bounded periodic retry of
* the re-exec. A successful retry replaces the process image entirely (this
* function never returns in that case); a repeated failure re-arms itself
* via ha_self_restart's own failure path. */
static int ha_self_restart_retry_fn(void *arg)
{
cluster_t *c = (cluster_t *)arg;
if (!c)
return -EINVAL;
return ha_self_restart(c);
}
#endif
/*
* ha_self_restart — the truncate-and-restart primitive's restart half.
*
* Production (Req 1.3, 1.4): flush logs, then execv() the saved argv so the
* Production (Req 1.3): flush logs, then execv() the saved argv so the
* controller rebuilds in-memory state purely from the surviving (committed)
* WAL prefix via boot recovery. If exec cannot be performed, terminate
* FATALLY (abort) — a demoted ex-primary must NEVER continue running with
* potentially divergent state. NEVER returns in production.
* WAL prefix via boot recovery. NEVER returns on success (execv replaces the
* process image). If exec cannot be performed, do NOT abort (see
* ha_self_restart_enter_fence_and_retry above) — enter a permanent
* non-promotable fence and keep retrying periodically, per the project's
* never-self-terminate invariant.
*
* Test (Req 1.6): under XCE_TESTING invoke ha_restart_hook(c) instead of
* execv and return its result on the success path; a non-zero hook return
* models an exec failure and routes to ha_fatal_hook in place of abort, so a
* test binary can assert the FATAL path was taken without dying (Req 1.4).
* models an exec failure and routes to ha_fatal_hook (retained as the test
* seam name — it still marks "restart could not be performed", now backed by
* the fence-and-retry response in production rather than abort), so a test
* binary can assert that failure path was taken without dying.
*/
int ha_self_restart(cluster_t *c)
{
@ -88,20 +142,26 @@ int ha_self_restart(cluster_t *c)
rc = ha_restart_hook(c);
if (rc != 0 || ha_restart_hook == NULL) {
/* Restart could not be performed → FATAL contract (Req 1.4). */
/* Restart could not be performed — same non-abort response as
* production (Req 1.6 mirrors ha_self_restart_enter_fence_and_retry's
* flag-setting without the real delaymgr retry, since a unit fixture
* has no live delaymgr to retry on). */
if (c)
atomic_store_explicit(&c->ha.self_restart_pending, 1,
memory_order_release);
if (ha_fatal_hook != NULL)
ha_fatal_hook("ha_self_restart: restart seam failed (FATAL)");
ha_fatal_hook("ha_self_restart: restart seam failed — fenced "
"non-promotable, staying alive");
return rc != 0 ? rc : -1;
}
return 0;
#else
(void)c;
char **argv = controller_saved_argv();
if (argv == NULL || argv[0] == NULL) {
ERROR("HA self-restart: no saved argv — cannot re-exec (FATAL)");
tlog_exit(); /* flush pending log buffer before dying */
abort();
ERROR("HA self-restart: no saved argv — cannot re-exec; entering "
"non-promotable fence and retrying periodically (stay alive)");
ha_self_restart_enter_fence_and_retry(c);
return -1;
}
INFO("HA self-restart: re-executing %s to rebuild from committed state",
@ -110,12 +170,16 @@ int ha_self_restart(cluster_t *c)
execv(argv[0], argv);
/* execv only returns on failure — FATAL (Req 1.4): never continue serving
* as a divergent ex-primary. */
ERROR("HA self-restart: execv(%s) failed: %d — aborting (FATAL)",
argv[0], errno);
abort();
return -1; /* unreachable */
/* execv only returns on failure. A demoted ex-primary must never continue
* serving with potentially divergent state (ha-convergence-recovery Req
* 1.4) — but per the project-wide fail-loud invariant (never self-
* terminate as a designed response to an operational condition), the
* correct response is a permanent non-promotable fence that stays alive
* and keeps retrying, not abort(). */
ERROR("HA self-restart: execv(%s) failed: %d — entering non-promotable "
"fence and retrying periodically (stay alive)", argv[0], errno);
ha_self_restart_enter_fence_and_retry(c);
return -1;
#endif
}

45
src/controller/join.c
View file

@ -5,7 +5,6 @@
#include <unistd.h>
#include "cluster.h"
#include "config_repl.h"
#include "ha.h"
#include "persist.h"
#include "xce/cluster_def.h"
@ -172,40 +171,36 @@ static int handle_scheduler_join(cluster_t* c, endpoint_t* ep,
return rc;
}
/*
* handle_controller_join: reject a peer-controller join request.
*
* Controller-to-controller connection establishment is NOT this path — per
* ha-async-io.md's Connection Ownership, it is always primary-initiated
* outbound (primary_heartbeat_fn -> on_standby_connected -> ctrlhello ->
* repl_add_standby), with the resulting endpoint owned by
* vote_ep_err_handler / standby_ep_err_handler_primary. This arm used to
* accept an INBOUND role=HostRole_controller join and store it directly
* into standby_endpoints, bypassing both documented owners and their
* err-handler-driven eviction — a live ownership-model violation, even
* though no current sender in the tree ever builds such a request
* (xce_join_req structurally rejects role==HostRole_controller). Reject
* outright rather than leave the bypass reachable from the wire.
*/
static int handle_controller_join(cluster_t* c, endpoint_t* ep,
xce_request_t* request)
{
xce_reply_t reply;
init_reply_by_req(&reply, request);
reply.status = 0;
reply.status = -E_XCE_BAD_ROLE;
reply.data_which = RequestReply_data_nodata;
xce_reply(ep, c->myhost, &reply);
/* Register this endpoint for OP_upgradeConfig fan-out to standbys */
if (c->ha.standby_endpoints) {
ep_private_t *priv = (ep_private_t *)ep->data;
if (priv)
priv->role = HostRole_controller;
WARN("%s rejecting controller-role join from %s — controller peering is "
"primary-initiated only, never via join",
__func__, request->from ? request->from->name : "?");
pthread_mutex_lock(&c->ha.lock);
zarray_store(c->ha.standby_endpoints, ep);
pthread_mutex_unlock(&c->ha.lock);
INFO("controller %s joined — %d standby(s) tracked",
request->from ? request->from->name : "?",
c->ha.standby_endpoints->count);
/* Immediately push current config to the newly-joined standby so it
* doesn't have to wait for the next scheduled push. Transport-only
* (no barrier wait — task 5.2 remediation r1): this is a best-effort
* catch-up stream; convergence is owned by the OP_replConfig
* anti-entropy pull and the heartbeat decision loop. */
if (ha_is_primary(&c->ha))
push_config_transport_only(c, NULL, 0, NULL, 0, NULL);
}
return 0;
return -E_XCE_BAD_ROLE;
}
static int handle_apiserver_join(cluster_t* c, endpoint_t* ep,

124
src/controller/persist.c
View file

@ -3441,6 +3441,55 @@ void persist_apply_pump_kick(persist_t* p, void* pool_arg)
}
}
/* persist_apply_pump_run_sync (2026-07-10, chaotic-soak regression fix,
* ROUND 2): synchronously drain the apply pump on THIS thread, GUARANTEEING
* last_applied_lsn reaches committed_lsn (as observed at entry) before
* returning -- regardless of what any concurrent caller is doing.
*
* Round 1 of this fix CAS-claimed apply_pump_scheduled and, on losing that
* claim to a concurrent caller (e.g. a heartbeat-driven kick that fired
* moments before promotion), just returned -- betting the OTHER caller's
* async vtask-dispatched drain would finish in time. A second live chaotic
* soak proved that bet wrong: job loss persisted even with the environmental
* fdatasync noise (bug-924) independently confirmed and eliminated, so this
* was a genuine, separate bug in the fix itself, not noise. The caller here
* (promotion) cannot outsource its guarantee to a drain it doesn't control
* the completion of.
*
* Fix: do NOT use the CAS-coalescing flag as a "someone else has it, I'm
* done" signal -- call persist_apply_pump_run() directly in a loop until
* THIS call observes last_applied_lsn caught up to committed_lsn. Multiple
* concurrent callers (this one and any vtask-dispatched persist_apply_pump_
* kick) safely interleave via persist_apply_pump_run's own internal
* apply_pump_lock and apply_committed_entry's per-LSN idempotency -- at
* worst a redundant no-op pass, never lost or duplicated work. Stops (does
* not busy-loop forever) on a genuine read failure, exactly mirroring
* apply_pump_drain's own error handling: persist_apply_pump_run() already
* logs loudly and leaves last_applied_lsn unadvanced on that path. */
void persist_apply_pump_run_sync(persist_t* p)
{
if (!p || !p->cluster)
return;
if (!atomic_load_explicit(&p->recovery_complete, memory_order_acquire))
return;
for (;;) {
uint64_t committed = atomic_load_explicit(&p->committed_lsn,
memory_order_acquire);
uint64_t applied = atomic_load_explicit(&p->last_applied_lsn,
memory_order_relaxed);
if (applied >= committed)
return; /* caught up -- by us or a concurrent drain */
int rc = persist_apply_pump_run(p);
if (rc < 0)
return; /* already logged loudly; do not busy-loop on the same
* unresolved entry (mirrors apply_pump_drain) */
/* Loop: re-check in case committed_lsn advanced further mid-walk, or
* a concurrent drain made partial progress this pass didn't see. */
}
}
/* Read WAL segment files in LSN order; replay entries with lsn > from_lsn.
*
* Segment filename format: seg-<start_lsn_hex16>.wal
@ -5312,6 +5361,81 @@ int persist_truncate_wal_above(persist_t* p, uint64_t cutoff_lsn)
return rc;
}
/* persist_committed_lsn_advance_atomic (2026-07-10, controller-invariant-audit
* finding): race-safe monotonic-max advance of p->committed_lsn via a CAS
* retry loop. Replaces the naive "load; if target>cur store" pattern, which
* can lose an update if two writers race: persist_promotion_advance_committed
* (below) and ha_standby_advance_committed_and_apply (ha.c) can both run
* during the promotion window before role flips to CTRL_PRIMARY (a
* still-in-flight WAL catch-up entry processed on another thread at the same
* instant as this node's own election completing). No lock required -- CAS
* only succeeds if the value hasn't changed since the read, so a losing
* racer retries with the freshly-observed value and can never clobber a
* larger value with a smaller one. */
uint64_t persist_committed_lsn_advance_atomic(persist_t* p, uint64_t target)
{
uint64_t cur = atomic_load_explicit(&p->committed_lsn, memory_order_acquire);
while (target > cur) {
if (atomic_compare_exchange_weak_explicit(&p->committed_lsn, &cur, target,
memory_order_release,
memory_order_acquire))
return target;
/* cur was refreshed to the current value by the failed CAS; retry. */
}
return cur;
}
/* persist_promotion_advance_committed (2026-07-10, promotion apply-lag fix):
* unconditionally advance p->committed_lsn toward `safe_lsn` at promotion
* time. persist_truncate_wal_above only sets committed_lsn on the branches
* of truncate_wal_above_locked that perform a REAL truncation (partial
* truncate, or the full-segment-purge demotion-convergence case); its no-op
* early return (nothing above cutoff_lsn to remove -- the common case for a
* newly-elected leader whose own log already covers safe_lsn) leaves
* committed_lsn untouched at its pre-election value. Without this call,
* committed_lsn -- and therefore apply, which is commit-gated
* (raft-apply-decoupling) -- stalls until the normal per-entry ack-driven
* commit path organically catches it up, which can take seconds while
* surviving standbys reconnect and re-ACK. During that gap a job already
* committed on a majority is invisible via expect-job/joblist on the very
* primary that is authoritative for it (bug-828 class).
*
* DESIGN NOTE -- why this function never forces a synchronous flush (round 3,
* 2026-07-10, controller-invariant-audit): an earlier round of this fix
* called flusher_flush_once(p) here when durable_lsn lagged safe_lsn, to
* force this node's own fsync to catch up immediately rather than waiting on
* fsync_worker's cadence. That was a real bug-1198-CLASS thread-role
* violation: the real multi-node winning-vote-reply path reaches
* raft_cb_become_leader (and so this function) via peer_ctrl_classify's
* VTASK_LIVENESS dispatch (ha.c ~5935-5940) -- a task class the design
* DOCUMENTS as "never admission-gated" / never expected to block (see the
* bug-1198 hazard box a few hundred lines above peer_ctrl_classify's
* definition, which closed the EXACT same class of bug for
* HA_REPL_SEND_SNAPSHOT's persist_checkpoint). A blocking fdatasync here
* would reintroduce that closed bug class on the pool's liveness-reserved
* workers. The visibility gap the force-flush was trying to close is instead
* closed non-blockingly by ha_start_primary_heartbeat sending its first
* heartbeat synchronously at promotion (bug-1432) -- that lets a reconnecting
* standby's real position get learned, and the entry gets committed via the
* normal ack-driven commit_advance_locked path, within one network RTT
* instead of waiting a full heartbeat interval. A live chaotic soak (10
* rounds) confirmed durable-capped-only + immediate-heartbeat together reach
* committed_job_loss=0 with zero blocking calls added to this function.
*
* Durable-capped (never advances committed_lsn past this node's own
* durable_lsn -- see persistence.md's "acknowledged implies durable on a
* majority" core invariant) and monotonic + race-safe via
* persist_committed_lsn_advance_atomic. */
void persist_promotion_advance_committed(persist_t* p, uint64_t safe_lsn)
{
if (!p)
return;
uint64_t durable = atomic_load_explicit(&p->durable_lsn,
memory_order_acquire);
uint64_t target = safe_lsn < durable ? safe_lsn : durable;
persist_committed_lsn_advance_atomic(p, target);
}
/* Lock-held core of persist_truncate_wal_above (see the forward declaration and
* the conflict-guard reuse near persist_wal_write_replicated). Caller MUST hold
* p->lock; this function never touches the lock so it is reusable from another

232
src/controller/testsuite/test_apply_pump.c
View file

@ -79,6 +79,7 @@
#include "cluster.h"
#include "commit.h"
#include "controller.capnp.h"
#include "ha.h"
#include "hook.h"
#include "jobj.h"
#include "persist.h"
@ -1058,6 +1059,235 @@ static void test_apply_pump_late_pending_apply_registration_self_heals(void)
"PASS\n");
}
/* ── 6. Standby-ingest commit-then-apply (2026-07-10 fix) ────────────────
*
* Regression coverage for ha.c's standby WAL-replication ingest path.
* Before this fix, ha_apply_single_entry / ha_apply_term_entry_via_library
* called apply_wal_entry() directly the instant a replicated entry was
* durably persisted — materializing the job into the job table regardless
* of whether the sending primary ever reached cluster-wide majority
* commit. If that primary died before reaching majority, a later
* (term,lsn) conflict truncate would remove the WAL bytes with no
* rollback of the already-materialized entity, leaving a permanent
* phantom job.
*
* The fix decouples durability from materialization: ha_apply_single_entry
* / ha_apply_term_entry_via_library now only persist + ACK. Entity
* materialization happens later, only once THIS standby's own
* persist_t.committed_lsn actually covers the entry — driven by
* ha_standby_advance_committed_and_apply (clamped to
* min(learned_committed_lsn, local durable tip)) via
* persist_apply_pump_kick -> persist_apply_pump_run -> apply_committed_entry
* (the same seam the primary's own online-apply/replay use).
*
* These tests drive the REAL standby ingest seam — ha_apply_wal_catchup_entry,
* the exact function every incoming WAL replication message routes through —
* directly against this file's real jobmgr/persist fixture, so "not yet
* visible" / "now visible" is observed via tap_find_job, the same job-table
* lookup every other test in this file uses, not a synthetic stand-in.
* g_tap_c.ha.raft_c is never set to &g_tap_c in this fixture, so
* ha_apply_wal_catchup_entry always takes the documented term-blind
* direct-persist bypass (ha_apply_single_entry) regardless of the entries'
* term — the same reachable-and-safe-by-guard path
* test_6_2_real_senders_never_produce_termblind_shape (test_ha_core_safety.c)
* documents. */
/* Build one WAL_JOB_SUBMIT sub-entry for jobid at lsn, via the same capnp
* encoding tap_encode_job already uses for the live command-entry test
* above. Caller owns *payload_out (free() after the entry is no longer
* needed) and must keep it alive across the ha_apply_wal_catchup_entry
* call — persist_wal_write_replicated copies the bytes to disk, so the
* pointer's lifetime need not outlive the call, but freeing before the
* call would read stale data mid-encode. */
static wal_catchup_entry_t tas_build_submit_entry(uint64_t lsn, uint64_t jobid,
uint32_t state,
uint8_t** payload_out)
{
memset(&g_tap_seed_submit, 0, sizeof(g_tap_seed_submit));
g_tap_seed_submit.queue = (char*)"q";
g_tap_seed_submit.user = (char*)"tester";
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = jobid;
jdef.state = state;
jdef.submit = &g_tap_seed_submit;
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tap_encode_job(&jdef, &payload, &plen));
wal_catchup_entry_t sub;
memset(&sub, 0, sizeof(sub));
sub.lsn = lsn;
sub.entry_type = WAL_JOB_SUBMIT;
sub.raw_payload = payload;
sub.raw_payload_len = (uint32_t)plen;
/* term == 0 deliberately: see the file-header comment above — this
* fixture always exercises the term-blind direct-persist bypass. */
*payload_out = payload;
return sub;
}
/* (a) Positive case: persisted-but-uncommitted stays invisible; a
* heartbeat-learned commit position materializes it. */
static void test_standby_apply_deferred_until_committed_then_materializes(
void)
{
tap_setup();
const uint64_t lsn = 1; /* first entry in a fresh persist_t */
const uint64_t jobid = 960;
uint8_t* payload = NULL;
wal_catchup_entry_t sub = tas_build_submit_entry(lsn, jobid, JOB_STAT_PEND,
&payload);
wal_catchup_entry_t* sub_arr[1] = { &sub };
wal_catchup_entry_t top;
memset(&top, 0, sizeof(top));
top.n_entries = 1;
top.entries = sub_arr;
/* Ingest via the REAL standby seam: persist + ACK only, no apply. */
int rc = ha_apply_wal_catchup_entry(&g_tap_c.ha, &g_tap_c.persist, &top,
NULL, &g_tap_c);
uint64_t current_lsn_after_persist = g_tap_c.persist.current_lsn;
uint64_t applied_lsn_after_persist =
atomic_load(&g_tap_c.ha.applied_lsn);
uint64_t committed_after_persist = atomic_load(&g_tap_c.persist.committed_lsn);
job_t* j_before_commit = tap_find_job(jobid);
/* Learn the commit position the way a real heartbeat does -- this is
* what drives the commit-then-apply wiring (ha_consume_heartbeat_committed
* -> ha_standby_advance_committed_and_apply -> persist_apply_pump_kick). */
atomic_store(&g_tap_c.ha.role, (int32_t)CTRL_STANDBY);
heartbeat_msg_t hb;
memset(&hb, 0, sizeof(hb));
hb.term = 0;
hb.committed_lsn = lsn;
ha_consume_heartbeat_committed(&g_tap_c, &hb);
/* See tap_wait_applied's doc comment: the walk may complete on a
* different thread than this call. */
tap_wait_applied(lsn, 2000);
uint64_t committed_after_hb = atomic_load(&g_tap_c.persist.committed_lsn);
job_t* j_after_commit = tap_find_job(jobid);
uint32_t j_state_after = j_after_commit ? j_after_commit->job.state : 0;
uint64_t last_applied_final = persist_last_applied_lsn(&g_tap_c.persist);
free(payload);
tap_teardown();
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn + 1, current_lsn_after_persist,
"the entry must be durably persisted by the standby ingest seam");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn, applied_lsn_after_persist,
"applied_lsn (the held/persisted watermark) advances on receipt -- "
"distinct from entity materialization");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, committed_after_persist,
"committed_lsn must NOT advance just because the entry was "
"persisted -- nothing has told this standby the entry reached "
"cluster majority yet");
TEST_ASSERT_NULL_MESSAGE(j_before_commit,
"REGRESSION GUARD: a replicated entry must not materialize into "
"the job table before this standby's own committed_lsn covers it "
"-- materializing here (the old eager-apply behavior) is exactly "
"the phantom-job hazard this fix closes");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn, committed_after_hb,
"the heartbeat-learned commit position must advance committed_lsn");
TEST_ASSERT_NOT_NULL_MESSAGE(j_after_commit,
"once committed_lsn covers the entry, the apply pump must "
"materialize it into the job table");
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_PEND, j_state_after);
TEST_ASSERT_EQUAL_UINT64(lsn, last_applied_final);
printf("test_standby_apply_deferred_until_committed_then_materializes: "
"PASS\n");
}
/* (b) Regression guard: an entry truncated away before ever being learned
* as committed must never leave a phantom job — even when a stale/racing
* heartbeat still advertises the old (now-truncated) committed position. */
static void test_standby_truncate_before_commit_leaves_no_phantom_job(void)
{
tap_setup();
const uint64_t lsn = 1;
const uint64_t jobid = 961;
const uint64_t cutoff_lsn = lsn - 1; /* truncate away lsn and above */
uint8_t* payload = NULL;
wal_catchup_entry_t sub = tas_build_submit_entry(lsn, jobid, JOB_STAT_PEND,
&payload);
wal_catchup_entry_t* sub_arr[1] = { &sub };
wal_catchup_entry_t top;
memset(&top, 0, sizeof(top));
top.n_entries = 1;
top.entries = sub_arr;
int rc = ha_apply_wal_catchup_entry(&g_tap_c.ha, &g_tap_c.persist, &top,
NULL, &g_tap_c);
job_t* j_before_truncate = tap_find_job(jobid);
/* The sending primary died before reaching majority: a new leader's
* conflict-truncate removes the entry before this standby ever learns
* it was committed. */
int trc = persist_truncate_wal_above(&g_tap_c.persist, cutoff_lsn);
uint64_t current_lsn_after_truncate = g_tap_c.persist.current_lsn;
/* A stale/racing heartbeat still carries the old (now-truncated)
* committed position -- ha_standby_advance_committed_and_apply's clamp
* to the local durable tip (current_lsn-1) must refuse to walk past
* what is actually on disk. */
atomic_store(&g_tap_c.ha.role, (int32_t)CTRL_STANDBY);
heartbeat_msg_t hb;
memset(&hb, 0, sizeof(hb));
hb.term = 0;
hb.committed_lsn = lsn;
ha_consume_heartbeat_committed(&g_tap_c, &hb);
/* There is nothing valid left to apply, so no walk will ever reach
* `lsn` -- a bounded settle (not a wait-for) is the correct shape here. */
usleep(50000);
uint64_t committed_after = atomic_load(&g_tap_c.persist.committed_lsn);
uint64_t last_applied_final = persist_last_applied_lsn(&g_tap_c.persist);
job_t* j_after = tap_find_job(jobid);
free(payload);
tap_teardown();
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NULL_MESSAGE(j_before_truncate,
"must not be materialized before commit (same guard as the "
"positive-case test)");
TEST_ASSERT_EQUAL_INT(0, trc);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(cutoff_lsn + 1, current_lsn_after_truncate,
"the entry must actually be gone from the WAL after truncation");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, committed_after,
"REGRESSION GUARD: the local-durable-tip clamp in "
"ha_standby_advance_committed_and_apply must refuse to advance "
"committed_lsn past a truncated-away entry, even when a stale "
"heartbeat still advertises it as committed");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, last_applied_final,
"nothing may be applied past the truncated tip");
TEST_ASSERT_NULL_MESSAGE(j_after,
"REGRESSION GUARD: no phantom job may appear -- the entry was "
"truncated before ever being committed, so it must never be "
"materialized");
printf("test_standby_truncate_before_commit_leaves_no_phantom_job: "
"PASS\n");
}
/* ── Test group entry point ───────────────────────────────────────────── */
void test_apply_pump(void)
@ -1068,5 +1298,7 @@ void test_apply_pump(void)
RUN_TEST(test_apply_pump_guard_does_not_break_job_ctrl_live_append);
RUN_TEST(test_apply_pump_fail_racing_finish_merges_against_current_state);
RUN_TEST(test_apply_pump_late_pending_apply_registration_self_heals);
RUN_TEST(test_standby_apply_deferred_until_committed_then_materializes);
RUN_TEST(test_standby_truncate_before_commit_leaves_no_phantom_job);
UNITY_END();
}