lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race

Browse source 
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).

bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.

Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
This commit is contained in:
Rongsong Shen 2026-07-11 12:16:28 +00:00
parent ea0ac81c92
commit 40f510fb14
25 changed files with 1617 additions and 226 deletions
Download patch file Download diff file Expand all files Collapse all files

102
src/controller/ha_convergence.c
View file

@ -65,19 +65,73 @@ void (*ha_quorum_transition_log_hook)(int lost, int connected,
int required) = NULL;
#endif
#ifndef XCE_TESTING
#define HA_SELF_RESTART_RETRY_MS 5000
static int ha_self_restart_retry_fn(void *arg);
/* ha_self_restart_enter_fence_and_retry: the non-abort response to a failed
* re-exec (2026-07-10, supersedes ha-convergence-recovery Req 1.4's original
* "terminate fatally" text — see requirements.md's superseding note). Sets
* the sticky, never-live-cleared ha->self_restart_pending flag (checked
* unconditionally by raft_cb_become_leader / promotion_fence_timeout_fn
* alongside apply_divergent, ha.c) so this node can never become primary
* while it still carries its stale pre-demotion in-memory state, then
* schedules a bounded periodic retry of the re-exec. This satisfies the same
* safety property Req 1.4 protects — a divergent ex-primary never serves
* authoritatively — via the project-wide fail-loud invariant (a controller
* MUST NEVER self-terminate as a designed response to an operational
* condition; CLAUDE.md "Process Lifecycle") instead of abort(): staying
* alive and retrying self-heals automatically once the transient condition
* (ENOMEM, EMFILE, a permissions change mid-upgrade) clears, whereas a dead
* process needs manual admin recovery and degrades quorum meanwhile. */
static void ha_self_restart_enter_fence_and_retry(cluster_t *c)
{
atomic_store_explicit(&c->ha.self_restart_pending, 1, memory_order_release);
if (c->delaymgr.running) {
int rc = delay_task_ms(&c->delaymgr, HA_SELF_RESTART_RETRY_MS, 1,
ha_self_restart_retry_fn, c);
if (rc < 0)
WARN("HA self-restart: retry scheduling failed: %d — node stays "
"fenced (non-promotable) with no further automatic retry "
"attempt", rc);
} else {
WARN("HA self-restart: delaymgr not running — node stays fenced "
"(non-promotable) with no automatic retry");
}
}
/* ha_self_restart_retry_fn: delaymgr task body — bounded periodic retry of
* the re-exec. A successful retry replaces the process image entirely (this
* function never returns in that case); a repeated failure re-arms itself
* via ha_self_restart's own failure path. */
static int ha_self_restart_retry_fn(void *arg)
{
cluster_t *c = (cluster_t *)arg;
if (!c)
return -EINVAL;
return ha_self_restart(c);
}
#endif
/*
* ha_self_restart — the truncate-and-restart primitive's restart half.
*
* Production (Req 1.3, 1.4): flush logs, then execv() the saved argv so the
* Production (Req 1.3): flush logs, then execv() the saved argv so the
* controller rebuilds in-memory state purely from the surviving (committed)
* WAL prefix via boot recovery. If exec cannot be performed, terminate
* FATALLY (abort) — a demoted ex-primary must NEVER continue running with
* potentially divergent state. NEVER returns in production.
* WAL prefix via boot recovery. NEVER returns on success (execv replaces the
* process image). If exec cannot be performed, do NOT abort (see
* ha_self_restart_enter_fence_and_retry above) — enter a permanent
* non-promotable fence and keep retrying periodically, per the project's
* never-self-terminate invariant.
*
* Test (Req 1.6): under XCE_TESTING invoke ha_restart_hook(c) instead of
* execv and return its result on the success path; a non-zero hook return
* models an exec failure and routes to ha_fatal_hook in place of abort, so a
* test binary can assert the FATAL path was taken without dying (Req 1.4).
* models an exec failure and routes to ha_fatal_hook (retained as the test
* seam name — it still marks "restart could not be performed", now backed by
* the fence-and-retry response in production rather than abort), so a test
* binary can assert that failure path was taken without dying.
*/
int ha_self_restart(cluster_t *c)
{
@ -88,20 +142,26 @@ int ha_self_restart(cluster_t *c)
rc = ha_restart_hook(c);
if (rc != 0 || ha_restart_hook == NULL) {
/* Restart could not be performed → FATAL contract (Req 1.4). */
/* Restart could not be performed — same non-abort response as
* production (Req 1.6 mirrors ha_self_restart_enter_fence_and_retry's
* flag-setting without the real delaymgr retry, since a unit fixture
* has no live delaymgr to retry on). */
if (c)
atomic_store_explicit(&c->ha.self_restart_pending, 1,
memory_order_release);
if (ha_fatal_hook != NULL)
ha_fatal_hook("ha_self_restart: restart seam failed (FATAL)");
ha_fatal_hook("ha_self_restart: restart seam failed — fenced "
"non-promotable, staying alive");
return rc != 0 ? rc : -1;
}
return 0;
#else
(void)c;
char **argv = controller_saved_argv();
if (argv == NULL || argv[0] == NULL) {
ERROR("HA self-restart: no saved argv — cannot re-exec (FATAL)");
tlog_exit(); /* flush pending log buffer before dying */
abort();
ERROR("HA self-restart: no saved argv — cannot re-exec; entering "
"non-promotable fence and retrying periodically (stay alive)");
ha_self_restart_enter_fence_and_retry(c);
return -1;
}
INFO("HA self-restart: re-executing %s to rebuild from committed state",
@ -110,12 +170,16 @@ int ha_self_restart(cluster_t *c)
execv(argv[0], argv);
/* execv only returns on failure — FATAL (Req 1.4): never continue serving
* as a divergent ex-primary. */
ERROR("HA self-restart: execv(%s) failed: %d — aborting (FATAL)",
argv[0], errno);
abort();
return -1; /* unreachable */
/* execv only returns on failure. A demoted ex-primary must never continue
* serving with potentially divergent state (ha-convergence-recovery Req
* 1.4) — but per the project-wide fail-loud invariant (never self-
* terminate as a designed response to an operational condition), the
* correct response is a permanent non-promotable fence that stays alive
* and keeps retrying, not abort(). */
ERROR("HA self-restart: execv(%s) failed: %d — entering non-promotable "
"fence and retrying periodically (stay alive)", argv[0], errno);
ha_self_restart_enter_fence_and_retry(c);
return -1;
#endif
}