lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): close standby election-freeze (bug-1562) + flusher retry storm (bug-1563)

Browse source 
bug-1562: ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c) returned
without re-arming the liveness watchdog when a debounced RDHUP landed after
the in-flight PreVote round had already concluded (e.g. a lone cold-booting
node whose first round loses in milliseconds with no peers up yet). That was
the last timer ever scheduled -- the node froze permanently in CTRL_STANDBY,
so a 3-node cluster with a long inter-node start stagger could never elect a
primary. Root-caused via a live local reproduction matching the reporting
user's exact timing (byte-for-byte match of their hung ctrl-1.log), fixed by
re-arming the watchdog in that branch (mirroring the function's own 5.2
branch), and confirmed end-to-end: the same scenario now reaches CTRL_PRIMARY
and full 3-node config convergence.

bug-1563: fsync_worker's retry loop (persist.c) only backs off when no work
is pending, but flusher_work_pending_locked() can't distinguish "fresh work
just landed" from "still stuck from the last failed fdatasync" -- a
sustained sync failure left durable_lsn permanently behind tip, so the loop
skipped its wait every iteration and spun fdatasync as fast as the CPU/lock
allowed (a 40-round chaotic /xce-test-ha soak hit 300k+ failed attempts on
one node in under a minute). Fixed by waiting out the normal wal_sync_ms
cadence when a flush was pending but made no progress, leaving the
legitimate idle and just-succeeded fast paths untouched.

Both fixed via TDD (RED test before the fix, GREEN after) and live-validated
by re-running 40-round sequential and chaotic HA soaks end-to-end: both pass
cleanly, and the chaotic run's excluded fdatasync-error count dropped from
304,689 to 0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-17 06:35:06 +00:00
parent f8a4b1e668
commit 846713ab92
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
5 changed files with 202 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

77
src/controller/testsuite/test_term_safety_adapter.c
View file

@ -2340,6 +2340,82 @@ static void test_every_callback_and_marker_path_end_to_end(void)
printf("test_every_callback_and_marker_path_end_to_end: PASS\n");
}
/* ================================================================== */
/* bug-1563 — fsync_worker must back off on a sustained fdatasync fail */
/* ================================================================== */
/*
* Live 40-round chaotic /xce-test-ha soak evidence: a freshly-promoted
* primary's local fdatasync started failing (errno=22) and never recovered.
* flusher_work_pending_locked() (durable_lsn < tip) stays true forever once
* fdatasync starts failing -- the fail-loud gate in flusher_flush_once()
* deliberately never advances durable_lsn on failure (that part is correct
* and unchanged). But fsync_worker's loop only waits (pthread_cond_timedwait)
* when NOT pending -- "pending" looks identical whether it means "genuine
* fresh work just landed" or "still stuck from last attempt", so a sustained
* failure makes the loop skip the wait every single iteration and spin
* flusher_flush_once() as fast as the CPU/lock allow: the live soak hit
* 300k+ failed attempts in well under a minute on one node (one core pinned,
* log flooded). This test reproduces the mechanism directly (force every
* fdatasync to fail via the existing test_force_fsync_fail seam, on a
* PRIMARY so the entry's only sync path is the async flusher under test --
* a standby's on_persist_entry does its own separate inline persist-before-
* ACK sync, see test_on_persist_entry_failure_propagates_no_ack) and asserts
* the attempt count over a fixed wall-clock window stays close to the
* wal_sync_ms cadence, not an unbounded spin.
*/
static void test_flusher_backs_off_on_sustained_fsync_failure(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_flusher_backoff_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
cluster_t* c = calloc(1, sizeof(cluster_t));
TEST_ASSERT_NOT_NULL(c);
persist_config_t cfg = persist_config_defaults();
cfg.wal_sync_ms = 20; /* short cadence so backoff is observable quickly */
cfg.snap_keep_n = 2;
persist_set_workdir(&c->persist, tmp, "ctrl-test");
TEST_ASSERT_EQUAL_INT(0, persist_init(&c->persist, &cfg, c));
TEST_ASSERT_EQUAL_INT(0, persist_start(&c->persist));
c->ha.raft_c = c;
atomic_store(&c->ha.role, CTRL_PRIMARY);
/* Force every fdatasync to fail BEFORE appending, so the entry created
* below is genuinely pending/undurable for the entire observation
* window -- the async fsync_worker thread is the only thing that could
* ever sync it. */
atomic_store(&c->persist.test_force_fsync_fail, 1);
uint8_t blob[64];
size_t n = pack_entry(blob, WAL_JOB_UPDATE, "y", 1);
TEST_ASSERT_EQUAL_INT(0,
ha_cb_persist_entry_for_test(&c->ha, /*index*/1, /*term*/1, blob, n));
struct timespec ts = { .tv_sec = 0, .tv_nsec = 300 * 1000 * 1000L };
nanosleep(&ts, NULL);
unsigned attempts =
atomic_load(&c->persist.test_flusher_flush_attempt_count);
atomic_store(&c->persist.test_force_fsync_fail, 0);
teardown_cluster(c);
/* 300ms / 20ms cadence -> ~15 attempts expected once backed off; a
* generous upper bound absorbs scheduling jitter without being loose
* enough to miss a genuine busy-spin (pre-fix: thousands of attempts in
* the same window). */
TEST_ASSERT_TRUE_MESSAGE(attempts < 60,
"fsync_worker must back off between failed fdatasync attempts "
"(bug-1563) -- a sustained sync failure must not spin the flusher "
"thread at full CPU with zero delay between retries");
printf("test_flusher_backs_off_on_sustained_fsync_failure: PASS "
"(attempts=%u)\n", attempts);
}
/* ------------------------------------------------------------------ */
/* Suite entry point */
/* ------------------------------------------------------------------ */
@ -2355,6 +2431,7 @@ void test_term_safety_adapter(void)
RUN_TEST(test_on_persist_entry_stamps_term_standby);
RUN_TEST(test_on_persist_entry_stamps_term_primary);
RUN_TEST(test_on_persist_entry_failure_propagates_no_ack);
RUN_TEST(test_flusher_backs_off_on_sustained_fsync_failure);
RUN_TEST(test_on_get_entry_term_present_legacy_absent);
RUN_TEST(test_on_truncate_from_drops_suffix);
RUN_TEST(test_on_truncate_from_removes_all);