wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

feat(raft-apply-decoupling): arm timeout for async commit continuations (task 1.3)

Browse source 
Converting COMMIT_MODE_WAIT away from a blocking wait removed the only
mechanism that ever detected a commit timing out (the caller's own
pthread_cond_timedwait deadline). Adds a dedicated per-commit-engine
sweep thread that arms an equivalent timeout for async-registered
continuations, resolving FAILED/-ETIMEDOUT within the same
DEFAULT_SYNC_TIMEOUT_MS window the old blocking path used, with no
thread parked to detect it.

Chose a private sweep thread over the shared delaymgr: wiring
delaymgr would touch cluster.c/ha.c (out of boundary) and its 200ms
poll granularity vs the 500ms default timeout would add jitter,
breaking the "identical failure timing" no-regression bar.

Round 1 review rejected a lost-wakeup race: the sweep loop computed
its next-deadline snapshot under lock, unlocked to dispatch, then
slept on the stale snapshot -- a registration landing mid-dispatch
had its broadcast lost (no thread parked on the condvar), so the
sweep thread could sleep on an absent/late deadline indefinitely.
Fixed by holding the lock continuously across scan-then-wait
decisions and always re-scanning immediately before ever waiting
(recheck-then-wait idiom). Round 2 review independently reproduced
the round-1 bug shape against the fix to confirm it actually closes
the race, not just narrows it.
This commit is contained in:
Rongsong Shen 2026-07-03 10:35:37 +00:00
parent fd7a0746eb
commit 530a02bda3
4 changed files with 461 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

209
src/controller/testsuite/test_commit_resolve.c
View file

@ -52,6 +52,21 @@
* caller-supplied outcome/rc faithfully, exactly once, off-thread.
*
* Requirements (task 1.2): 1.3, 9.3, 9.4
*
* Timeout-sweep half (task 1.3): commit_submit_async() no longer parks a
* thread that would otherwise notice its own pthread_cond_timedwait deadline
* expire, so a dedicated timeout-sweep thread (private to commit_engine_t,
* started in commit_engine_init / stopped+joined in commit_engine_fini) is
* the only remaining detector of "this registered lsn has been pending too
* long". The cases below prove: (a) a registered continuation for an lsn
* that never reaches majority and is NEVER manually resolved still resolves
* FAILED/-ETIMEDOUT on its own, near the existing DEFAULT_SYNC_TIMEOUT_MS
* default (Req 11.1 -- identical failure timing/vocabulary to today's
* blocking path); and (b) the majority-satisfied outcome always wins a race
* against an already-expired deadline, mirroring commit_submit's own
* last-look-before-timeout tie-break.
*
* Requirements (task 1.3): 1.3, 1.4, 9.3, 9.4
*/
#include <errno.h>
#include <pthread.h>
@ -165,6 +180,11 @@ typedef struct {
int last_rc;
void* last_ctx;
pthread_t last_thread;
/* raft-apply-decoupling task 1.3 lost-wakeup regression: when nonzero,
* recorder_fn sleeps this long BEFORE recording. Zero (the default
* zero-initializer every existing test uses) preserves today's
* immediate-recording behavior. */
unsigned sleep_us;
} continuation_recorder_t;
static void recorder_fn(void* ctx, uint64_t lsn, commit_outcome_t outcome,
@ -172,6 +192,9 @@ static void recorder_fn(void* ctx, uint64_t lsn, commit_outcome_t outcome,
{
continuation_recorder_t* r = (continuation_recorder_t*)ctx;
if (r->sleep_us > 0) {
usleep(r->sleep_us);
}
r->last_lsn = lsn;
r->last_outcome = outcome;
r->last_rc = rc;
@ -589,6 +612,189 @@ static void test_commit_resolve_falls_back_to_inline_without_pool(void)
printf("test_commit_resolve_falls_back_to_inline_without_pool: PASS\n");
}
/* ════════════════════════════════════════════════════════════════════════
* Task 1.3 — timeout-sweep: async continuations resolve FAILED/-ETIMEDOUT
* on their own, with no manual commit_resolve() call.
* ════════════════════════════════════════════════════════════════════════ */
/* ------------------------------------------------------------------ *
* (11) A registered continuation for an lsn that never reaches *
* majority resolves FAILED/-ETIMEDOUT on its own, near the *
* existing DEFAULT_SYNC_TIMEOUT_MS default -- no manual *
* commit_resolve() call, no thread parked to detect it. *
* ------------------------------------------------------------------ */
static void test_commit_submit_async_resolves_failed_on_timeout(void)
{
continuation_recorder_t rec = {0};
uint64_t t0;
uint64_t elapsed;
/* n_peers=3, no standby ACKs ever delivered: lsn=1 can never commit on
* its own. Nothing in this test ever calls commit_resolve() -- the ONLY
* thing that can resolve this continuation is the timeout-sweep
* mechanism built by this task. */
resolve_pool_setup(/*n_peers=*/3, /*n_standbys=*/2, /*appended_lsn=*/1,
/*n_vcpus=*/4);
t0 = mono_ms_r();
TEST_ASSERT_EQUAL_INT(0, commit_submit_async(&g_ce, 1, recorder_fn, &rec));
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, poll_calls(&rec.calls, 1, DEFAULT_SYNC_TIMEOUT_MS + 1500),
"a registered continuation that never reaches majority must resolve "
"FAILED/-ETIMEDOUT on its own within the configured timeout window");
elapsed = mono_ms_r() - t0;
TEST_ASSERT_EQUAL_UINT64(1, rec.last_lsn);
TEST_ASSERT_EQUAL_INT(COMMIT_OUTCOME_FAILED, rec.last_outcome);
TEST_ASSERT_EQUAL_INT(-ETIMEDOUT, rec.last_rc);
TEST_ASSERT_EQUAL_INT(1, atomic_load(&rec.calls));
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_ce.inflight_waits,
"the timeout resolve must release the continuation's admission slot");
/* Req 11.1 -- identical failure timing to today's blocking path:
* generous bounds absorb scheduler jitter and the sweep thread's own
* bounded scan overhead without masking a mistimed implementation. */
TEST_ASSERT_TRUE_MESSAGE(
elapsed + 50 >= DEFAULT_SYNC_TIMEOUT_MS,
"must not resolve meaningfully before the configured default "
"timeout");
TEST_ASSERT_TRUE_MESSAGE(
elapsed < DEFAULT_SYNC_TIMEOUT_MS + 1000,
"must resolve within the configured timeout window, not drastically "
"late");
resolve_pool_teardown();
printf("test_commit_submit_async_resolves_failed_on_timeout: PASS\n");
}
/* ------------------------------------------------------------------ *
* (12) Race: majority-satisfied always wins over an already-expired *
* deadline observed in the SAME sweep pass -- mirrors *
* commit_submit's own last-look-before-timeout tie-break. White- *
* box: force the deadline into the past and satisfy the majority *
* predicate directly (bypassing commit_on_ack / the already- *
* tested commit_resolve_pending ack path) so ONLY the timeout- *
* sweep thread's own scan discovers both conditions at once. *
* ------------------------------------------------------------------ */
static void test_commit_submit_async_majority_wins_race_at_expired_deadline(void)
{
continuation_recorder_t rec = {0};
int found = 0;
resolve_pool_setup(/*n_peers=*/3, /*n_standbys=*/2, /*appended_lsn=*/1,
/*n_vcpus=*/4);
TEST_ASSERT_EQUAL_INT(0, commit_submit_async(&g_ce, 1, recorder_fn, &rec));
pthread_mutex_lock(&g_ce.lock);
for (int i = 0; i < g_ce.n_continuations; i++) {
if (g_ce.continuations[i].lsn == 1) {
g_ce.continuations[i].deadline.tv_sec = 0;
g_ce.continuations[i].deadline.tv_nsec = 0;
found = 1;
}
}
/* Satisfy wait_done_locked directly -- no ack path involved. */
atomic_store(&g_p.committed_lsn, 1);
pthread_cond_broadcast(&g_ce.cond); /* wake the sweep thread promptly */
pthread_mutex_unlock(&g_ce.lock);
TEST_ASSERT_TRUE_MESSAGE(found, "lsn=1 registration must be tracked");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, poll_calls(&rec.calls, 1, 2000),
"the sweep thread must independently notice and resolve this lsn");
TEST_ASSERT_EQUAL_INT_MESSAGE(
COMMIT_OUTCOME_COMMITTED, rec.last_outcome,
"majority-satisfied must win over an already-expired deadline "
"observed in the same scan");
TEST_ASSERT_EQUAL_INT(0, rec.last_rc);
TEST_ASSERT_EQUAL_INT(1, atomic_load(&rec.calls));
resolve_pool_teardown();
printf("test_commit_submit_async_majority_wins_race_at_expired_deadline: "
"PASS\n");
}
/* ------------------------------------------------------------------ *
* (13) Lost-wakeup regression: a registration landing WHILE the sweep *
* thread is mid-dispatch for a previous continuation must not be *
* missed. commit_sweep_scan_locked()'s classification runs under *
* ce->lock, but the actual commit_resolve() dispatch for whatever *
* it found must run UNLOCKED (lock-ordering rule) -- a *
* commit_submit_async() broadcast landing in that unlocked window *
* finds no thread parked in cond_wait to receive it (POSIX *
* condvars do not latch a signal sent to no waiter). If the sweep *
* worker then trusted the have_next/next_deadline it computed *
* BEFORE that dispatch, it would commit to an unbounded *
* pthread_cond_wait() (or a stale deadline) and never notice the *
* new registration until some UNRELATED future broadcast -- *
* unbounded delay instead of DEFAULT_SYNC_TIMEOUT_MS. *
* *
* Deterministic reproduction: use the no-pool inline-dispatch *
* fallback, whose continuation function runs SYNCHRONOUSLY on the *
* thread that called commit_resolve() -- the sweep thread itself, *
* for a timeout-triggered resolve. Continuation A's recorder *
* deliberately sleeps, pinning the sweep thread outside ce->lock *
* for a wide, fully-controlled window (this IS the dispatch *
* window the bug lived in, just stretched from microseconds to *
* ~200ms so a concurrent registration can land inside it with *
* certainty instead of chasing a race). Continuation B is *
* registered mid-sleep and must still resolve within *
* DEFAULT_SYNC_TIMEOUT_MS of ITS OWN registration, not stall on *
* whatever unrelated broadcast happens to come next. *
* ------------------------------------------------------------------ */
static void test_commit_submit_async_registration_during_dispatch_not_lost(void)
{
continuation_recorder_t recA = {0};
continuation_recorder_t recB = {0};
/* Plain (no-pool) setup: dispatch_continuation's fail-loud inline
* fallback invokes the continuation function directly, unlocked, on
* the calling thread -- the sweep thread, here. */
resolve_setup(/*n_peers=*/3, /*n_standbys=*/2, /*appended_lsn=*/1,
/*n_vcpus=*/8);
TEST_ASSERT_NULL(g_ce.pool);
recA.sleep_us = 200 * 1000; /* 200ms: sweep thread is unlocked, "busy
* dispatching" A for this long */
TEST_ASSERT_EQUAL_INT(0, commit_submit_async(&g_ce, 1, recorder_fn, &recA));
/* Let A's DEFAULT_SYNC_TIMEOUT_MS deadline pass and the sweep thread
* begin dispatching it (enter recorder_fn's sleep). Comfortably past
* the deadline, comfortably before the 200ms sleep would elapse on its
* own. */
usleep((DEFAULT_SYNC_TIMEOUT_MS + 100) * 1000);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, atomic_load(&recA.calls),
"sanity: A must still be inside its 200ms sleep, not yet recorded, "
"when B is registered below -- otherwise this test isn't actually "
"landing inside the dispatch window");
/* Register B while the sweep thread is (with certainty, given the
* 200ms window) unlocked and blocked inside recorder_fn(A)'s sleep --
* exactly the window a real registration could race in production,
* just widened for a deterministic test. */
TEST_ASSERT_EQUAL_INT(0, commit_submit_async(&g_ce, 2, recorder_fn, &recB));
TEST_ASSERT_EQUAL_INT_MESSAGE(1, poll_calls(&recA.calls, 1, 2000),
"continuation A must still resolve");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, poll_calls(&recB.calls, 1, DEFAULT_SYNC_TIMEOUT_MS + 2000),
"continuation B, registered while the sweep thread was mid-dispatch "
"for A, must still resolve within the configured timeout window -- "
"not delayed indefinitely waiting for some unrelated future "
"broadcast (the lost-wakeup regression)");
TEST_ASSERT_EQUAL_UINT64(2, recB.last_lsn);
TEST_ASSERT_EQUAL_INT(COMMIT_OUTCOME_FAILED, recB.last_outcome);
TEST_ASSERT_EQUAL_INT(-ETIMEDOUT, recB.last_rc);
TEST_ASSERT_EQUAL_INT(0, g_ce.inflight_waits);
resolve_teardown();
printf("test_commit_submit_async_registration_during_dispatch_not_lost: "
"PASS\n");
}
void test_commit_resolve(void)
{
RUN_TEST(test_commit_submit_async_returns_immediately_before_commit);
@ -603,4 +809,7 @@ void test_commit_resolve(void)
RUN_TEST(test_commit_resolve_second_resolve_is_noop);
RUN_TEST(test_commit_resolve_unregistered_lsn_is_safe_noop);
RUN_TEST(test_commit_resolve_falls_back_to_inline_without_pool);
RUN_TEST(test_commit_submit_async_resolves_failed_on_timeout);
RUN_TEST(test_commit_submit_async_majority_wins_race_at_expired_deadline);
RUN_TEST(test_commit_submit_async_registration_during_dispatch_not_lost);
}