wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

feat(raft-apply-decoupling): commit engine resolve primitive + majority-ack wiring (task 1.2)

Browse source 
commit_resolve() delivers a registered continuation's outcome exactly
once (idempotent re-resolve), dispatched off-thread via
vtask_enq_classified(VTASK_LIVE_BLOCK) so it never runs inline on the
replication-ACK or iomgr path. Wired into all 6 commit.c-internal
paths that can advance committed_lsn/writable_lsn.

Review caught a real task-graph gap, not a defect: converting
COMMIT_MODE_WAIT away from a blocking wait also removes the only
mechanism that ever detected a commit timing out (the caller's own
cond_timedwait deadline). Nothing yet arms an equivalent timeout for
async-registered continuations, and nothing wires demotion/log-truncation
detection to commit_resolve(FAILED, ...) -- left as-is, a pending async
command during quorum loss would hang forever instead of failing,
which is worse than today. Split into new tasks 1.3 (timeout arming)
and 1.4 (demotion/truncation wiring), renumbered the old concurrency
test to 1.5, and made 3.2 depend on both since its failure/timeout
observable is meaningless without them. design.md's File Structure
Plan updated to reflect that ha.c/persist.c need a narrow touch for
1.4's detection hooks.
This commit is contained in:
Rongsong Shen 2026-07-03 09:03:50 +00:00
parent 10d157f8c5
commit fd7a0746eb
5 changed files with 558 additions and 19 deletions
Download patch file Download diff file Expand all files Collapse all files

317
src/controller/testsuite/test_commit_resolve.c
View file

@ -1,17 +1,15 @@
/*
* test_commit_resolve.c — raft-apply-decoupling task 1.1.
* test_commit_resolve.c — raft-apply-decoupling tasks 1.1 + 1.2.
*
* Keystone conversion, registration half: the commit engine gains
* Keystone conversion, registration half (task 1.1): the commit engine gains
* commit_submit_async(), a non-blocking sibling of
* commit_submit(..., COMMIT_MODE_WAIT, ...) that registers a continuation
* (callback + context) against an appended LSN instead of parking the
* calling thread. This file covers ONLY the registration mechanism —
* resolution (invoking the continuation on majority-ack / timeout /
* demotion / truncation) is wired by a follow-up task (1.2) and is not
* exercised here: a registered continuation is never invoked in these
* tests, by design.
* calling thread. The task-1.1 cases below cover ONLY the registration
* mechanism — a registered continuation is never invoked in those tests, by
* design (resolution did not exist yet).
*
* Asserts:
* Asserts (task 1.1):
* (1) commit_submit_async returns immediately (0), before the LSN
* commits, and does NOT invoke the continuation inline.
* (2) Bad-argument rejection: -EINVAL for NULL engine, LSN 0, NULL fn.
@ -24,7 +22,36 @@
* same inflight_waits pool.
* (5) Distinct concurrent registrations occupy distinct per-LSN slots.
*
* Requirements: 1.3, 1.4, 9.1, 9.2
* Requirements (task 1.1): 1.3, 1.4, 9.1, 9.2
*
* Resolution half (task 1.2): commit_resolve() delivers a registered
* continuation's outcome exactly once, dispatched off the calling thread via
* vtask_enq_classified (VTASK_LIVE_BLOCK), never inline on the ACK/iomgr
* path in the real (pool-wired) case. Covers the four Observable triggers
* from tasks.md 1.2 -- {majority ack, timeout, demotion, truncation} -- plus
* idempotency, unregistered-lsn safety, admission-slot release, and the
* fail-loud inline fallback when no vtaskmgr pool is wired.
*
* On "timeout"/"demotion"/"truncation": commit_resolve() is a generic
* outcome+rc delivery primitive (design §Commit Engine Resolve) -- it does
* not itself detect any of these three conditions. Tracing the existing
* codebase (persist.c truncate path, ha.c raft_cb_become_follower demotion
* path, repl_wait_majority_ack's error vocabulary) found NO existing
* distinct rc for demotion or truncation anywhere today: commit_submit's
* ONLY negative "did not commit" outcome is -ETIMEDOUT, because the current
* blocking-WAIT model doesn't distinguish these causes either -- a
* demoted/truncated pending entry simply never satisfies its predicate and
* dies at the caller's own deadline. Wiring an ACTUAL demotion/truncation
* detector to call commit_resolve() belongs to later tasks that touch
* ha.c/persist.c/repl.c (out of this task's commit.c-only boundary; see
* task 3.2 in tasks.md). These tests therefore "force" each condition the
* way a future detector will: by calling commit_resolve(FAILED, rc)
* directly, using representative existing rc vocabulary (-ETIMEDOUT for
* timeout, -XCE_ERR_STANDBY -- "op is primary-only" -- for demotion, -ENOENT
* for a log-truncated-away entry) to prove commit_resolve delivers ANY
* caller-supplied outcome/rc faithfully, exactly once, off-thread.
*
* Requirements (task 1.2): 1.3, 9.3, 9.4
*/
#include <errno.h>
#include <pthread.h>
@ -40,6 +67,7 @@
#include "commit.h"
#include "repl.h"
#include "vtaskmgr.h"
/* ── Fixture ──────────────────────────────────────────────────────────────
* Static: persist_t embeds large path buffers, and statics survive Unity's
@ -77,15 +105,66 @@ static void resolve_teardown(void)
repl_fini(&g_p.repl);
}
/* ── Pool-wired fixture (task 1.2) ───────────────────────────────────────────
* A real, RUNNING vtaskmgr_t wired via commit_set_pool, mirroring
* test_commit_admission_pool.c's pool_setup but additionally calling
* vtaskmgr_start so enqueued vtasks are actually picked up and executed —
* required to observe commit_resolve's real (non-fallback) off-thread
* dispatch, not just the admission-accounting seam. */
static vtaskmgr_t g_mgr;
static void resolve_pool_setup(int n_peers, int n_standbys,
uint64_t appended_lsn, int n_vcpus)
{
resolve_setup(n_peers, n_standbys, appended_lsn, n_vcpus);
TEST_ASSERT_EQUAL_INT(0, vtaskmgr_init(&g_mgr, n_vcpus));
TEST_ASSERT_EQUAL_INT(0, vtaskmgr_start(&g_mgr));
commit_set_pool(&g_ce, &g_mgr);
}
static void resolve_pool_teardown(void)
{
commit_engine_fini(&g_ce);
vtaskmgr_fini(&g_mgr);
repl_fini(&g_p.repl);
}
static uint64_t mono_ms_r(void)
{
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return (uint64_t)ts.tv_sec * 1000u + (uint64_t)ts.tv_nsec / 1000000u;
}
/* Spin-poll (bounded) until *calls reaches want; avoids correctness-by-sleep
* on the assertion path (pattern from test_vtask_admission.c wait_atomic_ge /
* test_commit_engine.c poll_inflight). */
static int poll_calls(_Atomic int* calls, int want, uint32_t deadline_ms)
{
uint64_t t0 = mono_ms_r();
int v;
for (;;) {
v = atomic_load(calls);
if (v >= want || mono_ms_r() - t0 >= deadline_ms) {
return v;
}
usleep(500);
}
}
/* Continuation recorder — proves (non-)invocation and captures the args a
* future commit_resolve() would deliver. Not exercised as a real resolve
* path in this file (task 1.2's job); only used to assert it stays quiet. */
* commit_resolve() delivers, including the thread it ran on (task 1.2: proves
* dispatch is off the calling thread, never inline on the real pool-wired
* path). */
typedef struct {
_Atomic int calls;
uint64_t last_lsn;
commit_outcome_t last_outcome;
int last_rc;
void* last_ctx;
pthread_t last_thread;
} continuation_recorder_t;
static void recorder_fn(void* ctx, uint64_t lsn, commit_outcome_t outcome,
@ -97,6 +176,7 @@ static void recorder_fn(void* ctx, uint64_t lsn, commit_outcome_t outcome,
r->last_outcome = outcome;
r->last_rc = rc;
r->last_ctx = ctx;
r->last_thread = pthread_self();
atomic_fetch_add(&r->calls, 1);
}
@ -301,6 +381,214 @@ static void test_commit_submit_async_tracks_distinct_lsns(void)
printf("test_commit_submit_async_tracks_distinct_lsns: PASS\n");
}
/* ════════════════════════════════════════════════════════════════════════
* Task 1.2 — commit_resolve(): majority-commit and failure-path resolution.
* ════════════════════════════════════════════════════════════════════════ */
/* ------------------------------------------------------------------ *
* (6) Majority ack: a registered continuation is resolved COMMITTED, *
* exactly once, dispatched OFF the ACK-driving thread (real, *
* pool-wired path — never inline). *
* ------------------------------------------------------------------ */
static void test_commit_resolve_majority_ack_dispatches_committed_off_thread(void)
{
continuation_recorder_t rec = {0};
pthread_t self = pthread_self();
/* 3-member cluster, entry 1 appended+durable, no ACKs yet. */
resolve_pool_setup(/*n_peers=*/3, /*n_standbys=*/2, /*appended_lsn=*/1,
/*n_vcpus=*/4);
TEST_ASSERT_EQUAL_INT(0, commit_submit_async(&g_ce, 1, recorder_fn, &rec));
TEST_ASSERT_EQUAL_INT(1, g_ce.inflight_waits);
/* Majority ACK arrives on THIS thread (mirrors commit_on_ack running on
* the iomgr thread in production). */
repl_on_ack(&g_p.repl, &g_eps[0], 1);
commit_on_ack(&g_ce, 0, 1);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, poll_calls(&rec.calls, 1, 2000),
"commit_resolve must deliver COMMITTED once the majority ack lands");
TEST_ASSERT_EQUAL_UINT64(1, rec.last_lsn);
TEST_ASSERT_EQUAL_INT(COMMIT_OUTCOME_COMMITTED, rec.last_outcome);
TEST_ASSERT_EQUAL_INT(0, rec.last_rc);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, atomic_load(&rec.calls),
"the continuation must be invoked exactly once");
TEST_ASSERT_FALSE_MESSAGE(
pthread_equal(self, rec.last_thread),
"the continuation must run off the calling (ACK-driving) thread, "
"never inline");
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, g_ce.inflight_waits,
"resolve must release the continuation's admission slot");
resolve_pool_teardown();
printf("test_commit_resolve_majority_ack_dispatches_committed_off_thread: "
"PASS\n");
}
/* ------------------------------------------------------------------ *
* (7) Failure outcomes: commit_resolve delivers ANY caller-supplied *
* outcome/rc faithfully, off-thread, for the three failure causes *
* tasks.md 1.2 names — timeout, demotion, truncation. commit.c has *
* no existing distinct detector for demotion/truncation (see file *
* header); these calls simulate a future detector invoking the *
* generic resolve primitive directly. *
* ------------------------------------------------------------------ */
static void test_commit_resolve_delivers_failed_outcome_for_timeout(void)
{
continuation_recorder_t rec = {0};
pthread_t self = pthread_self();
resolve_pool_setup(/*n_peers=*/3, /*n_standbys=*/2, /*appended_lsn=*/1,
/*n_vcpus=*/4);
TEST_ASSERT_EQUAL_INT(0, commit_submit_async(&g_ce, 1, recorder_fn, &rec));
commit_resolve(&g_ce, 1, COMMIT_OUTCOME_FAILED, -ETIMEDOUT);
TEST_ASSERT_EQUAL_INT(1, poll_calls(&rec.calls, 1, 2000));
TEST_ASSERT_EQUAL_UINT64(1, rec.last_lsn);
TEST_ASSERT_EQUAL_INT(COMMIT_OUTCOME_FAILED, rec.last_outcome);
TEST_ASSERT_EQUAL_INT(-ETIMEDOUT, rec.last_rc);
TEST_ASSERT_EQUAL_INT(1, atomic_load(&rec.calls));
TEST_ASSERT_FALSE(pthread_equal(self, rec.last_thread));
TEST_ASSERT_EQUAL_INT(0, g_ce.inflight_waits);
resolve_pool_teardown();
printf("test_commit_resolve_delivers_failed_outcome_for_timeout: PASS\n");
}
static void test_commit_resolve_delivers_failed_outcome_for_demotion(void)
{
continuation_recorder_t rec = {0};
resolve_pool_setup(/*n_peers=*/3, /*n_standbys=*/2, /*appended_lsn=*/1,
/*n_vcpus=*/4);
TEST_ASSERT_EQUAL_INT(0, commit_submit_async(&g_ce, 1, recorder_fn, &rec));
/* -XCE_ERR_STANDBY ("op is primary-only") is the existing vocabulary code
* closest to "this controller is no longer primary" — a future
* ha.c demotion detector's natural choice of rc. */
commit_resolve(&g_ce, 1, COMMIT_OUTCOME_FAILED, -XCE_ERR_STANDBY);
TEST_ASSERT_EQUAL_INT(1, poll_calls(&rec.calls, 1, 2000));
TEST_ASSERT_EQUAL_INT(COMMIT_OUTCOME_FAILED, rec.last_outcome);
TEST_ASSERT_EQUAL_INT(-XCE_ERR_STANDBY, rec.last_rc);
TEST_ASSERT_EQUAL_INT(1, atomic_load(&rec.calls));
resolve_pool_teardown();
printf("test_commit_resolve_delivers_failed_outcome_for_demotion: PASS\n");
}
static void test_commit_resolve_delivers_failed_outcome_for_truncation(void)
{
continuation_recorder_t rec = {0};
resolve_pool_setup(/*n_peers=*/3, /*n_standbys=*/2, /*appended_lsn=*/1,
/*n_vcpus=*/4);
TEST_ASSERT_EQUAL_INT(0, commit_submit_async(&g_ce, 1, recorder_fn, &rec));
/* -ENOENT ("no such entry") represents a pending lsn whose log tail was
* truncated out from under it — a future persist.c truncation detector's
* natural choice of rc. */
commit_resolve(&g_ce, 1, COMMIT_OUTCOME_FAILED, -ENOENT);
TEST_ASSERT_EQUAL_INT(1, poll_calls(&rec.calls, 1, 2000));
TEST_ASSERT_EQUAL_INT(COMMIT_OUTCOME_FAILED, rec.last_outcome);
TEST_ASSERT_EQUAL_INT(-ENOENT, rec.last_rc);
TEST_ASSERT_EQUAL_INT(1, atomic_load(&rec.calls));
resolve_pool_teardown();
printf("test_commit_resolve_delivers_failed_outcome_for_truncation: PASS\n");
}
/* ------------------------------------------------------------------ *
* (8) Idempotency: a second resolve() for an already-resolved lsn is a *
* no-op — the FIRST outcome/rc stands, no second invocation. *
* ------------------------------------------------------------------ */
static void test_commit_resolve_second_resolve_is_noop(void)
{
continuation_recorder_t rec = {0};
resolve_pool_setup(/*n_peers=*/3, /*n_standbys=*/2, /*appended_lsn=*/1,
/*n_vcpus=*/4);
TEST_ASSERT_EQUAL_INT(0, commit_submit_async(&g_ce, 1, recorder_fn, &rec));
commit_resolve(&g_ce, 1, COMMIT_OUTCOME_FAILED, -ETIMEDOUT);
TEST_ASSERT_EQUAL_INT(1, poll_calls(&rec.calls, 1, 2000));
/* A second resolve for the SAME lsn, with a DIFFERENT outcome/rc, must
* be a silent no-op: no second invocation, and the recorded outcome/rc
* from the first (and only) delivery must stand unchanged. */
commit_resolve(&g_ce, 1, COMMIT_OUTCOME_COMMITTED, 0);
usleep(50 * 1000); /* give a wrongly-dispatched second call a chance to land */
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, atomic_load(&rec.calls),
"a second resolve for an already-resolved lsn must not re-invoke");
TEST_ASSERT_EQUAL_INT_MESSAGE(
COMMIT_OUTCOME_FAILED, rec.last_outcome,
"the first delivery's outcome must stand, not be overwritten");
TEST_ASSERT_EQUAL_INT(-ETIMEDOUT, rec.last_rc);
resolve_pool_teardown();
printf("test_commit_resolve_second_resolve_is_noop: PASS\n");
}
/* ------------------------------------------------------------------ *
* (9) Resolving an lsn with no registered continuation is a safe *
* no-op — never crashes, never invokes anything. *
* ------------------------------------------------------------------ */
static void test_commit_resolve_unregistered_lsn_is_safe_noop(void)
{
resolve_pool_setup(/*n_peers=*/3, /*n_standbys=*/2, /*appended_lsn=*/1,
/*n_vcpus=*/4);
commit_resolve(&g_ce, /*lsn=*/1, COMMIT_OUTCOME_COMMITTED, 0);
commit_resolve(NULL, 1, COMMIT_OUTCOME_FAILED, -ETIMEDOUT);
commit_resolve(&g_ce, /*lsn=*/0, COMMIT_OUTCOME_FAILED, -ETIMEDOUT);
/* No crash is the primary assertion; give any errant dispatch a moment
* then confirm nothing was admitted/released unexpectedly. */
usleep(50 * 1000);
TEST_ASSERT_EQUAL_INT(0, g_ce.inflight_waits);
resolve_pool_teardown();
printf("test_commit_resolve_unregistered_lsn_is_safe_noop: PASS\n");
}
/* ------------------------------------------------------------------ *
* (10) Fail-loud inline fallback: with no vtaskmgr pool wired (the *
* "commit-only unit fixtures" case, commit.h field comment), the *
* continuation must STILL be resolved — never silently dropped — *
* via the documented inline fallback. *
* ------------------------------------------------------------------ */
static void test_commit_resolve_falls_back_to_inline_without_pool(void)
{
continuation_recorder_t rec = {0};
/* Plain resolve_setup: no commit_set_pool call, so g_ce.pool == NULL. */
resolve_setup(/*n_peers=*/3, /*n_standbys=*/2, /*appended_lsn=*/1,
/*n_vcpus=*/4);
TEST_ASSERT_NULL(g_ce.pool);
TEST_ASSERT_EQUAL_INT(0, commit_submit_async(&g_ce, 1, recorder_fn, &rec));
commit_resolve(&g_ce, 1, COMMIT_OUTCOME_FAILED, -ETIMEDOUT);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, atomic_load(&rec.calls),
"without a pool the continuation must still resolve, via the "
"fail-loud inline fallback (never silently dropped)");
TEST_ASSERT_EQUAL_INT(COMMIT_OUTCOME_FAILED, rec.last_outcome);
TEST_ASSERT_EQUAL_INT(-ETIMEDOUT, rec.last_rc);
TEST_ASSERT_EQUAL_INT(0, g_ce.inflight_waits);
resolve_teardown();
printf("test_commit_resolve_falls_back_to_inline_without_pool: PASS\n");
}
void test_commit_resolve(void)
{
RUN_TEST(test_commit_submit_async_returns_immediately_before_commit);
@ -308,4 +596,11 @@ void test_commit_resolve(void)
RUN_TEST(test_commit_submit_async_respects_admission_cap);
RUN_TEST(test_commit_submit_async_shares_admission_budget_with_wait);
RUN_TEST(test_commit_submit_async_tracks_distinct_lsns);
RUN_TEST(test_commit_resolve_majority_ack_dispatches_committed_off_thread);
RUN_TEST(test_commit_resolve_delivers_failed_outcome_for_timeout);
RUN_TEST(test_commit_resolve_delivers_failed_outcome_for_demotion);
RUN_TEST(test_commit_resolve_delivers_failed_outcome_for_truncation);
RUN_TEST(test_commit_resolve_second_resolve_is_noop);
RUN_TEST(test_commit_resolve_unregistered_lsn_is_safe_noop);
RUN_TEST(test_commit_resolve_falls_back_to_inline_without_pool);
}