lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(raft-apply-decoupling): convert job submit to continuation-based commit (task 3.2)

Browse source 
Replace handle_jobsubmit()'s blocking commit call (task 3.1) with a
real async registration against commit_submit_async(). On
FAILED/timeout, replies failure immediately with no job constructed.
On COMMITTED, hands off to a new pending-apply registration
(persist_register_pending_apply) for the ordered apply pump (task
6.1, not yet built) to consume -- does not apply or reply directly,
per design.

Deviates from design.md's stated integration site: appends via the
same ha_primary_log_append/persist_wal_async primitive the RETAIN-fact
path already uses and calls commit_submit_async directly on
p->repl.ce, bypassing repl_wait_majority_ack/repl_submit_majority_commit
in repl.c. Verified safe -- majority-ack detection is driven entirely
by (ce, lsn) events inside the commit engine, agnostic to which
wrapper registered the continuation -- but narrows the reusable
primitive design.md intended; documented in tasks.md.

Deletes (not preserves-dead) task 3.1's bug-1408 notification fix --
apply_committed_entry()'s WAL_JOB_SUBMIT case (task 2.2) already
fires the notification per-entry, immune to the batch-suppression
class bug-1408 fixed.

IMPORTANT: no live path applies a successfully-committed job
submission until task 6 lands (only restart replay does) -- every
successful submission commits durably but gets zero reply until
then. This commit range must not be soak-tested or deployed against
live traffic before task 6 completes.
This commit is contained in:
Rongsong Shen 2026-07-04 05:35:35 +00:00
parent 6f1fa97fab
commit df4d2fa0ed
5 changed files with 689 additions and 299 deletions
Download patch file Download diff file Expand all files Collapse all files

337
src/controller/testsuite/test_commit_barrier_reply.c
View file

@ -834,13 +834,23 @@ static void test_worker_join_no_hostconfig_error_reply_immediate(void)
printf("test_worker_join_no_hostconfig_error_reply_immediate: PASS\n");
}
/* ── handle_jobsubmit (raft-apply-decoupling task 3.1: apply-after-commit) ──
/* ── handle_jobsubmit (raft-apply-decoupling task 3.1/3.2) ───────────────
*
* Req 2.1/2.3: a job submission whose commit fails or times out must leave
* NO job object in c->jobmgr — construction is deferred until after the
* commit barrier succeeds. Req 2.2: a successful submission still replies
* synchronously with the jobid, and by the time that reply is observable the
* job is already visible in c->jobmgr. */
* task 3.1 (Req 2.1/2.3): a job submission whose commit fails or times out
* must leave NO job object in c->jobmgr — construction is deferred until
* after the commit barrier succeeds.
*
* task 3.2 (Req 1.1, 1.4, 2.1, 2.3): the synchronous commit call from 3.1 is
* replaced with a registration against the commit engine's continuation
* mechanism (commit_submit_async, via persist_submit_job_payloads_async).
* handle_jobsubmit now returns BEFORE any commit outcome is known. A
* failed/timed-out outcome still replies failure directly, with no job ever
* constructed (same observable as 3.1). A COMMITTED outcome is handed off to
* the not-yet-built ordered apply pump (task 6.1/6.2) via
* persist_register_pending_apply() instead of applying or replying directly
* — so a successful submission has NO independently observable behavior
* (no reply, no job in c->jobmgr) until task 6 exists; the test below
* asserts only that the hand-off itself was registered for the correct lsn. */
static job_startinfo_t g_jobsub_startinfo;
static job_submit_data_t g_jobsub_data;
@ -902,121 +912,24 @@ static void test_jobsubmit_withheld_acks_no_phantom_job(void)
printf("test_jobsubmit_withheld_acks_no_phantom_job: PASS\n");
}
static void test_jobsubmit_success_reply_and_job_visible_after_commit(void)
/* raft-apply-decoupling task 3.2: supersedes the pre-3.2
* test_jobsubmit_success_reply_and_job_visible_after_commit (which drove
* handle_jobsubmit through cbrp_drive_acked, a helper that asserts the
* handler blocks until its reply is observable — no longer true here by
* this task's own explicit contract). This test asserts ONLY what task 3.2
* itself makes observable on the committed-outcome path: the calling
* thread returns before the commit resolves, and the continuation hands
* the entry off to persist's pending-apply registry keyed by the correct
* lsn — NOT a reply, NOT a constructed job (task 6 owns both; neither
* exists yet). */
static void test_jobsubmit_returns_before_commit_and_registers_pending_apply(void)
{
cbrp_setup(1500);
uint64_t lsn_before, watermark, t0, join_elapsed_ms, t1;
cbrp_reply_obs_t obs = { 0 };
bool has_pending, job_exists;
cbrp_setup(2000); /* generous timeout: must never be needed */
cbrp_arrange_jobsubmit(9101, 1);
cbrp_drive_acked(handle_jobsubmit, &g_jobsub_req);
TEST_ASSERT_TRUE_MESSAGE(cbrp_job_exists(9101),
"a successful job submission must construct the job object once "
"its commit succeeds, and still reply synchronously with the "
"jobid (raft-apply-decoupling task 3.1, Req 2.2)");
cbrp_teardown();
printf("test_jobsubmit_success_reply_and_job_visible_after_commit: PASS\n");
}
/* Captures every jobid delivered via the AFTER_EVENT/HOOK_DETAIL_JOB_NEW
* notification hook (job.c's post-commit notify list). Needed by the
* array-partial-construction-failure remediation test below to observe
* that a durably committed AND successfully constructed sibling is still
* notified even when another array element in the same submission fails to
* construct (raft-apply-decoupling task 3.1 remediation). xce_run_hooks
* also fires this same hook TYPE for HOOK_BEFORE_EVENT with a different
* vararg shape (xce_job_submit_t*, not xlist_t*) — filtered out below
* before touching ap. */
#define CBRP_MAX_JOBNEW_NOTIFY 8
static int g_jobnew_notify_count;
static uint64_t g_jobnew_notify_jobids[CBRP_MAX_JOBNEW_NOTIFY];
static void cbrp_jobnew_capture_reset(void)
{
g_jobnew_notify_count = 0;
memset(g_jobnew_notify_jobids, 0, sizeof(g_jobnew_notify_jobids));
}
static int cbrp_jobnew_capture_fn(int type, int situation, int detail,
void *data, va_list ap)
{
(void)type; (void)data;
if (detail != HOOK_DETAIL_JOB_NEW || situation != HOOK_AFTER_EVENT)
return 0;
{
xlist_t *jlist;
int i;
(void)va_arg(ap, cluster_t *);
jlist = va_arg(ap, xlist_t *);
if (jlist == NULL)
return 0;
for (i = 0; i < jlist->count
&& g_jobnew_notify_count < CBRP_MAX_JOBNEW_NOTIFY; i++) {
job_t *j = (job_t *)xlist_get(jlist, i);
if (j != NULL)
g_jobnew_notify_jobids[g_jobnew_notify_count++] = j->job.jobid;
}
}
return 0;
}
static bool cbrp_jobnew_notified(uint64_t jobid)
{
int i;
for (i = 0; i < g_jobnew_notify_count; i++) {
if (g_jobnew_notify_jobids[i] == jobid)
return true;
}
return false;
}
/* Array-partial-construction-failure remediation (round-2 fix, task 3.1):
* when ONE array element's post-commit create_job() fails (a jobid
* collision that slipped past jobid_range_exists()'s pre-check, or any
* other construct-time failure), the OTHER array elements that constructed
* successfully — and are already durably committed — must still fire the
* AFTER_EVENT/JOB_NEW notification hook. Both scheduler and apiserver are
* purely event-driven off this hook with no periodic reconciliation, so a
* withheld notification strands an otherwise-healthy, already-durable job
* until the next controller restart replays the WAL.
*
* The colliding element's failure is reproduced deterministically (no real
* race) by exploiting the actual TOCTOU window between
* jobid_range_exists()'s pre-check (a READ-locked snapshot taken BEFORE any
* WAL work) and makeJob_'s own key_find (a WRITE-locked check made fresh at
* construction time, AFTER the commit barrier): a stand-in job for the
* second array element's jobid is inserted directly into c->jobmgr from the
* test thread strictly after the pre-check has already run (its own WAL
* entry is visible, which cannot happen until after the pre-check passes)
* and strictly before the construction loop can run (its commit is still
* pending on a withheld ACK). */
static void test_jobsubmit_array_partial_construct_failure_notifies_survivors(void)
{
cbrp_reply_obs_t obs = { 0 };
uint64_t lsn_before, wm0, wm1;
job_submit_data_t collide_submit;
bool elem0_wal_appeared, elem1_wal_appeared;
bool elem0_exists, elem0_notified;
/* Every TEST_ASSERT_* below is deferred until after cbrp_teardown().
* A failing Unity assertion longjmps out of this function (TEST_ABORT),
* which would otherwise skip cbrp_teardown() and leak live
* persist/commit-engine background threads into the NEXT test's setup,
* deadlocking the whole group (see bug-1408's round-2 correction: this
* exact failure mode was originally reproduced as a hang, not a clean
* FAIL, under the standard --run test_commit_barrier_reply invocation).
* Capture every observable into a local first, teardown unconditionally,
* then assert on the captured locals. */
cbrp_setup(2000);
cbrp_jobnew_capture_reset();
TEST_ASSERT_EQUAL_INT(0, xce_add_hook(&g_hc.hooksys, XCE_CONTROLLER_HOOK_JOB,
cbrp_jobnew_capture_fn, NULL));
cbrp_arrange_jobsubmit(9200, 2); /* 9200 survives, 9201 collides */
pthread_mutex_lock(&g_hc.persist.lock);
lsn_before = g_hc.persist.current_lsn;
@ -1028,70 +941,146 @@ static void test_jobsubmit_array_partial_construct_failure_notifies_survivors(vo
TEST_ASSERT_EQUAL_INT(0, pthread_create(&g_op_tid, NULL,
cbrp_op_thread_fn, &g_op_arg));
/* Element 0's (jobid 9200) WAL entry appears first; ACK it so the
* sequential payload loop advances to element 1. The settling sleep
* before each ACK mirrors cbrp_drive_acked's proven pattern — it gives
* the handler thread time to actually register its commit wait for the
* just-appended LSN before the ACK is delivered. */
wm0 = cbrp_poll_append(&g_hc.persist, lsn_before, 2000);
elem0_wal_appeared = (wm0 > 0);
cbrp_sleep_ms(150);
if (elem0_wal_appeared)
repl_on_ack(&g_hc.persist.repl, &g_hep_a, wm0);
/* No ACK is delivered before this join. If handle_jobsubmit still
* blocked on the majority-commit barrier (the pre-3.2 behavior), this
* join could not return before the 2000ms sync_timeout elapses; a fast
* return proves registration replaced blocking (Req 1.1, 1.4). */
t0 = cbrp_mono_ms();
TEST_ASSERT_EQUAL_INT(0, pthread_join(g_op_tid, NULL));
join_elapsed_ms = cbrp_mono_ms() - t0;
/* Element 1's (jobid 9201) WAL entry appears once element 0 commits.
* Its own ACK is deliberately withheld until the collision stand-in
* below is in place. cbrp_poll_append's lsn_before is a current_lsn
* SNAPSHOT (see its first use above, seeded from persist.current_lsn),
* not an entry's assigned LSN — element 0 already advanced current_lsn
* to wm0 + 1, so that (not wm0 itself) is the correct baseline to wait
* past for element 1's append. */
wm1 = cbrp_poll_append(&g_hc.persist, wm0 + 1, 2000);
elem1_wal_appeared = (wm1 > wm0);
cbrp_sleep_ms(150);
watermark = cbrp_poll_append(&g_hc.persist, lsn_before, 2000);
/* Deterministic TOCTOU-outcome injection (see block comment above). */
memset(&collide_submit, 0, sizeof(collide_submit));
cbrp_make_job(9201, &collide_submit);
/* Deliver the covering ACK, then poll for the hand-off artifact
* (raft-apply-decoupling task 3.2's producer-side registration) —
* NOT for a reply or a constructed job, neither of which this task
* produces on the committed-outcome path. */
repl_on_ack(&g_hc.persist.repl, &g_hep_a, watermark);
if (elem1_wal_appeared)
repl_on_ack(&g_hc.persist.repl, &g_hep_a, wm1);
has_pending = false;
t1 = cbrp_mono_ms();
while (cbrp_mono_ms() - t1 < 2000) {
if (persist_has_pending_apply(&g_hc.persist, watermark)) {
has_pending = true;
break;
}
cbrp_sleep_ms(5);
}
cbrp_poll_reply(&g_req_ep, 2000, &obs);
pthread_join(g_op_tid, NULL);
elem0_exists = cbrp_job_exists(9200);
elem0_notified = cbrp_jobnew_notified(9200);
/* Give the (non-existent) rest of the pipeline a beat, then confirm
* nothing leaked through — the task's own explicit "no independently
* observable behavior" contract for the committed-outcome path. */
cbrp_sleep_ms(200);
cbrp_scan_replies(&g_req_ep, &obs);
job_exists = cbrp_job_exists(9101);
cbrp_teardown();
TEST_ASSERT_TRUE_MESSAGE(elem0_wal_appeared,
"array element 0's WAL entry must appear");
TEST_ASSERT_TRUE_MESSAGE(elem1_wal_appeared,
"array element 1's WAL entry must appear");
TEST_ASSERT_TRUE_MESSAGE(join_elapsed_ms < 500,
"handle_jobsubmit must return before any commit outcome is known "
"(raft-apply-decoupling task 3.2, Req 1.1/1.4) — no ACK was ever "
"delivered before this join");
TEST_ASSERT_TRUE_MESSAGE(watermark > 0,
"job submission must still append its WAL entry");
TEST_ASSERT_TRUE_MESSAGE(has_pending,
"a committed entry must be handed off to the ordered apply pump "
"via persist_register_pending_apply, keyed by its lsn "
"(raft-apply-decoupling task 3.2)");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, obs.n_replies,
"no reply may be sent on the committed-outcome path until task 6's "
"apply pump exists (raft-apply-decoupling task 3.2's own contract)");
TEST_ASSERT_FALSE_MESSAGE(job_exists,
"no job may be constructed on the committed-outcome path until "
"task 6's apply pump exists (raft-apply-decoupling task 3.2)");
TEST_ASSERT_TRUE_MESSAGE(obs.n_replies > 0,
"the requester must still receive a reply when one array element "
"fails post-commit construction");
TEST_ASSERT_TRUE_MESSAGE(obs.last_status < 0,
"a partially-failed array submission must report the construction "
"failure to the client");
TEST_ASSERT_TRUE_MESSAGE(elem0_exists,
"the non-colliding array element must still be constructed and "
"present in c->jobmgr despite its sibling failing to construct");
TEST_ASSERT_TRUE_MESSAGE(elem0_notified,
"the successfully constructed sibling must fire the AFTER_EVENT/"
"JOB_NEW notification hook even though array element 9201 failed "
"to construct post-commit (raft-apply-decoupling task 3.1 "
"remediation) — otherwise it is durably committed and constructed "
"but invisible to the purely event-driven scheduler/apiserver "
"until the next controller restart replays the WAL");
printf("test_jobsubmit_array_partial_construct_failure_notifies_survivors:"
printf("test_jobsubmit_returns_before_commit_and_registers_pending_apply:"
" PASS\n");
}
/* raft-apply-decoupling task 3.2: the failure/timeout path must ALSO prove
* the handler returns immediately — cbrp_drive_withheld (used by
* test_jobsubmit_withheld_acks_no_phantom_job below) only proves the
* EVENTUAL reply/no-phantom-job observable, not that the calling thread
* stopped blocking well before the commit timeout elapses. */
static void test_jobsubmit_returns_before_commit_on_failure_path_too(void)
{
uint64_t lsn_before, watermark, t0, join_elapsed_ms;
cbrp_reply_obs_t obs = { 0 };
bool job_exists;
cbrp_setup(1000);
cbrp_arrange_jobsubmit(9104, 1);
pthread_mutex_lock(&g_hc.persist.lock);
lsn_before = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
g_op_arg.fn = handle_jobsubmit;
g_op_arg.req = &g_jobsub_req;
g_op_arg.rc = -9999;
TEST_ASSERT_EQUAL_INT(0, pthread_create(&g_op_tid, NULL,
cbrp_op_thread_fn, &g_op_arg));
/* No ACK is EVER delivered in this test. The 1000ms sync_timeout_ms is
* how long the ENTRY takes to resolve as FAILED — the HANDLER itself
* must return immediately, well before that timeout. */
t0 = cbrp_mono_ms();
TEST_ASSERT_EQUAL_INT(0, pthread_join(g_op_tid, NULL));
join_elapsed_ms = cbrp_mono_ms() - t0;
watermark = cbrp_poll_append(&g_hc.persist, lsn_before, 500);
/* Let the timeout sweep resolve the entry FAILED and confirm the
* (eventual) failure reply + no-phantom-job observable — same
* contract test_jobsubmit_withheld_acks_no_phantom_job already pins,
* verified here alongside the non-blocking-return proof. */
cbrp_poll_reply(&g_req_ep, 1000 + 1500, &obs);
job_exists = cbrp_job_exists(9104);
cbrp_teardown();
TEST_ASSERT_TRUE_MESSAGE(join_elapsed_ms < 500,
"handle_jobsubmit must return immediately after registering its "
"continuation, without waiting for the commit timeout to elapse "
"(raft-apply-decoupling task 3.2, Req 1.1/1.4)");
TEST_ASSERT_TRUE_MESSAGE(watermark > 0,
"job submission must still append its WAL entry before the "
"handler returns");
TEST_ASSERT_TRUE_MESSAGE(obs.n_replies > 0,
"a forced timeout must still eventually reply to the client");
TEST_ASSERT_EQUAL_INT_MESSAGE(-ETIMEDOUT, obs.last_status,
"a timed-out commit must reply with -ETIMEDOUT, the same "
"vocabulary the synchronous path already used (Req 2.3)");
TEST_ASSERT_FALSE_MESSAGE(job_exists,
"no job may ever be constructed for an entry whose commit timed "
"out (raft-apply-decoupling task 3.2, Req 2.3)");
printf("test_jobsubmit_returns_before_commit_on_failure_path_too: PASS\n");
}
/* REMOVED (raft-apply-decoupling task 3.2): the array-partial-construction-
* failure remediation test (task 3.1 round-2 fix, bug-1408) that lived here
* drove handle_jobsubmit's OLD sequential, per-element BLOCKING append loop
* — it ACKed array element 0 and only then observed element 1's WAL entry
* appear, exploiting that strict ordering to inject a deterministic jobid
* collision between the two elements' commits. Task 3.2 appends every array
* element back-to-back, non-blocking, with no per-element ACK gate between
* them, so that ordering assumption no longer holds — the test's whole
* premise is gone, not just its expected outcome.
*
* More importantly, the CODE this test exercised (handle_jobsubmit's own
* post-commit create_job()-loop + jlist-build + xce_run_hooks(AFTER_EVENT)
* block) no longer exists in job.c at all (see the "REMOVED" comment above
* handle_jobsubmit's async registration block) — it is superseded, not
* relocated, by apply_committed_entry()'s existing WAL_JOB_SUBMIT case
* (persist.c, task 2.2), which already does restore_job() + fires
* HOOK_DETAIL_JOB_NEW per entry, independently per array element, and is
* therefore structurally immune to the bug-1408 class already. Coverage for
* that invariant now belongs to persist.c's own apply-seam test suite
* (test_apply_seam.c), not here. See this task's status report for the
* full reasoning; git history (commit 6f1fa97) preserves the removed test
* verbatim. */
/* ════════════════════════════════════════════════════════════════════════
* Receipt-ACK verdict pins (ha-commit-barrier-rollout task 3.3, Req 2.3)
*
@ -2133,8 +2122,8 @@ void test_commit_barrier_reply(void)
RUN_TEST(test_worker_join_no_hostconfig_error_reply_immediate);
/* handle_jobsubmit (raft-apply-decoupling task 3.1, Req 2.1/2.2/2.3) */
RUN_TEST(test_jobsubmit_withheld_acks_no_phantom_job);
RUN_TEST(test_jobsubmit_success_reply_and_job_visible_after_commit);
RUN_TEST(test_jobsubmit_array_partial_construct_failure_notifies_survivors);
RUN_TEST(test_jobsubmit_returns_before_commit_and_registers_pending_apply);
RUN_TEST(test_jobsubmit_returns_before_commit_on_failure_path_too);
/* Receipt-ACK verdict pins (task 3.3, Req 2.3) — green on landing */
RUN_TEST(test_jobdispatch_receipt_ack_no_wal_append);
RUN_TEST(test_set_preason_receipt_reply_fact_retained);