wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

feat(raft-apply-decoupling): ordered live apply pump activates apply_committed_entry (task 6.1)

Browse source 
Build persist_apply_pump_run()/persist_apply_pump_kick() -- the
ordered walk from last_applied_lsn+1 through committed_lsn, applying
every entry via apply_committed_entry() regardless of commit mode.
This is the first live (non-replay) consumer of that dispatch
function. Kick is wired into commit_resolve_pending() (the single
choke point all 6 internal committed_lsn-advancing paths funnel
through, discovered in task 1.2), dispatched via
vtask_enq_classified(VTASK_LIVE_BLOCK) with a CAS-coalesced
fire-and-forget trigger.

Fixes the two hazards flagged since task 4.1 as required before
going live:

1. Double-append: a thread-local call-provenance guard
   (t_apply_seam_notifying) prevents persist_job_hook_fun/
   persist_queue_hook_fun/persist_worker_hook_fun from re-appending
   entries the apply pump itself just applied, while preserving
   job_ctrl's legitimate live command-path appends (a naive
   detail-based filter would have broken SIGCONT/SIGSTOP). Also
   closes an undocumented instance of the same hazard in the
   queue/worker hook functions.

2. Merge-staleness: record_job_fail_fact/postdone's fact recording
   previously baked a receipt-time-computed final job state into the
   WAL entry, which goes stale under deferred apply -- a FINISH
   followed shortly by an ordinary post-exec FAIL could silently
   clobber a correct DONE completion with a stale RUN|EXIT|PERR, no
   guard catching it. Fixed by repurposing wal_jobstate_t's unused
   _pad field to carry just enough data (FAIL's stage) to recompute
   against the job's actual current state at apply time via the real
   apply_job_fail_transition()/new apply_job_postdone_transition()
   guard cores, instead of overwriting with a stale absolute value.
   Backward compatible: _pad==0 (all prior data) routes to the same
   legacy overwrite behavior.

Also fixes bug-1409 (unbounded busy-loop in apply_pump_drain when
committed_lsn advances without a corresponding on-disk entry) and a
genuine test-flakiness hazard (a second, independent apply-pump
kicker from the fsync_worker thread that existing tests didn't
account for).

Two pre-existing tests in test_commit_barrier_reply.c now fail as
expected -- their own assertions state "no job may be constructed
until task 6's apply pump exists," now correctly invalidated by this
task landing; updating them is task 7.3's explicit job.
This commit is contained in:
Rongsong Shen 2026-07-04 19:00:43 +00:00
parent d3da3ca8bd
commit 2fa955b92e
11 changed files with 1687 additions and 135 deletions
Download patch file Download diff file Expand all files Collapse all files

715
src/controller/testsuite/test_apply_pump.c Normal file
View file

@ -0,0 +1,715 @@
/* test_apply_pump.c — raft-apply-decoupling task 6.1
*
* Proves the ordered apply pump: the strict-order walk that drives
* apply_committed_entry() (task 2.2) off live commit-advance for the very
* first time — before this task, only restart replay (task 2.3) ever called
* it. Covers:
*
* 1. The task's own named observable: a burst of entries appended in one
* order, whose majority commit becomes known all at once (a single
* covering ACK advancing committed_lsn past all of them in one jump),
* is applied in strict ascending LSN order — never reordered, never
* skipped — regardless of commit mode (a WAL_JOB_SUBMIT WAIT-mode entry
* mixed with WAL_JOB_{START,FINISH} RETAIN-mode facts in the same
* burst). Also folds in the Hazard 1 (double-append) regression check:
* current_lsn must not grow beyond the entries this test itself
* appended, proving the live pump's own post-mutation notification
* never re-appends an already-committed entry.
* 2. persist_apply_pump_run() called directly (white-box), confirming the
* walk applies exactly the committed-but-unapplied range and advances
* last_applied_lsn accordingly.
* 3. A non-regression check for the Hazard 1 fix's precision: job_ctrl's
* SIGCONT/SIGSTOP path (mark_job_update) still durably appends its own
* WAL_JOB_UPDATE entry — proving the double-append guard is scoped by
* call provenance (apply-seam-only), not by hook detail value, since
* job_ctrl fires the identical HOOK_DETAIL_JOB_UPDATE detail for its
* own, still-legitimate live append.
*
* Fixture mirrors test_commit_barrier_reply.c's cbrp_setup (real persist_init
* + persist_start over a private tmpdir so persist_start registers the
* production hooks, a real 3-member commit_engine_t with two stub standbys,
* ACKs delivered explicitly via repl_on_ack), trimmed of the worker-join
* scaffolding this file does not need. commit_set_pool is deliberately never
* called, so ce->pool stays NULL — persist_apply_pump_kick's documented
* NULL-pool fallback then runs the walk INLINE, synchronously, on whichever
* thread's kick wins the CAS-coalescing race.
*
* IMPORTANT (round-2 review correction): that does NOT mean every assertion
* below is directly observable immediately after a triggering call returns
* with no polling needed — persist_start's fsync_worker thread independently
* drives its own kick (commit_on_durable_advance) whenever durable_lsn
* advances, racing with this test's own repl_on_ack-driven kick. The
* coalescing guarantees exactly one kick actually runs the walk, but it can
* be the FSYNC THREAD's, asynchronously, after repl_on_ack has already
* returned on the main test thread. Tests that trigger a commit and then
* need to observe its APPLIED effect (not just committed_lsn) use
* tap_wait_applied() to poll persist_last_applied_lsn() before reading
* final state — see that helper's doc comment for the full mechanism and
* the flake it fixes.
*
* Every test below captures ALL assertion-relevant observables into local
* variables and calls tap_teardown() UNCONDITIONALLY before running any
* TEST_ASSERT — a failed assertion's longjmp must never skip teardown, or it
* hangs the next test in this binary (the known longjmp-past-teardown
* pathology from task 3.1's round-2 lesson, tasks.md Implementation Notes).
*/
#include <rtti.h>
#include <capnp_c.h>
#include <errno.h>
#include <signal.h>
#include <stdbool.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <unity/unity.h>
#include "cluster.h"
#include "commit.h"
#include "controller.capnp.h"
#include "hook.h"
#include "jobj.h"
#include "persist.h"
#include "repl.h"
#include "wal_types.h"
#include "xce/cluster_def.h"
#include "xce/types.h"
#include "xstring.h"
/* jobj.c's jobsub table and the process-global xstring table are shared
* across every test file in the controller_unit_test binary — guard
* idempotently, mirroring every other raft-apply-decoupling test file. */
static int g_tap_jobobj_ready;
static cluster_t g_tap_c;
static commit_engine_t g_tap_hce;
static int g_tap_hep_a, g_tap_hep_b; /* dummy standby endpoints */
static char g_tap_tmpdir[256];
/* Test-injectable send hook (repl.c, XCE_TESTING) — keeps repl_broadcast from
* dereferencing the dummy standby endpoints above, mirroring
* test_commit_barrier_reply.c's identical cbrp_drop_send_hook. */
extern int (*repl_broadcast_send_hook)(void *ep, wal_catchup_entry_t **entries,
int n_entries);
static int tap_drop_send_hook(void *ep, wal_catchup_entry_t **entries,
int n_entries)
{
(void)ep; (void)entries; (void)n_entries;
return 0; /* pretend success, drop the data */
}
/* ── Spy hook: records the exact (detail, jobid) sequence of every
* AFTER_EVENT firing on XCE_CONTROLLER_HOOK_JOB, in the order they occur. ── */
#define TAP_SPY_CAP 32
typedef struct {
int detail;
uint64_t jobid;
} tap_spy_entry_t;
static tap_spy_entry_t g_tap_spy[TAP_SPY_CAP];
static int g_tap_spy_n;
static int tap_spy_hook(int type, int situation, int detail, void* data,
va_list ap)
{
(void)type; (void)data;
if (situation != HOOK_AFTER_EVENT)
return 0;
uint64_t jobid = 0;
switch (detail) {
case HOOK_DETAIL_JOB_NEW: {
/* args: cluster_t*, xlist_t* (apply_committed_entry's WAL_JOB_SUBMIT
* branch) -- consume the leading cluster_t* first, mirroring
* persist_job_hook_fun's own va_arg pattern for this hook type. */
(void)va_arg(ap, cluster_t*);
xlist_t* jlist = va_arg(ap, xlist_t*);
if (jlist && jlist->count > 0) {
job_t* j = (job_t*)((void**)jlist->data)[0];
jobid = j->job.jobid;
}
break;
}
case HOOK_DETAIL_JOB_UPDATE: {
/* args: cluster_t*, job_t* -- same leading-arg shape. */
(void)va_arg(ap, cluster_t*);
job_t* j = va_arg(ap, job_t*);
if (j)
jobid = j->job.jobid;
break;
}
default:
return 0; /* not relevant to this file's assertions */
}
if (g_tap_spy_n < TAP_SPY_CAP) {
g_tap_spy[g_tap_spy_n].detail = detail;
g_tap_spy[g_tap_spy_n].jobid = jobid;
g_tap_spy_n++;
}
return 0;
}
static void tap_spy_reset(void)
{
g_tap_spy_n = 0;
memset(g_tap_spy, 0, sizeof(g_tap_spy));
}
static void tap_setup(void)
{
kvmgr_params_t str_params = { .key_type = KVMGR_KEY_STRING,
.nslots = 32,
.delta = 16 };
kvmgr_params_t int_params = { .key_type = KVMGR_KEY_INT,
.nslots = 32,
.delta = 16 };
if (makeString((char*)"tap-probe") == NULL)
xstring_table_init(64);
if (!g_tap_jobobj_ready) {
TEST_ASSERT_EQUAL_INT(0, job_obj_init(32, 32));
g_tap_jobobj_ready = 1;
}
memset(&g_tap_c, 0, sizeof(g_tap_c));
g_tap_c.myhost = &g_tap_c.myhost_info;
g_tap_c.myhost_info.name = (char*)"ctrl-test";
g_tap_c.myhost_info.key = (char*)"";
TEST_ASSERT_EQUAL_INT(0,
xce_hook_init(&g_tap_c.hooksys, XCE_CONTROLLER_HOOK_MAX));
g_tap_c.jobmgr = makeInstance(&xkvmgr_type, &int_params);
g_tap_c.qmgr = makeInstance(&xkvmgr_type, &str_params);
g_tap_c.workermgr = makeInstance(&xkvmgr_type, &str_params);
g_tap_c.resmgr = makeInstance(&xkvmgr_type, &str_params);
TEST_ASSERT_NOT_NULL(g_tap_c.jobmgr);
TEST_ASSERT_NOT_NULL(g_tap_c.qmgr);
TEST_ASSERT_NOT_NULL(g_tap_c.workermgr);
TEST_ASSERT_NOT_NULL(g_tap_c.resmgr);
zarray_init(&g_tap_c.sharedinst, 16, NULL);
persist_config_t cfg = persist_config_defaults();
cfg.wal_sync_ms = 5000;
cfg.sync_timeout_ms = 2000;
snprintf(g_tap_tmpdir, sizeof(g_tap_tmpdir), "/tmp/xce_tap61_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(g_tap_tmpdir));
persist_set_workdir(&g_tap_c.persist, g_tap_tmpdir, "ctrl");
TEST_ASSERT_EQUAL_INT(0, persist_init(&g_tap_c.persist, &cfg, &g_tap_c));
TEST_ASSERT_EQUAL_INT(0, persist_start(&g_tap_c.persist));
/* 3-member cluster: 2 stub standbys, engine wired, ACKs withheld until a
* test explicitly delivers them via repl_on_ack. commit_set_pool is
* deliberately never called -- see file header for why. */
TEST_ASSERT_TRUE(
repl_add_standby(&g_tap_c.persist.repl, &g_tap_hep_a, "sb_a", 7500) >= 0);
TEST_ASSERT_TRUE(
repl_add_standby(&g_tap_c.persist.repl, &g_tap_hep_b, "sb_b", 7501) >= 0);
TEST_ASSERT_EQUAL_INT(0,
commit_engine_init(&g_tap_hce, &g_tap_c.persist, 3, 8));
repl_set_commit_engine(&g_tap_c.persist.repl, &g_tap_hce);
repl_broadcast_send_hook = tap_drop_send_hook;
TEST_ASSERT_EQUAL_INT(0,
xce_add_hook(&g_tap_c.hooksys, XCE_CONTROLLER_HOOK_JOB, tap_spy_hook,
&g_tap_c));
tap_spy_reset();
}
static void tap_teardown(void)
{
repl_broadcast_send_hook = NULL;
commit_engine_fini(&g_tap_hce);
persist_stop(&g_tap_c.persist);
persist_fini(&g_tap_c.persist);
zarray_fini(&g_tap_c.sharedinst);
if (g_tap_c.resmgr) dereference(g_tap_c.resmgr);
if (g_tap_c.workermgr) dereference(g_tap_c.workermgr);
if (g_tap_c.qmgr) dereference(g_tap_c.qmgr);
if (g_tap_c.jobmgr) dereference(g_tap_c.jobmgr);
xce_hook_fini(&g_tap_c.hooksys);
}
/* Creates a job directly in the fixture's jobmgr, bypassing WAL entirely —
* for tests that only need a pre-existing job for a fact to transition,
* mirroring test_apply_reapply_idempotency.c's tari_seed_job. */
static job_submit_data_t g_tap_seed_submit;
static job_t* tap_seed_job(uint64_t jobid, uint32_t state)
{
xce_job_t jdef;
memset(&g_tap_seed_submit, 0, sizeof(g_tap_seed_submit));
g_tap_seed_submit.queue = (char*)"q";
g_tap_seed_submit.user = (char*)"tester";
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = jobid;
jdef.state = state;
jdef.submit = &g_tap_seed_submit;
return makeJob(g_tap_c.jobmgr, &jdef);
}
static job_t* tap_find_job(uint64_t jobid)
{
job_t* j = NULL;
lock(g_tap_c.jobmgr, RTTI_LOCK_READ);
int found = key_find(g_tap_c.jobmgr, (void*)jobid, (void**)&j);
unlock(g_tap_c.jobmgr);
return (found > 0) ? j : NULL;
}
/* A standby ACK only counts toward majority once this node's OWN copy is
* durable (persist_durable_lsn, capped by the group-commit flusher's own
* fdatasync cadence -- ha-commit-durability's majority_candidate_locked
* caps self's contribution at durable_lsn, never the raw appended tip).
* persist_wal_async signals the flusher's condvar immediately after each
* append, but the flusher still runs on its own thread -- poll briefly
* rather than assume it has already caught up by the time this function
* returns, mirroring cbrp_poll_append's identical polling idiom in
* test_commit_barrier_reply.c. */
static void tap_wait_durable(uint64_t lsn, uint32_t deadline_ms)
{
for (uint32_t waited = 0; waited < deadline_ms; waited += 5) {
if (atomic_load(&g_tap_c.persist.durable_lsn) >= lsn)
return;
usleep(5000);
}
}
/* raft-apply-decoupling task 6.1 round 2 review: waits for the ordered apply
* pump to actually finish applying up through `lsn` after a triggering event
* (repl_on_ack) returns. This is needed even though ce->pool is NULL here
* (persist_apply_pump_kick's documented inline-fallback runs the walk on the
* CALLING thread rather than a vtask worker) -- a SEPARATE real thread, the
* persist group-commit flusher (fsync_worker, started by persist_start in
* tap_setup), ALSO independently drives a kick via commit_on_durable_advance
* every time durable_lsn itself advances, regardless of any ACK. That kick
* races with this test's own repl_on_ack-driven kick on the main thread.
* persist_apply_pump_kick's CAS-coalescing (apply_pump_scheduled) guarantees
* exactly one of the two ever actually runs the walk at a time -- but the
* LOSING side's caller returns immediately without having applied anything
* itself, handing responsibility to whichever thread already holds the
* flag. That is by design: the kick is "fire and forget" (coalesced,
* eventually-consistent dispatch), never "kick and block until this
* specific walk completes". So a test that calls repl_on_ack and then reads
* job state or hook-fire counts immediately cannot assume the walk that
* will apply its entries ran synchronously on ITS OWN thread, even with
* ce->pool NULL -- it may instead complete asynchronously, moments later, on
* the fsync worker thread. Poll persist_last_applied_lsn() (which only
* advances after a real apply pass, from whichever thread ran it) rather
* than assume repl_on_ack's return implies completion -- confirmed via
* review to reproduce as a genuine, reproducible flake (observed at ~7.5%,
* both in this file's burst-order test and the FAIL/FINISH race test)
* without this wait. */
static void tap_wait_applied(uint64_t lsn, uint32_t deadline_ms)
{
for (uint32_t waited = 0; waited < deadline_ms; waited += 5) {
if (persist_last_applied_lsn(&g_tap_c.persist) >= lsn)
return;
usleep(5000);
}
}
/* ── 1. The task's named observable + Hazard 1 regression ───────────────── */
static void test_apply_pump_kick_applies_committed_burst_in_ascending_lsn_order(
void)
{
tap_setup();
job_t* j901 = tap_seed_job(901, JOB_STAT_PEND);
job_t* j902 = tap_seed_job(902, JOB_STAT_PEND);
int seeded_ok = (j901 != NULL) && (j902 != NULL);
uint64_t first_lsn = g_tap_c.persist.current_lsn;
/* Append three facts, deliberately in an order that is NOT sorted by
* jobid and NOT the order a human might narrate the story in — job 902
* (START then FINISH) interleaved around job 901's single START. LSN
* order is exactly the append order below: 902/RUN, 901/RUN, 902/DONE. */
int rc1 = persist_submit_job_fact(&g_tap_c.persist, WAL_JOB_START, 902,
JOB_STAT_RUN);
int rc2 = persist_submit_job_fact(&g_tap_c.persist, WAL_JOB_START, 901,
JOB_STAT_RUN);
int rc3 = persist_submit_job_fact(&g_tap_c.persist, WAL_JOB_FINISH, 902,
JOB_STAT_DONE);
uint64_t watermark = g_tap_c.persist.current_lsn - 1;
/* Snapshot the pre-commit state before delivering any ACK: nothing may
* have mutated or notified yet (Req 5.1, 5.3, 6.1). */
uint32_t j901_state_precommit = seeded_ok ? j901->job.state : 0;
uint32_t j902_state_precommit = seeded_ok ? j902->job.state : 0;
int spy_n_precommit = g_tap_spy_n;
/* One covering ACK jumps committed_lsn past all three in a single
* contiguous-prefix advance -- commit_resolve_pending's unconditional
* persist_apply_pump_kick(ce->p, ce->pool) call (task 6.1's commit.c
* wiring) fires here; ce->pool is NULL in this fixture, so the walk
* runs inline, synchronously, before repl_on_ack returns. Self's own
* copy must be durable up to `watermark` first, or the ACK cannot lift
* committed_lsn that far (self caps its own contribution at
* durable_lsn, never the raw appended tip). */
tap_wait_durable(watermark, 2000);
repl_on_ack(&g_tap_c.persist.repl, &g_tap_hep_a, watermark);
/* See tap_wait_applied's doc comment: the fsync-worker thread's own
* durable-advance-driven kick can win the coalescing race and apply this
* burst asynchronously, after repl_on_ack already returned here. */
tap_wait_applied(watermark, 2000);
uint64_t committed_after = atomic_load(&g_tap_c.persist.committed_lsn);
int spy_n_after = g_tap_spy_n;
tap_spy_entry_t spy_snap[TAP_SPY_CAP];
memcpy(spy_snap, g_tap_spy, sizeof(spy_snap));
uint32_t j901_state_final = seeded_ok ? j901->job.state : 0;
uint32_t j902_state_final = seeded_ok ? j902->job.state : 0;
uint64_t last_applied_final = persist_last_applied_lsn(&g_tap_c.persist);
uint64_t current_lsn_final = g_tap_c.persist.current_lsn;
tap_teardown();
TEST_ASSERT_TRUE(seeded_ok);
TEST_ASSERT_EQUAL_INT(0, rc1);
TEST_ASSERT_EQUAL_INT(0, rc2);
TEST_ASSERT_EQUAL_INT(0, rc3);
TEST_ASSERT_EQUAL_UINT64(first_lsn + 2, watermark);
/* Nothing has committed yet (ACKs withheld) -- nothing may be applied. */
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_PEND, j901_state_precommit,
"a fact must not mutate state before it commits");
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_PEND, j902_state_precommit,
"a fact must not mutate state before it commits");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, spy_n_precommit,
"no notification may fire before any entry in the burst commits");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(watermark, committed_after,
"the covering ACK must advance committed_lsn to the full watermark");
/* Strict order, no skipping or reordering (Req 3.3, 7.3): the spy's
* recorded sequence must be EXACTLY the append/LSN order, never sorted
* by jobid, never reflecting any other ordering scheme. */
TEST_ASSERT_EQUAL_INT_MESSAGE(3, spy_n_after,
"the burst must apply as exactly three notifications, one per "
"committed entry, in a single pump pass");
TEST_ASSERT_EQUAL_INT(HOOK_DETAIL_JOB_UPDATE, spy_snap[0].detail);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(902, spy_snap[0].jobid,
"entry 1 (lowest LSN) must apply first, regardless of jobid");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(901, spy_snap[1].jobid,
"entry 2 must apply second, in strict LSN order");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(902, spy_snap[2].jobid,
"entry 3 (highest LSN) must apply last, even though it targets the "
"SAME job as entry 1 -- LSN order, not jobid grouping, governs");
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_RUN, j901_state_final,
"job 901 must reflect its single applied fact");
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_DONE, j902_state_final,
"job 902 must reflect its LAST applied fact (FINISH), proving "
"entry 2 (job 901, in between) did not get skipped or cause "
"job 902's later entry to apply out of turn");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(watermark, last_applied_final,
"last_applied_lsn must advance to exactly the applied watermark");
/* Hazard 1 regression: applying these facts live must not re-append
* them. Without the t_apply_seam_notifying guard, each HOOK_DETAIL_
* JOB_UPDATE notification above would have looped back through
* persist_job_hook_fun and appended a duplicate WAL_JOB_UPDATE entry,
* growing current_lsn past this test's own three appends. */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(watermark + 1, current_lsn_final,
"the live apply pump must never re-append an already-committed "
"entry (Hazard 1) -- current_lsn must reflect only this test's own "
"three appends, nothing more");
printf("test_apply_pump_kick_applies_committed_burst_in_ascending_lsn_"
"order: PASS\n");
}
/* ── 2. persist_apply_pump_run() direct call: mixed command + fact ──────── */
static int tap_encode_job(const xce_job_t* j, uint8_t** out, size_t* out_len)
{
struct capn c;
struct capn_segment* cs;
struct Job cj;
Job_ptr jp;
capn_init_malloc(&c);
cs = capn_root(&c).seg;
encode_Job(cs, &cj, (xce_job_t*)j);
jp = new_Job(cs);
write_Job(&cj, jp);
capn_setp(capn_root(&c), 0, jp.p);
*out_len = capn_size(&c);
*out = malloc(*out_len);
if (!*out) { capn_free(&c); return -ENOMEM; }
capn_write_mem(&c, *out, *out_len, 0);
capn_free(&c);
return 0;
}
static void test_apply_pump_run_applies_mixed_submit_and_fact_entries(void)
{
tap_setup();
job_t* j903 = tap_seed_job(903, JOB_STAT_PEND);
int seeded_ok = (j903 != NULL);
/* Entry 1 (lower LSN): a fact for the pre-existing job. */
int rc1 = persist_submit_job_fact(&g_tap_c.persist, WAL_JOB_START, 903,
JOB_STAT_RUN);
/* Entry 2 (higher LSN): a WAL_JOB_SUBMIT command entry for a brand-new
* job, appended directly via persist_wal_async (bypassing the full
* commit_submit_async/continuation pipeline, which is task 3.2/6.2's
* concern, not this walk's) -- this file only needs the entry's bytes
* to exist on disk at a known LSN, exactly what the live pump consumes
* regardless of how the entry got there. */
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = 904;
jdef.state = JOB_STAT_PEND;
jdef.submit = &g_tap_seed_submit;
uint8_t* payload = NULL;
size_t plen = 0;
int enc_rc = tap_encode_job(&jdef, &payload, &plen);
int rc2 = persist_wal_async(&g_tap_c.persist, WAL_JOB_SUBMIT,
payload, (uint32_t)plen);
free(payload);
uint64_t watermark = g_tap_c.persist.current_lsn - 1;
/* White-box: mark the whole range committed directly (this test targets
* the walk's own dispatch logic, not commit-advance itself -- that is
* test 1's job) and drive the walk with the exposed, directly-callable
* entry point. */
atomic_store(&g_tap_c.persist.committed_lsn, watermark);
int applied = persist_apply_pump_run(&g_tap_c.persist);
int applied2 = persist_apply_pump_run(&g_tap_c.persist);
uint32_t j903_state_final = seeded_ok ? j903->job.state : 0;
job_t* j904 = tap_find_job(904);
uint32_t j904_state_final = j904 ? j904->job.state : 0;
uint64_t last_applied_final = persist_last_applied_lsn(&g_tap_c.persist);
tap_teardown();
TEST_ASSERT_TRUE(seeded_ok);
TEST_ASSERT_EQUAL_INT(0, rc1);
TEST_ASSERT_EQUAL_INT(0, enc_rc);
TEST_ASSERT_EQUAL_INT(0, rc2);
/* raft-apply-decoupling task 6.1 round 2 review: this fixture's
* commit_engine_t has a real fsync_worker thread wired (tap_setup), and
* persist_submit_job_fact/persist_wal_async above are real appends --
* once each becomes durable, commit_on_durable_advance (persist.c) fires
* on the FSYNC THREAD and unconditionally calls persist_apply_pump_kick,
* completely independent of this test never calling repl_on_ack. That
* background kick can race this test's own two direct
* persist_apply_pump_run() calls below and win the apply_pump_lock
* first -- applying both entries itself before either of this test's
* own calls gets a turn (persist_apply_pump_run is mutex-serialized and
* fully synchronous, so whichever caller actually performs the walk
* always completes it before returning -- there is no partial/in-flight
* state to poll for here, unlike tests 1/4's tap_wait_applied
* situation). When the background thread wins BEFORE either of this
* test's own calls, BOTH `applied` and `applied2` legitimately read 0
* (nothing was left for this test's own calls to do) even though every
* entry did get applied, correctly, by the background thread's own
* separate call -- so neither the individual values nor their sum can
* be asserted exactly in this shared-thread fixture without a deeper
* synchronization change. What IS deterministic, and what this test
* exists to actually prove, is a purely state-based invariant: every
* committed entry ends up applied by SOMEONE exactly once, regardless
* of commit mode, and the final materialized state is correct --
* checked below via last_applied_lsn and the two jobs' live state,
* never via the racy call-arity split. Observed empirically via review:
* the original strict `applied==2 && applied2==0` split flaked at
* ~1.6% under this exact race. */
TEST_ASSERT_TRUE_MESSAGE(applied >= 0 && applied <= 2 &&
applied2 >= 0 && applied2 <= 2,
"the walk must never report a negative or over-large apply count");
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_RUN, j903_state_final,
"the fact entry must have applied");
TEST_ASSERT_NOT_NULL_MESSAGE(j904,
"the WAL_JOB_SUBMIT command entry must have applied too -- the walk "
"does not skip command-mode entries");
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_PEND, j904_state_final);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(watermark, last_applied_final,
"last_applied_lsn must reach the watermark no matter which thread "
"performed the walk -- this, not the applied/applied2 call-arity "
"split, is the authoritative proof both entries were applied "
"exactly once");
printf("test_apply_pump_run_applies_mixed_submit_and_fact_entries: "
"PASS\n");
}
/* ── 3. Hazard 1 guard precision: job_ctrl's own live append must survive ── */
static void test_apply_pump_guard_does_not_break_job_ctrl_live_append(void)
{
tap_setup();
/* A suspended, not-yet-started job (allocs == NULL) -- job_ctrl's
* SIGCONT branch calls mark_job_update() directly, synchronously
* mutating state and firing BEFORE/AFTER_EVENT HOOK_DETAIL_JOB_UPDATE
* itself, exactly the same detail apply_committed_entry() now also
* fires for live-applied facts (test 1 above). This is the dual-origin
* case the t_apply_seam_notifying guard must NOT break. */
job_t* j = tap_seed_job(905, JOB_STAT_PSUSP);
int seeded_ok = (j != NULL) && (j->job.allocs == NULL);
uint64_t first_lsn = g_tap_c.persist.current_lsn;
xce_signal_t sig = { .signo = SIGCONT, .force = 0 };
xce_jobctrl_t ctrl = { .action = JobControl_u_siginfo, .jobid = 905 };
ctrl.u.siginfo = &sig;
int rc = job_ctrl(&g_tap_c, &ctrl);
uint32_t state_final = seeded_ok ? j->job.state : 0xffffffffu;
uint64_t current_lsn_final = g_tap_c.persist.current_lsn;
tap_teardown();
TEST_ASSERT_TRUE(seeded_ok);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_UINT32_MESSAGE(0, state_final & JOB_STAT_PSUSP,
"SIGCONT must clear the suspend bit synchronously, as it always "
"has -- job_ctrl's command path is untouched by task 6.1");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(first_lsn + 1, current_lsn_final,
"job_ctrl's own live append via persist_job_hook_fun must still "
"happen exactly once -- the apply-seam double-append guard is "
"scoped to apply_committed_entry()'s own call stack (a thread-local "
"flag), not to the HOOK_DETAIL_JOB_UPDATE value itself, so it must "
"never suppress this legitimate, unrelated live append");
printf("test_apply_pump_guard_does_not_break_job_ctrl_live_append: "
"PASS\n");
}
/* ── 4. Merge-staleness regression: FAIL must not clobber a racing FINISH ─
*
* Reproduces the round-1 corruption bug found by review: a FINISH fact
* (deterministic, state-independent -- exit_code alone decides DONE vs
* EXIT) and a POST_EXEC-stage FAIL fact for the SAME job are received in
* two separate commit rounds, with the FAIL's receipt happening BEFORE
* either fact has applied (both are durably appended and registered, but
* apply is deferred past the commit barrier -- exactly task 6.1's point).
* Driven through the REAL receipt handlers (on_job_finish/on_job_fail), not
* seeded directly, so the genuine staleness window is exercised: at FAIL
* receipt time, the job's real in-memory state is still RUN (FINISH has
* not applied yet). Old (round-1) code baked a snapshot of that STALE RUN
* state into the FAIL's WAL entry, so once both entries applied in LSN
* order the FAIL's blind overwrite clobbered FINISH's DONE with a stray
* RUN bit, corrupting the job to RUN|EXIT|PERR instead of the correct
* merged terminal state DONE|PERR. */
static void test_apply_pump_fail_racing_finish_merges_against_current_state(
void)
{
tap_setup();
job_t* j906 = tap_seed_job(906, JOB_STAT_RUN);
int seeded_ok = (j906 != NULL);
uint64_t first_lsn = g_tap_c.persist.current_lsn;
/* Round 1: worker reports FINISH (exit_code 0 -> DONE). Durably
* appended only -- record_job_finish_fact fires no AFTER_EVENT and
* mutates nothing, so j906->job.state must still read RUN right after
* this call returns. */
xce_jobfinish_t finish = { .jobid = 906, .exit_code = 0 };
int rc_finish = on_job_finish(&g_tap_c, &finish);
/* Round 2: BEFORE either fact has applied, a POST_EXEC-stage FAIL
* arrives for the same job (an ordinary HPC lifecycle event -- e.g. an
* epilogue script failing shortly after the main process exits). At
* this exact instant, job.state is STILL RUN -- the genuine staleness
* window this test reproduces, not a seeded/simulated one. */
uint32_t state_at_fail_receipt = seeded_ok ? j906->job.state : 0;
job_failinfo_t failinfo = { .jobid = 906,
.stage = JOB_EXEC_STAGE_POST_EXEC,
.code = 1,
.detail = NULL };
int rc_fail = on_job_fail(&g_tap_c, &failinfo);
uint32_t state_before_commit = seeded_ok ? j906->job.state : 0;
uint64_t watermark = g_tap_c.persist.current_lsn - 1;
/* One covering ACK commits both entries at once; the real
* commit_resolve_pending -> persist_apply_pump_kick production path
* applies them in strict LSN order (FINISH first, then FAIL), exactly
* as task 6.1's pump guarantees. */
tap_wait_durable(watermark, 2000);
repl_on_ack(&g_tap_c.persist.repl, &g_tap_hep_a, watermark);
/* See tap_wait_applied's doc comment: the fsync-worker thread's own
* durable-advance-driven kick can win the coalescing race and apply
* these two entries asynchronously, after repl_on_ack already returned
* here -- this is the exact mechanism behind the round-2 self-disclosed
* flake, not a corruption bug in the merge-staleness fix itself. */
tap_wait_applied(watermark, 2000);
uint64_t committed_after = atomic_load(&g_tap_c.persist.committed_lsn);
uint32_t state_final = seeded_ok ? j906->job.state : 0;
tap_teardown();
TEST_ASSERT_TRUE(seeded_ok);
TEST_ASSERT_EQUAL_INT(0, rc_finish);
TEST_ASSERT_EQUAL_INT(0, rc_fail);
TEST_ASSERT_EQUAL_UINT64(first_lsn + 1, watermark);
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_RUN, state_at_fail_receipt,
"the FAIL fact must be received while the job is still genuinely "
"RUN -- proving FINISH has not applied yet, the real staleness "
"window, not a simulated one");
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_RUN, state_before_commit,
"neither fact may mutate state before its entry commits (Req 5.1)");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(watermark, committed_after,
"the covering ACK must advance committed_lsn past both entries");
TEST_ASSERT_EQUAL_UINT32_MESSAGE((JOB_STAT_DONE | JOB_STAT_PERR),
state_final,
"FAIL's PERR merge must be computed against FINISH's already-"
"applied DONE (correct terminal state) -- not a stale RUN snapshot "
"taken at FAIL's receipt time");
TEST_ASSERT_EQUAL_UINT32_MESSAGE(0, state_final & JOB_STAT_RUN,
"the stray RUN bit from the round-1 corruption bug must never "
"appear in the final state");
TEST_ASSERT_TRUE_MESSAGE((state_final & JOB_STAT_DONE) != 0,
"FINISH's DONE bit must survive -- it must never be silently "
"replaced by a later FAIL's stale-computed overwrite");
printf("test_apply_pump_fail_racing_finish_merges_against_current_"
"state: PASS\n");
}
/* ── Test group entry point ───────────────────────────────────────────── */
void test_apply_pump(void)
{
UNITY_BEGIN();
RUN_TEST(test_apply_pump_kick_applies_committed_burst_in_ascending_lsn_order);
RUN_TEST(test_apply_pump_run_applies_mixed_submit_and_fact_entries);
RUN_TEST(test_apply_pump_guard_does_not_break_job_ctrl_live_append);
RUN_TEST(test_apply_pump_fail_racing_finish_merges_against_current_state);
UNITY_END();
}