lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-apply-decoupling): bug-1419 real fix — replay tally, not dedup-guard

Browse source 
Root cause (confirmed via targeted DIAG1419 logging on a live chaotic soak):
a client's commit-timeout retry during a genuine quorum-loss window appends
a second WAL_JOB_SUBMIT for the same jobid; the timed-out entry is never
retracted, so if it later achieves majority commit anyway both entries end
up committed. This is a legitimate, harmless duplicate — restore_job()'s own
-EEXIST correctly no-ops the second one — but replay_wal() tallied every raw
WAL_JOB_SUBMIT occurrence instead of genuine creations, so the duplicate
made new_job_submits disagree with jobmgr's real growth, firing a false
APPLY_DIVERGENT and deadlocking promotion.

Reverts the layer-3 claim/release machinery (job_wal_commit_claim/release,
claim_committed_job_submits, and their wiring): three attempts at preventing
the duplicate at submit/dispatch/commit time all failed to close the gap,
because the duplicate isn't a bug to prevent — a WAL entry can't be
retracted once appended, so any submit-time guard is structurally the wrong
layer.

Real fix: apply_committed_entry()'s WAL_JOB_SUBMIT case now returns 1 for a
genuine creation vs 0 for an idempotent no-op (restore_job() -EEXIST), and
replay_wal() only tallies new_job_submits_out on rc==1. Updates the affected
apply-seam/reapply-idempotency tests for the new return contract, and
reverses two apply-correctness tests whose premise (duplicate-jobid replay
must diverge) is now understood to be the actual bug rather than the
expected behavior — they now force divergence directly via the same
production fencing seam instead of relying on that trigger.

Verified: two consecutive 10-round chaotic soaks, zero divergence, zero job
loss, no election stall (previously reproduced within ~5 rounds).
This commit is contained in:
Rongsong Shen 2026-07-08 12:48:44 +00:00
parent f28ce57993
commit 11244dd654
7 changed files with 197 additions and 232 deletions
Download patch file Download diff file Expand all files Collapse all files

20
src/controller/testsuite/test_apply_committed_entry.c
View file

@ -223,7 +223,9 @@ static void test_ace_job_submit_constructs_and_notifies(void)
int rc = apply_committed_entry(&g_tace_p, WAL_JOB_SUBMIT, 10, payload,
(uint32_t)plen);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, rc,
"a genuine new job creation must return 1 (bug-1419: distinguishes "
"creation from an idempotent duplicate no-op)");
job_t* j = tace_find_job(501);
TEST_ASSERT_NOT_NULL_MESSAGE(j,
@ -258,16 +260,20 @@ static void test_ace_job_submit_duplicate_is_noop_no_extra_notify(void)
xce_add_hook(&g_tace_c.hooksys, XCE_CONTROLLER_HOOK_JOB, tace_spy_hook,
NULL);
TEST_ASSERT_EQUAL_INT(0, apply_committed_entry(&g_tace_p, WAL_JOB_SUBMIT,
11, payload,
(uint32_t)plen));
TEST_ASSERT_EQUAL_INT_MESSAGE(1, apply_committed_entry(&g_tace_p,
WAL_JOB_SUBMIT, 11,
payload,
(uint32_t)plen),
"a genuine new job creation must return 1");
TEST_ASSERT_EQUAL_INT(1, g_tace_hook_fires);
/* Re-apply the SAME entry (replay / resync redelivery) — must be a
* no-op: no error, no second job, no second notification (Req 7.1). */
TEST_ASSERT_EQUAL_INT(0, apply_committed_entry(&g_tace_p, WAL_JOB_SUBMIT,
11, payload,
(uint32_t)plen));
TEST_ASSERT_EQUAL_INT_MESSAGE(0, apply_committed_entry(&g_tace_p,
WAL_JOB_SUBMIT, 11,
payload,
(uint32_t)plen),
"an idempotent duplicate must return 0, not 1");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_tace_hook_fires,
"an idempotent duplicate submit must not fire a second notification");

215
src/controller/testsuite/test_apply_correctness_verification.c
View file

@ -629,29 +629,28 @@ static void test_apply_divergent_loud_fence_sets_sticky_flag_and_stays_alive(voi
}
/*
* End-to-end: a REAL materialization mismatch produced by an actual
* persist_recover() replay (not the pure functions tested above) must ITSELF
* set the sticky non-promotable fence, with no test-only shortcut faking the
* verdict.
*
* Forces a genuine mismatch: replay_wal's new_job_submits_out counts every
* RAW WAL_JOB_SUBMIT header it scans (persist.c), unconditionally -- it does
* NOT know whether apply_committed_entry()'s restore_job() actually created a
* new job or silently no-op'd on a duplicate jobid (restore_job is
* idempotent, Req 7.1). Appending the identical jobid twice therefore drives
* new_job_submits=2 while the live jobmgr table only grows by 1 -- a real,
* reproducible APPLY_DIVERGENT with nothing about
* persist_verify_apply_correctness() itself mocked or shortcut.
* bug-1419: a duplicate WAL_JOB_SUBMIT for the SAME jobid is a legitimate,
* harmless occurrence, not a hidden apply defect -- a client's local
* commit-timeout retry can append a second entry for a jobid while the first
* is still in flight during a quorum-loss window; the timed-out entry is
* never retracted, so if it later achieves majority commit anyway, both
* entries end up committed and replayed. replay_wal()'s new_job_submits_out
* tallies only GENUINE creations (apply_committed_entry()'s return value
* == 1), not raw WAL_JOB_SUBMIT occurrences, so this scenario must replay as
* APPLY_CLEAN with no fence -- restore_job()'s own idempotent -EEXIST
* correctly prevents a second job object, and the corrected count now
* agrees with jobmgr's real growth of 1.
*/
static void test_real_replay_divergence_sets_unrecoverable_fence(void)
static void test_real_replay_duplicate_jobid_is_apply_clean_no_fence(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_tacv_divergent_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
/* Phase 1: append the SAME jobid twice as two separate WAL_JOB_SUBMIT
* entries -- a real replay counts both, but restore_job() only
* materializes the job once. */
* entries -- simulating a commit-timeout retry whose original entry
* later also achieves commit. restore_job() only materializes the job
* once; the corrected replay tally must agree. */
{
persist_t* p = tacv_make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
@ -681,9 +680,10 @@ static void test_real_replay_divergence_sets_unrecoverable_fence(void)
}
/* Phase 2: fresh persist_t + fresh cluster_t, recover from disk. The
* duplicate-jobid replay must produce a genuine APPLY_DIVERGENT verdict,
* which persist_recover() must now route through the existing
* non-promotable loud-fence mechanism (task 5.2). */
* duplicate-jobid replay must now replay APPLY_CLEAN -- the corrected
* new_job_submits_out tally counts only apply_committed_entry()'s
* genuine creations (rc==1), so it agrees with jobmgr's real growth of
* 1 and never trips the loud-fence mechanism (task 5.2). */
{
persist_t* p2 = tacv_make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p2);
@ -695,22 +695,18 @@ static void test_real_replay_divergence_sets_unrecoverable_fence(void)
persist_recovery_unrecoverable(p2),
"sanity: the sticky flag starts clear before recovery runs");
/* persist_recover() itself must RETURN normally (0) -- no
* abort()/exit() anywhere on this path (controller-never-self-
* terminate invariant). Reaching the assertions below is itself
* part of the proof the process stayed alive. */
TEST_ASSERT_EQUAL_INT(0, persist_recover(p2));
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_DIVERGENT,
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_CLEAN,
persist_apply_verdict(p2),
"fixture sanity: the duplicate-jobid replay must genuinely "
"diverge (new_job_submits=2, jobmgr only grew by 1)");
"bug-1419: a duplicate-jobid replay is a harmless idempotent "
"no-op (new_job_submits=1, jobmgr grew by 1) -- it must NOT "
"report APPLY_DIVERGENT");
TEST_ASSERT_TRUE_MESSAGE(
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(p2),
"a real APPLY_DIVERGENT verdict from persist_recover() must set "
"the sticky non-promotable fence via the existing loud-fence "
"mechanism (Req 8.4)");
"an APPLY_CLEAN verdict must never set the non-promotable "
"fence");
tacv_teardown_cluster();
tacv_teardown_persist(p2);
@ -721,7 +717,7 @@ static void test_real_replay_divergence_sets_unrecoverable_fence(void)
if (system(cmd) != 0)
fprintf(stderr, "cleanup warning: rm -rf %s failed\n", tmp);
printf("test_real_replay_divergence_sets_unrecoverable_fence: PASS\n");
printf("test_real_replay_duplicate_jobid_is_apply_clean_no_fence: PASS\n");
}
/*
@ -850,15 +846,26 @@ static void test_apply_divergent_loud_fence_log_omits_false_peer_hwm_clause(void
* exercised anywhere in this file). ─────────────────────────────────── */
/*
* More explicit "stays alive" pin than 5.2's implicit evidence. Forces a
* REAL, end-to-end APPLY_DIVERGENT verdict via an actual persist_recover()
* replay (the duplicate-jobid trigger, same as
* test_real_replay_divergence_sets_unrecoverable_fence), then performs
* MULTIPLE, clearly UNRELATED, real operations on the SAME now-fenced
* persist_t afterward, in the SAME test function and thread -- proving
* control genuinely returned to the caller and the process remains fully
* live and functional, not merely "hasn't crashed yet by the time one
* boolean was checked."
* More explicit "stays alive" pin than 5.2's implicit evidence. Performs a
* REAL, end-to-end clean replay via an actual persist_recover() call on a
* live persist_t + cluster, then force-trips the SAME production fencing
* function persist_recover() itself invokes on a genuine divergence
* (persist_recover_loud_fence_if_unrecoverable, with in.apply_divergent =
* true) -- and performs MULTIPLE, clearly UNRELATED, real operations on the
* SAME now-fenced persist_t afterward, in the SAME test function and
* thread -- proving control genuinely returned to the caller and the
* process remains fully live and functional, not merely "hasn't crashed yet
* by the time one boolean was checked."
*
* bug-1419: an earlier version of this test forced the fence via a
* duplicate-jobid replay (the same trigger
* test_real_replay_duplicate_jobid_is_apply_clean_no_fence exercises) --
* that trigger is now understood to be a legitimate, harmless idempotent
* no-op, not a real divergence, so it can no longer serve as a genuine
* end-to-end divergence source. Forcing the SAME fencing function real
* divergence would call is the faithful substitute: it exercises the exact
* fence-then-stays-alive contract this test targets, independent of any one
* specific (and, as it turned out, buggy) trigger mechanism.
*
* The follow-up WAL append models a real operational fact: fencing
* (recovery_unrecoverable) only refuses PROMOTION (ha.c's promotion gate,
@ -873,38 +880,37 @@ static void test_apply_divergent_process_stays_alive_and_functional_after_fence(
strcpy(tmp, "/tmp/xce_tacv_alive_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
/* Phase 1: duplicate-jobid WAL -- the same genuine divergence trigger as
* test_real_replay_divergence_sets_unrecoverable_fence. */
/* Phase 1: a single, ordinary WAL_JOB_SUBMIT -- proves the replay this
* fixture recovers from is genuinely clean before the fence is forced. */
{
persist_t* p = tacv_make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
for (int i = 0; i < 2; i++) {
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = 76001;
jdef.state = JOB_STAT_PEND;
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = 76001;
jdef.state = JOB_STAT_PEND;
job_submit_data_t submit;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"rq1";
submit.user = (char*)"tester";
jdef.submit = &submit;
job_submit_data_t submit;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"rq1";
submit.user = (char*)"tester";
jdef.submit = &submit;
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tacv_encode_job(&jdef, &payload, &plen));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_SUBMIT, payload, (uint32_t)plen));
free(payload);
}
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tacv_encode_job(&jdef, &payload, &plen));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_SUBMIT, payload, (uint32_t)plen));
free(payload);
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
tacv_teardown_persist(p);
}
/* Phase 2: recover into divergence, then prove the process stays alive
* and functional with several explicit, unrelated follow-up steps. */
/* Phase 2: recover cleanly, force-trip the fence via the same production
* seam a genuine divergence would invoke, then prove the process stays
* alive and functional with several explicit, unrelated follow-up steps. */
{
persist_t* p2 = tacv_make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p2);
@ -917,10 +923,27 @@ static void test_apply_divergent_process_stays_alive_and_functional_after_fence(
* everything below is this task's requested MORE EXPLICIT pin on
* top of that. */
TEST_ASSERT_EQUAL_INT(0, persist_recover(p2));
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_DIVERGENT,
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_CLEAN,
persist_apply_verdict(p2),
"fixture sanity: the duplicate-jobid replay must genuinely "
"diverge");
"fixture sanity: a single ordinary submit must replay clean");
TEST_ASSERT_FALSE_MESSAGE(persist_recovery_unrecoverable(p2),
"sanity: a clean replay must not fence on its own");
recover_policy_inputs_t force_divergent_in = {
.verdict = RECOVER_CLEAN,
.durable_below_hwm = false,
.healthy_peer_reachable = true,
.catch_up_succeeded = true,
.durable_floor_lsn = 100,
.cluster_hwm_lsn = 100,
.apply_divergent = true,
};
recover_decision_t d =
persist_recover_loud_fence_if_unrecoverable(p2,
&force_divergent_in);
TEST_ASSERT_EQUAL_INT_MESSAGE(RECOVER_DECISION_FENCE_UNRECOVERABLE, d,
"forcing apply_divergent must decide the loud, non-terminating "
"fence");
TEST_ASSERT_TRUE_MESSAGE(persist_recovery_unrecoverable(p2),
"the fence must be set before this test's stays-alive pin means "
"anything");
@ -934,9 +957,6 @@ static void test_apply_divergent_process_stays_alive_and_functional_after_fence(
"the fence must remain observably set across repeated "
"reads -- each read requires the calling thread to still "
"be alive");
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_DIVERGENT,
persist_apply_verdict(p2),
"the verdict must remain stable across repeated reads");
}
/* Explicit pin #2: a genuinely unrelated, real persist operation
@ -1094,30 +1114,36 @@ static void test_apply_divergent_fence_does_not_survive_a_clean_restart(void)
tacv_build_cluster();
/* "Boot 1": a duplicate-jobid WAL genuinely diverges and fences. */
/* "Boot 1": a clean replay, then the fence is force-tripped via the same
* production seam a genuine divergence would invoke (bug-1419: a
* duplicate-jobid replay is a legitimate no-op, not a real divergence,
* so it can no longer serve as this fixture's trigger -- see
* test_apply_divergent_process_stays_alive_and_functional_after_fence
* for the same substitution and its rationale). What this test actually
* checks -- a fenced instance's flag surviving independently of a later,
* separate instance's clean outcome -- does not depend on which real
* mechanism tripped the fence. */
{
persist_t* p = tacv_make_fresh_persist(tmp_divergent);
TEST_ASSERT_NOT_NULL(p);
for (int i = 0; i < 2; i++) {
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = 77001;
jdef.state = JOB_STAT_PEND;
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = 77001;
jdef.state = JOB_STAT_PEND;
job_submit_data_t submit;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"rq1";
submit.user = (char*)"tester";
jdef.submit = &submit;
job_submit_data_t submit;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"rq1";
submit.user = (char*)"tester";
jdef.submit = &submit;
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tacv_encode_job(&jdef, &payload, &plen));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_SUBMIT, payload, (uint32_t)plen));
free(payload);
}
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tacv_encode_job(&jdef, &payload, &plen));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_SUBMIT, payload, (uint32_t)plen));
free(payload);
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
tacv_teardown_persist(p);
}
@ -1126,9 +1152,22 @@ static void test_apply_divergent_fence_does_not_survive_a_clean_restart(void)
TEST_ASSERT_NOT_NULL(p_boot1);
p_boot1->cluster = &g_tacv_c;
TEST_ASSERT_EQUAL_INT(0, persist_recover(p_boot1));
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_DIVERGENT,
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_CLEAN,
persist_apply_verdict(p_boot1),
"fixture sanity: boot 1 must genuinely diverge");
"fixture sanity: boot 1's own replay is clean before the fence is "
"forced");
recover_policy_inputs_t boot1_force_divergent_in = {
.verdict = RECOVER_CLEAN,
.durable_below_hwm = false,
.healthy_peer_reachable = true,
.catch_up_succeeded = true,
.durable_floor_lsn = 100,
.cluster_hwm_lsn = 100,
.apply_divergent = true,
};
persist_recover_loud_fence_if_unrecoverable(p_boot1,
&boot1_force_divergent_in);
TEST_ASSERT_TRUE_MESSAGE(persist_recovery_unrecoverable(p_boot1),
"boot 1 must be fenced");
@ -1208,7 +1247,7 @@ void test_apply_correctness_verification(void)
RUN_TEST(test_verify_apply_correctness_clean_when_no_cluster);
RUN_TEST(test_apply_divergent_decision_forces_unconditional_fence);
RUN_TEST(test_apply_divergent_loud_fence_sets_sticky_flag_and_stays_alive);
RUN_TEST(test_real_replay_divergence_sets_unrecoverable_fence);
RUN_TEST(test_real_replay_duplicate_jobid_is_apply_clean_no_fence);
RUN_TEST(test_apply_divergent_loud_fence_log_omits_false_peer_hwm_clause);
RUN_TEST(test_apply_divergent_process_stays_alive_and_functional_after_fence);
RUN_TEST(test_apply_divergent_fence_clears_via_existing_clear_mechanism);

19
src/controller/testsuite/test_apply_reapply_idempotency.c
View file

@ -203,9 +203,10 @@ static void test_tari_job_submit_reapplied_thrice_is_idempotent(void)
tari_spy_hook, NULL));
/* First application: real mutation + one notification. */
TEST_ASSERT_EQUAL_INT(0, apply_committed_entry(&g_tari_p, WAL_JOB_SUBMIT,
100, payload,
(uint32_t)plen));
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
apply_committed_entry(&g_tari_p, WAL_JOB_SUBMIT, 100, payload,
(uint32_t)plen),
"a genuine new job creation must return 1 (bug-1419)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_tari_hook_fires,
"the first application must fire exactly one notification");
@ -218,9 +219,10 @@ static void test_tari_job_submit_reapplied_thrice_is_idempotent(void)
* observable state must not change again, and the notification must not
* fire again, on EITHER repeat. */
for (int i = 0; i < 2; i++) {
TEST_ASSERT_EQUAL_INT(0,
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
apply_committed_entry(&g_tari_p, WAL_JOB_SUBMIT, 100, payload,
(uint32_t)plen));
(uint32_t)plen),
"an idempotent duplicate must return 0, not 1 (bug-1419)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_tari_hook_fires,
"re-applying an identical committed SUBMIT must never fire a "
"second notification, no matter how many times it repeats");
@ -313,9 +315,10 @@ test_tari_finish_then_stale_start_preserves_terminal_state_via_dispatch(void)
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0, tari_encode_job(&jdef, &payload, &plen));
TEST_ASSERT_EQUAL_INT(0, apply_committed_entry(&g_tari_p, WAL_JOB_SUBMIT,
300, payload,
(uint32_t)plen));
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
apply_committed_entry(&g_tari_p, WAL_JOB_SUBMIT, 300, payload,
(uint32_t)plen),
"a genuine new job creation must return 1 (bug-1419)");
free(payload);
job_t* j = tari_find_job(703);

42
src/controller/testsuite/test_commit_barrier_reply.c
View file

@ -1381,47 +1381,6 @@ static void test_jobsubmit_resubmit_after_full_resolution_rejected_no_second_job
" PASS\n");
}
/* bug-1419 layer 3: a jobid claimed by something OTHER than this node's own
* handle_jobsubmit call -- simulating the apply pump processing a
* WAL_JOB_SUBMIT entry received via replication/replay, never locally
* submitted -- must still block a same-jobid submit arriving through
* handle_jobsubmit. Before this fix, job_wal_commit_claim did not exist and
* apply_committed_entry() never registered an in-flight claim for an
* externally-sourced entry, so jobid_range_exists() (c->jobmgr, not yet
* applied) and the in-flight registry (only ever populated by
* handle_jobsubmit itself) both missed it -- exactly the cross-primary
* retry gap a live chaotic soak reproduced (waldump showed two committed
* WAL_JOB_SUBMIT entries for one jobid, same term, no fresh idgen collision
* -- the same generated jobid resent by the client after a connection drop,
* landing on a newly elected primary that never saw the original). */
static void test_jobsubmit_rejected_when_jobid_claimed_by_foreign_commit(void)
{
const uint64_t jobid = 9160;
int rc;
cbrp_setup(2000);
cbrp_arrange_jobsubmit(jobid, 1);
/* Simulate: the apply pump is mid-dispatch on a WAL_JOB_SUBMIT entry for
* this jobid that this node never itself submitted (e.g. replicated
* from a prior primary that committed it before dying). */
TEST_ASSERT_TRUE_MESSAGE(job_wal_commit_claim(jobid),
"fixture sanity: claiming a fresh jobid must succeed");
rc = handle_jobsubmit(&g_hc, &g_req_ep, &g_jobsub_req);
job_wal_commit_release(jobid);
cbrp_teardown();
TEST_ASSERT_EQUAL_INT_MESSAGE(-EEXIST, rc,
"bug-1419 layer 3: a submit for a jobid claimed by a foreign "
"(non-self-originated) commit must be rejected -EEXIST, closing "
"the cross-primary retry gap");
printf("test_jobsubmit_rejected_when_jobid_claimed_by_foreign_commit:"
" PASS\n");
}
/* The explicit-FAILED (non-timeout) sibling of the repeated-timeout test
* above: repeats the truncation-forced FAILED path
* CBRP_JOBSUBMIT_REPEAT_N times in the same fixture, proving the
@ -3388,7 +3347,6 @@ void test_commit_barrier_reply(void)
RUN_TEST(test_jobsubmit_repeated_timeout_failures_no_phantom_jobs);
RUN_TEST(test_jobsubmit_resubmit_before_resolution_rejected_no_dup_wal_entry);
RUN_TEST(test_jobsubmit_resubmit_after_full_resolution_rejected_no_second_job);
RUN_TEST(test_jobsubmit_rejected_when_jobid_claimed_by_foreign_commit);
RUN_TEST(test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs);
RUN_TEST(test_jobsubmit_array_mixed_outcome_no_phantom_job_for_failed_elements);
/* handle_jobsubmit success path via the ordered apply pump