wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(job-array-single-wal-record): replay_wal's job-count tally was blind to array submissions

Browse source 
replay_wal() (persist.c ~3902) tallied *new_job_submits_out only for
hdr.type == WAL_JOB_SUBMIT. WAL_JOB_ARRAY_SUBMIT entries were
dispatched and correctly materialized N real jobmgr entries via
apply_committed_entry()'s array case, but never counted --
persist_verify_apply_correctness() then saw real jobmgr growth exceed
the undercounted expectation and declared APPLY_DIVERGENT, engaging a
PERMANENT, non-self-healing fence (distinct from task 4.2's
self-healing Live Apply Resource Fence).

Live-reproduced during task 6.1's soak attempt on 2 of 3 nodes in a
single small-array (N=2) sub-round via two different trigger paths.
Deterministic, not size-dependent -- the first array submission to a
live cluster would fence any node that restarts or hits an
election-path recovery fallback before its next checkpoint, shrinking
the promotable pool. A pre-existing comment at persist.c ~3863-3874
(predating this spec) had already flagged this exact code region as
needing re-verification before tasks 4.1/6.1+ went live; this is that
re-verification, and it failed.

Fix adds a WAL_JOB_ARRAY_SUBMIT branch tallying apply_rc (the genuine
per-record element count apply_committed_entry()'s array case already
returns) when apply_rc > 0, mirroring the existing dedup-safe pattern
for WAL_JOB_SUBMIT. Traced the full return-value space of
restore_job_array()/apply_committed_entry()'s array case to confirm
every tallied value is provably equal to real jobmgr growth -- the fix
cannot overcount and so cannot mask a genuine divergence.

Verified via a from-scratch live 3-node HA soak re-run (--array-soak),
independently by both implementer and reviewer: zero APPLY_DIVERGENT
across all three original trigger paths, one-WAL-record/one-commit-
barrier confirmed for N=2 and N=10000, per-element post-failover
status/cancel/completion confirmed.

Task 6.0a of job-array-single-wal-record (Requirements 3.1, 3.2, 4.1,
9.4). Inserted mid-run after live soak testing surfaced the bug; user
confirmed sequencing (fix now, blocking task 6.1+).
This commit is contained in:
Rongsong Shen 2026-07-16 04:09:43 +00:00
parent b7b90e9d21
commit 16747aa96d
3 changed files with 292 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

11
.kiro/specs/job-array-single-wal-record/tasks.md
View file

@ -157,11 +157,21 @@
- _Boundary: Compact Scheduler/API-Server Push_
- [ ] 6. Integration & non-regression soak
- [x] 6.0a Fix replay_wal()'s job-count tally for WAL_JOB_ARRAY_SUBMIT
- `replay_wal()` (`src/controller/persist.c` ~3902) tallies `*new_job_submits_out` only for `hdr.type == WAL_JOB_SUBMIT`. `WAL_JOB_ARRAY_SUBMIT` (task 1.1) is dispatched to `apply_committed_entry()` and correctly materializes N real jobmgr entries, but contributes 0 to this tally. `persist_verify_apply_correctness()` then compares real jobmgr growth (correct, N-sized) against the undercounted expectation, produces a false `APPLY_DIVERGENT` verdict, and engages `persist_recover_loud_fence_if_unrecoverable()` — a PERMANENT, non-self-healing fence (distinct from task 4.2's Live Apply Resource Fence, which self-heals; this one has no live-clear path short of a later restart landing after a checkpoint advances past the array's LSN)
- Live-reproduced during task 6.1's soak attempt (2026-07-15) on 2 of 3 nodes in a single small-array (N=2) sub-round, via two different trigger paths (ordinary boot-time `persist_recover()`, and a standby's election-path recovery fallback) — confirmed deterministic, not racy, not size-dependent. A pre-existing comment at persist.c ~3863-3874 (predates this spec, from `raft-apply-decoupling` task 2.3) already flagged this exact code region as needing re-verification "before tasks 4.1/6.1/6.2/6.3 make this path broadly live" — this is that re-verification, and it failed
- Fix: add a `WAL_JOB_ARRAY_SUBMIT` branch to the tally that adds `apply_rc` (the genuine new-element count `apply_committed_entry()`'s array case already returns) to `*new_job_submits_out` when `apply_rc > 0`, mirroring the existing dedup-safe pattern used for `WAL_JOB_SUBMIT`
- Observable: the exact live-soak reproduction (small array submit, then restart/re-elect a node) no longer produces `APPLY_DIVERGENT`; the node boots/rejoins cleanly and remains promotable
- _Requirements: 3.1, 3.2, 4.1, 9.4_
- _Boundary: Array Apply Expansion (`persist.c`'s replay-side job-count accounting) — discovered during task 6.1's soak (2026-07-15), inserted per explicit user decision; blocks 6.1/6.2/6.3, all of which require a cluster that survives a restart after an array submission_
- [ ] 6.1 Large-array soak scenario
- Extend the existing HA soak harness with a large-array submission scenario asserting: one WAL record and one commit barrier regardless of array size, failover replay time governed by the record's size rather than element count, and every array element individually addressable (status, cancel, completion) after a failover
- Observable: a soak run reporting the one-record/one-barrier assertion, a replay-time measurement independent of array size, and successful per-element status checks post-failover
- _Requirements: 4.3, 5.2, 5.3, 8.2, 8.3_
- _Boundary: Non-Regression & Testing_
- _Depends: 6.0a_
- [ ] 6.2 Chaotic-failover array retry scenario
- Extend the soak harness with a scenario submitting an array immediately before killing the primary, then retrying the identical submission against the new primary
@ -194,3 +204,4 @@
- **Task 5.3b**: Added `case Data_data_jobArray:` to `extract_data()` (`request.c`) — sets `dataset->rows`/`nrows=1` directly (no pagination, matching the always-one-shot wire shape) and nulls `getdata->data.job_array` after transferring it. The null is NOT cosmetic — independently verified `free_Request()`'s `Request_body_pushmsg` case calls `free_Data()`, which has a live `Data_data_jobArray` branch that would double-free/UAF without it. Proven end-to-end with a real `xce_encode()`/`xce_decode()` round-trip through `extract_data()` → `sched_handle_dataset()` → `sched_on_job_array()` (task 5.3, unmodified), not just a unit-level check. Also independently confirmed the alternate hypothesis some earlier notes floated — that `scheduler_req_handler()` itself has a `pushmsg`-rejecting branch — is dead code on the live wire path, since `vtask_entry()` always converts `body_which` to `datamsg` before invoking any handler. **New residual gap, self-flagged, NOT fixed (pre-existing since task 5.3, not introduced here, genuinely out of this task's `request.c`-only boundary)**: `dataset->rows` for the jobArray case is never deep-freed downstream — `free_Dataset()` is an empty no-op for all table types, and `sched_handle_dataset()`'s jobArray case (task 5.3) doesn't null it after consuming it. Net effect: one bounded `xce_job_submit_t` + embedded spec leaks per successfully-delivered array push commit (not per element, not unbounded) — real but small; worth a dedicated follow-up before this spec is considered fully closed, alongside the other self-flagged residual gaps (task 3.1's bulk-insert orphan prefix, task 4.2's boot-replay fence gap).
- **Task 5.4**: Go/CGO receive path — `rawJobArrayToModels()` (pure-Go N-way expansion), `cJobArrayToModels()` (CGO adapter sibling to `cJobToModel`), `dispatchJobArray()` (dedup via the SAME `store.Jobs.Has()` the ordinary `Data_data_jobs` path writes through, one confirmation log only on ≥1 newly-dispatched element), new `case C.Data_data_jobArray:` in `handleDataset`. Elements route through `OnJobUpdated` (not `OnJobCreated`), matching the ordinary single-job push's own routing — verified consistent, not a divergence. CGO boundary needed zero changes — `C.xce_job_submit_t` and `C.Data_data_jobArray` were already exposed/generated from prior tasks. A structural TOCTOU exists between the `Has()` dedup check and the eventual store write (two separate lock acquisitions, no compare-and-set primitive) but is narrower than task 5.3's accepted C-side gap (no intervening async hop) and strictly safer than the pre-existing baseline (the ordinary `Data_data_jobs` path has NO dedup check at all today) — not fixed, judged acceptable. `cJobArrayToModels()` has zero direct unit coverage at the CGO-struct level, but this exactly mirrors the pre-existing, repo-wide precedent for `cJobToModel()` itself (also never unit-tested with a real C struct) — reviewer independently reconstructed a full `CGO_ENABLED=1` build+link+test run against the real generated headers to confirm correctness beyond a compile-only check. `store.ArrayJobs` (array-level rollup aggregate) deliberately left unpopulated — confirmed genuinely out of this spec's scope per requirements.md's Boundary Context, which explicitly defers whole-array bulk query/cancel to a future spec; zero code anywhere in the codebase populates it today, including this diff.
- **Task 5.5**: Cross-daemon integration test, test-only (zero net production diff, confirmed on `request.c` and every task 5.1-5.4 file). Drives real `xce_encode()`/`xce_decode()` wire bytes through the REAL production dispatch chain on both daemons — `extract_data()` → `scheduler_req_handler()` → `sched_handle_dataset()` → `sched_on_job_array()` (C, via `#include "../src/handler.c"` in the test) and `go_push_handler()` → `handleDataset()` → `cJobArrayToModels()`/`dispatchJobArray()` (Go/CGO) — skipping only `vtask_entry()`'s ~10 lines of `static` glue, which the test mirrors verbatim (independently verified line-by-line accurate on the one behavior that mattered: `if (callnext)` treating any nonzero `extract_data()` return, including a negative errno, as "proceed"). Redelivery is tested via genuinely independent fresh decodes of separately-encoded bytes, not a cheaper decode-once-call-twice shortcut. Bug-catching capability independently proven twice (implementer, then reviewer): temporarily reverting task 5.3b's fix makes both the C and Go tests fail meaningfully; restoring returns both to green. Go/cgo toolchain constraint discovered: `import "C"` is disallowed in any `_test.go` file, forcing a two-file split (`wire_dispatch_support.go` for cgo-preamble helpers, `wire_dispatch_test.go` for the actual tests) — logged as bug-1440/1441/1442. One low-severity, non-blocking nit (not fixed): the C test file's own header comment misdescribes which assertion fails first under the RED-phase revert (claims the `callnext != 0` check; actually a downstream `>= 0` check) — cosmetic only, the test's real bug-catching property is independently confirmed accurate.
- **Task 6.0a**: Fixed `replay_wal()`'s job-count tally (persist.c ~3902) to add a `WAL_JOB_ARRAY_SUBMIT` branch that tallies `apply_rc` (the genuine per-record element count `apply_committed_entry()`'s array case already returns) when `apply_rc > 0`. Independently traced the FULL return-value space of `restore_job_array()`/`apply_committed_entry()`'s array case (6 distinct cases: 3 negative pre-mutation failures, `-EEXIST`→`apply_rc=0` idempotent no-op, the task-3.1 bulk-insert partial-failure case (stays negative, correctly still NOT tallied — a pre-existing, out-of-boundary undercount that still surfaces as a detectable anomaly rather than masking one), and positive `created` on genuine success) and confirmed every positive `apply_rc` this fix tallies is PROVABLY equal to real jobmgr growth for that record — the fix cannot overcount, and therefore cannot weaken `persist_verify_apply_correctness()`'s ability to catch a genuine divergence. Verified end-to-end via a from-scratch live 3-node HA soak re-run (`--array-soak`, both by the implementer and independently by the reviewer): zero `APPLY_DIVERGENT` across all three of the original bug's trigger paths (restart-mid-small-array, standby election-path recovery fallback, restart-mid-N=10000-array), one-WAL-record/one-commit-barrier confirmed for both N=2 and N=10000, per-element post-failover status/cancel/completion confirmed. This soak run also directly satisfies task 6.1's own Observable — its results should be reused rather than re-run from scratch. **New non-blocking finding (pre-existing, NOT introduced by this task)**: `controller_unit_test`'s single outer `UNITY_BEGIN()` is silently reset by ~19 of its called test-group functions' OWN nested `UNITY_BEGIN()/UNITY_END()` calls, making the process exit code / meson `Ok:/Fail:` summary line unreliable for detecting a regression in any group running before the last such nested call — both the implementer and reviewer worked around this by grepping raw `testlog.txt` for `:FAIL:` lines directly rather than trusting the summary, and this should be the standard practice going forward for this binary until a dedicated follow-up fixes the harness itself.

24
src/controller/persist.c
View file

@ -3903,6 +3903,30 @@ static int replay_wal(persist_t* p, uint64_t from_lsn,
apply_rc == 1)
(*new_job_submits_out)++;
/* job-array-single-wal-record task 6.0a: the array analog of the
* WAL_JOB_SUBMIT tally immediately above. apply_committed_entry()'s
* WAL_JOB_ARRAY_SUBMIT case (restore_job_array()'s return,
* generalized to N elements) returns apply_rc == the genuine
* count of NEWLY created elements (1..array_size) on a real
* apply, or 0 for an idempotent whole-record re-apply (every
* index already existed -- restore_job_array() returned
* -EEXIST, which this same dispatch maps to 0, mirroring
* WAL_JOB_SUBMIT's own dedup return). Tallying apply_rc (not a
* flat +1) is required here, unlike the single-job case above,
* because one WAL_JOB_ARRAY_SUBMIT entry can materialize many
* jobmgr entries at once; guarding on apply_rc > 0 excludes the
* idempotent 0 case from double-counting a re-apply, exactly as
* apply_rc == 1 excludes WAL_JOB_SUBMIT's duplicate-jobid case
* (bug-1419) above. Without this branch, replay_wal() silently
* undercounts every array record's real N-element jobmgr growth
* to 0, and persist_verify_apply_correctness() then reports a
* false APPLY_DIVERGENT verdict that engages the PERMANENT,
* non-self-healing persist_recover_loud_fence_if_unrecoverable()
* fence (live-reproduced task 6.1 soak, 2026-07-15). */
if (hdr.type == WAL_JOB_ARRAY_SUBMIT && new_job_submits_out &&
apply_rc > 0)
(*new_job_submits_out) += (uint64_t)apply_rc;
/* Advance current_lsn to one past this entry */
if (hdr.lsn + 1 > p->current_lsn)
p->current_lsn = hdr.lsn + 1;

257
src/controller/testsuite/test_apply_correctness_verification.c
View file

@ -131,6 +131,43 @@ static int tacv_encode_job(const xce_job_t* j, uint8_t** out, size_t* out_len)
return 0;
}
/* Encodes a (base_jobid, array_size, spec) JobSubmit payload -- the exact
* shape a committed WAL_JOB_ARRAY_SUBMIT record carries (mirrors
* test_job_array_apply.c's taj_encode_job_submit and persist.c's
* persist_submit_job_array_async encode side). Caller frees *out. */
static int tacv_encode_job_submit(uint64_t base_jobid, uint64_t array_size,
job_submit_data_t* spec, uint8_t** out,
size_t* out_len)
{
struct capn c;
struct capn_segment* cs;
struct JobSubmit cjs;
JobSubmit_ptr jsp;
xce_job_submit_t s;
memset(&s, 0, sizeof(s));
s.jobid = base_jobid;
s.array_size = array_size;
s.submit = spec;
capn_init_malloc(&c);
cs = capn_root(&c).seg;
encode_JobSubmit(cs, &cjs, &s);
jsp = new_JobSubmit(cs);
write_JobSubmit(&cjs, jsp);
capn_setp(capn_root(&c), 0, jsp.p);
*out_len = capn_size(&c);
*out = malloc(*out_len);
if (!*out) {
capn_free(&c);
return -ENOMEM;
}
capn_write_mem(&c, *out, *out_len, 0);
capn_free(&c);
return 0;
}
static persist_t* tacv_make_fresh_persist(const char* tmpdir)
{
persist_t* p = calloc(1, sizeof(persist_t));
@ -720,6 +757,224 @@ static void test_real_replay_duplicate_jobid_is_apply_clean_no_fence(void)
printf("test_real_replay_duplicate_jobid_is_apply_clean_no_fence: PASS\n");
}
/*
* job-array-single-wal-record task 6.0a (Req 3.1, 3.2, 4.1, 9.4): a
* committed WAL_JOB_ARRAY_SUBMIT record's real N-element jobmgr growth must
* be counted in replay_wal()'s new_job_submits_out tally, mirroring the
* WAL_JOB_SUBMIT tally above. Before the fix, replay_wal() only checked
* hdr.type == WAL_JOB_SUBMIT, so an array record's genuine N-element growth
* (apply_committed_entry()'s WAL_JOB_ARRAY_SUBMIT case correctly returns
* apply_rc == N) contributed 0 to the tally -- persist_verify_apply_
* correctness() then compared real growth (N) against the undercounted
* expectation (0), producing a false APPLY_DIVERGENT verdict that engages
* the PERMANENT, non-self-healing persist_recover_loud_fence_if_
* unrecoverable() fence (distinct from task 4.2's self-healing Live Apply
* Resource Fence). Live-reproduced during task 6.1's soak (2026-07-15) on 2
* of 3 nodes after a small (N=2) array submit followed by a node restart.
*
* This test is the exact live-soak reproduction: an ordinary WAL_JOB_SUBMIT
* followed by a small WAL_JOB_ARRAY_SUBMIT, replayed on restart. A correct
* tally must report APPLY_CLEAN and must never trip the permanent
* apply-divergent fence; every array element must be genuinely
* materialized.
*/
static void
test_real_replay_array_submit_tallies_correctly_no_false_divergence(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_tacv_array_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t base_jobid = 81000;
const uint64_t array_size = 2;
/* Phase 1: one ordinary WAL_JOB_SUBMIT, then one small WAL_JOB_ARRAY_
* SUBMIT -- mirrors the live soak's mixed ordinary-submit + array-submit
* WAL, replayed whole on restart. */
{
persist_t* p = tacv_make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = 80999;
jdef.state = JOB_STAT_PEND;
job_submit_data_t plain_submit;
memset(&plain_submit, 0, sizeof(plain_submit));
plain_submit.queue = (char*)"rq1";
plain_submit.user = (char*)"tester";
jdef.submit = &plain_submit;
uint8_t* plain_payload = NULL;
size_t plain_len = 0;
TEST_ASSERT_EQUAL_INT(0,
tacv_encode_job(&jdef, &plain_payload, &plain_len));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_SUBMIT, plain_payload,
(uint32_t)plain_len));
free(plain_payload);
job_submit_data_t array_spec;
memset(&array_spec, 0, sizeof(array_spec));
array_spec.queue = (char*)"rq1";
array_spec.user = (char*)"tester";
uint8_t* array_payload = NULL;
size_t array_len = 0;
TEST_ASSERT_EQUAL_INT(0,
tacv_encode_job_submit(base_jobid, array_size, &array_spec,
&array_payload, &array_len));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_ARRAY_SUBMIT, array_payload,
(uint32_t)array_len));
free(array_payload);
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
tacv_teardown_persist(p);
}
/* Phase 2: fresh persist_t + fresh cluster_t, recover from disk. */
{
persist_t* p2 = tacv_make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p2);
tacv_build_cluster();
p2->cluster = &g_tacv_c;
TEST_ASSERT_EQUAL_INT(0, persist_recover(p2));
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_CLEAN,
persist_apply_verdict(p2),
"task 6.0a: a WAL_JOB_ARRAY_SUBMIT's real N-element jobmgr "
"growth must be counted in replay_wal's tally -- an "
"undercounted tally must NOT produce a false APPLY_DIVERGENT "
"verdict");
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(p2),
"task 6.0a: a correctly-tallied array replay must never trip "
"the permanent, non-self-healing recovery_unrecoverable fence");
job_t* j0 = NULL;
job_t* j1 = NULL;
lock(g_tacv_c.jobmgr, RTTI_LOCK_READ);
key_find(g_tacv_c.jobmgr, (void*)(base_jobid + 0), (void**)&j0);
key_find(g_tacv_c.jobmgr, (void*)(base_jobid + 1), (void**)&j1);
unlock(g_tacv_c.jobmgr);
TEST_ASSERT_NOT_NULL_MESSAGE(j0,
"array element 0 must be materialized");
TEST_ASSERT_NOT_NULL_MESSAGE(j1,
"array element 1 must be materialized");
tacv_teardown_cluster();
tacv_teardown_persist(p2);
}
char cmd[512];
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp);
if (system(cmd) != 0)
fprintf(stderr, "cleanup warning: rm -rf %s failed\n", tmp);
printf("test_real_replay_array_submit_tallies_correctly_no_false_"
"divergence: PASS\n");
}
/*
* Idempotency (mirrors test_real_replay_duplicate_jobid_is_apply_clean_
* no_fence's single-job pattern, extended to a whole array as one unit --
* bug-1419's array analog, the trigger task 6.2's future soak scenario
* exercises at the wire level): the SAME array record committed twice as two
* separate WAL_JOB_ARRAY_SUBMIT entries (a commit-timeout retry whose
* original entry also later achieves commit) must replay APPLY_CLEAN.
* restore_job_array()'s per-index idempotency check materializes the array
* only once (first application: apply_rc == array_size; second application:
* every index already exists -> restore_job_array() returns -EEXIST ->
* apply_committed_entry() returns 0) -- replay_wal's corrected tally must
* agree with jobmgr's real growth of exactly array_size, not
* 2 * array_size.
*/
static void
test_real_replay_duplicate_array_submit_is_apply_clean_no_fence(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_tacv_array_dup_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t base_jobid = 82000;
const uint64_t array_size = 3;
/* Phase 1: append the SAME array record twice as two separate
* WAL_JOB_ARRAY_SUBMIT entries. */
{
persist_t* p = tacv_make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
for (int attempt = 0; attempt < 2; attempt++) {
job_submit_data_t spec;
memset(&spec, 0, sizeof(spec));
spec.queue = (char*)"rq1";
spec.user = (char*)"tester";
uint8_t* payload = NULL;
size_t len = 0;
TEST_ASSERT_EQUAL_INT(0,
tacv_encode_job_submit(base_jobid, array_size, &spec,
&payload, &len));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_ARRAY_SUBMIT, payload,
(uint32_t)len));
free(payload);
}
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
tacv_teardown_persist(p);
}
/* Phase 2: fresh persist_t + fresh cluster_t, recover from disk. */
{
persist_t* p2 = tacv_make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p2);
tacv_build_cluster();
p2->cluster = &g_tacv_c;
TEST_ASSERT_EQUAL_INT(0, persist_recover(p2));
TEST_ASSERT_EQUAL_INT_MESSAGE(APPLY_CLEAN,
persist_apply_verdict(p2),
"task 6.0a idempotency: a retried (duplicate) array commit "
"must replay APPLY_CLEAN -- the second, idempotent "
"whole-record re-apply (-EEXIST, apply_rc==0) must not be "
"double-counted into the tally");
TEST_ASSERT_FALSE_MESSAGE(
persist_recovery_unrecoverable(p2),
"a duplicate array commit must never trip the permanent "
"apply-divergent fence");
for (uint64_t i = 0; i < array_size; i++) {
job_t* j = NULL;
lock(g_tacv_c.jobmgr, RTTI_LOCK_READ);
key_find(g_tacv_c.jobmgr, (void*)(base_jobid + i), (void**)&j);
unlock(g_tacv_c.jobmgr);
TEST_ASSERT_NOT_NULL_MESSAGE(j,
"every array index must be materialized exactly once");
}
tacv_teardown_cluster();
tacv_teardown_persist(p2);
}
char cmd[512];
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp);
if (system(cmd) != 0)
fprintf(stderr, "cleanup warning: rm -rf %s failed\n", tmp);
printf("test_real_replay_duplicate_array_submit_is_apply_clean_no_"
"fence: PASS\n");
}
/*
* raft-apply-decoupling task 5.2 round 2 (Req 8.3 remediation): the
* apply_divergent ERROR log must NOT append the shared structural-fence
@ -1248,6 +1503,8 @@ void test_apply_correctness_verification(void)
RUN_TEST(test_apply_divergent_decision_forces_unconditional_fence);
RUN_TEST(test_apply_divergent_loud_fence_sets_sticky_flag_and_stays_alive);
RUN_TEST(test_real_replay_duplicate_jobid_is_apply_clean_no_fence);
RUN_TEST(test_real_replay_array_submit_tallies_correctly_no_false_divergence);
RUN_TEST(test_real_replay_duplicate_array_submit_is_apply_clean_no_fence);
RUN_TEST(test_apply_divergent_loud_fence_log_omits_false_peer_hwm_clause);
RUN_TEST(test_apply_divergent_process_stays_alive_and_functional_after_fence);
RUN_TEST(test_apply_divergent_fence_clears_via_existing_clear_mechanism);