test_persist_truncate.c's three continuation tests called
commit_engine_fini(&ce) before persist_stop(p), backwards from
commit_engine_fini's own documented contract (persist_stop must run
first). With the fsync_worker flusher thread still alive, it could be
mid-flight in commit_on_durable_advance -> commit_advance_locked
(no lock held) while commit_engine_fini concurrently destroyed
ce->lock/ce->cond and nulled ce->p -- a TOCTOU race where the guard
check reads ce->p as valid but a later re-read observes NULL, SEGV at
exactly offsetof(persist_t, committed_lsn) = 0x38.
This was misdiagnosed as caused by g_drive_async_demotion/catchup_offload
(bug-1450/1451): flipping those flags only perturbed thread-scheduling
timing enough to trigger the always-latent race more reliably. The same
crash signature was independently observed on 2026-07-07, before those
flags existed, proving they were never causal.
Fix: call persist_stop(p) before commit_engine_fini(&ce) in all three
tests. teardown_persist's own internal persist_stop becomes a safe
no-op on the second call. No production code changed.
Verified: full ASAN controller_unit_test suite, 976/976 tests, 6 runs
(1 via meson + 5 standalone direct binary runs), zero ASAN reports.
test_demotion_convergence.c has the same commit_engine_fini-before-
persist_stop ordering (line 233/235) against a real persist_start'd
fixture -- not fixed here, flagged for a follow-up sweep.
Only 21 of 86 test_*.c group functions called their own nested
UNITY_BEGIN()/UNITY_END(). Each nested call reset Unity's global
NumberOfTests/TestFailures counters, so controller_test.c's final
`return UNITY_END();` -- the value meson test actually gates on --
only ever reflected whichever small window of tests ran since the
LAST nested reset, silently discarding pass/fail evidence for
everything before it (measured: 16 of 972 total tests in one run).
The same nested-BEGIN calls also caused PASS/FAIL banners to show
the wrong source file for tests in files that don't reset it
themselves (Unity.TestFile just carries over from whichever group
last called UNITY_BEGIN()).
Fix: replace every nested UNITY_BEGIN() with UnitySetTestFile(__FILE__)
(sets file attribution only, no counter reset) and drop the matching
UNITY_END(); add the same UnitySetTestFile(__FILE__) as the first
statement to the other group functions that had neither. The 2
genuinely standalone binaries (test_job_array_sched_push.c,
test_job_array_apiserver_push.c -- each has its own int main(), not
registered via controller_test.c's TEST() macro) are untouched.
Verified: single consolidated summary now reads
"972 Tests 0 Failures 0 Ignored OK", exactly matching the
ground-truth RUN_TEST call-site count across all 83 registered
group files (976 total minus 4 belonging to the 2 standalone
binaries). Every PASS/FAIL banner now names its own true source
file. Full xce-build.sh all regression pass clean (Ok: 9, Fail: 0).
No production code touched -- test-harness plumbing only.
Closes out task-group 1 (the commit-engine event-driven-resolve
keystone). Pure test-only: real pthreads racing the admission gate,
the sweep thread, and commit_on_ack, not sequential calls dressed as
concurrent. Adds the genuine two-real-thread ack-vs-sweep-thread race
both task-1.3 review rounds flagged as a nice-to-have (the existing
race test only proved the tie-break via single-thread white-box state
manipulation).
Also resolves the bug-1407 saga: task 1.4's "closed, 70/70" and this
task's initial "reopened, 10/10 failing" were both correct,
contradictory measurements of the identical source commit. Root cause
was a stale incremental-build artifact in the shared build/ directory
(ninja -C build rebuilt incrementally across 5 sequential tasks
without ever a fresh meson setup) -- not a source regression. Proven
by isolating each measurement in a throwaway git worktree with a
fresh build. bug-1407 is genuinely closed; the incremental-build
staleness gotcha is now documented in cerebrum.md for the remaining
task series.
commit_resolve_pending_above(ce, lsn_floor, rc) fails every registered
continuation for lsn > lsn_floor, reusing commit_resolve()'s existing
exactly-once/idempotent guarantee (no parallel bookkeeping path).
Truncation: a new on_truncate/truncate_cb_ctx callback on persist_t
(same inversion-of-control shape as the existing on_durable_advance
hook) fires from persist_truncate_wal_above after releasing p->lock,
only when entries were actually removed. This one wrapper covers all
three real truncation call sites: raft_cb_become_leader's
post-election truncate, the library's on_truncate_from
conflict-reconcile, and ha_demotion_convergence_fn's post-demotion
truncate.
Demotion: raft_cb_become_follower calls
commit_resolve_pending_above(&c->commit, 0, -XCE_ERR_STANDBY) --
lsn_floor=0 fails everything, since a demoted node can never
authoritatively confirm anything still pending (CP posture).
Without this, a demoted or truncated node would leave any pending
async command permanently unresolved -- the requester would simply
never get a reply, worse than today's bounded-timeout behavior.
Also closes bug-1407 (pre-existing test_config_repl flake, unrelated
to this feature) -- confirmed fixed as a side effect of prior tasks
in this series, 70/70 passing across 3 independent runs.
