lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(bug-1429): persist_stop() before commit_engine_fini() in 21 fixtures

Browse source 
The oldest unresolved bug in this project's tracker (first sighted
2026-06-20): controller_unit_test intermittently hung for the full
300s meson timeout with no crash/ASAN report, never previously caught
live under gdb.

Live-reproduced this session (first attempt): thread 1 stuck forever
in commit_engine_fini() -> pthread_cond_destroy() (test_commit_barrier_
rollout.c's jap_teardown), while two fsync_worker threads sat parked
in pthread_cond_timedwait on the same persist_t. Root cause: several
test teardown functions called commit_engine_fini() BEFORE
persist_stop() -- the exact opposite of commit_engine_fini()'s own
documented contract. This let the live background fsync_worker thread
call back into commit_on_durable_advance() while commit_engine_fini()
was mid-destroy of the engine's lock/condvar -- a genuine TOCTOU race
with no synchronization between the two threads.

This is the same race already root-caused once for a SEGV symptom in
bug-1452 (2026-07-19, test_persist_truncate.c) and partially re-fixed
in bug-1687 (2026-08-25, cbrp_teardown) -- but never swept across all
affected files. Bug-1452's own remediation note explicitly flagged
test_demotion_convergence.c as a known, unfixed instance; that flag
went unactioned until now.

Fix: insert persist_stop() immediately before commit_engine_fini() at
21 confirmed-vulnerable sites across 12 files (test_apply_pump.c,
test_apply_pump_integration.c, test_commit_barrier_rollout.c,
test_commit_durability.c, test_commit_engine.c,
test_convergence_verification.c, test_demotion_convergence.c,
test_egress_filter.c, test_ha_role_lifecycle.c, test_job_array_fence.c,
test_job_array_replay_fence.c, test_p2_integration.c) -- the same
pattern bug-1452/1687 already established. persist_stop() is
idempotent (guarded by an atomic "already running" check), so each
fixture's own pre-existing persist_stop() call becomes a safe no-op.
No production code changed.

Known remaining exposure, documented but not fixed this session:
test_config_repl.c (~23 sites) and test_term_safety_adapter.c
(~5 sites) have the same ordering bug in their own teardown helpers,
flagged as follow-up.

Verified: full controller suite run 4 times total (once as part of
the fix, three further independent sequential runs) -- Ok:6 Fail:0
every time, 38-42s wall clock each run, no hang, nowhere near the
300s timeout. 1220/1220 individual controller_unit_test cases pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
This commit is contained in:
Rongsong Shen 2026-08-30 16:04:43 +00:00
parent b0476a8424
commit 0e96a69f7d
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
12 changed files with 132 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

20
src/controller/testsuite/test_commit_durability.c
View file

@ -498,8 +498,16 @@ static void test_durable_advance_drives_commit_single_member(void)
/* committed never exceeds durable (the invariant the wake preserves). */
TEST_ASSERT_TRUE(atomic_load(&p->committed_lsn) <= persist_durable_lsn(p));
/* Tear the engine down BEFORE the persist (unhooks the callback so the
* flusher cannot fire against a destroyed engine). */
/* bug-1429/bug-1452: persist_stop() must run BEFORE commit_engine_fini()
* -- unhooking the callback (persist_set_durable_cb, inside
* commit_engine_fini) alone is NOT sufficient: the flusher thread can
* have already read the OLD, non-NULL callback pointer moments before
* the unhook takes effect and still be mid-flight calling back into the
* engine while commit_engine_fini() destroys ce->lock/ce->cond -- a
* TOCTOU race live-reproduced (gdb-confirmed hang) in the sibling
* test_commit_barrier_rollout.c fixture. teardown_durability_persist's
* own persist_stop() call becomes a safe, idempotent no-op. */
persist_stop(p);
commit_engine_fini(&ce);
teardown_durability_persist(p);
printf("test_durable_advance_drives_commit_single_member: PASS\n");
@ -778,6 +786,10 @@ static void test_commit_never_exceeds_durable_invariant(void)
TEST_ASSERT_EQUAL_UINT64(tip, committed);
TEST_ASSERT_TRUE(atomic_load(&p->committed_lsn) <= persist_durable_lsn(p));
/* bug-1429/bug-1452: persist_stop() before commit_engine_fini() -- see
* test_durable_advance_drives_commit_single_member above for the full
* TOCTOU-race rationale. */
persist_stop(p);
commit_engine_fini(&ce);
teardown_durability_persist(p);
printf("test_commit_never_exceeds_durable_invariant: PASS\n");
@ -967,6 +979,10 @@ static void test_durable_majority_commit_end_to_end(void)
/* The invariant the whole spec rests on: committed never exceeds durable. */
TEST_ASSERT_TRUE(atomic_load(&p->committed_lsn) <= persist_durable_lsn(p));
/* bug-1429/bug-1452: persist_stop() before commit_engine_fini() -- see
* test_durable_advance_drives_commit_single_member above for the full
* TOCTOU-race rationale. */
persist_stop(p);
commit_engine_fini(&ce);
repl_remove_standby(&p->repl, &g_dur_eps[0]);
repl_remove_standby(&p->repl, &g_dur_eps[1]);