13 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | fix_commits | legacy | gitea_issue | gitea_synced_status | gitea_synced_hash | ||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1429 | fixed | 2026-09-25 | b5d59253 |
controller | crash | 2026-07-10 | 2026-09-24T10:51:31Z | 5 |
|
|
|
|
true | 264 | fixed | 3e1232c31137 |
bug-1429: controller_unit_test suite TIMEOUT (300s, killed by SIGTERM) during /xce-build controller run…
Description
controller_unit_test suite TIMEOUT (300s, killed by SIGTERM) during /xce-build controller run immediately after uncommitted edits to persist.c, ha_convergence.c, config_repl.c (delaymgr-inline blocking I/O moved to VTASK_LIVE_BLOCK vtasks with inline-fallback-on-failure). Compile succeeded with zero errors (only pre-existing warnings). meson test summary: Ok:0 Fail:0 Timeout:1. Last test output before the hang was test_demotion_replay_yields_committed_only (test_commit_barrier_reply.c) printing overlayfs-SKIP, followed by a burst of HA FSM / persist_apply_pump_run log lines ending in "HA quorum lost: connected=0 required=1", then total silence for the remaining ~294s until SIGTERM. No ASAN/crash report -- looks like a genuine hang/deadlock, not a crash.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
ROOT-CAUSED 2026-08-30, LIVE-REPRODUCED with a full gdb thread backtrace (the first time in this bug's history any session captured a live hang instead of just reporting the SIGTERM/timeout). The original delaymgr/VTASK_LIVE_BLOCK hypothesis carried by this bug since 2026-07-10 was WRONG -- a static read of every vtask_enq_classified(..., VTASK_LIVE_BLOCK) call site in ha.c/persist.c/config_repl.c/commit.c found every one of them NULL-checks its pool pointer (or the trampoline is gated behind c->delaymgr.running, which test fixtures never set) and falls back inline on failure -- there is no path where a controller_unit_test fixture leaves a non-NULL-but-never-started vtaskmgr_t that a VTASK_LIVE_BLOCK task could be silently swallowed into. That entire subsystem is a red herring for this bug.
The REAL root cause is the same TOCTOU test-teardown race already root-caused once before, for a DIFFERENT symptom, in bug-1452 (2026-07-19: an ASAN SEGV in test_persist_truncate.c) and partially re-applied once more in bug-1687 (2026-08-25: cbrp_teardown in test_commit_barrier_reply.c). Every affected test fixture calls commit_engine_fini(&ce) BEFORE persist_stop(&persist) in its teardown helper -- the exact opposite of the contract documented in commit_engine_fini's own comment (commit.c: 'Callers are expected to have stopped the flusher (persist_stop) first'). Because persist_stop had not yet run, the persist_t's live fsync_worker background thread (persist.c:800, started by persist_start) can still be mid-flight calling p->on_durable_advance(p->durable_cb_ctx) -- i.e. commit_on_durable_advance(ce) -- with NO lock held (persist.c:866-868), concurrently with commit_engine_fini destroying ce->lock/ce->cond (commit.c:476-477) and freeing ce->continuations. commit_engine_fini's own persist_set_durable_cb(ce->p, NULL, NULL) unhook is 'defensive and cheap' per its own comment but is NOT sufficient on its own: the flusher thread can have already read the OLD non-NULL callback pointer a few instructions before the unhook takes effect. Depending on exact thread interleaving this TOCTOU race manifests as either bug-1452's SEGV (a NULL/freed dereference inside commit_advance_locked) OR -- newly confirmed this session -- a hang: pthread_cond_destroy(&ce->cond) blocking forever because the racing fsync_worker thread is concurrently inside pthread_cond_broadcast(&ce->cond) / holding ce->lock from commit_on_durable_advance, which glibc's cond-destroy implementation cannot safely proceed past. This explains every symptom previously logged against this bug: the varying hang location across sessions (different tests hit their OWN teardown's copy of this same race, at whatever moment the fsync_worker thread happened to be flushing), the total silence after the hang (both racing threads are genuinely blocked in futex/pthread_cond primitives, not spinning), zero CPU time accruing, and "both threads parked in futex_do_wait" from the 2026-08-23 sighting.
LIVE REPRO (this session): meson test -C src/xce.build.bug1429 --suite controller --verbose hung 40s into the run with 3 threads (main + 2x fsync_worker) all in futex_do_wait, 0 CPU time advancing over a 3s recheck. gdb -p -batch -ex 'thread apply all bt full' (attached BEFORE the 300s SIGTERM, while the process was still alive) showed: Thread 1 (main) stuck in pthread_cond_destroy() called from commit_engine_fini (commit.c:476) <- jap_teardown() (test_commit_barrier_rollout.c:1877) <- test_job_array_submit_payload_size_bounded_at_max_array_idx (test_commit_barrier_rollout.c:2294) <- UnityDefaultTestRun <- test_commit_barrier_rollout() <- main(). Threads 2 and 3 both stuck in pthread_cond_timedwait at fsync_worker (persist.c:827), BOTH with p = &g_jap (the same global persist_t) -- confirming jap_teardown's exact commit_engine_fini-before-persist_stop ordering bug (test_commit_barrier_rollout.c:1877-1879, at the time of capture) as the live mechanism. This is the 4th distinct hang location logged against this bug (previously: test_demotion_replay_yields_committed_only, the test_configack_* group, test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs) -- all four are different call sites of the identical teardown-ordering bug, which is why the location never repeated and why every prior static-inference pass aimed at the vtaskmgr/delaymgr subsystem instead never found it.
Fix
FIXED 2026-08-30. Test-fixture-only fix -- no production code changed (commit.c and persist.c are exactly as documented; the bug is every affected fixture violating commit_engine_fini's own documented calling contract). Inserted an explicit persist_stop(&persist); call immediately before every confirmed-vulnerable commit_engine_fini(&ce); call (mirroring bug-1452/bug-1687's established fix pattern exactly), leaving each function's pre-existing later persist_stop()/teardown-helper call in place as a safe, idempotent no-op (persist_stop's own fsync_thread_running and wal_fd>=0 guards make the second call a clean skip). Where a fixture also tears down a real multi-vcpu vtaskmgr_t pool (test_apply_pump.c's tpr_teardown, test_apply_pump_integration.c's tpi_teardown), vtaskmgr_fini() was left in its original position AFTER commit_engine_fini() (a separate, unrelated invariant: no new vtask may be dispatched onto the pool mid-teardown) -- only persist_stop()'s position moved earlier.
21 sites fixed across 12 files, all confirmed vulnerable by verifying (a) the fixture's persist_t is genuinely persist_start()'d (has a live fsync_worker thread) and (b) the teardown called commit_engine_fini() before persist_stop(): test_commit_barrier_rollout.c (3 sites -- g_fce/g_fp inline teardown, cb5_teardown, jap_teardown -- the last being the exact live-reproduced hang site), test_demotion_convergence.c (dc_teardown -- this EXACT site was flagged but explicitly left unfixed by bug-1452's own remediation note as a known follow-up, never actioned until now), test_convergence_verification.c (cv_teardown), test_egress_filter.c (ef_teardown), test_apply_pump.c (2 sites -- tap_teardown, tpr_teardown), test_apply_pump_integration.c (tpi_teardown), test_job_array_fence.c (tf_teardown), test_ha_role_lifecycle.c (lifecycle_cluster_teardown), test_job_array_replay_fence.c (1 site of its 4 persist_start'd fixtures -- the other 3 never wire a commit engine), test_commit_durability.c (3 sites, including one with a comment that incorrectly claimed persist_set_durable_cb's unhook alone was sufficient), test_commit_engine.c (3 sites, all sharing sync_persist_teardown), test_p2_integration.c (3 sites, 2 sharing var p + 1 using g_p9). Confirmed SAFE and left unchanged: barrier_teardown (test_commit_barrier_rollout.c), engine_teardown (test_commit_engine.c), and the fixtures in test_repl_majority.c/test_commit_admission_pool.c/test_coldstart_lsn0.c/test_commit_resolve.c -- all use a 'stub' persist_t that is never actually persist_start()'d (no live fsync_worker thread), confirmed by grep for persist_start against each fixture's global before touching it.
KNOWN REMAINING EXPOSURE (not fixed this session, explicit follow-up for a future pass, same as bug-1452 left one for this session): test_config_repl.c has ~23 call sites following the identical inline commit_engine_fini(&c->commit); t42_teardown(c); pattern (t42_teardown, line 629, confirmed calls persist_stop THEN persist_fini -- so every one of these is the same bug) -- not fixed here because the file is large (~7000 lines) with some sites (e.g. the c_control/c_raced pair around line 2911-2912) not yet individually verified for uniformity, and a bulk edit without that verification was judged too risky for this session's budget. test_term_safety_adapter.c has persist_start'd fixtures (lines 59, 245, 2382) and 5 commit_engine_fini call sites (676, 930, 1776, 1950, 2338) not individually checked. Neither file's exposure has been observed to hang or crash yet (this bug's own history shows the race is intermittent and timing-dependent -- absence of a prior sighting there is not evidence of safety). Recommended next step: a dedicated /xce-audit-invariants or grep-driven sweep of these two files using the exact same verification method used here (confirm persist_start presence, confirm fini-before-stop order, insert persist_stop() immediately before commit_engine_fini()).
VERIFICATION: full meson test -C xce.build.bug1429 --suite controller run 3 times back-to-back after the settled fix (no edits during any of these three runs) -- 39.01s / 38.86s / 38.91s, Ok:6 Fail:0 every time, zero hangs, zero timeouts (previously: TIMEOUT 300.08s-300.30s on effectively every attempt this session before the fix, reproduced live on the very first try). No regression test added (per the task's own documented exception for a suite-level timing-dependent teardown race): the race requires a live fsync_worker thread mid-flush at the exact moment of teardown, which is not reliably triggerable in a bounded, fast unit test without reintroducing the same flakiness this fix removes; the 3x full-suite re-run is the verification method this project's own bug-1452 precedent already established for this exact bug class.
UPDATE (2026-09-24): 2026-09-24 ROOT CAUSE of the residual 300s TIMEOUT (live-captured with gdb under a 6-way parallel stress harness, unshare -rn per instance): any failing TEST_ASSERT longjmps out and SKIPS the fixture teardown; the fixture's commit engine sweep thread (and persist fsync_worker) stay asleep on the static engine's ce->cond / p->fsync_cond; the next test re-inits the same static struct underneath them (pthread_cond_init on a condvar with a live waiter), corrupting the waiter count; a later teardown then blocks forever in pthread_cond_destroy (commit_engine_fini / persist_fini) or pthread_join (cb5_teardown) -- or, for a stack-local engine, the sweep thread reads a dead frame (ASAN stack-use-after-return in commit_sweep_worker, test_persist_truncate.c). The failing assertion itself was never seen because Unity's stdout was fully buffered and lost on SIGTERM/abort. Triggering flaky assertions found once visible: test_job_array_submit_one_entry_one_continuation_across_sizes (the test in flight when the 11:34 meson run hung; continuation resolved on the fsync_worker thread, test read a plain int immediately), test_pump_mixed_facts_and_commands_catchup_original_order (reply count read once before trailing continuations), test_persist_flush_fires_durable_advance_callback (fsync_worker credited a concurrent persist_flush advance to itself -> double fire). FIX (uncommitted): (1) controller_test.c line-buffers stdout; (2) XCE_TESTING registries: tearDown() calls persist_test_reap_unstopped() then commit_test_reap_unfinalized_engines(), stopping/joining whatever a failed test leaked (clean full run reaps 0; injected failure now = 1 FAIL in 2s instead of a hang); (3) persist.c flusher_flush_once reports only its own advance (fsync_worker no longer double-fires; regression test test_flusher_pass_does_not_claim_concurrent_flush_advance); (4) the two flaky tests wait (bounded) for cross-thread resolution.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1452, bug-1687, bug-924
- Gitea: #264