chore: update OpenWolf session state (bug-1452 fix)

buglog.json: bug-1452 marked FIXED with root cause and fix writeup.
cerebrum.md: Do-Not-Repeat entry for the commit_engine_fini/persist_stop
ordering contract, plus the flag-noted latent duplicate in
test_demotion_convergence.c.
memory.md/anatomy.md/session state: routine session bookkeeping.
This commit is contained in:
Rongsong Shen 2026-07-19 09:27:56 +00:00
parent d67ccc2280
commit 619f41f70e
6 changed files with 862 additions and 617 deletions

View file

@ -13491,21 +13491,25 @@
"timestamp": "2026-07-18T00:00:00Z",
"error_message": "AddressSanitizer: SEGV on unknown address 0x000000000038 in commit_advance_locked (controller/commit.c:208), called from commit_on_durable_advance (commit.c:341) <- fsync_worker (persist.c:399), a background ASAN thread started by persist_start (persist.c:723) inside test_persist_truncate.c's make_fresh_persist / test_truncate_above_resolves_pending_continuations_as_failed. Reproduced by temporarily flipping BOTH g_drive_async_demotion_offload and g_drive_async_catchup_offload (bug-1450/1451) from their default 0 to 1 and running the standalone controller_unit_test binary directly (976-test suite) -- crashed immediately, before completing the suite. Confirmed absent with both flags at their committed default of 0: the same binary/build (just reverted) completed cleanly, 976 Tests 0 Failures 0 Ignored, twice.",
"file": "src/controller/commit.c (crash site, commit_advance_locked); src/controller/persist.c (fsync_worker background thread, apply_committed_entry/restore_job/makeJob chain also appears in a related non-crashing stack from the same investigation); src/controller/ha_convergence.c + config_repl.c (g_drive_async_demotion_offload / g_drive_async_catchup_offload -- the flags whose flip triggers this)",
"root_cause": "NOT ROOT-CAUSED -- discovered as a side effect of an xce-audit-invariants Step 4 live-regression attempt (2026-07-18), not investigated further, deliberately out of scope for that session (see bug-1450/1451: flipping either offload flag's default to ON is explicitly scoped to a LATER, dedicated compatibility-audit pass, matching the original rollout discipline of the snapshot/config-push/reconnect offloads). bug-1450/1451 already documented a HANG from flipping these flags ON against pre-existing fixtures that never call slowlane_init (uninitialized c->slowlane silently queues work that's never run). This is a DIFFERENT and more severe symptom -- a hard SEGV, not a hang -- in an apparently-unrelated code path (commit.c's durability-advance callback, invoked from the fsync background thread against a WAL-truncation test fixture), suggesting the two offload flags' interaction with the broader pre-existing test-fixture surface is not limited to the single hang mechanism already documented. Not confirmed whether this reproduces with only ONE of the two flags flipped (both were flipped together when this was found) or whether it is fixture-specific (test_persist_truncate.c) vs a broader live-relevant race -- the crash is inside commit.c/persist.c machinery that IS also exercised by the real live daemon, so this should NOT be assumed to be test-only until someone actually checks whether a live cluster with both flags forced on hits the same SEGV.",
"fix": "NOT FIXED -- out of scope for the xce-audit-invariants session that found it (which correctly kept both flags at their default of 0, matching bug-1450/1451's scoping decision; this finding is additional evidence supporting that decision, discovered incidentally while verifying it). Whoever picks up the deferred 'flip demotion/catchup offload to default ON' follow-up audit (referenced in bug-1450/1451) MUST reproduce and root-cause this SEGV as part of that work, in addition to the already-documented hang -- do not treat the hang as the only pre-flip blocker. Suggested next steps: (1) reproduce with each flag flipped individually to isolate which one (or the combination) triggers it; (2) reproduce running only test_persist_truncate.c in isolation (narrow binary via meson test args or a --run filter) to rule out cross-test global-state pollution as a confound; (3) once isolated, read commit_advance_locked (commit.c:208) against what's at offset 0x38 in whatever struct is being dereferenced -- a near-null read at a small fixed offset strongly suggests an uninitialized or already-freed context pointer, consistent with the same 'offload path assumes c->slowlane/commit-engine machinery is live-initialized' class of issue as the documented hang, just manifesting as a crash instead of a stall in this particular fixture.",
"root_cause": "ROOT-CAUSED 2026-07-19: the g_drive_async_demotion_offload/g_drive_async_catchup_offload flags were a RED HERRING -- this exact SEGV signature (NULL deref in the fsync_worker -> commit_on_durable_advance -> commit_advance_locked chain) was already independently observed on 2026-07-07, long before those flags existed (see memory: 'Pre-existing ASAN SEGV in test_persist_truncate Unrelated to Bug-1417 Work'). The flags only perturbed thread-scheduling timing enough to make a PRE-EXISTING, always-latent test-teardown race fire more reliably; they do not participate in the crash's causal chain at all. Actual root cause: in src/controller/testsuite/test_persist_truncate.c, the three continuation-related tests (test_truncate_above_resolves_pending_continuations_as_failed, test_truncate_noop_does_not_resolve_pending_continuations, test_truncate_concurrent_burst_resolves_above_cutoff_exactly_once) called `commit_engine_fini(&ce)` BEFORE `teardown_persist(p)` (which calls `persist_stop(p)`) -- the exact opposite of the contract documented in commit_engine_fini's own comment (commit.c:464-468: 'Callers are expected to have stopped the flusher (persist_stop) first'). Because persist_stop had not yet run, the fsync_worker background thread was still alive and could be mid-flight invoking the durable-advance callback (fsync_worker, persist.c:398-399, calls p->on_durable_advance(p->durable_cb_ctx) with NO lock held) concurrently with commit_engine_fini destroying ce->lock/ce->cond and, as its final step, setting ce->p = NULL. This produces a genuine TOCTOU data race with no synchronization between the two threads: commit_on_durable_advance's own guard (`if (ce == NULL || ce->p == NULL) return;`) can read ce->p as still non-NULL (racing ahead of or concurrent with commit_engine_fini's teardown), pass the guard, and then commit_advance_locked's SEPARATE, later re-read of ce->p (commit.c:208, `atomic_load(&ce->p->committed_lsn)`) observes the NOW-NULLED value written by the concurrently-running commit_engine_fini on the main thread -- producing a NULL-pointer dereference at exactly offset 0x38 (offsetof(persist_t, committed_lsn)), matching the crash address bit-for-bit. persist_set_durable_cb's unhook (called from commit_engine_fini) is 'defensive and cheap' per its own comment but is NOT sufficient on its own without persist_stop's pthread_join guaranteeing the flusher thread cannot be in-flight at all -- confirmed by the fact that persist_set_durable_cb existed and ran in this path the whole time and the crash still occurred.",
"fix": "FIXED 2026-07-19 (commit pending): added an explicit `persist_stop(p);` call immediately before `commit_engine_fini(&ce);` in all three affected test functions in test_persist_truncate.c (previously commit_engine_fini ran first, then teardown_persist -> persist_stop ran second -- backwards from the documented contract). persist_stop joins the fsync_worker thread, so by the time commit_engine_fini tears down ce->lock/ce->cond/ce->p, no thread can be mid-callback referencing them. teardown_persist's own internal persist_stop(p) call becomes a safe idempotent no-op on the second invocation (fsync_thread_running is already 0 and wal_fd is already -1, so both of its internal branches skip). No production code changed -- this was purely a test-fixture ordering bug, not a bug in commit.c/persist.c itself. Verified: full ASAN controller_unit_test suite (976 tests) run 6 times total (1 via meson + 5 standalone direct binary runs per project convention of not trusting meson's aggregate alone) -- 976 Tests 0 Failures 0 Ignored on every run, zero ASAN reports (SEGV/UAF/etc.), all 8 test_persist_truncate.c tests PASS every run. NOTE for a future session: the identical commit_engine_fini-before-persist_stop ordering also exists in test_demotion_convergence.c (line 233 commit_engine_fini(&g_dc.commit) before line 235 persist_stop(&g_dc.persist), against a real persist_start'd fixture) -- same latent race, not yet observed to crash there (likely a narrower window against a static/global fixture rather than a per-test stack fixture), out of scope for this fix but worth a follow-up sweep of all commit_engine_fini call sites.",
"tags": [
"ha_convergence.c",
"config_repl.c",
"commit.c",
"demotion-offload",
"catchup-offload",
"persist.c",
"test_persist_truncate.c",
"teardown-ordering",
"toctou-race",
"commit_engine_fini",
"persist_stop",
"raft-drive-async-pattern",
"xce-audit-invariants",
"controller-tests",
"ASAN",
"SEGV",
"NOT-FIXED",
"not-root-caused"
"FIXED",
"root-caused"
],
"related_bugs": [
"bug-1429",
@ -13513,6 +13517,6 @@
"bug-1451"
],
"occurrences": 1,
"last_seen": "2026-07-18"
"last_seen": "2026-07-19"
}
]