lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

test(raft-apply-decoupling): genuine multi-threaded concurrency tests for commit resolve (task 1.5)

Browse source 
Closes out task-group 1 (the commit-engine event-driven-resolve
keystone). Pure test-only: real pthreads racing the admission gate,
the sweep thread, and commit_on_ack, not sequential calls dressed as
concurrent. Adds the genuine two-real-thread ack-vs-sweep-thread race
both task-1.3 review rounds flagged as a nice-to-have (the existing
race test only proved the tie-break via single-thread white-box state
manipulation).

Also resolves the bug-1407 saga: task 1.4's "closed, 70/70" and this
task's initial "reopened, 10/10 failing" were both correct,
contradictory measurements of the identical source commit. Root cause
was a stale incremental-build artifact in the shared build/ directory
(ninja -C build rebuilt incrementally across 5 sequential tasks
without ever a fresh meson setup) -- not a source regression. Proven
by isolating each measurement in a throwaway git worktree with a
fresh build. bug-1407 is genuinely closed; the incremental-build
staleness gotcha is now documented in cerebrum.md for the remaining
task series.
This commit is contained in:
Rongsong Shen 2026-07-03 13:07:44 +00:00
parent 9b4b805a27
commit 2d9f1e947a
6 changed files with 586 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

14
.wolf/buglog.json
View file

@ -12350,19 +12350,19 @@
"timestamp": "2026-07-03T08:00:00.000Z",
"error_message": "controller_unit_test: test_commit_advance_retry_clears_push_before_commit_fence FAILS (\"Expected TRUE Was FALSE\") when running --run test_config_repl (RUN_TEST site: test_config_repl.c:4008). NOTE: Unity's own printed file:line prefix for this failure is misleading/stale (shows test_repl_unit.c:1543 or controller_test.c:1543 depending on which other suites ran earlier in the same process) -- Unity.TestFile is apparently not re-set per test_xxx(void) group in this harness, only Unity.CurrentTestLineNumber is accurate. The true source is test_config_repl.c.",
"file": "src/controller/testsuite/test_config_repl.c",
"root_cause": "Not yet root-caused at discovery (3/3 tries on raft_tla @ 3ecb724, before any raft-apply-decoupling commit.c/commit.h changes). Re-confirmed present through task 1.2 (commit_resolve wired onto commit_advance_locked) and task 1.3 (async timeout sweep thread). Flagged as fixed as a side effect once task 1.4 landed (demotion/truncation resolve wiring, ha.c+persist.c+commit.c) -- exact mechanism not isolated, but 3 independent runs of `controller_unit_test --run test_config_repl` on the task-1.4 diff show 70/70 passing, 0 failures, where the prior baseline consistently showed 1 failure.",
"fix": "CLOSED 2026-07-03 -- test_commit_advance_retry_clears_push_before_commit_fence now passes reliably (70/70, 3/3 consecutive runs) as of the raft-apply-decoupling task 1.4 commit. Verified independently by task 1.4's adversarial reviewer, not just re-asserted. Root mechanism not isolated (candidate: task 1.2's commit_resolve wiring on commit_advance_locked, or task 1.4's demotion-path changes); if it resurfaces, re-open and bisect between 3ecb724 and the task-1.4 commit.",
"root_cause": "DEFINITIVELY ROOT-CAUSED 2026-07-03: this was never a source-code bug. It was a stale/corrupted incremental-build artifact in the main worktree's shared `build/` directory, which had been incrementally `ninja -C build`-rebuilt across 5 sequential tasks (1.1-1.5) without ever a fresh `meson setup`. Proof (four independent measurements, isolated via separate `git worktree add` + fresh `meson setup` + `ninja`, never mutating the main worktree in place): (1) pre-feature commit 17b210c, fresh build: 0 failures, 3/3 runs. (2) commit 9b4b805 (task-1.4 HEAD, IDENTICAL source to the failing main-worktree binary), fresh build in a separate worktree: 0 failures, 3/3 runs. (3) main worktree's stale incrementally-rebuilt `build/`, same 9b4b805 commit: 100% failure, 8/8 runs across two investigation sessions. (4) main worktree after `rm -rf build && meson setup --buildtype=debug -Db_sanitize=address && ninja -C build` (full clean rebuild, same source, same commit): 0 failures, 3/3 runs. This conclusively isolates the failure to the incremental build state, not the code. This is a DIFFERENT gotcha from bug-924 (which is about the `meson test` driver hanging on overlayfs fdatasync) -- this one is `ninja -C build`'s incremental compilation silently producing a wrong/stale binary despite reporting success, likely a header-dependency tracking gap on this overlayfs-backed worktree. The prior 'CLOSED' (task 1.4, 70/70) and 'REOPENED' (task 1.5, 10/10 fail) statuses were BOTH real, contradictory, correct measurements of the SAME source code in DIFFERENT incremental-build states -- neither implementer did anything wrong; the build itself was non-deterministic relative to source content.",
"fix": "CLOSED 2026-07-03 (high confidence, this time backed by cross-checked fresh-build isolation, not just an in-place re-run). No source-code fix needed -- `test_commit_advance_retry_clears_push_before_commit_fence` has apparently been correct since at least commit 3ecb724 (pre-feature). ACTION ITEM for all future tasks in this series: do not trust `ninja -C build` incremental results alone when a test result looks surprising or contradicts a very recent prior measurement on the same commit -- rebuild with `rm -rf build && meson setup build --buildtype=debug -Db_sanitize=address && ninja -C build` (or isolate via a throwaway `git worktree add` + fresh setup) before concluding a regression exists. Added to `.wolf/cerebrum.md` Do-Not-Repeat.",
"tags": [
"testsuite",
"controller",
"commit-engine",
"flaky-or-broken",
"pre-existing",
"build-environment",
"false-positive",
"raft-apply-decoupling",
"closed"
],
"related_bugs": [],
"occurrences": 3,
"last_seen": "2026-07-03T08:00:00.000Z"
"related_bugs": ["bug-924"],
"occurrences": 5,
"last_seen": "2026-07-03T13:00:00.000Z"
}
]