chore: update OpenWolf session state (xce-audit-invariants controller pass)

This commit is contained in:
Rongsong Shen 2026-07-19 08:14:19 +00:00
parent d8d1901901
commit e1f4d676de
6 changed files with 6009 additions and 28 deletions

View file

@ -503,3 +503,8 @@ tasks.md showed 18/18 (12 leaf + 6 group-header) checked since 2026-06-16, and r
## Key Learning (2026-07-18, bug-1449 fixed) — run-ha-soak.sh had TWO independent convergence signals conflated into one; round 1 was only guarded by the wrong one
`wait_for_full_convergence`/`cluster_committed_converged` (gates every round AFTER round 1) checks role-count (1 primary + 2 standby) + committed-LSN catch-up — it says nothing about `configVersion` (the config "tag" a standby needs before `ha_on_primary_ctrl_fd_rdhup`'s ha-config-bootstrap Req 2.8 voter-only guard will let it self-elect). Round 1 had NO wait for that second, independent signal before its first kill, so an unlucky-but-legitimate slow cold-start election (observed 10-12s across 3 verification runs, vs. the ~1.7s job-submit budget the old code implicitly relied on) could let the kill land before config-push finished, correctly triggering the tag=0 guard and producing a false "no primary elected" round-1 failure. Fix mirrors `wait_for_full_convergence`'s exact polling primitive (alive-count via `pgrep -c -f "$(run_pattern)"`, STABLE_TICKS=2, sleep 2, ~90s bound) but gates on a NEW `cluster_config_converged` (all 3 nodes' `configVersion` via ctrlstatus non-zero, probed by a new `node_config_tag()` mirroring `node_probe`'s single-node client-probe technique) — added right after `wait_for_full_convergence` (~line 556) and called once, right before the round loop (~line 1897), so it applies ONLY to sequential mode's round-1 kill (chaotic mode intentionally tolerates this exact race elsewhere — a stalled election there is a COUNTED outcome, never a hard fail, so it was correctly left untouched). **Do-not-repeat**: when a soak/test driver has more than one "is the cluster ready" signal server-side (committed-prefix vs. config-tag here), a wait function gating on ONE does not imply the other is also satisfied — check what the actual guard/precondition being raced depends on before assuming an existing convergence wait already covers it. Verified 3/3 back-to-back clean `--sequential 6` runs on genuine ext4 (`/home/claude/tmp`, confirmed via `mount`/`df -T`, not overlayfs); cross-checked cold-start timing from controller logs across all 3 runs (election 10.3-11.7s after process start, config-push landing only 40-46ms after election every time) to confirm the new gate's ~4s minimum dwell / ~90s ceiling has ~2 orders of magnitude of margin, not just enough for that day's luck.
## Do-Not-Repeat (2026-07-18, bugs 1450/1451) — a new slow-lane offload's own non-blocking-submit contract is not enough; defaulting the flag ON can STILL hang controller_unit_test via a DIFFERENT mechanism than the vtask-pool deadlock it replaces
Fixed two `xce-audit-invariants controller` thread-role violations (`ha_demotion_convergence_fn` ha_convergence.c, `ha_catchup_continuation_fn` config_repl.c — both ran blocking work directly on the delaymgr thread) by adding a 4th/5th instance of the established slow-lane pattern (`slowlane_submit`/`*_lane_fn`, mirroring `ha_snapshot_send_dispatch`/`ha_config_push_offloadable`/`ha_connect_to_peer` in ha.c) — NOT the VTASK_LIVE_BLOCK vtask-pool approach a prior session tried and reverted (bug-1429) after it deadlocked `controller_unit_test`. Confirmed via `git log`/memory why that attempt failed: `c->ha.pool` was never initialized in the lightweight unit-test fixtures it ran against, so `vtask_enq_classified(..., VTASK_LIVE_BLOCK)` blocked forever waiting for a worker that didn't exist. `slowlane_submit` is architecturally immune to THAT failure mode (it is unconditionally non-blocking — with zero live workers it just returns `-EAGAIN`/`0` and leaves the work queued-but-never-run, never blocks the caller). BUT defaulting the new offload flags to ON (`g_drive_async_demotion_offload`/`g_drive_async_catchup_offload = 1`) reproduced a REAL hang anyway, via a DIFFERENT mechanism: the production callback being offloaded (`raft_cb_become_follower` for demotion, `ha_walack_on_ack`/`ha_catchup_continuation_fn` for catch-up) is exercised by a VERY BROAD set of PRE-EXISTING `controller_unit_test` fixtures across many files (confirmed hit: `test_convergence_verification.c`'s ACK-driven catch-up suite) that never call `slowlane_init` — so the offloaded work is silently queued on an uninitialized lane and NEVER RUNS, and any downstream code (in THOSE pre-existing tests, not the new ones) that waits — even boundedly — for the now-never-happening completion stalls/times out. Reproduced with a direct standalone binary run (`timeout -s QUIT 90 ./controller_unit_test`, bypassing meson) BOTH before (hang, SIGQUIT after 90s) and after (clean, "976 Tests 0 Failures 0 Ignored") the flag-default flip to OFF. **General rule**: when adding a NEW slow-lane offload flag for an existing, widely-exercised callback, default it OFF and add your own dedicated tests that explicitly flip it ON against a fixture you control (with `slowlane_init` called) — do NOT default a new offload flag ON just because the lane mechanism itself is provably non-blocking; the blast radius of "which pre-existing fixtures reach this code path without lane infra" is the real risk, and it can only be ruled out by a full test-suite compatibility audit (exactly how the ORIGINAL three raft-drive-async offloads were rolled out: shipped OFF, flipped ON only in a later dedicated task-3.4-style pass). Always verify a "fixed hang" claim by running the STANDALONE `controller_unit_test` binary directly with an external `timeout -s QUIT`, not just `meson test` (whose `Ok:N/Fail:0` aggregate can hide a TIMEOUT line entirely if you only grep for `Fail:`) — grep for `Timeout` explicitly too.
## Key Learning (2026-07-18) — `test_convergence_verification.c` is the direct, pre-existing test surface for `ha_walack_on_ack`/`ha_catchup_continuation_fn` (the ACK-driven catch-up decision, ha-convergence-recovery task 4.x); it drives them via `ha_walack_continuation_for_test` against a real-but-un-started `delaymgr` and has NO `slowlane_init` — any future change that makes catch-up continuation offload-capable must either default OFF or update this file's fixture.