docs(buglog): bug-1485 and bug-1725 recurred during controller-peers-env-fallback task 4.2

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-03 07:45:06 +00:00

5.9 KiB

Raw Blame History

status

status_updated

status_note

component

severity

timestamp

last_seen

occurrences

files

bug-1485: test_e2e_live_ha_push.py::test_live_config_push_against_primary_reaches_success_path flaky…

Description

test_e2e_live_ha_push.py::test_live_config_push_against_primary_reaches_success_path flaky (near-100% fail rate across ~25 live runs during admin-config-tool task 6.3 round-2 remediation): TimeoutError: controller never opened localhost:: [Errno 111] Connection refused, raised from a fresh wait_for_port re-probe of the elected primary's own address performed AFTER _wait_for_primary (CTRL_PRIMARY marker) + a new _wait_for_standbys (both other nodes show CTRL_STANDBY) + a 3s settle sleep all already succeeded.

Environment

Not recorded (migrated from buglog.json).

Reproduction

Attempts: 0/40 repro attempts (2026-08-01)

Root cause

NOT a pure test-side readiness race as round-1 review assumed. Live forensic capture (ps aux mid-run) caught the freshly-elected ctrl-1 as a zombie process moments after its own log recorded the full raft_cb_become_leader promotion sequence ending at 'HA: repl registered standby ep[1] (slot 1, ...)' -- the log then goes silent (no crash trace, no further lines) and the process is gone. stderr was DEVNULL'd in the fixture so no ASAN/signal diagnostic was captured from a real failing pytest run. Confusingly, 3 separate standalone (non-pytest) repro scripts using byte-identical spawn code (same xce_controller build.dbg binary, same deploy-ha.sh cluster, same peers/certs/workdir args) ran the exact same primary-promotion sequence successfully every single time with the primary staying alive and the port reachable for 20-30s post-election -- while pytest-invoked runs of the identical fixture code failed roughly 24/25 times. Ruled out during investigation: pytest's default fd-capture (reproduced with -s / capturing disabled, still failed 3/3), ulimits (NPROC/NOFILE/AS/STACK identical between nix-develop plain-shell and pytest-invoked-via-python contexts), and orphaned-process port contention (ps aux clean before each batch). True root cause not isolated -- prime suspects are a genuine timing-sensitive controller-side crash/race in the CTRL_STANDBY->CTRL_PRIMARY raft promotion path (raft_cb_become_leader / repl registration) that pytest's process tree somehow makes far more likely to trigger, or a process-group/session/signal-mask difference inherited by subprocess.Popen children when the parent is pytest vs a bare python3 script.

Fix

UPDATE 2026-08-01 -- investigated per this entry's own "next step" (capture stderr instead of DEVNULL, re-attempt). Redirected each node's stderr to deploy_dir/logs/.stderr.log (permanent change to test_e2e_live_ha_push.py, replacing the DEVNULL redirect that left this entry's original investigation with no diagnostic) and added an XCE_KEEP_DEPLOY_DIR=1 debug env var to preserve deploy_dir across the fixture's own cleanup for inspection. Ran the live test 40 times total in this environment: 25 baseline (idle sandbox, load avg ~2/22 cores) + 15 more with a concurrent xce-build.sh controller debug compile racing for CPU (separate build.dbg dir, so it could not clobber the xce.build controller binary the test depends on -- unlike bug-1486's same-dir clobber shape). Result: 40/40 PASSED, 0/120 captured stderr logs (3 nodes x 40 runs) non-empty -- no crash, no ASAN report, in any run. Could NOT reproduce the ~24/25 failure rate from the original investigation despite matching the documented repro shape exactly (same binaries, same fixture code, same pytest invocation).

NOT marking fixed -- absence of reproduction here is not proof of absence of the underlying bug, and this session's single concurrent compile is a much lighter contention scenario than the real multi-session sandbox reality bug-1486 documents (several concurrent AGENT SESSIONS, potentially each running their own live 3-node cluster). Leading hypothesis (unconfirmed): the original ~24/25 failure rate was itself a symptom of resource contention from OTHER concurrent sessions on the shared sandbox during that investigation (CPU starvation delaying the primary's post-election warmup past _POST_ELECTION_SETTLE_S, or an OOM-adjacent condition killing the freshly-forked primary outright) rather than a bug in the promotion path reachable in isolation -- consistent with the original investigation's own "pytest's process tree somehow makes far more likely" framing, since pytest's subprocess tree carries more memory/fd overhead than the bare python3 scripts that never reproduced it there either.

Downgraded to low-priority/environment-flake, left OPEN. The stderr-capture infrastructure is now permanently in place (not reverted) so the next real live failure -- in pytest, in a genuinely contended multi-session sandbox, or in CI -- captures an actual ASAN report or signal instead of DEVNULL silence, which is what would actually close this out.

Verification