--- id: "bug-1726" status: "cannot-reproduce" status_updated: "2026-09-24" status_note: "single occurrence, no surviving logs" component: "xceadmin" severity: "unknown" timestamp: "2026-08-28T08:15:00Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 1 files: ["src/xceadmin/tests/test_e2e_live_ha_push.py (live_ha_cluster fixture", "_wait_for_primary)"] tags: ["flaky-test", "ha", "raft-election", "live-ha-push", "unresolved", "unreproduced", "low-priority", "distinct-from-bug-1485", "found-live"] related: ["bug-1485", "bug-1486"] repro_attempts: "5 immediate retries, 0 reproduced" legacy: true --- # bug-1726: test_e2e_live_ha_push.py::test_live_config_push_against_standby_redirects_and_succeeds ERRORed at… ## Description test_e2e_live_ha_push.py::test_live_config_push_against_standby_redirects_and_succeeds ERRORed at live_ha_cluster fixture setup (single occurrence, user-run via `nix develop ./src` + direct pytest against the built build.dbg mirror, not via any Claude-driven build): _wait_for_primary(nodes, 30.0) raised TimeoutError -- none of the 3 freshly-spawned xce_controller processes ever printed the CTRL_PRIMARY election-complete marker within the 30s timeout (normal election completes ~10.2s per this file's own docstring, keyed to HA_DEFAULT_HEARTBEAT_TIMEOUT_MS=10000). ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction - Attempts: 5 immediate retries, 0 reproduced ## Root cause NOT ISOLATED -- deliberately distinct from bug-1485 (do not merge into it): bug-1485's signature is election COMPLETING (all 3 markers print) then the freshly-promoted primary going /unreachable shortly after; this occurrence is election never visibly settling AT ALL within 3x the normal budget, a different symptom with a different plausible cause. XCE_KEEP_DEPLOY_DIR was not set for this run, so the 3 nodes' stdout/stderr logs from the actual failure were deleted by the fixture's own cleanup before this could be forensically investigated -- no diagnostic evidence survives from this specific occurrence. Immediate reruns (5/5, by Claude directly after the report, same build.dbg binaries, same test, quiet system: load avg ~0.6-1.5/22 cores, no concurrent xce_controller/meson/nix processes found) all passed cleanly in ~14-15s each, well inside the 30s budget -- consistent with (but not proof of) transient CPU/IO contention from another concurrent session on this shared multi-agent sandbox (the established pattern documented at length in bug-1486 and this same file's bug-1485 investigation) delaying all 3 nodes' heartbeat timers past the window, rather than a deterministic protocol liveness bug. Not ruled out: a genuine, rare election-liveness edge case (e.g. a PreVote/RequestVote split-vote condition) that happens not to leave a marker before some other transient condition times it out. ## Fix NOT FIXED -- single occurrence, unreproduced on 5 immediate retries, no surviving log evidence to root-cause further. Logged per this project's own established convention (bug-1485) of tracking an inconclusive live-cluster-election flake rather than silently dismissing it. Next step if this recurs: re-run with XCE_KEEP_DEPLOY_DIR=1 (already supported by the fixture, added for exactly this class of investigation per bug-1485's own remediation) to preserve deploy_dir/logs/*.log and *.stderr.log for real forensic inspection instead of losing them to the fixture's own cleanup. ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-1485, bug-1486