| id |
status |
status_updated |
status_note |
component |
severity |
timestamp |
last_seen |
occurrences |
files |
tags |
related |
fix_commits |
repro_attempts |
legacy |
| bug-1485 |
fixed |
2026-10-03 |
|
xceadmin |
unknown |
2026-07-30T14:35:00Z |
2026-10-03T07:44:48Z |
26 |
| src/xceadmin/tests/test_e2e_live_ha_push.py (live_ha_cluster fixture); root cause is in src/controller (exact site not yet isolated -- C-side investigation out of this task's boundary) |
|
| flaky-test |
| ha |
| raft-election |
| controller-crash |
| admin-config-tool |
| task-6.3 |
| e2e-live |
| unresolved |
| pytest-specific |
|
| bug-1477 |
| bug-1479 |
| bug-606 |
|
|
0/40 repro attempts (2026-08-01) |
true |
bug-1485: test_e2e_live_ha_push.py::test_live_config_push_against_primary_reaches_success_path flaky…
Description
test_e2e_live_ha_push.py::test_live_config_push_against_primary_reaches_success_path flaky (near-100% fail rate across ~25 live runs during admin-config-tool task 6.3 round-2 remediation): TimeoutError: controller never opened localhost:: [Errno 111] Connection refused, raised from a fresh wait_for_port re-probe of the elected primary's own address performed AFTER _wait_for_primary (CTRL_PRIMARY marker) + a new _wait_for_standbys (both other nodes show CTRL_STANDBY) + a 3s settle sleep all already succeeded.
Environment
Not recorded (migrated from buglog.json).
Reproduction
- Attempts: 0/40 repro attempts (2026-08-01)
Root cause
Corrected 2026-10-03, superseding the earlier zombie-primary / CPU-starvation hypotheses (kept in git history; the zombie symptom was never reproduced). The 2026-10-03 symptom version_tag == 0 is an error reply, not a bad success: push.py maps any non-zero status to versionTag=0 ackedCount=0, and the controller rejected the push with -110 (config_repl.c ~1839-1845, repl_wait_majority_ack 500 ms). It happens after a "late-joiner" election: the own-term no-op (lsn 1) is appended before any standby is registered (ha.c ~5178, repl.c ~306-311), the standbys register later via on_standby_connected (ha.c ~9643-9660) without the no-op being resent, and heartbeat-driven catch-up commits it only ~6 s (two 3 s cycles) after the election. A push in that window is NACKed by both standbys and times out. The fixture's readiness check (role markers + 3 s settle) did not cover that window. The controller behaviour itself (fail-loud rejection) is correct; its ~6 s write-unavailability is tracked as bug-1913.
Fix
Test-side (src/xceadmin/tests): new wait_for_primary_write_ready() / primary_write_ready() in _live_e2e_common.py wait until a commit logged after the latest "own-term noop appended lsn=N" reaches N; the live_ha_cluster fixture calls it (30 s) after waiting for standbys; versionTag assertions now print the CLI's stdout/stderr. Fix commit 362098b1.
Verification
- Test:
test_live_config_push_against_primary_reaches_success_path in src/xceadmin/tests/test_e2e_live_ha_push.py — RED before under CPU load (40 busy loops): 7/28 runs failed with expected a non-zero versionTag, got 0 (... stderr='Error: controller returned status -110'), all late-joiner elections; GREEN after: 95 looped runs incl. 24 under the same load (7 late-joiner elections, all passed). Helper unit tests in test_live_e2e_common.py (no-op not committed → not ready; latest no-op only; commits before the latest no-op ignored).
- Suites: xceadmin 373 passed, 2026-10-03.
- Live: residual, different mechanism: under heavy oversubscription a normally-elected primary can miss the 500 ms marker barrier (1/70 HA loops).
References
- Related: bug-1477, bug-1479, bug-606
