42 lines
3.6 KiB
Markdown
42 lines
3.6 KiB
Markdown
---
|
|
id: "bug-1485"
|
|
status: "fixed"
|
|
status_updated: "2026-10-03"
|
|
status_note: ""
|
|
component: "xceadmin"
|
|
severity: "unknown"
|
|
timestamp: "2026-07-30T14:35:00Z"
|
|
last_seen: "2026-10-03T07:44:48Z"
|
|
occurrences: 26
|
|
files: ["src/xceadmin/tests/test_e2e_live_ha_push.py (live_ha_cluster fixture); root cause is in src/controller (exact site not yet isolated -- C-side investigation out of this task's boundary)"]
|
|
tags: ["flaky-test", "ha", "raft-election", "controller-crash", "admin-config-tool", "task-6.3", "e2e-live", "unresolved", "pytest-specific"]
|
|
related: ["bug-1477", "bug-1479", "bug-606"]
|
|
fix_commits: ["362098b1"]
|
|
repro_attempts: "0/40 repro attempts (2026-08-01)"
|
|
legacy: true
|
|
---
|
|
|
|
# bug-1485: test_e2e_live_ha_push.py::test_live_config_push_against_primary_reaches_success_path flaky…
|
|
|
|
## Description
|
|
test_e2e_live_ha_push.py::test_live_config_push_against_primary_reaches_success_path flaky (near-100% fail rate across ~25 live runs during admin-config-tool task 6.3 round-2 remediation): TimeoutError: controller never opened localhost:<port>: [Errno 111] Connection refused, raised from a fresh wait_for_port re-probe of the elected primary's own address performed AFTER _wait_for_primary (CTRL_PRIMARY marker) + a new _wait_for_standbys (both other nodes show CTRL_STANDBY) + a 3s settle sleep all already succeeded.
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
- Attempts: 0/40 repro attempts (2026-08-01)
|
|
|
|
## Root cause
|
|
Corrected 2026-10-03, superseding the earlier zombie-primary / CPU-starvation hypotheses (kept in git history; the zombie symptom was never reproduced). The 2026-10-03 symptom `version_tag == 0` is an error reply, not a bad success: `push.py` maps any non-zero status to `versionTag=0 ackedCount=0`, and the controller rejected the push with -110 (`config_repl.c` ~1839-1845, `repl_wait_majority_ack` 500 ms). It happens after a "late-joiner" election: the own-term no-op (lsn 1) is appended before any standby is registered (ha.c ~5178, repl.c ~306-311), the standbys register later via `on_standby_connected` (ha.c ~9643-9660) without the no-op being resent, and heartbeat-driven catch-up commits it only ~6 s (two 3 s cycles) after the election. A push in that window is NACKed by both standbys and times out. The fixture's readiness check (role markers + 3 s settle) did not cover that window. The controller behaviour itself (fail-loud rejection) is correct; its ~6 s write-unavailability is tracked as bug-1913.
|
|
|
|
## Fix
|
|
Test-side (`src/xceadmin/tests`): new `wait_for_primary_write_ready()` / `primary_write_ready()` in `_live_e2e_common.py` wait until a commit logged after the latest "own-term noop appended lsn=N" reaches N; the `live_ha_cluster` fixture calls it (30 s) after waiting for standbys; versionTag assertions now print the CLI's stdout/stderr. Fix commit 362098b1.
|
|
|
|
## Verification
|
|
- Test: `test_live_config_push_against_primary_reaches_success_path` in `src/xceadmin/tests/test_e2e_live_ha_push.py` — RED before under CPU load (40 busy loops): 7/28 runs failed with `expected a non-zero versionTag, got 0 (... stderr='Error: controller returned status -110')`, all late-joiner elections; GREEN after: 95 looped runs incl. 24 under the same load (7 late-joiner elections, all passed). Helper unit tests in `test_live_e2e_common.py` (no-op not committed → not ready; latest no-op only; commits before the latest no-op ignored).
|
|
- Suites: xceadmin 373 passed, 2026-10-03.
|
|
- Live: residual, different mechanism: under heavy oversubscription a normally-elected primary can miss the 500 ms marker barrier (1/70 HA loops).
|
|
|
|
## References
|
|
- Related: bug-1477, bug-1479, bug-606
|