lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(buglog): bug-1485 and bug-1725 recurred during controller-peers-env-fallback task 4.2

Browse source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-03 07:45:06 +00:00
parent 1fd934ca33
commit bdfa6051c7
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
3 changed files with 20 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

20
.wolf/buglog.json
View file

@ -6084,14 +6084,14 @@
},
{
"id": "bug-1485",
"status": "cannot-reproduce",
"status_updated": "2026-09-24",
"status_note": "stderr capture added a5560256, never re-seen",
"status": "investigating",
"status_updated": "2026-10-03",
"status_note": "2026-10-03, controller-peers-env-fallback task 4.2 review: one run of 'bash src/tools/xce-build.sh xceadmin --incremental' failed this test with a NEW symptom, assert version_tag > 0 got 0 (not a refused connection); passed on the immediate re-run and on a run with XCE_CONTROLLER_PEERS exported. The diff under test touched only xceadmin cli.py/test_cli.py and this test passes --controller explicitly, so unrelated.",
"component": "xceadmin",
"severity": "unknown",
"timestamp": "2026-07-30T14:35:00Z",
"last_seen": "2026-09-19T09:00:00Z",
"occurrences": 25,
"last_seen": "2026-10-03T07:44:48Z",
"occurrences": 26,
"error_message": "test_e2e_live_ha_push.py::test_live_config_push_against_primary_reaches_success_path flaky…\n\ntest_e2e_live_ha_push.py::test_live_config_push_against_primary_reaches_success_path flaky (near-100% fail rate across ~25 live runs during admin-config-tool task 6.3 round-2 remediation): TimeoutError: controller never opened localhost:<port>: [Errno 111] Connection refused, raised from a fresh wait_for_port re-probe of the elected primary's own address performed AFTER _wait_for_primary (CTRL_PRIMARY marker) + a new _wait_for_standbys (both other nodes show CTRL_STANDBY) + a 3s settle sleep all already succeeded.",
"file": "src/xceadmin/tests/test_e2e_live_ha_push.py (live_ha_cluster fixture); root cause is in src/controller (exact site not yet isolated -- C-side investigation out of this task's boundary)",
"root_cause": "NOT a pure test-side readiness race as round-1 review assumed. Live forensic capture (ps aux mid-run) caught the freshly-elected ctrl-1 as a <defunct> zombie process moments after its own log recorded the full raft_cb_become_leader promotion sequence ending at 'HA: repl registered standby ep[1] (slot 1, ...)' -- the log then goes silent (no crash trace, no further lines) and the process is gone. stderr was DEVNULL'd in the fixture so no ASAN/signal diagnostic was captured from a real failing pytest run. Confusingly, 3 separate standalone (non-pytest) repro scripts using byte-identical spawn code (same xce_controller build.dbg binary, same deploy-ha.sh cluster, same peers/certs/workdir args) ran the exact same primary-promotion sequence successfully every single time with the primary staying alive and the port reachable for 20-30s post-election -- while pytest-invoked runs of the identical fixture code failed roughly 24/25 times. Ruled out during investigation: pytest's default fd-capture (reproduced with -s / capturing disabled, still failed 3/3), ulimits (NPROC/NOFILE/AS/STACK identical between nix-develop plain-shell and pytest-invoked-via-python contexts), and orphaned-process port contention (ps aux clean before each batch). True root cause not isolated -- prime suspects are a genuine timing-sensitive controller-side crash/race in the CTRL_STANDBY->CTRL_PRIMARY raft promotion path (raft_cb_become_leader / repl registration) that pytest's process tree somehow makes far more likely to trigger, or a process-group/session/signal-mask difference inherited by subprocess.Popen children when the parent is pytest vs a bare python3 script.",
@ -11051,14 +11051,14 @@
},
{
"id": "bug-1725",
"status": "fixed",
"status_updated": "2026-09-24",
"status_note": "",
"status": "open",
"status_updated": "2026-10-03",
"status_note": "2026-10-03, controller-peers-env-fallback task 4.2 (RED run): test_e2e_live_push.py::test_live_config_push_against_real_single_node_controller failed once — 'config activated from committed marker tag=...' not yet in the controller log after the push returned; passed on the next two full xceadmin runs. Diff under test touched only xceadmin cli.py/test_cli.py and the test passes --controller explicitly, so unrelated.",
"component": "xceadmin",
"severity": "unknown",
"timestamp": "2026-08-28T00:00:00Z",
"last_seen": "2026-09-19T09:00:00Z",
"occurrences": 1,
"last_seen": "2026-10-03T07:45:06Z",
"occurrences": 2,
"error_message": "After bug-1724's fix made…\n\nAfter bug-1724's fix made test_live_config_push_against_standby_returns_standby_rejection_not_redirect actually execute (previously always skipped), it FAILED for real: pushing config to a standby now redirects to the primary and succeeds, contradicting the test's own hard-coded 'standby flatly rejects, no redirect' assertions. Separately, the rewritten success-path test's reused _assert_primary_committed() helper intermittently failed its 4th assertion ('config activated from committed marker tag=...' not found in the primary's log immediately after the push CLI returned success).",
"file": "src/xceadmin/tests/test_e2e_live_ha_push.py",
"root_cause": "(1) The test's assumption was stale, not wrong test infra: bug-1700/1701 (fixed 2026-08-26) closed the exact standby-redirect gap this test was written to regression-lock, at the generic ha_dispatch_gate() rejection level in cluster.c -- the test's own docstring had explicitly predicted this exact failure as the correct signal to replace it with a success-path assertion. design.md ('Boundary Commitments'/'Revalidation Triggers') and roadmap.md still described the gap as open. (2) config_activate_committed() on the primary runs off the WAL apply-pump dispatch (persist.c ~5111, the bug-1469 fix), asynchronously relative to the majority-ack-gated client reply (commitIndex vs lastApplied are distinct steps) -- the CLI can legitimately return success before the primary's own local apply of that marker has logged its activation line, a latent race in the existing single-shot log read shared by the direct-to-primary test too.",

10
.wolf/bugs/bug-1485.md
View file

@ -1,13 +1,13 @@
---
id: "bug-1485"
status: "cannot-reproduce"
status_updated: "2026-09-24"
status_note: "stderr capture added a5560256, never re-seen"
status: "investigating"
status_updated: "2026-10-03"
status_note: "2026-10-03, controller-peers-env-fallback task 4.2 review: one run of 'bash src/tools/xce-build.sh xceadmin --incremental' failed this test with a NEW symptom, assert version_tag > 0 got 0 (not a refused connection); passed on the immediate re-run and on a run with XCE_CONTROLLER_PEERS exported. The diff under test touched only xceadmin cli.py/test_cli.py and this test passes --controller explicitly, so unrelated."
component: "xceadmin"
severity: "unknown"
timestamp: "2026-07-30T14:35:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 25
last_seen: "2026-10-03T07:44:48Z"
occurrences: 26
files: ["src/xceadmin/tests/test_e2e_live_ha_push.py (live_ha_cluster fixture); root cause is in src/controller (exact site not yet isolated -- C-side investigation out of this task's boundary)"]
tags: ["flaky-test", "ha", "raft-election", "controller-crash", "admin-config-tool", "task-6.3", "e2e-live", "unresolved", "pytest-specific"]
related: ["bug-1477", "bug-1479", "bug-606"]

10
.wolf/bugs/bug-1725.md
View file

@ -1,13 +1,13 @@
---
id: "bug-1725"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
status: "open"
status_updated: "2026-10-03"
status_note: "2026-10-03, controller-peers-env-fallback task 4.2 (RED run): test_e2e_live_push.py::test_live_config_push_against_real_single_node_controller failed once — 'config activated from committed marker tag=...' not yet in the controller log after the push returned; passed on the next two full xceadmin runs. Diff under test touched only xceadmin cli.py/test_cli.py and the test passes --controller explicitly, so unrelated."
component: "xceadmin"
severity: "unknown"
timestamp: "2026-08-28T00:00:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
last_seen: "2026-10-03T07:45:06Z"
occurrences: 2
files: ["src/xceadmin/tests/test_e2e_live_ha_push.py"]
tags: ["xceadmin", "config-push", "redirect", "OPCode_upgradeConfig", "stale-test", "design-doc-sync", "roadmap-sync", "race-condition", "config_activate_committed", "found-live", "root-caused", "live-verified"]
related: ["bug-1724", "bug-1700", "bug-1701", "bug-1468"]