diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 8e2b4e93..69c127a0 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -6084,14 +6084,14 @@ }, { "id": "bug-1485", - "status": "cannot-reproduce", - "status_updated": "2026-09-24", - "status_note": "stderr capture added a5560256, never re-seen", + "status": "investigating", + "status_updated": "2026-10-03", + "status_note": "2026-10-03, controller-peers-env-fallback task 4.2 review: one run of 'bash src/tools/xce-build.sh xceadmin --incremental' failed this test with a NEW symptom, assert version_tag > 0 got 0 (not a refused connection); passed on the immediate re-run and on a run with XCE_CONTROLLER_PEERS exported. The diff under test touched only xceadmin cli.py/test_cli.py and this test passes --controller explicitly, so unrelated.", "component": "xceadmin", "severity": "unknown", "timestamp": "2026-07-30T14:35:00Z", - "last_seen": "2026-09-19T09:00:00Z", - "occurrences": 25, + "last_seen": "2026-10-03T07:44:48Z", + "occurrences": 26, "error_message": "test_e2e_live_ha_push.py::test_live_config_push_against_primary_reaches_success_path flaky…\n\ntest_e2e_live_ha_push.py::test_live_config_push_against_primary_reaches_success_path flaky (near-100% fail rate across ~25 live runs during admin-config-tool task 6.3 round-2 remediation): TimeoutError: controller never opened localhost:: [Errno 111] Connection refused, raised from a fresh wait_for_port re-probe of the elected primary's own address performed AFTER _wait_for_primary (CTRL_PRIMARY marker) + a new _wait_for_standbys (both other nodes show CTRL_STANDBY) + a 3s settle sleep all already succeeded.", "file": "src/xceadmin/tests/test_e2e_live_ha_push.py (live_ha_cluster fixture); root cause is in src/controller (exact site not yet isolated -- C-side investigation out of this task's boundary)", "root_cause": "NOT a pure test-side readiness race as round-1 review assumed. Live forensic capture (ps aux mid-run) caught the freshly-elected ctrl-1 as a zombie process moments after its own log recorded the full raft_cb_become_leader promotion sequence ending at 'HA: repl registered standby ep[1] (slot 1, ...)' -- the log then goes silent (no crash trace, no further lines) and the process is gone. stderr was DEVNULL'd in the fixture so no ASAN/signal diagnostic was captured from a real failing pytest run. Confusingly, 3 separate standalone (non-pytest) repro scripts using byte-identical spawn code (same xce_controller build.dbg binary, same deploy-ha.sh cluster, same peers/certs/workdir args) ran the exact same primary-promotion sequence successfully every single time with the primary staying alive and the port reachable for 20-30s post-election -- while pytest-invoked runs of the identical fixture code failed roughly 24/25 times. Ruled out during investigation: pytest's default fd-capture (reproduced with -s / capturing disabled, still failed 3/3), ulimits (NPROC/NOFILE/AS/STACK identical between nix-develop plain-shell and pytest-invoked-via-python contexts), and orphaned-process port contention (ps aux clean before each batch). True root cause not isolated -- prime suspects are a genuine timing-sensitive controller-side crash/race in the CTRL_STANDBY->CTRL_PRIMARY raft promotion path (raft_cb_become_leader / repl registration) that pytest's process tree somehow makes far more likely to trigger, or a process-group/session/signal-mask difference inherited by subprocess.Popen children when the parent is pytest vs a bare python3 script.", @@ -11051,14 +11051,14 @@ }, { "id": "bug-1725", - "status": "fixed", - "status_updated": "2026-09-24", - "status_note": "", + "status": "open", + "status_updated": "2026-10-03", + "status_note": "2026-10-03, controller-peers-env-fallback task 4.2 (RED run): test_e2e_live_push.py::test_live_config_push_against_real_single_node_controller failed once — 'config activated from committed marker tag=...' not yet in the controller log after the push returned; passed on the next two full xceadmin runs. Diff under test touched only xceadmin cli.py/test_cli.py and the test passes --controller explicitly, so unrelated.", "component": "xceadmin", "severity": "unknown", "timestamp": "2026-08-28T00:00:00Z", - "last_seen": "2026-09-19T09:00:00Z", - "occurrences": 1, + "last_seen": "2026-10-03T07:45:06Z", + "occurrences": 2, "error_message": "After bug-1724's fix made…\n\nAfter bug-1724's fix made test_live_config_push_against_standby_returns_standby_rejection_not_redirect actually execute (previously always skipped), it FAILED for real: pushing config to a standby now redirects to the primary and succeeds, contradicting the test's own hard-coded 'standby flatly rejects, no redirect' assertions. Separately, the rewritten success-path test's reused _assert_primary_committed() helper intermittently failed its 4th assertion ('config activated from committed marker tag=...' not found in the primary's log immediately after the push CLI returned success).", "file": "src/xceadmin/tests/test_e2e_live_ha_push.py", "root_cause": "(1) The test's assumption was stale, not wrong test infra: bug-1700/1701 (fixed 2026-08-26) closed the exact standby-redirect gap this test was written to regression-lock, at the generic ha_dispatch_gate() rejection level in cluster.c -- the test's own docstring had explicitly predicted this exact failure as the correct signal to replace it with a success-path assertion. design.md ('Boundary Commitments'/'Revalidation Triggers') and roadmap.md still described the gap as open. (2) config_activate_committed() on the primary runs off the WAL apply-pump dispatch (persist.c ~5111, the bug-1469 fix), asynchronously relative to the majority-ack-gated client reply (commitIndex vs lastApplied are distinct steps) -- the CLI can legitimately return success before the primary's own local apply of that marker has logged its activation line, a latent race in the existing single-shot log read shared by the direct-to-primary test too.", diff --git a/.wolf/bugs/bug-1485.md b/.wolf/bugs/bug-1485.md index 182ff63b..de8dc6b1 100644 --- a/.wolf/bugs/bug-1485.md +++ b/.wolf/bugs/bug-1485.md @@ -1,13 +1,13 @@ --- id: "bug-1485" -status: "cannot-reproduce" -status_updated: "2026-09-24" -status_note: "stderr capture added a5560256, never re-seen" +status: "investigating" +status_updated: "2026-10-03" +status_note: "2026-10-03, controller-peers-env-fallback task 4.2 review: one run of 'bash src/tools/xce-build.sh xceadmin --incremental' failed this test with a NEW symptom, assert version_tag > 0 got 0 (not a refused connection); passed on the immediate re-run and on a run with XCE_CONTROLLER_PEERS exported. The diff under test touched only xceadmin cli.py/test_cli.py and this test passes --controller explicitly, so unrelated." component: "xceadmin" severity: "unknown" timestamp: "2026-07-30T14:35:00Z" -last_seen: "2026-09-19T09:00:00Z" -occurrences: 25 +last_seen: "2026-10-03T07:44:48Z" +occurrences: 26 files: ["src/xceadmin/tests/test_e2e_live_ha_push.py (live_ha_cluster fixture); root cause is in src/controller (exact site not yet isolated -- C-side investigation out of this task's boundary)"] tags: ["flaky-test", "ha", "raft-election", "controller-crash", "admin-config-tool", "task-6.3", "e2e-live", "unresolved", "pytest-specific"] related: ["bug-1477", "bug-1479", "bug-606"] diff --git a/.wolf/bugs/bug-1725.md b/.wolf/bugs/bug-1725.md index df8d3057..bcd5748f 100644 --- a/.wolf/bugs/bug-1725.md +++ b/.wolf/bugs/bug-1725.md @@ -1,13 +1,13 @@ --- id: "bug-1725" -status: "fixed" -status_updated: "2026-09-24" -status_note: "" +status: "open" +status_updated: "2026-10-03" +status_note: "2026-10-03, controller-peers-env-fallback task 4.2 (RED run): test_e2e_live_push.py::test_live_config_push_against_real_single_node_controller failed once — 'config activated from committed marker tag=...' not yet in the controller log after the push returned; passed on the next two full xceadmin runs. Diff under test touched only xceadmin cli.py/test_cli.py and the test passes --controller explicitly, so unrelated." component: "xceadmin" severity: "unknown" timestamp: "2026-08-28T00:00:00Z" -last_seen: "2026-09-19T09:00:00Z" -occurrences: 1 +last_seen: "2026-10-03T07:45:06Z" +occurrences: 2 files: ["src/xceadmin/tests/test_e2e_live_ha_push.py"] tags: ["xceadmin", "config-push", "redirect", "OPCode_upgradeConfig", "stale-test", "design-doc-sync", "roadmap-sync", "race-condition", "config_activate_committed", "found-live", "root-caused", "live-verified"] related: ["bug-1724", "bug-1700", "bug-1701", "bug-1468"]