docs(buglog): bug-1683, bug-1485, bug-1725 fixed with corrected root causes; log bug-1913

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.wolf/bugs/bug-1485.md

@@ -1,8 +1,8 @@
 ---
 id: "bug-1485"
-status: "investigating"
+status: "fixed"
 status_updated: "2026-10-03"
-status_note: "2026-10-03, controller-peers-env-fallback task 4.2 review: one run of 'bash src/tools/xce-build.sh xceadmin --incremental' failed this test with a NEW symptom, assert version_tag > 0 got 0 (not a refused connection); passed on the immediate re-run and on a run with XCE_CONTROLLER_PEERS exported. The diff under test touched only xceadmin cli.py/test_cli.py and this test passes --controller explicitly, so unrelated."
+status_note: ""
 component: "xceadmin"
 severity: "unknown"
 timestamp: "2026-07-30T14:35:00Z"
@@ -11,6 +11,7 @@ occurrences: 26
 files: ["src/xceadmin/tests/test_e2e_live_ha_push.py (live_ha_cluster fixture); root cause is in src/controller (exact site not yet isolated -- C-side investigation out of this task's boundary)"]
 tags: ["flaky-test", "ha", "raft-election", "controller-crash", "admin-config-tool", "task-6.3", "e2e-live", "unresolved", "pytest-specific"]
 related: ["bug-1477", "bug-1479", "bug-606"]
+fix_commits: ["362098b1"]
 repro_attempts: "0/40 repro attempts (2026-08-01)"
 legacy: true
 ---
@@ -27,17 +28,15 @@ _Not recorded (migrated from buglog.json)._
 - Attempts: 0/40 repro attempts (2026-08-01)
 
 ## Root cause
-NOT a pure test-side readiness race as round-1 review assumed. Live forensic capture (ps aux mid-run) caught the freshly-elected ctrl-1 as a <defunct> zombie process moments after its own log recorded the full raft_cb_become_leader promotion sequence ending at 'HA: repl registered standby ep[1] (slot 1, ...)' -- the log then goes silent (no crash trace, no further lines) and the process is gone. stderr was DEVNULL'd in the fixture so no ASAN/signal diagnostic was captured from a real failing pytest run. Confusingly, 3 separate standalone (non-pytest) repro scripts using byte-identical spawn code (same xce_controller build.dbg binary, same deploy-ha.sh cluster, same peers/certs/workdir args) ran the exact same primary-promotion sequence successfully every single time with the primary staying alive and the port reachable for 20-30s post-election -- while pytest-invoked runs of the identical fixture code failed roughly 24/25 times. Ruled out during investigation: pytest's default fd-capture (reproduced with -s / capturing disabled, still failed 3/3), ulimits (NPROC/NOFILE/AS/STACK identical between nix-develop plain-shell and pytest-invoked-via-python contexts), and orphaned-process port contention (ps aux clean before each batch). True root cause not isolated -- prime suspects are a genuine timing-sensitive controller-side crash/race in the CTRL_STANDBY->CTRL_PRIMARY raft promotion path (raft_cb_become_leader / repl registration) that pytest's process tree somehow makes far more likely to trigger, or a process-group/session/signal-mask difference inherited by subprocess.Popen children when the parent is pytest vs a bare python3 script.
+Corrected 2026-10-03, superseding the earlier zombie-primary / CPU-starvation hypotheses (kept in git history; the zombie symptom was never reproduced). The 2026-10-03 symptom `version_tag == 0` is an error reply, not a bad success: `push.py` maps any non-zero status to `versionTag=0 ackedCount=0`, and the controller rejected the push with -110 (`config_repl.c` ~1839-1845, `repl_wait_majority_ack` 500 ms). It happens after a "late-joiner" election: the own-term no-op (lsn 1) is appended before any standby is registered (ha.c ~5178, repl.c ~306-311), the standbys register later via `on_standby_connected` (ha.c ~9643-9660) without the no-op being resent, and heartbeat-driven catch-up commits it only ~6 s (two 3 s cycles) after the election. A push in that window is NACKed by both standbys and times out. The fixture's readiness check (role markers + 3 s settle) did not cover that window. The controller behaviour itself (fail-loud rejection) is correct; its ~6 s write-unavailability is tracked as bug-1913.
 
 ## Fix
-UPDATE 2026-08-01 -- investigated per this entry's own "next step" (capture stderr instead of DEVNULL, re-attempt). Redirected each node's stderr to deploy_dir/logs/<name>.stderr.log (permanent change to test_e2e_live_ha_push.py, replacing the DEVNULL redirect that left this entry's original investigation with no diagnostic) and added an XCE_KEEP_DEPLOY_DIR=1 debug env var to preserve deploy_dir across the fixture's own cleanup for inspection. Ran the live test 40 times total in this environment: 25 baseline (idle sandbox, load avg ~2/22 cores) + 15 more with a concurrent `xce-build.sh controller debug` compile racing for CPU (separate build.dbg dir, so it could not clobber the xce.build controller binary the test depends on -- unlike bug-1486's same-dir clobber shape). Result: 40/40 PASSED, 0/120 captured stderr logs (3 nodes x 40 runs) non-empty -- no crash, no ASAN report, in any run. Could NOT reproduce the ~24/25 failure rate from the original investigation despite matching the documented repro shape exactly (same binaries, same fixture code, same pytest invocation).
-
-NOT marking fixed -- absence of reproduction here is not proof of absence of the underlying bug, and this session's single concurrent compile is a much lighter contention scenario than the real multi-session sandbox reality bug-1486 documents (several concurrent AGENT SESSIONS, potentially each running their own live 3-node cluster). Leading hypothesis (unconfirmed): the original ~24/25 failure rate was itself a symptom of resource contention from OTHER concurrent sessions on the shared sandbox during that investigation (CPU starvation delaying the primary's post-election warmup past _POST_ELECTION_SETTLE_S, or an OOM-adjacent condition killing the freshly-forked primary outright) rather than a bug in the promotion path reachable in isolation -- consistent with the original investigation's own "pytest's process tree somehow makes far more likely" framing, since pytest's subprocess tree carries more memory/fd overhead than the bare python3 scripts that never reproduced it there either.
-
-Downgraded to low-priority/environment-flake, left OPEN. The stderr-capture infrastructure is now permanently in place (not reverted) so the next real live failure -- in pytest, in a genuinely contended multi-session sandbox, or in CI -- captures an actual ASAN report or signal instead of DEVNULL silence, which is what would actually close this out.
+Test-side (`src/xceadmin/tests`): new `wait_for_primary_write_ready()` / `primary_write_ready()` in `_live_e2e_common.py` wait until a commit logged after the latest "own-term noop appended lsn=N" reaches N; the `live_ha_cluster` fixture calls it (30 s) after waiting for standbys; versionTag assertions now print the CLI's stdout/stderr. Fix commit 362098b1.
 
 ## Verification
-_Not recorded (migrated from buglog.json)._
+- Test: `test_live_config_push_against_primary_reaches_success_path` in `src/xceadmin/tests/test_e2e_live_ha_push.py` — RED before under CPU load (40 busy loops): 7/28 runs failed with `expected a non-zero versionTag, got 0 (... stderr='Error: controller returned status -110')`, all late-joiner elections; GREEN after: 95 looped runs incl. 24 under the same load (7 late-joiner elections, all passed). Helper unit tests in `test_live_e2e_common.py` (no-op not committed → not ready; latest no-op only; commits before the latest no-op ignored).
+- Suites: xceadmin 373 passed, 2026-10-03.
+- Live: residual, different mechanism: under heavy oversubscription a normally-elected primary can miss the 500 ms marker barrier (1/70 HA loops).
 
 ## References
 - Related: bug-1477, bug-1479, bug-606