docs(admin-config-tool): mark task 6.3 blocked pending flake reproduction

Standby-redirect coverage is done and correct (redirect wiring for
OP_upgradeConfig genuinely doesn't exist server-side yet, matches
design.md's own documented gap). Primary-push success-path test is
flaky (bug-1485) -- a dedicated debug round could not reproduce the
crash in 26+ clean runs, so root cause is unconfirmed. Test files left
uncommitted in the worktree pending re-verification on an idle host.
This commit is contained in:
Rongsong Shen 2026-07-30 14:57:32 +00:00
parent 1075f4529a
commit 9232d8fa6e

View file

@ -159,6 +159,17 @@
- _Boundary: xceadmin (test-only, cross-cutting with the built controller binary)_
- _Depends: 6.1_
- [ ] 6.3 Add a live 3-node HA E2E test for `config push` (success path + redirect)
- Task 6.2 proved a single-node deployment always returns `ackedCount == 0` and exits 1 even on full server-side success (no standbys exist to ACK) — the `acked_count > 0` exit-0 success path, and Requirement 3's redirect-following, have never been exercised against a real controller, only against mocks (task 5.6) or a topology structurally unable to reach them (task 6.2). This task closes both gaps in one live 3-node deployment.
- Deploys a real 3-node HA `xce_controller` cluster (reuse `deploy-ha.sh`'s existing 3-node pattern, e.g. `localhost:19991/19992/19993`, staggered start for clean election per this repo's existing convention), waits for a primary to be elected
- Pushing to the primary directly: real `xceadmin config push` exits 0, reports `ackedCount > 0` (a real positive count from real standby ACKs, not the single-node `0` case), and the controller logs confirm commit/activate on the primary and replication/ack on at least one standby
- Pushing to a standby node: `xceadmin` follows the redirect to the primary (Requirement 3.1, 3.3), the push still succeeds end-to-end, and the redirect is visible either in `xceadmin`'s own stderr redirect message (`discovery.py`'s existing "redirected: host:port -> host:port" line) or equivalent captured evidence
- Skips cleanly (not a hard failure) when no controller binary is available, matching task 6.2's convention; reuses task 6.2's cleanup discipline (kill all node processes, remove scratch deploy dir, on both pass and failure paths) — check whether task 6.2's fixture/helpers can be extracted and reused for the multi-node case rather than duplicated
- _Requirements: 2.1, 2.5, 2.6, 3.1, 3.2, 3.3_
- _Boundary: xceadmin (test-only, cross-cutting with the built controller binary)_
- _Depends: 6.2_
- _Blocked: standby-redirect half is DONE and correct (round-1 reviewer independently re-verified: OP_upgradeConfig has no server-side redirect wiring today, matches design.md's own pre-existing "Out of Boundary" gap; test correctly locks in real current rejection behavior). Primary-push success-path test is flaky (~24/25 live fails in round-2, see bug-1485) — elected primary observed as a `<defunct>` zombie shortly after promotion, but a dedicated debug round could NOT reproduce this in 26+ clean live runs on an idle host and could not capture any crash evidence (stderr was DEVNULL'd in the original failing runs). Leading theory: round-2's failures were environment contamination (stale venv, concurrent build/test load) rather than a deterministic product bug, but this is unconfirmed either way. Uncommitted work (test_e2e_live_ha_push.py, _live_e2e_common.py, test_e2e_live_push.py refactor) is left in the worktree, not discarded — re-run the primary-push test 20-25x standalone on an otherwise-idle host with real stderr capture (not DEVNULL) before deciding whether it's mergeable as-is, needs the settle-window-to-real-poll fix noted in bug-1485, or points to a real controller-side race requiring a separate task._
## Implementation Notes
- **1.3**: nix's bare `python3` does not put `pip` on PATH — `python3Packages.pip` had to be added to `nativeBuildInputs` alongside the packages design.md's Technology Stack table lists; not a spec gap, just an operational dependency the table didn't need to spell out.