lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

test(admin-config-tool): update standby-redirect E2E for bug-1700/1701

Browse source 
test_live_config_push_against_standby_returns_standby_rejection_not_redirect
locked in the OLD controller behavior (a standby flatly rejects
OP_upgradeConfig with no redirect fields). That gap is now closed:
myreq_handler's gate-rejection path calls fill_redirect_reply() on
every -XCE_ERR_STANDBY rejection (bug-1700), and handle_ctrlhello() now
actually populates primary_host/primary_port from the static --peers
list (bug-1701) -- a standby pushing config now redirects to the
primary and the push succeeds. Renamed to
test_live_config_push_against_standby_redirects_and_succeeds and
rewritten to assert the new behavior.

_assert_primary_committed() changed from a single-shot log read to a
poll loop (up to 5s) for the "config activated from committed
marker..." line -- accounts for the documented async gap between the
majority-ack-gated client reply and the primary's own local WAL-apply-
pump activation, a real race the old single-shot read could hit.

meson-build.sh: wire src/tools and each build profile's controller
subdir into the isolated repo-mirror the E2E tests run against, and
export XCEADMIN_BIN explicitly, so these live tests actually find real
binaries instead of always skipping.

Verified: xceadmin suite 288/288 (incl. both tests in this file);
controller suite 1220/1222 (only the pre-existing bug-1561 flake).
Updated admin-config-tool design.md and roadmap.md to mark the
dispatch-gate and redirect-wiring gaps RESOLVED, matching this test's
new assertions and the live evidence.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
This commit is contained in:
Rongsong Shen 2026-08-29 12:10:28 +00:00
parent 14f95b06fe
commit 6193fa7ea1
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
4 changed files with 168 additions and 100 deletions
Download patch file Download diff file Expand all files Collapse all files

36
src/xceadmin/meson-build.sh
View file

@ -63,6 +63,29 @@ mkdir -p "$MIRROR_DIR/src"
ln -s "$REPO_ROOT/ha" "$MIRROR_DIR/ha"
ln -s "$SRC_DIR/configs" "$MIRROR_DIR/src/configs"
ln -s "$SRC_DIR/schema" "$MIRROR_DIR/src/schema"
# tests/_live_e2e_common.py's DEPLOY_HA_SH points at a sibling tools/
# dir -- same fixed-relative-path assumption as configs/schema above.
ln -s "$SRC_DIR/tools" "$MIRROR_DIR/src/tools"
# tests/_live_e2e_common.py's CONTROLLER_CANDIDATES look for a real
# xce_controller binary under a sibling xce.build/build.release/build.dbg
# dir (xce-build.sh's own profile-dir naming, src/tools/xce-build.sh) next
# to this file's own SRC_DIR -- same fixed-relative-path assumption as
# ha/configs/schema above, so it needs the same out-of-tree symlink
# treatment or the live E2E tests can never find a controller binary that
# was actually built, no matter what's really on disk. Only the `controller`
# subdir is linked (not the whole profile-root xce.build/etc.) because
# `controller` and `xceadmin` are always siblings under a profile root --
# unlike the profile root itself, this is never an ancestor of $MIRROR_DIR
# (which nests only under <profile>/xceadmin), so it can't form a
# self-referential symlink loop the way linking the whole root would.
for _live_e2e_build_dir in xce.build build.release build.dbg; do
_live_e2e_controller_dir="$SRC_DIR/$_live_e2e_build_dir/controller"
if [ -d "$_live_e2e_controller_dir" ]; then
mkdir -p "$MIRROR_DIR/src/$_live_e2e_build_dir"
ln -s "$_live_e2e_controller_dir" "$MIRROR_DIR/src/$_live_e2e_build_dir/controller"
fi
done
cp -r "$SOURCE_DIR" "$WORK_DIR"
# Never trust a stray dev-loop artifact from the live checkout (xce-build.sh's
@ -115,7 +138,18 @@ rm -rf "$VENV_DIR"
"$VENV_DIR/bin/pip" install "$WORK_DIR" --quiet
echo "=== xceadmin (meson): running tests ==="
"$VENV_DIR/bin/python" -m pytest "$WORK_DIR/tests" -v
# tests/_live_e2e_common.py's XCEADMIN_CANDIDATES look for a sibling
# xce.build/build.release/build.dbg/xceadmin/.venv/bin/xceadmin next to this
# file's own SRC_DIR -- but for THIS profile that path is $BUILD_DIR itself
# (the xceadmin subdir currently being built), which is an ancestor of
# $MIRROR_DIR and so can't be symlinked into the mirror without a
# self-referential loop (unlike the sibling `controller` dir linked above).
# Route around it the same way a caller with a custom/explicit build dir
# already would: XCE_CONTROLLER_BIN/XCEADMIN_BIN
# (resolve_binary_candidates in _live_e2e_common.py) override discovery
# entirely, and this build's own freshly pip-installed binary is exactly
# what the live E2E tests should exercise.
XCEADMIN_BIN="$VENV_DIR/bin/xceadmin" "$VENV_DIR/bin/python" -m pytest "$WORK_DIR/tests" -v
# Build an installable wheel from the SAME schema-synced $WORK_DIR the pinned
# install above just tested -- not a second, separately-assembled tree --

214
src/xceadmin/tests/test_e2e_live_ha_push.py
View file

@ -12,48 +12,59 @@ pattern -- see ``live_ha_cluster`` below):
``test_live_config_push_against_primary_reaches_success_path`` genuinely
exits 0 with a positive ``ackedCount`` from real standby ACKs.
REAL FINDING, not a test bug -- the redirect scenario does NOT close the
way the task text originally anticipated: pushing `config push` to a
*standby* does NOT redirect to the primary against a real controller
today, and this is not something a test-only change can fix. This is an
ALREADY-DOCUMENTED, pre-existing gap in design.md's own "Boundary
Commitments" section (`.kiro/specs/admin-config-tool/design.md`, "Out of
Boundary" -- "Redirect wiring for `OP_upgradeConfig`") and "Revalidation
Triggers" (`fill_redirect_reply()` extended to `OPCode_upgradeConfig` is
listed there as a FUTURE, not-yet-done trigger): `fill_redirect_reply()`
(`join.c:27-31`) is only ever invoked from the `OPCode_join` path
(`join.c:277`); nothing calls it for `OPCode_upgradeConfig`.
`RequestReply.redirectHost`/`redirectPort` are therefore never populated
UPDATE (2026-08-28): the redirect gap described below WAS closed, exactly
as this docstring's own original prediction anticipated -- see
``test_live_config_push_against_standby_redirects_and_succeeds`` (renamed
from ``..._returns_standby_rejection_not_redirect``), which now live-
verifies the success-via-redirect path instead. The investigation history
below is kept for context (why the task's original text and the
mid-investigation finding diverged, and exactly what changed to close the
gap); treat every "does NOT redirect" / "no redirect line, no success"
statement below as describing REAL, live-verified controller behavior AS
OF 2026-08-26 (the day before bug-1700/1701 landed), not current behavior.
ORIGINAL FINDING (now superseded) -- the redirect scenario did NOT close
the way the task text originally anticipated: pushing `config push` to a
*standby* did NOT redirect to the primary against a real controller as of
this file's original writing, and this was not something a test-only
change could fix. This was an ALREADY-DOCUMENTED, pre-existing gap in
design.md's own "Boundary Commitments" section
(`.kiro/specs/admin-config-tool/design.md`, "Out of Boundary" -- "Redirect
wiring for `OP_upgradeConfig`") and "Revalidation Triggers"
(`fill_redirect_reply()` extended to `OPCode_upgradeConfig` was listed
there as a FUTURE, not-yet-done trigger): `fill_redirect_reply()`
(`join.c:27-31`) was only ever invoked from the `OPCode_join` path
(`join.c:277`); nothing called it for `OPCode_upgradeConfig`.
`RequestReply.redirectHost`/`redirectPort` were therefore never populated
for an `upgradeConfig` reply. Confirmed directly in `cluster.c`'s
`myreq_handler`: a standby's `OP_upgradeConfig` request is rejected by the
`myreq_handler`: a standby's `OP_upgradeConfig` request was rejected by the
generic `ha_dispatch_gate()` role check (`ha.c`, opcode absent from the
`CTRL_STANDBY` allow-list) BEFORE the opcode-specific `case
OPCode_upgradeConfig:` handler ever runs -- the gate-rejection reply path
(`cluster.c` ~line 273-280) sends `reply.status = -XCE_ERR_STANDBY`
OPCode_upgradeConfig:` handler ever ran -- the gate-rejection reply path
sent `reply.status = -XCE_ERR_STANDBY`
(`errnox.h`: `E_XCE_STANDBY == XCE_ERRNO_BASE + 13 == 10014`) with
`data_which = RequestReply_data_nodata` and never touches
`data_which = RequestReply_data_nodata` and never touched
`redirectHost`/`redirectPort` at all. Live-verified against a real 3-node
cluster: pushing to a standby returns `versionTag=0 ackedCount=0`, exit 1,
stderr `"Error: controller returned status -10014"` -- no redirect line,
no success. `xceadmin.discovery.follow_redirects`/`decode_reply` behave
exactly per Requirement 3.1's contract given this reply (both fields
absent/empty => no redirect => the bare rejection is returned as-is); the
gap is entirely server-side and out of this task's test-only boundary to
fix (design.md: "This is a controller defect, not an xceadmin defect;
xceadmin implements the client side of the documented contract... and
cannot itself fix the gate"). Task 5.6 already covers the SUCCESSFUL
redirect-follow path against mocks (`test_config_push_mock.py`), which is
still the only place that path is exercised.
``test_live_config_push_against_standby_returns_standby_rejection_not_redirect``
below live-verifies and regression-locks this REAL current behavior
instead of the hypothetical "redirect then succeed" outcome -- the same
resolution task 6.2 used for its own real-vs-literal-task-text mismatch
(single-node `ackedCount == 0`/exit 1). The moment a future controller-side
task wires `fill_redirect_reply()` for `OPCode_upgradeConfig` (already
tracked as a Revalidation Trigger in design.md), THIS test will start
failing on the "no redirect line" assertion -- that failure is the signal
to replace it with a real success-path assertion, matching what this
docstring originally described before investigation.
cluster at the time: pushing to a standby returned `versionTag=0
ackedCount=0`, exit 1, stderr `"Error: controller returned status
-10014"` -- no redirect line, no success.
WHAT CLOSED IT (bug-1700/1701, fixed 2026-08-26): `cluster.c`'s generic
`myreq_handler` dispatch-gate rejection path (~`cluster.c:411-437`) now
calls `fill_redirect_reply(&reply, &c->ha)` whenever `ha_dispatch_gate()`
rejects with `-XCE_ERR_STANDBY` -- a fix at the GATE itself, covering
every opcode that hits that rejection, not an `OPCode_upgradeConfig`-
specific patch (broader than what the original Revalidation Trigger
anticipated). A second, independently-discovered blocker (bug-1701:
`ha_state_t.primary_host`/`primary_port` were never actually written by
any production code path on a standby, so `fill_redirect_reply()` had
nothing to fill) was closed the same day via a local lookup in
`handle_ctrlhello()` against the statically-known `--peers` list. Together
these make a standby's `OP_upgradeConfig` rejection carry a real, populated
redirect hint, and `xceadmin.discovery.follow_redirects()` -- already
generic and already tested against mocks (`test_config_push_mock.py::
test_push_config_follows_one_redirect_to_primary`) -- transparently
retries against the primary with zero client-side changes required.
Shares binary discovery / port / stdout-parsing helpers with
``test_e2e_live_push.py`` via ``_live_e2e_common`` (extracted there as part
@ -68,10 +79,16 @@ that merging them would cost more readability than it would save.
Skip convention: identical to task 6.2 -- an imperative ``pytest.skip``
inside the fixture (no controller binary / no ``openssl``), plus an
in-test skip for a missing ``xceadmin`` binary, both matching this suite's
existing cert-dependent skip precedent (task 5.3). Skipping is the NORMAL
outcome under the fast ``xce-build.sh xceadmin`` / meson path, which runs
pytest from an isolated ``repo-mirror`` with no sibling
``xce.build``/``build.dbg``/``build.release`` controller binaries.
existing cert-dependent skip precedent (task 5.3). UPDATE (2026-08-28):
skipping is no longer the normal outcome under the ``xce-build.sh
xceadmin`` / meson path -- ``meson-build.sh`` now symlinks each profile's
``controller`` subdirectory (and ``src/tools/``, for ``deploy-ha.sh``) into
the isolated ``repo-mirror`` it runs pytest from, and sets ``XCEADMIN_BIN``
to that build's own freshly-installed binary, so these tests find and
exercise a real controller/xceadmin whenever one exists anywhere in the
repo's ``xce.build``/``build.dbg``/``build.release`` dirs. Skipping now
correctly means what it always should have: no such binary was ever built
in this checkout.
Process-leak fix (task 6.2's own finding, reused here for all 3 nodes):
each controller is launched as the real ``xce_controller`` binary
@ -413,11 +430,36 @@ def _assert_no_decode_crash(push_result: subprocess.CompletedProcess) -> None:
assert "Traceback" not in push_result.stderr, push_result.stderr
def _assert_primary_committed(log_text: str, version_tag_hex: str) -> None:
# config_activate_committed() (config_repl.c) runs off the WAL apply-pump
# dispatch (persist.c, the on_commit_advanced-adjacent seam bug-1469 added),
# asynchronously relative to the majority-ack-gated client reply
# (commitIndex/lastApplied are distinct steps -- controller/CLAUDE.md's TLA+
# mapping) -- a client can legitimately see its push's "success" reply
# before the primary's OWN local apply of that same marker has logged
# "config activated...". Poll instead of a single-shot read to close this
# real race rather than assume an ordering the design never guarantees.
_ACTIVATION_POLL_TIMEOUT_S = 5.0
_ACTIVATION_POLL_INTERVAL_S = 0.2
def _assert_primary_committed(log_file: Path, version_tag_hex: str) -> str:
marker_line = f"config activated from committed marker tag={version_tag_hex}"
deadline = time.monotonic() + _ACTIVATION_POLL_TIMEOUT_S
log_text = log_file.read_text()
while marker_line not in log_text and time.monotonic() < deadline:
time.sleep(_ACTIVATION_POLL_INTERVAL_S)
log_text = log_file.read_text()
assert f"tag={version_tag_hex}" in log_text, log_text
assert "Config change committed" in log_text
assert f"Config upgraded to tag={version_tag_hex}" in log_text
assert f"config activated from committed marker tag={version_tag_hex}" in log_text
assert marker_line in log_text, (
f"{marker_line!r} never appeared within {_ACTIVATION_POLL_TIMEOUT_S}s "
f"of the push replying success -- see this function's own comment "
f"for why activation is asynchronous relative to the reply. "
f"log_text={log_text!r}"
)
return log_text
def test_live_config_push_against_primary_reaches_success_path(
@ -464,8 +506,7 @@ def test_live_config_push_against_primary_reaches_success_path(
assert push_result.returncode == 0, push_result.stderr
version_tag_hex = f"{version_tag:016x}"
primary_log_text = live_ha_cluster.primary.log_file.read_text()
_assert_primary_committed(primary_log_text, version_tag_hex)
primary_log_text = _assert_primary_committed(live_ha_cluster.primary.log_file, version_tag_hex)
# Majority-ACK barrier evidence on the primary's own log: at least one
# standby genuinely acked this exact tag (push_config_to_standbys).
@ -494,33 +535,29 @@ def test_live_config_push_against_primary_reaches_success_path(
assert standby_saw_it, [s.log_file.read_text() for s in live_ha_cluster.standbys]
# errnox.h: XCE_ERRNO_BASE=10001, E_XCE_STANDBY is the 14th entry (0-indexed
# 13) in that enum => 10001+13=10014. "Controller in CTRL_STANDBY; op is
# primary-only" -- exactly the rejection a standby's OP_upgradeConfig hits
# (module docstring). Named here instead of a bare magic number per this
# project's error-handling convention of surfacing XCE's own error
# vocabulary rather than an unexplained literal.
_E_XCE_STANDBY = 10014
def test_live_config_push_against_standby_returns_standby_rejection_not_redirect(
def test_live_config_push_against_standby_redirects_and_succeeds(
live_ha_cluster: HaCluster, tmp_path: Path
) -> None:
"""Pushing `config push` directly to a standby node against a REAL
controller today does NOT redirect to the primary -- it is flatly
rejected with ``E_XCE_STANDBY`` (module docstring: a pre-existing,
already-documented controller-side gap, `fill_redirect_reply()` never
wired for `OPCode_upgradeConfig`; out of this test-only task's
boundary to fix). This test live-verifies and regression-locks that
REAL current behavior: no redirect line, no success, and specifically
the `E_XCE_STANDBY` status rather than some other/unexpected failure
mode, so a future change to the rejection path itself would also be
caught here even before the redirect gap is closed.
controller now redirects to the primary and succeeds (bug-1700/1701,
fixed 2026-08-26 -- module docstring's "WHAT CLOSED IT" section): the
same success outcome as pushing straight to the primary
(`test_live_config_push_against_primary_reaches_success_path`), reached
via one transparent redirect hop instead of zero.
Renamed from `test_live_config_push_against_standby_returns_standby_
rejection_not_redirect` -- that test's own assertions predicted exactly
this outcome the moment the redirect gap closed (see module docstring
for the investigation history behind both the original gap and its
fix).
Confirms via ``discovery.py``'s own stderr line (`follow_redirects`:
"redirected: host:port -> host:port") being ABSENT -- direct evidence
no redirect was attempted, not an inference from the push merely
failing for some other reason.
"redirected: host:port -> host:port") being PRESENT -- direct evidence
a redirect was actually followed, not an inference from the push merely
succeeding some other way (e.g. `standby_addr` accidentally being the
primary after an election reshuffle; the fixture's own settle window
exists precisely to close that race, see module docstring's
"Election-wait" section).
"""
xceadmin_bin = find_first_executable(XCEADMIN_CANDIDATES)
if xceadmin_bin is None:
@ -542,33 +579,30 @@ def test_live_config_push_against_standby_returns_standby_rejection_not_redirect
)
_assert_no_decode_crash(push_result)
# No redirect was attempted -- the standby's reply never carries
# redirectHost/redirectPort for this opcode (module docstring), so
# discovery.py's follow_redirects has nothing to follow and never
# prints its "redirected: ..." line.
unexpected_redirect_line = f"redirected: {standby_addr} -> {primary_addr}"
assert unexpected_redirect_line not in push_result.stderr, (
"a redirect line appeared -- if the controller-side redirect gap "
"documented in this file's module docstring (design.md 'Boundary "
"Commitments' / 'Revalidation Triggers') has been closed, this "
"test is stale and should be replaced with a real success-path "
f"assertion: {push_result.stderr!r}"
# Positive evidence the redirect was actually followed -- the standby's
# reply now carries redirectHost/redirectPort (bug-1700/1701), so
# discovery.py's follow_redirects retries against the primary and prints
# this line before the retried push itself is attempted.
expected_redirect_line = f"redirected: {standby_addr} -> {primary_addr}"
assert expected_redirect_line in push_result.stderr, (
f"expected follow_redirects() to redirect from the standby to the "
f"primary, got stderr={push_result.stderr!r}"
)
match = VERSION_TAG_RE.search(push_result.stdout)
assert match, f"no versionTag/ackedCount found in stdout: {push_result.stdout!r}"
version_tag, acked_count = int(match.group(1)), int(match.group(2))
assert version_tag > 0
# Real, live-verified standby-contact behavior today: the reply is a
# bare gate rejection (versionTag=0 ackedCount=0), not a redirect or a
# partial success.
assert version_tag == 0
assert acked_count == 0
assert push_result.returncode == 1
assert f"controller returned status -{_E_XCE_STANDBY}" in push_result.stderr, push_result.stderr
# Same success bar as the direct-to-primary test: a genuine positive
# ackedCount from real standby ACKs, exit 0 -- the redirect hop changes
# HOW the primary was reached, not what a successful push looks like.
assert acked_count > 0, (
f"expected a positive ackedCount from real standby ACKs after the "
f"redirect, got {acked_count} (stdout={push_result.stdout!r} "
f"stderr={push_result.stderr!r})"
)
assert push_result.returncode == 0, push_result.stderr
# The rejected push never reached the primary at all -- its log shows
# no upgrade activity from this attempt (independent confirmation this
# is a clean gate rejection, not a landed-then-lost push).
primary_log_text = live_ha_cluster.primary.log_file.read_text()
assert "Config marker failed to reach committed prefix" not in primary_log_text
version_tag_hex = f"{version_tag:016x}"
_assert_primary_committed(live_ha_cluster.primary.log_file, version_tag_hex)