test(admin-config-tool): update standby-redirect E2E for bug-1700/1701
test_live_config_push_against_standby_returns_standby_rejection_not_redirect locked in the OLD controller behavior (a standby flatly rejects OP_upgradeConfig with no redirect fields). That gap is now closed: myreq_handler's gate-rejection path calls fill_redirect_reply() on every -XCE_ERR_STANDBY rejection (bug-1700), and handle_ctrlhello() now actually populates primary_host/primary_port from the static --peers list (bug-1701) -- a standby pushing config now redirects to the primary and the push succeeds. Renamed to test_live_config_push_against_standby_redirects_and_succeeds and rewritten to assert the new behavior. _assert_primary_committed() changed from a single-shot log read to a poll loop (up to 5s) for the "config activated from committed marker..." line -- accounts for the documented async gap between the majority-ack-gated client reply and the primary's own local WAL-apply- pump activation, a real race the old single-shot read could hit. meson-build.sh: wire src/tools and each build profile's controller subdir into the isolated repo-mirror the E2E tests run against, and export XCEADMIN_BIN explicitly, so these live tests actually find real binaries instead of always skipping. Verified: xceadmin suite 288/288 (incl. both tests in this file); controller suite 1220/1222 (only the pre-existing bug-1561 flake). Updated admin-config-tool design.md and roadmap.md to mark the dispatch-gate and redirect-wiring gaps RESOLVED, matching this test's new assertions and the live evidence. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
This commit is contained in:
parent
14f95b06fe
commit
6193fa7ea1
4 changed files with 168 additions and 100 deletions
|
|
@ -87,7 +87,7 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o
|
|||
- [ ] validate_host() real authorization — currently a stub returning success for all peers (`join.c:13-16`, `return 1;` unconditionally). **`CONF_ENABLE_UPGRADE_CONFIG` is already `1` in the standard build** (`config_repl.h:27-28`, no build-system override found) — the precondition this item was originally framed to guard against has already been crossed; `OP_upgradeConfig` (and `OPCode_schedAdmin`, added 2026-08-02, `scheduler-core`) are live and unauthenticated TODAY, not a future risk to prevent. Minimum fix: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` and `OPCode_schedAdmin` with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_ **Part of the broader `peer-authentication-hardening` item above.**
|
||||
- [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_ **Part of the broader `peer-authentication-hardening` item above.**
|
||||
- [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_
|
||||
- [x] ~~OP_upgradeConfig standby redirect verification~~ — **RESOLVED 2026-08-19 (finding: item's own premise doesn't match production behavior, but coverage is already complete).** A standby does NOT return the primary's address for `OPCode_upgradeConfig` — `ha_dispatch_gate` rejects it before the handler's own dead defensive check ever runs, replying flat `status=-XCE_ERR_STANDBY, data_which=nodata` with no redirect fields; `fill_redirect_reply()` (`join.c:27`) is never called for this opcode. This is a documented, deliberate feature gap (see `admin-config-tool` design.md's "Revalidation Triggers"), not a missing test. Coverage is already complete from both directions: the client-side redirect-following mechanism is unit-tested against mocks (`src/xceadmin/tests/test_discovery.py`, `test_config_push_mock.py::test_push_config_follows_one_redirect_to_primary`), and the real server-side current behavior (no redirect, flat rejection) is live-verified against a real 3-node cluster and regression-locked (`src/xceadmin/tests/test_e2e_live_ha_push.py::test_live_config_push_against_standby_returns_standby_rejection_not_redirect`). The remaining real work — wiring `fill_redirect_reply()` for `OPCode_upgradeConfig` — is a feature task already tracked under `admin-config-tool`'s Revalidation Triggers, not this item.
|
||||
- [x] ~~OP_upgradeConfig standby redirect verification~~ — **RESOLVED 2026-08-19, then the underlying gap itself CLOSED 2026-08-26 (bug-1700/1701).** As of 2026-08-19, a standby did NOT return the primary's address for `OPCode_upgradeConfig` — `ha_dispatch_gate` rejected it before the handler's own dead defensive check ever ran, replying flat `status=-XCE_ERR_STANDBY, data_which=nodata` with no redirect fields; `fill_redirect_reply()` (`join.c:27`) was never called for this opcode, a documented, deliberate feature gap (`admin-config-tool` design.md's "Revalidation Triggers"). That gap is now closed: `cluster.c`'s generic dispatch-gate rejection path calls `fill_redirect_reply()` on every `-XCE_ERR_STANDBY` rejection (bug-1700/1701, gate-level fix, not opcode-specific), and a standby pushing `config push` now redirects to the primary and succeeds. Coverage: client-side redirect-following remains unit-tested against mocks (`test_discovery.py`, `test_config_push_mock.py::test_push_config_follows_one_redirect_to_primary`); the real server-side behavior is live-verified against a real 3-node cluster (`src/xceadmin/tests/test_e2e_live_ha_push.py::test_live_config_push_against_standby_redirects_and_succeeds`, renamed from `..._returns_standby_rejection_not_redirect` to match the now-correct outcome).
|
||||
- [x] steering sync — `ha-async-io.md` synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.
|
||||
- [ ] xce_test WAL/snapshot inspection tooling — extend the offline inspector family in `xce_test` (sibling of the shipped `waldump`, `src/controller/testsuite/waldump.{c,h}` + `cmd_waldump`). Two remaining commands: **(a) `snapdump <snap_dir|snap_file>`** — decode the `XCESNAP\0` header (8-byte LSN; `persist.c:1266` `load_snapshot`) and report whether the snapshot carries job/queue entities or is an LSN-only stub. This directly settles Open Fork #3 of the failover job-loss investigation (`.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md`): if snapshots are entity-less, any empty-rejoin/WAL-prune below committed is unrecoverable BY DESIGN. **(b) `walcompare <dirA> <dirB>`** — diff two nodes' WALs by `lsn → crc/jobid` to flag divergence/prefix-gaps between members instantly. Reuse the `waldump.c` reader/decoder; same zero-alloc capnp jobid decode + `c_capnproto_dep`/`zlib_dep` wiring already added to `xce_test_deps`. Nonzero exit on entity-less snapshot / divergence so both double as E2E assertions. _(recorded 2026-06-20; `waldump` already done.)_
|
||||
- [x] ~~xce_test shell command support~~ — **IMPLEMENTED** (task 2.1, spec `xce-test-e2e-primitives`, closed 2026-06-19; stale entry corrected 2026-08-19). `preprocess_and_run()` (`src/controller/testsuite/test.c:950`) detects a leading `!` and runs the remainder via `system()`; the shell command's exit status becomes `g_last_status` (the `$?` auto-variable) and a non-zero status sets `g_session_failed`, aborting the script the same way any other failing command does — `system()`'s child inherits the parent's stdout/stderr fds, so output lands in whatever the invoking `xce_test` session's own output is already redirected to. Works in every REPL context (no `leave_ctx()`/disconnect), including mid-session inside `worker`/`client` contexts — used this session to fix bug-1579 (resource-value-verification's nested controller-check pattern). No parser/harness-layer change remains to do.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue