myxce/.kiro/specs/admin-config-tool/tasks.md

# Implementation Plan

- [x] 1. Foundation: package scaffolding, shared errors, build/toolchain integration
- [x] 1.1 Set up the xceadmin Python package skeleton
  - Editable install registers a real `xceadmin` console-script entry point backed by an empty, working click group (no subcommands yet — `config build`/`config push` are added by task 4.1, not this task)
  - `xcecommon` is importable as a standalone package independent of `xceadmin`'s CLI
  - Observable: running the entry point after an editable install prints usage/help text without error, showing zero commands
  - _Requirements: 5.4_

- [x] 1.2 Establish a shared error-handling foundation
  - One common base exception type carries a human-readable message
  - Every specific failure mode (schema not found, validation failure, reserved name, duplicate entry, connection timeout, corrupt reply, too many redirects, no primary available) has its own subclass
  - Observable: raising any of these subclasses and catching the common base type recovers the message
  - _Requirements: 7.1, 7.2, 7.3, 7.4_

- [x] 1.3 Wire xceadmin into the project's build and dev-shell tooling
  - Nix dev shell provides the Python interpreter and all dependencies needed to build and test xceadmin
  - Running the build command for xceadmin performs an editable install by default and a pinned non-editable install under the release profile, running the test suite after whichever install just happened
  - The build step keeps the packaged copies of the wire-protocol schema and the validation schemas in sync with their source locations elsewhere in the repo, and packages both correctly under either install mode
  - Build/test command documentation, permission allowlist entries, and automatic formatting/linting on Python file edits all exist for the new component
  - Observable: running the build command for this component from a clean checkout succeeds and reports test pass/fail
  - _Requirements: 5.5, 8.1, 8.2, 8.6_

- [x] 2. Core: xcecommon shared library
- [x] 2.1 (P) Validate operator-authored YAML against the bundled schemas
  - Each of the seven config file types can be checked against its own bundled validation schema, independent of anything the operator supplies
  - Reserved queue/host names are rejected even when otherwise schema-valid
  - Duplicate names within a single file/section are rejected
  - Cross-references to undefined groups or resources are flagged as warnings without blocking the build
  - Observable: validating a config directory with no schema files present anywhere in it still succeeds
  - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 5.5_
  - _Boundary: xcecommon.validation_

- [x] 2.2 (P) Load the cluster wire-protocol schema
  - The wire-protocol schema loads once per process and exposes every message type xceadmin needs to build or decode
  - Loading works identically whether xceadmin was installed in editable or pinned mode
  - A missing or corrupted schema copy produces a clear, actionable error rather than a crash
  - Observable: loading the schema under a pinned (non-editable) install still resolves the bundled copy correctly
  - _Requirements: 5.1, 5.5, 7.4_
  - _Boundary: xcecommon.schema_

- [x] 2.3 (P) Map validated YAML into cluster configuration fields
  - Every field the cluster configuration binary format defines is either populated from validated YAML or deliberately left untouched, with no field silently dropped
  - List-based sections preserve the exact order they appeared in the YAML
  - No timestamp, hostname, or username value ever enters the mapped output
  - Building the same input twice produces identical output
  - _Requirements: 1.1, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 5.3, 9.1, 9.2, 9.3_
  - _Boundary: xcecommon.mapping_
  - _Depends: 2.2_

- [x] 2.4 (P) Establish the TLS connection and wire framing to the controller
  - A single request can be sent and a single reply received over a TLS connection using the cluster's existing certificate trust model
  - Message framing matches the controller's own byte-for-byte, verified against known-good byte sequences
  - The connection always closes cleanly, including on error or interruption
  - _Requirements: 2.1, 2.3, 2.4, 5.2, 6.1, 7.1, 7.3, 7.6_
  - _Boundary: xcecommon.wire_
  - _Depends: 2.2_

- [x] 3. Core: xceadmin CLI orchestration
- [x] 3.1 (P) Compile a configs directory into a deployable cluster binary
  - Running the build command against a directory of YAML files produces a binary output file with a printed checksum
  - A missing individual YAML file is treated as an empty section, not an error
  - Unrecognized-but-valid YAML fields produce a warning, not a failure
  - Output can be directed to a specific path or to standard output
  - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.9, 8.3, 8.4_
  - _Boundary: xceadmin.build_
  - _Depends: 2.1, 2.3_

- [x] 3.2 (P) Follow controller redirects to find the primary
  - A reply pointing at a different host/port is followed automatically, up to a bounded number of hops
  - Exceeding the hop limit or waiting past the connection timeout produces a clear error naming the last host contacted
  - Observable: a chain of up to three redirects resolves to a final destination without operator intervention
  - _Requirements: 3.1, 3.2, 3.3, 3.4_
  - _Boundary: xceadmin.discovery_
  - _Depends: 2.4_

- [x] 3.3 Push a cluster binary to the controller, with a dry-run mode
  - A real push reports the assigned configuration version, acknowledgement count, and any error, exiting non-zero on failure
  - A dry-run reports whether the controller would accept the config without ever committing it
  - Every connection failure, protocol-level rejection, corrupt input, or missing schema maps to a specific, distinct error
  - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 6.1, 6.2, 6.3, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6_
  - _Boundary: xceadmin.push_
  - _Depends: 2.4, 3.2_

- [x] 4. Integration: wire the CLI entry point and hook xceadmin into cluster deployment
- [x] 4.1 Expose config build and config push as a single command-line tool
  - `xceadmin config build` and `xceadmin config push` (with `--dry-run`) are both reachable from one installed command
  - Any failure surfaces as a specific, human-readable message with a non-zero exit code, not a raw traceback
  - Interrupting a push mid-flight leaves no dangling connection and reports a clean cancellation
  - _Requirements: 1.2, 1.3, 2.2, 2.4, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 8.1, 8.2_
  - _Boundary: xceadmin.cli_
  - _Depends: 3.1, 3.3_

- [x] 4.2 Build cluster configuration automatically during HA deployment
  - Deploying a cluster with the build-config option produces a fresh binary from the current YAML before startup
  - Deploying with no binary present and no flag still produces one automatically
  - Deploying with an existing binary and no flag leaves it untouched, matching prior behavior
  - _Requirements: 8.3, 8.4, 8.5_
  - _Boundary: deploy-ha.sh_
  - _Depends: 4.1_

- [x] 5. Validation: test coverage across build, push, and packaging
- [x] 5.1 (P) Unit test the field-mapping logic in isolation
  - Both mapper shapes (list sections and the scalar section) are verified against in-memory input with no file I/O
  - The one file that feeds two separate output fields is verified explicitly
  - _Requirements: 1.1, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 5.3, 9.1, 9.2, 9.3_
  - _Boundary: xcecommon.mapping_
  - _Depends: 2.3_

- [x] 5.2 (P) Unit test wire framing
  - Framing round-trips correctly against a fixed, known byte sequence and against a local loopback connection
  - _Requirements: 2.1_
  - _Boundary: xcecommon.wire_
  - _Depends: 2.4_

- [x] 5.3 (P) Verify bundled schemas resolve under every install mode
  - Both the wire-protocol schema and the validation schemas resolve to real, parseable files under the currently active install
  - A deliberately corrupted or absent packaged schema produces the expected installation-hint error rather than a crash
  - Meaningful confirmation of the pinned install path requires running this under the release build profile
  - _Requirements: 5.5, 7.4_
  - _Boundary: xcecommon.schema, xcecommon.validation_
  - _Depends: 2.1, 2.2, 1.3_

- [x] 5.4 (P) Golden-file test config build determinism
  - Building from a minimal fixture, a full fixture covering all seven file types, and a set of intentionally invalid fixtures each produce the expected result
  - Building the same fixture twice produces byte-identical output and an identical checksum
  - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 9.1, 9.2, 9.3_
  - _Boundary: xceadmin.build_
  - _Depends: 3.1_

- [x] 5.5 (P) Test every YAML validation failure mode
  - Each invalid-fixture case (schema violation, reserved name, duplicate name including a cross-type collision, undefined reference) produces the documented outcome — hard failure or warning, matching the requirement
  - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
  - _Boundary: xcecommon.validation_
  - _Depends: 2.1_

- [x] 5.6 (P) Test config push and dry-run against a mock controller
  - A mock TLS server standing in for the controller confirms the full push flow: successful push, redirect-following, and every documented error path
  - The same mock confirms dry-run reports both an accepted and a rejected outcome correctly
  - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 3.1, 3.2, 3.3, 3.4, 6.1, 6.2, 6.3, 7.1, 7.2, 7.3_
  - _Boundary: xceadmin.push, xceadmin.discovery_
  - _Depends: 3.3_

- [x] 6. Fix reply envelope decode and add live E2E push coverage
- [x] 6.1 Unwrap the `Request` reply envelope before decoding as `RequestReply`
  - Live-verified 2026-07-30 against a real deployed single-node controller: `config push` crashes every time with `capnp.lib.capnp.KjException: ... non-list pointer where text was expected`. Root cause: `xce_reply()` (`src/lib/request.c`) wraps every reply — including `config push`'s — in an outer `Request{opcode: OPCode_reply, body: union{replymsg @4: RequestReply}}` envelope (`src/schema/controller.capnp:645-651`) before writing it to the wire. Two call sites decode the raw wire bytes directly as a bare `RequestReply`, skipping the outer envelope, which misaligns every pointer offset: `xceadmin.discovery._redirect_target()` (crashes first, on every reply) and `xceadmin.push.push_config()`'s own `mod.RequestReply.from_bytes(reply_bytes)` (same latent bug, currently unreached only because discovery crashes first — fixing discovery alone is not sufficient). The controller side is not at fault: server logs confirm the same request's dispatch, commit, and activation all succeed for the exact bytes that then crash the client on decode.
  - Both decode sites unwrap `Request.body.replymsg` before treating the bytes as a `RequestReply` — introduce one shared decode helper (e.g. in `xceadmin.discovery`, which `xceadmin.push` already imports from) rather than duplicating the unwrap logic at each call site
  - `config push` against a real controller (mock or live) succeeds end to end: exits 0, prints `versionTag`/`ackedCount`, no `KjException`
  - Redirect-following (standby → primary) still decodes correctly through the fixed unwrap — existing task 5.6 mock-based redirect tests continue to pass unmodified after their mock reply fixtures are updated to the real envelope shape
  - _Requirements: 2.1, 2.5, 3.1, 3.2, 7.2, 7.3_
  - _Boundary: xceadmin.discovery, xceadmin.push_
  - _Depends: 3.2, 3.3_

- [x] 6.2 Add a live single-node E2E test for `config push`
  - New test deploys one real `xce_controller` (no HA peers needed — reachability/decode correctness is what's under test, not consensus), runs `xceadmin config build` against `src/configs/`, then a real `xceadmin config push` against it over the actual TLS+capnp wire protocol — no mock server
  - Test asserts exit 0, a parsed `versionTag`/`ackedCount` in the output, and confirms the controller's own log shows the config committed and activated
  - Skips cleanly (not a hard failure) when no controller binary is available in the current build profile, matching this suite's existing cert-dependent skip convention (see task 5.3's Implementation Notes)
  - _Requirements: 2.1, 2.5, 2.6, 3.1_
  - _Boundary: xceadmin (test-only, cross-cutting with the built controller binary)_
  - _Depends: 6.1_

- [x] 6.3 Add a live 3-node HA E2E test for `config push` (success path + redirect)
  - Task 6.2 proved a single-node deployment always returns `ackedCount == 0` and exits 1 even on full server-side success (no standbys exist to ACK) — the `acked_count > 0` exit-0 success path, and Requirement 3's redirect-following, have never been exercised against a real controller, only against mocks (task 5.6) or a topology structurally unable to reach them (task 6.2). This task closes both gaps in one live 3-node deployment.
  - Deploys a real 3-node HA `xce_controller` cluster (reuse `deploy-ha.sh`'s existing 3-node pattern, e.g. `localhost:19991/19992/19993`, staggered start for clean election per this repo's existing convention), waits for a primary to be elected
  - Pushing to the primary directly: real `xceadmin config push` exits 0, reports `ackedCount > 0` (a real positive count from real standby ACKs, not the single-node `0` case), and the controller logs confirm commit/activate on the primary and replication/ack on at least one standby
  - Pushing to a standby node: `xceadmin` follows the redirect to the primary (Requirement 3.1, 3.3), the push still succeeds end-to-end, and the redirect is visible either in `xceadmin`'s own stderr redirect message (`discovery.py`'s existing "redirected: host:port -> host:port" line) or equivalent captured evidence
  - Skips cleanly (not a hard failure) when no controller binary is available, matching task 6.2's convention; reuses task 6.2's cleanup discipline (kill all node processes, remove scratch deploy dir, on both pass and failure paths) — check whether task 6.2's fixture/helpers can be extracted and reused for the multi-node case rather than duplicated
  - _Requirements: 2.1, 2.5, 2.6, 3.1, 3.2, 3.3_
  - _Boundary: xceadmin (test-only, cross-cutting with the built controller binary)_
  - _Depends: 6.2_
  - _Closed: standby-redirect half is DONE and correct (round-1 reviewer independently re-verified: OP_upgradeConfig has no server-side redirect wiring today, matches design.md's own pre-existing "Out of Boundary" gap; test correctly locks in real current rejection behavior). Primary-push success-path test was flaky (~24/25 live fails in round-2, see bug-1485) — elected primary observed as a `<defunct>` zombie shortly after promotion. Per this note's own next step, `test_e2e_live_ha_push.py`'s DEVNULL stderr redirect was replaced with real per-node log capture (commit `a5560256`) and the test re-run 40 times standalone (25 idle-sandbox + 15 with a concurrent competing build): 40/40 PASSED, 0/120 captured stderr logs non-empty — no crash, no ASAN report, in any run. Could not reproduce the original failure rate under matching conditions; leading hypothesis is that the original ~24/25 rate was itself caused by OTHER concurrent sandbox sessions contending for resources during that investigation, not a deterministic controller bug reachable in isolation. Decision: **mergeable as-is** — bug-1485 stays open, downgraded to low-priority/environment-flake (absence of reproduction here is not proof of absence), with the now-permanent stderr-capture infrastructure ready to catch a real ASAN report or signal if it recurs in a genuinely contended sandbox or in CI._

## Implementation Notes

- **1.3**: nix's bare `python3` does not put `pip` on PATH — `python3Packages.pip` had to be added to `nativeBuildInputs` alongside the packages design.md's Technology Stack table lists; not a spec gap, just an operational dependency the table didn't need to spell out.
- **1.3**: Claude Code `PostToolUse` hooks receive tool input via stdin JSON (`tool_input.file_path`), not a `$FILE_PATH` env var — brief.md's ruff-hook sketch used the env-var form, which would have silently no-op'd. The actual hooks added to `.claude/settings.json` follow this repo's existing `.wolf/hooks/post-write.js` stdin-JSON convention instead. Any future hook work in this repo should follow the same pattern, not brief.md's original sketch.
- **1.3**: `src/xceadmin/src/xcecommon/schema_files/` (synced capnp + JSON schema copies) is a build artifact, regenerated every `xce-build.sh xceadmin` run — never hand-edit it, never commit it (`src/xceadmin/.gitignore` excludes it).
- **2.1**: `check_undefined_refs` deliberately does not parse resource names out of `res_req`/`gpu_req` free-form LSF-style expression strings (e.g. `"mem>1000 rusage[mem=1000]"`) — design.md's `xcecommon.validation` component scopes this check to queue→host-group and limit→group/`resname` only. `limits.yaml`'s `projects` field and `scaler.yaml` name collisions are likewise out of scope — no `projects.yaml` catalog exists among the 7 config types, and Requirement 4.3 only enumerates queue/host/resource/group/limit names. `src/configs/hosts.yaml` has 2 pre-existing undefined refs (resource `mg`, host `h3`) that now surface as warnings — expected, not a regression; fixing the example fixture is outside this task's boundary.
- **2.2**: The real `controller.capnp` names its reply struct `RequestReply`, not `Reply` as design.md's postcondition states — an inexact design pseudo-type, corrected empirically (verified directly against `src/schema/controller.capnp`, not assumed). pycapnp 2.0.0/capnproto 1.2.0 abort the whole process (uncaught C++ exception, not a Python exception) on a missing cross-file import or malformed `.capnp` content passed to `capnp.load()` — a bare `try/except` cannot catch this. `xcecommon.schema.load_schema()` works around it by pre-validating in an isolated subprocess before ever calling `capnp.load()` in the caller's own process, converting any abort into a clean `SchemaNotFoundError`. Any future code that calls `capnp.load()` directly (bypassing this loader) reintroduces the crash risk — always go through `xcecommon.schema.load_schema()`.
- **2.3**: `map_all_sections()` (a `SECTION_MAP`-driven orchestrator, not explicitly named in design.md's Service Interface but consistent with its Contracts) is the single call `xceadmin.build` (task 3.1) needs — it treats a missing YAML file identically to a present-but-empty section (byte-for-byte), satisfying Requirement 1.5 structurally. `target.init(field, 0)` and never touching an empty/absent list field produce *different* serialized bytes for the same logical "nothing here" value (verified empirically) — the mapper always picks "never touch" for byte-determinism (9.1). Calling `.to_bytes()` twice on the same capnp builder triggers a spurious "message already written" `UserWarning` even when no field was mutated between calls — capture the bytes once and reuse them rather than calling `.to_bytes()` again.
- **2.4**: `errors.py`'s exception subclasses (task 1.2) all take `message` as an explicit first constructor argument in addition to their structured fields (e.g. `ConnectionTimeoutError(message, host, port)`, `ReplyDecodeError(message, reason)`) — design.md's abbreviated `Service Interface` type stubs (`class ConnectionTimeoutError(XceAdminError): host: str; port: int`) read as if only the structured fields were needed; always check the real constructor in `errors.py`, not the design doc's shorthand. `xcecommon.wire` is deliberately opcode-agnostic — `send_request`/`send_frame`/`recv_frame` move raw `bytes` only and never import a capnp type; callers own `.to_bytes()`/`.from_bytes()`. Reused `ha/data/xce.crt`/`xce.key` (the repo's existing dev cert pair, same one `deploy-ha.sh` already copies for controller TLS) as the test fixture for both the TLS test server and the client `cert_dir` — no synthetic cert generation needed.
- **3.1**: Requirement 1.7's "unrecognized-but-schema-valid field" warning is real and fires on essentially every real build using `src/configs/resources.yaml` — `resources.yaml`'s `type` key is *required* by `resources.schema.json` (`required: [name, type]`) but has no `ClusterConfig` destination (per `xcecommon.mapping`'s documented decision not to guess the `valtype`/`ordtype`/`flags` integer encoding), so every valid resource entry triggers a "field 'type' ... ignored" warning unconditionally, not just for unusual input. This is a disclosed spec tradeoff, not a bug — flagged here so a future pass over `xceadmin.build`'s warning UX (e.g. only warning once per field-name per run instead of once per item) knows the noise is expected, not a regression. `xceadmin.build`'s unmapped-field table (`_UNMAPPED_BUT_VALID_ITEM_KEYS`) was derived by cross-referencing `xcecommon.mapping`'s per-item builder functions against the real `src/configs/schema/*.schema.json` property lists directly (not from prose alone) to avoid drift.
- **3.2**: design.md's Dependencies list for `xceadmin.discovery` names only `xcecommon.wire`, but decoding a reply to inspect `RequestReply.redirectHost`/`.redirectPort` genuinely requires `xcecommon.schema` too (verified real field names against `controller.capnp:594-595`, which matched design.md exactly this time — unlike task 2.2's `Reply`/`RequestReply` struct-name slip). Treated as a design-doc omission, not license to hand-roll capnp decoding. `follow_redirects`'s `cert_dir` parameter is accepted for signature parity with design.md but unused internally — all I/O, including TLS setup, is delegated to the caller's `send_fn` closure, which is expected to already carry its own `cert_dir`. A connection timeout to any hop (including a non-initial one reached via redirect) raises `NoPrimaryError` naming that specific hop, not the original target.
- **bug-1461 (out-of-band, between 3.2 and 3.3)**: `controller.capnp` had a real, confirmed schema bug — a locally-declared `struct Data` (an `OPCode_get` query-result tagged union) shadowed Cap'n Proto's builtin `Data` blob type for every later `: Data` field in the file, miscompiling `UpgradeConfigRequest.sha256`/`.bytes` (and 5 other fields across `ReplConfigReply`/`ReplConfigPushRequest`/`ReplEnvelope`/`SnapshotTransfer`/`WalCatchupEntry`/`CtrlHelloRequest`, several HA-critical) as a struct-pointer type instead of a byte blob — blocking `push_config()` from constructing a valid request via pycapnp. Fixed at the root (renamed the local struct to `XceData`, `$C.mapname("xce_data_t")` unchanged so the C typedef is stable) across 20 files spanning controller/lib/scheduler/Go — see commit `e53c086` and `.wolf/buglog.json` bug-1461 for the full investigation and fix. This was NOT an admin-config-tool-boundary change; it's a pre-existing, cross-component schema defect that happened to block this task. `push_config()`'s real controller counterpart was, separately, blocked by a dispatch-gate defect (`ha_dispatch_gate()`/`cluster.c`) — **RESOLVED 2026-07-25 (bug-1468, commit `959b0cda`)**, fixed by `config-upgrade-snapshot` task 8.6 hitting the same wall building a live `OP_upgradeConfig` sender; that path has since been exercised live and successfully many times via that spec's `--config-upgrade` soak. Nothing blocks a live `push_config()` E2E run anymore, though `xceadmin`'s own TLS/redirect/framing code has not yet had one specifically.
- **3.3**: `probe_config()` does NOT go through `xceadmin.discovery.follow_redirects` — `ProbeConfigTagReply` is a direct struct with only `configTag`, no `redirectHost`/`.Port` fields exist on it at all, so there is no redirect signal for `follow_redirects` (which unconditionally decodes replies as `RequestReply`) to read; only `push_config()`'s `Request`/`RequestReply`-wrapped exchange supports redirects. `ProbeConfigTagReply` also carries no accept/reject signal — `DryRunResult.accepted` is a client-side inference (true on any successful round-trip), matching the real C reference client (`config_repl.c`'s `xce_probe_config_tag`, which also always sends an empty `name` and returns only a bare tag). pycapnp union field assignment for Void/Text-typed variants (e.g. `reply.data.nodata = None`, `reply.data.debugmsg = "..."`) must use direct assignment, not `.init(name)` — `.init()` without a size is only valid for struct/list/data-typed variants, confirmed via `KjException: init() without a size is only valid for struct and object fields`.
- **4.1**: `xcecommon.wire.connect()` deliberately lets `ConnectionRefusedError`/other bare `OSError`s propagate unwrapped (documented in `wire.py`'s own module docstring) — `xceadmin.cli` is the layer that catches `OSError` and formats it per Requirement 7.1, alongside its `except XceAdminError` dispatch surface; a task 4.1 implementer must not assume `XceAdminError` alone covers every network failure mode. click 8.3.1's `CliRunner` no longer accepts `mix_stderr=`; `result.stdout`/`.stderr` are separated by default, but `CliRunner`'s output capture decodes as text and is **lossy for arbitrary binary** (confirmed: `bytes(range(256))` does not round-trip through the captured `.output` string) — do not assert exact byte content of a `-o -` binary write through `CliRunner`; that's already covered at the `write_output()` unit level (task 3.1's `test_config_build.py`). A CLI test that omits `-o` falls through to the real default (`cluster.bin`, relative to CWD) — always pass an explicit `-o <tmp_path>` in tests that reach the write step, or the test will write a stray artifact into the actual repo working directory (caught via `git status` before commit, not by the test itself).
- **4.2**: `deploy-ha.sh` resolves both `xceadmin` (`$SCRIPT_DIR/../xceadmin/.venv/bin/xceadmin`) and the YAML configs source (`$SCRIPT_DIR/../configs`) relative to the script's own location (`$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)`), not the caller's CWD — matches the project's other relative-path scripts and keeps the deploy working regardless of where it's invoked from. `--build-config` is parsed out of `"$@"` before positional-arg assignment so it can appear anywhere in the argument list, not just a fixed position. Verified live against the real `xceadmin` build and `src/configs/`: missing `cluster.bin` auto-builds (8.4), an existing one with no flag is left byte-for-byte untouched — same mtime, same SHA (8.5), and `--build-config` against an existing `cluster.bin` forces a rebuild that reproduces byte-identical content (8.3, confirming 9.1's determinism guarantee holds through this path too).
- **5.1, 5.2**: Already fully satisfied by task 2.3's `test_mapping.py` and task 2.4's `test_wire.py` respectively — same requirement lists, same acceptance bullets (in-memory-only mapper coverage + the explicit two-field case for 5.1; a fixed known byte sequence + real loopback round-trips for 5.2). No new tests written; would have been pure duplication.
- **5.3**: Found and fixed a real gap while writing `test_schema_bundling.py` — `xcecommon.validation.validate_yaml()`'s `json.loads(schema_path.read_text())` call (loading the *bundled JSON validation schema itself*, not the operator's YAML) had no error handling, so a corrupted bundled JSON schema crashed with a raw `json.JSONDecodeError` instead of a clean `ValidationError` — the same "crash vs. clean error" class of bug task 2.2 already fixed for `controller.capnp`, just on the JSON-schema side, and not caught by task 2.1 since that task's tests only exercised operator-YAML failure modes, never a corrupted *schema* file. Fixed by wrapping it and raising `ValidationError(..., constraint="schema-corrupted")`, matching the existing missing-schema handling's pattern one function above it. **Gotcha for any future task touching `xcecommon`/`xceadmin` source**: editing a source file does NOT propagate to a running `.venv` until reinstalled — always re-run `xce-build.sh xceadmin` after a source edit before trusting a bare `.venv/bin/python -m pytest` result; a stale venv (especially one left in `release`/pinned mode by a prior profile run) will silently test the *old* code and produce a false failure (or false pass) that looks like a real bug.
- **5.4**: `tests/fixtures/{minimal,full}/configs/*.yaml` + checked-in `expected.bin` golden files added to `test_config_build.py` (not a new file — design.md's File Structure Plan lists the SAME requirement set for `test_config_build.py` under both 3.1 and 5.4). `full/configs/*.yaml` is a byte-for-byte copy of `src/configs/*.yaml`; its golden SHA (`860cfd4f59ef...`) independently matches the tag `deploy-ha.sh` produced in task 4.2's live verification against the same source files — an unplanned but welcome additional confirmation of Requirement 9's cross-context determinism guarantee. Golden-file tests catch representation drift (bytes change even though decoded content is still semantically correct) that purely structural/decode-based assertions elsewhere in the suite cannot detect; regenerate `expected.bin` deliberately (via `build_cluster_config()`, sanity-checked by decoding it back) whenever a real, intentional field-mapping change lands — never hand-edit the binary.
- **5.5**: Already fully satisfied by task 2.1's `test_validation.py` — every sub-case (schema violation, reserved name, cross-type duplicate collision via `test_check_duplicates_allows_limit_name_to_collide_with_queue_name`, undefined reference) already has an explicit test with the exact same requirement list. No new tests written.
- **5.6**: `probe_config()`'s dry-run has no reachable `accepted=False` path — `ProbeConfigTagReply` genuinely carries no rejection signal on the wire (see task 3.3's note). 5.6's bullet asking for both an accepted *and* a rejected outcome cannot be satisfied without inventing undocumented client-side rejection logic; instead added `test_probe_config_has_no_wire_level_rejection_signal_by_design` (parametrized over several `configTag` values) that locks in and documents the limitation explicitly rather than leaving it silently uncovered. Added two new end-to-end propagation tests through `push_config`'s real composition with `xceadmin.discovery` (not just discovery.py's own unit tests): a real 4-mock-server redirect chain proving `TooManyRedirectsError` surfaces correctly, and a `NoPrimaryError`-on-timeout test that monkeypatches `xcecommon.wire.connect` to raise `ConnectionTimeoutError` immediately rather than waiting out a real 30s socket timeout.
- **Post-implementation, out-of-band (2026-07-22)**: reopened and reversed the 2026-07-21 "not wired into meson" Design Review correction — see `design.md`'s File Structure Plan Amendment for full rationale and verification. Summary: added `src/xceadmin/meson.build` (`custom_target()`, stamp-file output, `build_always_stale: true`) + `src/xceadmin/meson-build.sh` (out-of-tree install+test, `repo-mirror/` symlink staging so `src/configs/`, `src/schema/`, `ha/data/` relative-path test assumptions still resolve from a copied location); `src/meson.build` gained `subdir('xceadmin')`; `src/tools/xce-build.sh`'s `xceadmin)` case now delegates to `meson compile -C "$BUILD_DIR" xceadmin` instead of its own pip/venv logic. Verified both `meson compile` (165/165 passed against a real dev checkout with certs present) and `nix build --option sandbox false .#xcecore_debug` (128 passed / 37 skipped — cert-dependent tests correctly skip under nix's git-tracked-only hermetic source, since `ha/data/xce.crt`/`.key` are intentionally not committed). AppImage bundling (README 3.5) explicitly deferred, not part of this change. Net effect on Requirement 8: the editable-vs-pinned distinction (8.1 vs. 8.2) collapses — every `xce-build.sh xceadmin` run is now the pinned/meson-driven path, a strictly stronger guarantee than the old default.
- **Post-implementation, out-of-band (2026-07-22, second pass)**: `meson-build.sh`/`meson-install.sh` no longer resolve `python3` via a bare PATH lookup — `meson.build` now pins it once via `import('python').find_installation('python3')` (same mechanism as `find_program('go')`/capnp/packcc elsewhere in this build, visible at `meson setup` time as `Program python3 found: YES (/nix/store/...)`) and threads the resolved path into both scripts explicitly. Verified: `meson setup` prints the pinned interpreter, rebuild stays 165/165 green, and the built venv's own `python3` resolves through to that exact store path.
- **Post-implementation, out-of-band (2026-07-22, third pass — removes `ha/data/` entirely)**: `deploy-ha.sh` no longer takes a `<data-dir>` argument at all (`--build-config` flag also removed) — every invocation now generates a fresh self-signed TLS cert/key (`openssl req -x509 -newkey rsa:2048 -nodes -days 3650`, 10y validity) directly into the deploy dir's `certs/`, and builds `cluster.bin` directly into `controller/ctrl-1/conf/` via `xceadmin`, with no intermediate staging directory. `run-ha-soak.sh` (`/xce-test-ha`) updated to match: dropped its own `XCE_HA_DATADIR`-based precondition entirely, since `deploy-ha.sh` now handles both cert and config generation itself. Updated 4 callers (`run-ha-soak.sh`, `test-heartbeat-tolerance.sh`, `test-commit-durability.sh`, plus both `xce-deploy-ha`/`xce-test-ha` SKILL.md docs) to the new 3-arg signature. `ha/data/{xce.crt,cluster.bin}` removed from git tracking (`cluster.bin` was already gitignored from the prior pass; the certs were a real checked-in TLS private key, now gone from history going forward).

- **6.2**: A genuinely single-node (zero-peer) controller always computes `ackedCount == 0` for `config push` — no standbys exist to ACK — even when the config change commits and activates successfully server-side (`config_repl.c`'s `push_config_to_standbys`/`do_upgrade_config_with_bytes`). `xceadmin.cli`'s postcondition (`acked_count > 0` required for exit 0) can't distinguish "no standbys exist" from "push failed," so a real single-node `config push` always exits 1 with `"push rejected (ackedCount == 0)"` even on full success — reproduced live and independently re-confirmed by the reviewer against real `config_repl.c`/`repl.c` source. Verified spec-consistent (matches `cli.py`/design.md's own exit-0-iff-`acked_count>0` postcondition as written), not a regression from this task. `test_e2e_live_push.py` asserts this real exit-1/ackedCount==0 behavior directly rather than the task text's literal "exit 0," and independently proves genuine server-side success via the controller's own commit/activate log lines. Out of this task's boundary to fix; a future task should decide whether single-node pushes deserve a distinct success signal in `xceadmin.cli`/`push.py` (e.g. treating a fully-connected zero-peer topology as trivially quorate) rather than always reading as a CLI failure.
- **6.1**: `decode_reply()` (new, `discovery.py`) is the single unwrap point for the `Request{opcode: reply, body.replymsg: RequestReply}` envelope every real controller reply arrives in (`xce_reply()`, `src/lib/request.c:384-418`) — both `discovery._redirect_target()` and `push.push_config()` go through it; `probe_config()` is untouched (`ProbeConfigTagReply` is never envelope-wrapped). Fixing the two documented call sites alone was insufficient — task 5.6's own mock-server test fixtures (`test_discovery.py`, `test_config_push_mock.py`) AND `test_cli.py`'s two `config push` mock handlers all constructed bare, unwrapped `RequestReply` bytes, meaning the entire existing test suite had been silently validating the wrong wire shape since task 5.6/task 4.1. All fixed via one shared `wrap_reply` fixture in `conftest.py`. Any future test that hand-builds a controller reply for a mock server must wrap it the same way, or it will pass against a mock while still being broken against a real controller.

  Investigating this surfaced a real, independent finding: `xcecommon.wire.connect()` was loading a **client-side** TLS cert (`context.load_cert_chain(...)` on the `PROTOCOL_TLS_CLIENT` context) — verified against the actual reference implementation (`netio.c`'s `create_ssl_context()`) that the real client presents **no** certificate at all when `cert_file`/`key_file` are `NULL` (bare `TLS_client_method()`, no `SSL_CTX_use_certificate_chain_file`/`SSL_CTX_use_PrivateKey_file` call). `xceadmin`'s client diverged from the reference client it's meant to mirror. Fixed: `connect()` no longer loads a client cert or takes a `cert_dir` parameter at all. Since nothing else consumed `cert_dir` (no CA verification happens either — `verify_mode=CERT_NONE` throughout), removed it end-to-end: `xceadmin.discovery.follow_redirects()`'s `cert_dir` param (already dead/`noqa`'d — signature-parity leftover), `xceadmin.push.push_config()`/`probe_config()`, and the `--cert-dir` CLI flag on `config push`. Server-side cert loading (the mock TLS servers in `test_wire.py`/`test_cli.py`/`test_config_push_mock.py`) is unaffected and still needed — moved into a shared session-scoped `tls_certs` fixture in a new `tests/conftest.py` (openssl-generated, same approach as `deploy-ha.sh`), replacing 3 duplicated module-level `REPO_HA_DATA_DIR`/`pytest.mark.skipif(not ha/data/xce.crt...)` blocks. Net result: those 3 files' TLS-dependent tests no longer skip under a hermetic `nix build` sandbox (which never had `ha/data/` in its git-tracked-only source) — verified 165/165 passed, 0 skipped, both via `meson compile` and via a full `nix build --option sandbox false .#xcecore_debug` rebuild (previously 128 passed / 37 skipped there). Requirements 2.3/2.4 and 8.3/8.4/8.5 marked superseded in `requirements.md` with dated correction notes rather than silently rewritten.