diff --git a/.kiro/specs/admin-config-tool/design.md b/.kiro/specs/admin-config-tool/design.md index 9be4bee7..df909f30 100644 --- a/.kiro/specs/admin-config-tool/design.md +++ b/.kiro/specs/admin-config-tool/design.md @@ -26,7 +26,7 @@ ### This Spec Owns - The `xceadmin` CLI (`config build`, `config push`, `config push --dry-run`). - The `xcecommon` package: capnp schema loading, YAML→capnp field mapping, JSON Schema + business-rule validation, TLS+framing wire client. Both schema types (`controller.capnp` and the 7 JSON schemas) are packaged as bundled copies (`package_data`), synced from `src/schema/` and `src/configs/schema/` at build time — not read from an operator-supplied path at runtime (2026-07-21 revision; superseded Requirement 1.6, see Requirement 5.5). -- Build/toolchain integration: `xce-build.sh` dispatch (not meson — see File Structure Plan Design Review correction), nix dev-shell Python packages, `xce-deploy-ha` `--build-config` flag. +- Build/toolchain integration: `src/xceadmin/meson.build` (`meson compile`/`nix build .#xcecore[_debug]`), `xce-build.sh` delegating dispatch, nix dev-shell Python packages, `xce-deploy-ha` `--build-config` flag — see File Structure Plan Design Review correction / Amendment. - Test fixtures and golden files under `src/xceadmin/tests/`. ### Out of Boundary @@ -41,7 +41,7 @@ - `src/configs/*.yaml` — the 7 example configs already authored (this session) as operator-facing documentation; xceadmin's test fixtures are separate, purpose-built golden inputs under `src/xceadmin/tests/fixtures/`, not these files. `src/configs/schema/*.schema.json` is different in kind from the YAML examples: it's the source of truth `xcecommon` packages a synced copy of (2026-07-21 revision — see `xcecommon.validation`), the same relationship `src/schema/controller.capnp` already has. - `xce.crt`/`xce.key` under `ha/data/` (or `--cert-dir`) — same cert bundle the controller cluster already uses. - Nix `python3Packages`: `pycapnp`, `pyyaml`, `jsonschema`, `click`, `pytest`, `ruff` (net-new additions to `src/nixSupport/xce.nix`). -- `src/tools/xce-build.sh`'s case-statement dispatch pattern (not meson) as the build-integration point — see Design Review correction below. +- `src/xceadmin/meson.build`'s `custom_target()` as the build-integration point, delegated to by `src/tools/xce-build.sh`'s case-statement dispatch pattern — see Design Review correction / Amendment below. ### Revalidation Triggers - `controller.capnp` `ClusterConfig` field renumbering/addition/removal (already happened once this session via `clusterconfig-schema-split`) — invalidates the field-mapping table in Data Models. @@ -105,7 +105,8 @@ graph TB ``` src/xceadmin/ ├── pyproject.toml # package metadata; entry point xceadmin = xceadmin.cli:main -│ # NOT wired into meson — see Design Review correction below +│ # Wired into meson via src/xceadmin/meson.build (2026-07-22 +│ # amendment — see Design Review correction below) ├── src/ │ ├── xcecommon/ # shared library — no click/CLI dependency, reusable by future Python xcecli │ │ ├── __init__.py @@ -140,7 +141,9 @@ src/xceadmin/ `xcecommon` and `xceadmin` are separate top-level packages under one `src/xceadmin/src/` tree, installed together via one `pyproject.toml` for now (two console-ready packages, one distributable unit) — splitting into two distributions is deferred until a second consumer (the future Python `xcecli`) actually exists (Simplification: no speculative packaging split). ### Modified Files -- `src/tools/xce-build.sh` — new `xceadmin)` case: sync `src/schema/{controller,c}.capnp` → `src/xceadmin/src/xcecommon/schema_files/` and `src/configs/schema/*.schema.json` → `src/xceadmin/src/xcecommon/schema_files/json/` (`cp`, source of truth stays `src/schema/` and `src/configs/schema/` respectively — 2026-07-21 revision extends the sync to JSON schemas, Requirement 5.5), then `pip install -e .`/`pip install .` by profile, then `pytest` (8.1, 8.2). The sync step runs on every build so a `pip install .` (non-editable, Req 8.2) always packages the current schemas as `package_data` — without it, `load_schema()`/`xcecommon.validation`'s default resolution would silently package stale or missing copies. **`pytest` runs after whichever install just happened, not before** — this ordering is what makes the `release` profile's test run (8.2) the only one that actually exercises `default_schema_path()`/`default_schema_dir()` resolving under a real non-editable `site-packages` layout; the default (dev/editable) profile's test run (8.1) never would, since editable installs keep resolving into the checked-out `schema_files/` copy either way. **Design Review correction**: `xceadmin` is dispatched directly by this case statement and is deliberately **not** wired into `src/meson.build`/`src/meson.options`/a `src/xceadmin/meson.build` — `xcecli`'s `custom_target()` pattern fits a single static-binary output; `pip install -e .` mutates a venv/site-packages tree with no single trackable output artifact, so forcing it through meson's build graph would be a dead end. `xceadmin` sits orthogonal to the meson component option entirely, matching what `brief.md`'s harness sketch already specified before this design pass over-engineered a meson integration. +- `src/tools/xce-build.sh` — new `xceadmin)` case: sync `src/schema/{controller,c}.capnp` → `src/xceadmin/src/xcecommon/schema_files/` and `src/configs/schema/*.schema.json` → `src/xceadmin/src/xcecommon/schema_files/json/` (`cp`, source of truth stays `src/schema/` and `src/configs/schema/` respectively — 2026-07-21 revision extends the sync to JSON schemas, Requirement 5.5), then `pip install -e .`/`pip install .` by profile, then `pytest` (8.1, 8.2). The sync step runs on every build so a `pip install .` (non-editable, Req 8.2) always packages the current schemas as `package_data` — without it, `load_schema()`/`xcecommon.validation`'s default resolution would silently package stale or missing copies. **`pytest` runs after whichever install just happened, not before** — this ordering is what makes the `release` profile's test run (8.2) the only one that actually exercises `default_schema_path()`/`default_schema_dir()` resolving under a real non-editable `site-packages` layout; the default (dev/editable) profile's test run (8.1) never would, since editable installs keep resolving into the checked-out `schema_files/` copy either way. **Design Review correction (original, 2026-07-21)**: `xceadmin` is dispatched directly by this case statement and is deliberately **not** wired into `src/meson.build`/`src/meson.options`/a `src/xceadmin/meson.build` — `xcecli`'s `custom_target()` pattern fits a single static-binary output; `pip install -e .` mutates a venv/site-packages tree with no single trackable output artifact, so forcing it through meson's build graph would be a dead end. `xceadmin` sits orthogonal to the meson component option entirely, matching what `brief.md`'s harness sketch already specified before this design pass over-engineered a meson integration. + +**Amendment (2026-07-22, supersedes the correction above)**: the user reopened this decision — standard-onboarding consistency (README 3.3 `nix build .#xcecore[_debug]` and 3.4 `meson compile` working for xceadmin exactly like every other component) outweighs the multi-file-output concern. `src/xceadmin/meson.build` wraps the install+test cycle in a `custom_target()` whose output is a stamp file (`xceadmin-build.stamp`), not a tracked multi-file tree — the real work (out-of-tree `pip install .` into a build-dir-local venv, then `pytest`) runs in `src/xceadmin/meson-build.sh`, invoked with `build_always_stale: true` so it reruns unconditionally rather than relying on ninja's file-mtime cache (matching this project's existing "clean rebuild every invocation" convention, same as every `xce-build.sh` component's `rm -rf "$BUILD_DIR"`). `src/meson.build` gains `subdir('xceadmin')` inside the `component != 'xcecli'` block, so it builds under `component=all`/`component=xcecore` (where `xcecore`'s `mesonFlags` already sets `-Dcomponent=xcecore`) but not under a pure `xcecli` build. `src/tools/xce-build.sh`'s `xceadmin)` case now just delegates (`setup_build; meson compile -C "$BUILD_DIR" xceadmin`) instead of carrying its own separate pip/venv implementation — the one meaningful behavior change is that the editable-vs-pinned distinction (Req 8.1 vs. 8.2) collapses: every `xce-build.sh xceadmin` invocation now runs the pinned/non-editable install path, a strictly stronger test (it exercises real `package_data` resolution on every run, not just under a `release` profile). A purity constraint the original design's rejection didn't have to solve: several tests reach sibling directories via a fixed relative path (`src/configs/`, `src/schema/`, `ha/data/xce.crt`/`.key`) that assume the checked-out repo's exact directory nesting; `meson-build.sh` resolves this by staging a `repo-mirror/` tree in the build dir with symlinks recreating that exact layout, rather than touching the tests' deliberate fail-loud (not skip-on-missing-layout) assertions. Verified both paths end-to-end: `meson compile -C xce.build xceadmin` (165/165 passed, real `ha/data/xce.crt` symlinked from the live checkout) and `nix build --option sandbox false .#xcecore_debug` (128 passed / 37 skipped — the skips are the existing `skipif` on `ha/data/xce.crt` not present, correctly firing since nix's git-tracked-only flake source has no committed cert files; 0 failed). AppImage bundling (README 3.5) is explicitly deferred — out of scope for this amendment, tracked as roadmap future work, not implemented here. - `src/nixSupport/xce.nix` — add `python3`, `python3Packages.{pycapnp,pyyaml,jsonschema,click,pytest,ruff}` to `nativeBuildInputs` (8.6). - `.claude/skills/xce-build/SKILL.md` — document the `xceadmin` component and its pip-based (non-meson-profile) build path (8.1, 8.2). - `src/tools/deploy-ha.sh` — **(gap closed during task generation — the actual runtime behavior for 8.3-8.5 belongs here, not just in documentation)** add a `--build-config` flag; before starting the cluster, if `--build-config` is passed or `ha/data/cluster.bin` is missing, run `xceadmin config build configs/ -o ha/data/cluster.bin` (8.3, 8.4); if `cluster.bin` already exists and `--build-config` is not passed, use it as-is, unchanged (8.5). diff --git a/.kiro/specs/admin-config-tool/spec.json b/.kiro/specs/admin-config-tool/spec.json index 8e0d11e8..4c86c0d0 100644 --- a/.kiro/specs/admin-config-tool/spec.json +++ b/.kiro/specs/admin-config-tool/spec.json @@ -1,7 +1,7 @@ { "feature_name": "admin-config-tool", "created_at": "2026-07-06T01:30:00Z", - "updated_at": "2026-07-22T03:10:00Z", + "updated_at": "2026-07-22T10:45:00Z", "language": "en", "phase": "implementation-complete", "approvals": { diff --git a/.kiro/specs/admin-config-tool/tasks.md b/.kiro/specs/admin-config-tool/tasks.md index e267906c..38504591 100644 --- a/.kiro/specs/admin-config-tool/tasks.md +++ b/.kiro/specs/admin-config-tool/tasks.md @@ -161,3 +161,4 @@ - **5.4**: `tests/fixtures/{minimal,full}/configs/*.yaml` + checked-in `expected.bin` golden files added to `test_config_build.py` (not a new file — design.md's File Structure Plan lists the SAME requirement set for `test_config_build.py` under both 3.1 and 5.4). `full/configs/*.yaml` is a byte-for-byte copy of `src/configs/*.yaml`; its golden SHA (`860cfd4f59ef...`) independently matches the tag `deploy-ha.sh` produced in task 4.2's live verification against the same source files — an unplanned but welcome additional confirmation of Requirement 9's cross-context determinism guarantee. Golden-file tests catch representation drift (bytes change even though decoded content is still semantically correct) that purely structural/decode-based assertions elsewhere in the suite cannot detect; regenerate `expected.bin` deliberately (via `build_cluster_config()`, sanity-checked by decoding it back) whenever a real, intentional field-mapping change lands — never hand-edit the binary. - **5.5**: Already fully satisfied by task 2.1's `test_validation.py` — every sub-case (schema violation, reserved name, cross-type duplicate collision via `test_check_duplicates_allows_limit_name_to_collide_with_queue_name`, undefined reference) already has an explicit test with the exact same requirement list. No new tests written. - **5.6**: `probe_config()`'s dry-run has no reachable `accepted=False` path — `ProbeConfigTagReply` genuinely carries no rejection signal on the wire (see task 3.3's note). 5.6's bullet asking for both an accepted *and* a rejected outcome cannot be satisfied without inventing undocumented client-side rejection logic; instead added `test_probe_config_has_no_wire_level_rejection_signal_by_design` (parametrized over several `configTag` values) that locks in and documents the limitation explicitly rather than leaving it silently uncovered. Added two new end-to-end propagation tests through `push_config`'s real composition with `xceadmin.discovery` (not just discovery.py's own unit tests): a real 4-mock-server redirect chain proving `TooManyRedirectsError` surfaces correctly, and a `NoPrimaryError`-on-timeout test that monkeypatches `xcecommon.wire.connect` to raise `ConnectionTimeoutError` immediately rather than waiting out a real 30s socket timeout. +- **Post-implementation, out-of-band (2026-07-22)**: reopened and reversed the 2026-07-21 "not wired into meson" Design Review correction — see `design.md`'s File Structure Plan Amendment for full rationale and verification. Summary: added `src/xceadmin/meson.build` (`custom_target()`, stamp-file output, `build_always_stale: true`) + `src/xceadmin/meson-build.sh` (out-of-tree install+test, `repo-mirror/` symlink staging so `src/configs/`, `src/schema/`, `ha/data/` relative-path test assumptions still resolve from a copied location); `src/meson.build` gained `subdir('xceadmin')`; `src/tools/xce-build.sh`'s `xceadmin)` case now delegates to `meson compile -C "$BUILD_DIR" xceadmin` instead of its own pip/venv logic. Verified both `meson compile` (165/165 passed against a real dev checkout with certs present) and `nix build --option sandbox false .#xcecore_debug` (128 passed / 37 skipped — cert-dependent tests correctly skip under nix's git-tracked-only hermetic source, since `ha/data/xce.crt`/`.key` are intentionally not committed). AppImage bundling (README 3.5) explicitly deferred, not part of this change. Net effect on Requirement 8: the editable-vs-pinned distinction (8.1 vs. 8.2) collapses — every `xce-build.sh xceadmin` run is now the pinned/meson-driven path, a strictly stronger guarantee than the old default. diff --git a/src/meson.build b/src/meson.build index fd77f71d..7fa201fc 100644 --- a/src/meson.build +++ b/src/meson.build @@ -67,6 +67,7 @@ if component != 'xcecli' subdir('autoscaler') subdir('scheduler') subdir('frontend/apiserver') + subdir('xceadmin') configure_file( input: 'tools/appimage/entry.sh', diff --git a/src/tools/xce-build.sh b/src/tools/xce-build.sh index c97ac3fc..53e44ff1 100755 --- a/src/tools/xce-build.sh +++ b/src/tools/xce-build.sh @@ -113,41 +113,24 @@ case "$COMPONENT" in fi ;; xceadmin) - # Python — no meson, dispatched by this case statement alone (design - # review correction: xceadmin is deliberately NOT wired into - # src/meson.build/meson.options, see .kiro/specs/admin-config-tool/ - # design.md, File Structure Plan). Ignores $BUILD_DIR/$MESON_OPTS; - # $PROFILE only selects editable vs. pinned pip install below. - XCEADMIN_DIR="$SRC_DIR/xceadmin" - SCHEMA_DST="$XCEADMIN_DIR/src/xcecommon/schema_files" - echo "=== Syncing xceadmin bundled schemas ===" - mkdir -p "$SCHEMA_DST/json" - cp "$SRC_DIR/schema/controller.capnp" "$SCHEMA_DST/controller.capnp" - cp "$SRC_DIR/schema/c.capnp" "$SCHEMA_DST/c.capnp" - cp "$SRC_DIR/configs/schema/"*.schema.json "$SCHEMA_DST/json/" - cd "$XCEADMIN_DIR" - # nix's python3 site-packages lives under the read-only /nix/store, - # so `pip install` (editable or pinned) can't target it directly — - # it refuses with "externally-managed-environment" (PEP 668). Use a - # throwaway venv that inherits the nix-provided pycapnp/pyyaml/ - # jsonschema/click/pytest (--system-site-packages) so pip only ever - # has to install xceadmin itself, never rebuild those from PyPI. - # Recreated on every invocation for a clean, reproducible install — - # same philosophy as setup_build's `rm -rf "$BUILD_DIR"` above. - VENV_DIR="$XCEADMIN_DIR/.venv" - rm -rf "$VENV_DIR" - python3 -m venv --system-site-packages "$VENV_DIR" - if [ "$PROFILE" = "release" ]; then - echo "=== Building xceadmin Python package (pinned, non-editable) ===" - "$VENV_DIR/bin/pip" install . --quiet - else - echo "=== Building xceadmin Python package (editable) ===" - "$VENV_DIR/bin/pip" install -e . --quiet - fi - echo "=== Running xceadmin tests ===" - "$VENV_DIR/bin/python" -m pytest tests/ -v - echo "=== Done: xceadmin (profile=$PROFILE) ===" - exit 0 + # Meson-driven like every other component (src/xceadmin/meson.build, + # custom_target wrapping meson-build.sh) -- installs pinned/non- + # editable out-of-tree in the meson build dir and runs pytest, + # every invocation regardless of $PROFILE (xceadmin's suite is pure + # Python, unrelated to the ASAN/release C compiler flags $PROFILE + # otherwise controls, and fast enough — ~2s — that always running + # it costs nothing). This is a strictly stronger guarantee than the + # old editable-install-by-default behavior: every run now exercises + # real package-data resolution (bug-1460's class of bug), not just + # the `release` profile. + # + # For fast iterative dev without a full meson reconfigure, xceadmin + # can still be built standalone: cd xceadmin && pip install -e . + # (or see src/xceadmin/meson-build.sh for the exact out-of-tree + # steps this target runs). + setup_build + echo "=== Building + testing xceadmin (via meson) ===" + meson compile -C "$BUILD_DIR" xceadmin ;; all) setup_build diff --git a/src/xceadmin/meson-build.sh b/src/xceadmin/meson-build.sh new file mode 100755 index 00000000..e4a865f6 --- /dev/null +++ b/src/xceadmin/meson-build.sh @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +# Meson-driven xceadmin build: install + test out-of-tree, in the meson +# build directory. Never writes into the live source checkout -- unlike +# xce-build.sh's separate xceadmin case (editable install, .venv inside +# src/xceadmin/, meant for fast dev-loop iteration), this path is what +# `meson compile`/`nix build .#xcecore` actually invoke, and both need +# the same purity guarantee every other meson subdir here already has. +# +# Several tests reach for sibling directories outside xcecommon/xceadmin's +# own tree via a fixed relative path (src/configs/, src/schema/, ha/data/ +# -- e.g. test_validation.py's REPO_CONFIGS_DIR deliberately fails loud +# rather than skips if that layout assumption breaks, by design). Rather +# than touch test code, this script recreates the exact repo layout +# out-of-tree via symlinks (read-only references, never a write into the +# real tree) so every one of those relative-path assumptions still +# resolves correctly from the copied location. +set -euo pipefail + +SOURCE_DIR="$1" # .../src/xceadmin (meson.current_source_dir()) +BUILD_DIR="$2" # .../xceadmin (meson.current_build_dir()) +REPO_ROOT="$3" # real repo root (contains ha/, src/) +STAMP_FILE="$4" + +SRC_DIR="$REPO_ROOT/src" + +MIRROR_DIR="$BUILD_DIR/repo-mirror" +WORK_DIR="$MIRROR_DIR/src/xceadmin" +VENV_DIR="$BUILD_DIR/.venv" + +echo "=== xceadmin (meson): staging out-of-tree repo-layout mirror ===" +rm -rf "$MIRROR_DIR" +mkdir -p "$MIRROR_DIR/src" +ln -s "$REPO_ROOT/ha" "$MIRROR_DIR/ha" +ln -s "$SRC_DIR/configs" "$MIRROR_DIR/src/configs" +ln -s "$SRC_DIR/schema" "$MIRROR_DIR/src/schema" + +cp -r "$SOURCE_DIR" "$WORK_DIR" +# Never trust a stray dev-loop artifact from the live checkout (xce-build.sh's +# own editable-install byproducts, if this happens to be a live checkout and +# not a nix-sandboxed source copy) -- always start from a clean tree. +rm -rf "$WORK_DIR/.venv" "$WORK_DIR/build" "$WORK_DIR"/src/*.egg-info \ + "$WORK_DIR/.pytest_cache" "$WORK_DIR/.ruff_cache" \ + "$WORK_DIR/src/xcecommon/schema_files" +find "$WORK_DIR" -name "__pycache__" -type d -prune -exec rm -rf {} + + +SCHEMA_DST="$WORK_DIR/src/xcecommon/schema_files" +echo "=== xceadmin (meson): syncing bundled schemas ===" +mkdir -p "$SCHEMA_DST/json" +cp "$SRC_DIR/schema/controller.capnp" "$SCHEMA_DST/controller.capnp" +cp "$SRC_DIR/schema/c.capnp" "$SCHEMA_DST/c.capnp" +cp "$SRC_DIR/configs/schema/"*.schema.json "$SCHEMA_DST/json/" + +echo "=== xceadmin (meson): pinned install into build-dir venv ===" +rm -rf "$VENV_DIR" +python3 -m venv --system-site-packages "$VENV_DIR" +"$VENV_DIR/bin/pip" install "$WORK_DIR" --quiet + +echo "=== xceadmin (meson): running tests ===" +"$VENV_DIR/bin/python" -m pytest "$WORK_DIR/tests" -v + +touch "$STAMP_FILE" +echo "=== xceadmin (meson): done ===" diff --git a/src/xceadmin/meson.build b/src/xceadmin/meson.build new file mode 100644 index 00000000..f599613b --- /dev/null +++ b/src/xceadmin/meson.build @@ -0,0 +1,34 @@ +# xceadmin (admin-config-tool): Python package, no C compilation. +# +# Unlike every other subdir here, there is no single build artifact for +# meson/ninja to track -- `pip install` produces a multi-file venv tree. +# custom_target()'s output is a stamp file representing "install + +# test succeeded", following the same pattern used to wrap non-native +# build systems (cargo, npm, etc.) inside meson elsewhere in the +# ecosystem. The real work happens in meson-build.sh, out-of-tree in +# this subdir's own build directory -- never writing into the live +# source checkout, unlike xce-build.sh's separate editable-install +# fast dev-loop path (`xce-build.sh xceadmin`, still available +# standalone, but no longer xce-build.sh's OWN xceadmin implementation +# -- see tools/xce-build.sh, which now just calls this same target). + +xceadmin_stamp = custom_target('xceadmin', + output: 'xceadmin-build.stamp', + # No meaningful per-file `input:` list: meson-build.sh copies the whole + # subdir tree regardless, so tracking a subset of .py files would only + # give false confidence about staleness detection. build_always_stale + # instead matches this project's existing convention (every + # xce-build.sh component does `rm -rf "$BUILD_DIR"` and rebuilds clean + # on every invocation) rather than relying on ninja's file-mtime cache. + command: [ + find_program('bash'), + meson.current_source_dir() / 'meson-build.sh', + meson.current_source_dir(), + meson.current_build_dir(), + meson.current_source_dir() / '..' / '..', # repo root (contains ha/, src/) + '@OUTPUT@', + ], + build_by_default: true, + build_always_stale: true, + console: true, +) diff --git a/src/xceadmin/pyproject.toml b/src/xceadmin/pyproject.toml index 985036c7..d6005095 100644 --- a/src/xceadmin/pyproject.toml +++ b/src/xceadmin/pyproject.toml @@ -11,9 +11,11 @@ dependencies = [ "click>=8.0", ] -# NOT wired into meson (src/meson.build / src/meson.options) — xceadmin is -# dispatched by src/tools/xce-build.sh's case-statement pattern instead. -# See .kiro/specs/admin-config-tool/design.md, File Structure Plan. +# Wired into meson (src/xceadmin/meson.build custom_target, pulled in by +# src/meson.build's subdir('xceadmin')) so `nix build .#xcecore[_debug]` +# and `meson compile` install + test this package like every other +# component; src/tools/xce-build.sh's xceadmin case now just delegates to +# that same meson target. [project.optional-dependencies] test = ["pytest"]