Cross-reference warnings ("references undefined entry") were genuine
gaps in the example configs, not validator bugs: src/configs/hosts.yaml
had a typo'd resource ref (mg -> mem) and referenced a nonexistent
host3/hostgroup pair; queues.yaml referenced hgroup1/hgroup2/ugroup1
instead of the real hg1/ug1 names; resources.yaml was missing the
r1m/r15s/tmp built-in worker load-index resources
(worker/src/resource_sync.c's init_builtin_load_indices()). Added the
missing host3 host, hg2 hostgroup, ug2 usergroup, and three resource
entries; fixed the typo'd refs.
"schema-valid but has no ClusterConfig destination" warnings for
resources.yaml's type/order/consumable/release fields were a real gap
in xcecommon/mapping.py's _build_resource, not an inherent limitation
-- the encoding (XCE_VAL_*/xce_order_t/XCE_RESF_CONSUMABLE/
XCE_RESF_RELEASE) is documented in include/xce/types.h and actively
read by the scheduler; the old docstring's claim that it was
undocumented was wrong. Only resourcemap remains genuinely unmapped
(its capnp destination fields are commented out).
Updated test_mapping.py/test_config_build.py for the new mapping
behavior and regenerated the full golden fixture's expected.bin
(legitimate representation-drift from the mapping fix, not a bug).
deploy-ha.sh no longer takes a <data-dir> argument or --build-config
flag. Every deploy now generates a fresh self-signed TLS cert/key
(openssl, RSA 2048, 10y validity) directly into the deploy dir's
certs/, and builds cluster.bin directly into controller/ctrl-1/conf/
via xceadmin -- no intermediate staging directory, nothing preserved
across runs. run-ha-soak.sh (/xce-test-ha) updated to match: dropped
its own XCE_HA_DATADIR-based precondition entirely. Updated 4 callers
(run-ha-soak.sh, test-heartbeat-tolerance.sh, test-commit-durability.sh,
both xce-deploy-ha/xce-test-ha SKILL.md docs).
Separately: xcecommon.wire.connect() was loading a client-side TLS
cert, diverging from the real reference client -- netio.c's
create_ssl_context() presents no certificate at all when acting as a
client (cert_file/key_file both NULL, bare TLS_client_method()).
Fixed connect() to match, and since nothing else consumed cert_dir
(verify_mode=CERT_NONE throughout, no CA check either), removed it
end-to-end: discovery.follow_redirects(), push.push_config()/
probe_config(), and the --cert-dir CLI flag.
Server-side cert loading (mock TLS test servers) is unaffected --
moved into a shared session-scoped tls_certs fixture (new
tests/conftest.py, openssl-generated) replacing 3 duplicated
REPO_HA_DATA_DIR/skipif blocks. Net result: those tests no longer skip
under a hermetic nix build sandbox (which never had ha/data/) --
verified 165/165 passed, 0 skipped, via both meson compile and a full
nix build --option sandbox false .#xcecore_debug rebuild (previously
128 passed / 37 skipped there).
ha/data/{xce.crt,xce.key} removed from git tracking. Requirements
2.3/2.4 and 8.3/8.4/8.5 marked superseded in requirements.md with
dated correction notes.
Both meson-build.sh (venv creation) and meson-install.sh (wrapper
shebang) resolved python3 independently at run time (bare `python3 -m
venv` / `command -v python3`) -- correct as long as invoked inside
nix develop/build, but nothing prevented a stray non-nix system
python3 earlier on PATH from shadowing it.
meson.build now resolves python3 once via import('python').
find_installation('python3') -- same mechanism as find_program('go')/
capnp/packcc elsewhere in this build, printed at `meson setup` time
as "Program python3 found: YES (/nix/store/...)" -- and threads that
exact resolved path into both scripts as an explicit argument instead
of a fresh lookup.
Verified: meson setup prints the pinned python3, xceadmin target
rebuilds clean (165/165 passed), and the built venv's own python3
resolves through to that exact nix store path.
xceadmin's custom_target only built+tested (stamp-file output, no
install:true) — nix build .#xcecore succeeded but shipped nothing in
result/bin, unlike every other component.
Add meson.add_install_script('meson-install.sh'): copies the
xceadmin/xcecommon package dirs (not the whole build-dir venv, whose
pip-generated shebang hardcodes a build path that stops existing after
the build) into $out/lib/xceadmin-site, then writes a wrapper at
$out/bin/xceadmin whose shebang and PYTHONPATH are resolved against
the install script's own nix build environment — real /nix/store
paths for python3 and each dependency (pycapnp, pyyaml, jsonschema,
click), so nix's reference scanner keeps them alive in the closure.
No venv activation needed at runtime.
Verified: meson install --destdir and nix build --option sandbox
false .#xcecore both produce a working result/bin/xceadmin, runnable
standalone with a stripped-down PATH-only environment.
Reopens the 2026-07-21 design decision to keep xceadmin outside meson.
Standard onboarding consistency (README 3.3 `nix build .#xcecore` and
3.4 `meson compile` working for xceadmin like every other component)
outweighs the earlier concern about meson's single-artifact model not
fitting a multi-file pip install.
- src/xceadmin/meson.build: custom_target() wrapping install+test,
stamp-file output, build_always_stale (matches this repo's existing
clean-rebuild-every-invocation convention).
- src/xceadmin/meson-build.sh: out-of-tree pinned install + pytest;
stages a repo-mirror/ symlink tree so tests' fixed-relative-path
assumptions (src/configs/, src/schema/, ha/data/) still resolve from
a copied location, without touching their deliberate fail-loud
behavior.
- src/meson.build: subdir('xceadmin') under component != 'xcecli'.
- src/tools/xce-build.sh: xceadmin case now delegates to
`meson compile -C "$BUILD_DIR" xceadmin` instead of its own
pip/venv implementation, like every other component.
Verified both build paths: meson compile (165/165 passed) and
nix build --option sandbox false .#xcecore_debug (128 passed / 37
skipped — cert-dependent tests correctly skip under nix's
git-tracked-only hermetic source). AppImage bundling (README 3.5)
is explicitly deferred to future work, not part of this change.
Adds end-to-end propagation tests through push_config's real
composition with xceadmin.discovery (not just discovery.py's own
unit tests): a 4-mock-server redirect chain proving TooManyRedirectsError
surfaces correctly, and a monkeypatched-timeout test proving
NoPrimaryError does too, without waiting out a real 30s socket
timeout. Also locks in and documents a real wire-contract limitation:
ProbeConfigTagReply carries no rejection signal, so dry-run's
accepted=False path is unreachable today by design, not an oversight.
This completes all 18 sub-tasks in tasks.md.
tests/fixtures/{minimal,full}/configs/*.yaml + checked-in expected.bin
golden files, added to test_config_build.py. Catches representation
drift (bytes changing even when decoded content stays correct) that
the existing structural/decode-based assertions can't detect.
5.1 (mapping) and 5.2 (wire framing) are already fully satisfied by
task 2.3's test_mapping.py and task 2.4's test_wire.py -- no new
tests, would have been pure duplication.
5.3: test_schema_bundling.py cross-checks both bundling mechanisms
(capnp schema, JSON validation schemas) resolve consistently under
the active install, and found a real gap: a corrupted bundled JSON
schema crashed validate_yaml() with a raw json.JSONDecodeError
instead of a clean ValidationError. Fixed in xcecommon/validation.py.
Wire config build/config push --dry-run onto the click group.
Thin layer only: one XceAdminError dispatch surface plus an OSError
catch (ConnectionRefusedError and friends propagate unwrapped from
xcecommon.wire by design and need formatting here), and a
KeyboardInterrupt handler for a clean "cancelled" message (socket
cleanup already happens via push.py's own try/finally).
push_config()/probe_config(): build/send Request{opcode:upgradeConfig}
and the direct-struct ProbeConfigTagRequest respectively, following
redirects to the primary for the former (ProbeConfigTagReply has no
redirect fields, so probe_config connects directly). Maps controller
status/reply-variant failures to typed results/errors. Tested against
a real local TLS mock controller built from the bundled schema, since
neither opcode has a live controller-side handler today (separately
documented, unrelated to this task).
Unblocked mid-task by bug-1461 (commit e53c086): controller.capnp had
a struct-name collision that miscompiled UpgradeConfigRequest.sha256/
.bytes, making this task's core request impossible to construct via
pycapnp until fixed at the schema root.
Generic, opcode-agnostic redirect-following against RequestReply's
redirectHost/redirectPort. Follows up to 3 hops, displaying each
redirect target; a 4th redirect raises TooManyRedirectsError. A
connection timeout to any hop is interpreted as "no primary
available" and raised as NoPrimaryError naming that hop.
config build pipeline: scan the 7 known YAML filenames, validate
each present file, run cross-file business-rule checks, map to
ClusterConfig, serialize. Missing files become empty sections;
schema-valid-but-unmapped fields and undefined references warn to
stderr instead of failing. write_output() handles -o <path>/-o -
and returns the hex SHA-256 digest (printing it is the CLI's job).
TLS connection + 8-byte native-order length-prefixed framing to the
controller, verified byte-for-byte against request.c's own framing
and myapp.c's client TLS posture (SSL_VERIFY_NONE, no CN check).
Opcode-agnostic: moves raw bytes only, so the same send_request()
serves both OP_upgradeConfig and ProbeConfigTag's direct-struct
variant. Connection always closes on error, including handshake
timeout.
Map validated YAML section dicts into ClusterConfig capnp field
values via a declarative SECTION_MAP driving two generic mapper
shapes (list/scalar). Preserves YAML source order, sources no
timestamp/hostname/username, and is proven byte-deterministic
against both synthetic and real src/configs/*.yaml fixtures.
Load controller.capnp once per process via a cached loader that
resolves identically under editable and pinned installs. Missing or
corrupted schema content converts a pycapnp uncaught-C++-abort into a
clean SchemaNotFoundError via isolated subprocess pre-validation.
Validate operator-authored YAML against the 7 bundled JSON schemas,
enforce reserved-name and duplicate-name rejection, and flag undefined
group/resource cross-references as non-blocking warnings.
Add python3 + pycapnp/pyyaml/jsonschema/click/pytest/ruff (+pip) to the
nix dev shell, wire a new `xceadmin` case into xce-build.sh (schema
sync from src/schema/ and src/configs/schema/, pip install by profile,
pytest), package-data config so bundled schemas survive a non-editable
install, xce-build SKILL.md docs, and ruff format/check hooks matching
this repo's stdin-JSON hook convention (not brief.md's env-var sketch,
which would have silently no-op'd).
Verified live end-to-end inside `nix develop ./src` for both the
default (editable) and release (pinned) profiles; found and fixed 3
real bugs along the way (missing pip, PEP 668 externally-managed-
environment, missing package-data for release-profile bundling) —
logged as bug-1458/1459/1460.
Requirements: 5.5, 8.1, 8.2, 8.6
Editable-installable xceadmin package with a real click-group entry
point (zero subcommands yet) and xcecommon as an independently
importable standalone package.
Requirements: 5.4
