rshen/Letmein was a live, checked-in cluster_admin credential in
auth.yaml. Renamed to a non-identifying username with a clearly-marked
placeholder password.
docs: add full ops deployment guide (src/docs/deployment.md)
Covers building and running all four daemons plus xceadmin, based on
direct source verification (CLI flags, config file shapes, TLS posture)
and a live single-node deploy. Documents real production gaps (no
process supervision, plaintext non-controller traffic, exposed dev
creds) rather than treating the repo as a finished runbook.
Standby-redirect coverage is done and correct (redirect wiring for
OP_upgradeConfig genuinely doesn't exist server-side yet, matches
design.md's own documented gap). Primary-push success-path test is
flaky (bug-1485) -- a dedicated debug round could not reproduce the
crash in 26+ clean runs, so root cause is unconfirmed. Test files left
uncommitted in the worktree pending re-verification on an idle host.
Deploys a real xce_controller (no HA peers) and runs real xceadmin
config build/push over actual TLS+capnp -- no mock server -- regression
locking task 6.1's envelope fix. Skips cleanly when no controller binary
is available. Surfaced a real, out-of-scope finding: single-node pushes
always report ackedCount==0/exit 1 even on full server-side success,
since there are no standbys to ACK.
config push crashed on every real controller with a capnp KjException --
xce_reply() wraps every reply in an outer Request{body.replymsg} envelope,
but discovery.py/push.py decoded raw wire bytes directly as a bare
RequestReply. Adds a shared decode_reply() unwrap point used by both call
sites, and fixes the same wrong-shape assumption baked into task 5.6's
and test_cli.py's mock-server fixtures.
admin-config-tool's spec.json blocked_on and tasks.md both still described
the OP_upgradeConfig dispatch-gate defect as an open blocker. It was fixed
2026-07-25 (bug-1468, commit 959b0cda) by config-upgrade-snapshot task 8.6,
which independently hit the same wall building a live OP_upgradeConfig
sender -- the fix's own comment in cluster.c already cross-references this
spec. The controller-side path has since been exercised live and
successfully many times via config-upgrade-snapshot's --config-upgrade
soak. All 23 admin-config-tool tasks are already [x]; this was pure
tracking-metadata debt, not a reopened implementation gap.
Also appends a cross-spec gap analysis (admin-config-tool x
config-upgrade-snapshot x clusterconfig-schema-split) to all three specs'
research.md, confirming clean schema integration between the latter two
and flagging the one remaining real gap: a new commit-timeout issue in
config-upgrade-snapshot task 8.6's force-elect step, surfaced now that
bug-1477 is fixed, not yet root-caused.
deploy-ha.sh no longer takes a <data-dir> argument or --build-config
flag. Every deploy now generates a fresh self-signed TLS cert/key
(openssl, RSA 2048, 10y validity) directly into the deploy dir's
certs/, and builds cluster.bin directly into controller/ctrl-1/conf/
via xceadmin -- no intermediate staging directory, nothing preserved
across runs. run-ha-soak.sh (/xce-test-ha) updated to match: dropped
its own XCE_HA_DATADIR-based precondition entirely. Updated 4 callers
(run-ha-soak.sh, test-heartbeat-tolerance.sh, test-commit-durability.sh,
both xce-deploy-ha/xce-test-ha SKILL.md docs).
Separately: xcecommon.wire.connect() was loading a client-side TLS
cert, diverging from the real reference client -- netio.c's
create_ssl_context() presents no certificate at all when acting as a
client (cert_file/key_file both NULL, bare TLS_client_method()).
Fixed connect() to match, and since nothing else consumed cert_dir
(verify_mode=CERT_NONE throughout, no CA check either), removed it
end-to-end: discovery.follow_redirects(), push.push_config()/
probe_config(), and the --cert-dir CLI flag.
Server-side cert loading (mock TLS test servers) is unaffected --
moved into a shared session-scoped tls_certs fixture (new
tests/conftest.py, openssl-generated) replacing 3 duplicated
REPO_HA_DATA_DIR/skipif blocks. Net result: those tests no longer skip
under a hermetic nix build sandbox (which never had ha/data/) --
verified 165/165 passed, 0 skipped, via both meson compile and a full
nix build --option sandbox false .#xcecore_debug rebuild (previously
128 passed / 37 skipped there).
ha/data/{xce.crt,xce.key} removed from git tracking. Requirements
2.3/2.4 and 8.3/8.4/8.5 marked superseded in requirements.md with
dated correction notes.
Reopens the 2026-07-21 design decision to keep xceadmin outside meson.
Standard onboarding consistency (README 3.3 `nix build .#xcecore` and
3.4 `meson compile` working for xceadmin like every other component)
outweighs the earlier concern about meson's single-artifact model not
fitting a multi-file pip install.
- src/xceadmin/meson.build: custom_target() wrapping install+test,
stamp-file output, build_always_stale (matches this repo's existing
clean-rebuild-every-invocation convention).
- src/xceadmin/meson-build.sh: out-of-tree pinned install + pytest;
stages a repo-mirror/ symlink tree so tests' fixed-relative-path
assumptions (src/configs/, src/schema/, ha/data/) still resolve from
a copied location, without touching their deliberate fail-loud
behavior.
- src/meson.build: subdir('xceadmin') under component != 'xcecli'.
- src/tools/xce-build.sh: xceadmin case now delegates to
`meson compile -C "$BUILD_DIR" xceadmin` instead of its own
pip/venv implementation, like every other component.
Verified both build paths: meson compile (165/165 passed) and
nix build --option sandbox false .#xcecore_debug (128 passed / 37
skipped — cert-dependent tests correctly skip under nix's
git-tracked-only hermetic source). AppImage bundling (README 3.5)
is explicitly deferred to future work, not part of this change.
design.md, research.md (new), brief.md and requirements.md (updated
during spec authoring: Req 1.6 resolved/superseded by bundling, Req
5.5 added, Req 4.3/4.4 extended for limits.yaml) had been sitting
uncommitted since before implementation began -- the entire 18-task
implementation was built against these documents without them ever
landing in git.
src/configs/*.yaml + schema/*.schema.json: the example config
directory and bundled JSON schemas most of the test suite reads as
"real repo fixtures" (test_config_build.py, test_mapping.py,
test_validation.py, etc.) were likewise never committed -- those
tests would silently skip (skipif guards) rather than run for anyone
else checking out this branch.
Adds end-to-end propagation tests through push_config's real
composition with xceadmin.discovery (not just discovery.py's own
unit tests): a 4-mock-server redirect chain proving TooManyRedirectsError
surfaces correctly, and a monkeypatched-timeout test proving
NoPrimaryError does too, without waiting out a real 30s socket
timeout. Also locks in and documents a real wire-contract limitation:
ProbeConfigTagReply carries no rejection signal, so dry-run's
accepted=False path is unreachable today by design, not an oversight.
This completes all 18 sub-tasks in tasks.md.
tests/fixtures/{minimal,full}/configs/*.yaml + checked-in expected.bin
golden files, added to test_config_build.py. Catches representation
drift (bytes changing even when decoded content stays correct) that
the existing structural/decode-based assertions can't detect.
5.1 (mapping) and 5.2 (wire framing) are already fully satisfied by
task 2.3's test_mapping.py and task 2.4's test_wire.py -- no new
tests, would have been pure duplication.
5.3: test_schema_bundling.py cross-checks both bundling mechanisms
(capnp schema, JSON validation schemas) resolve consistently under
the active install, and found a real gap: a corrupted bundled JSON
schema crashed validate_yaml() with a raw json.JSONDecodeError
instead of a clean ValidationError. Fixed in xcecommon/validation.py.
Add --build-config (parsed anywhere in the arg list) that rebuilds
<data-dir>/cluster.bin from configs/ via the venv's xceadmin before
deploying; a missing cluster.bin auto-builds the same way even
without the flag. An existing cluster.bin with no flag passed is
left byte-for-byte untouched, matching prior behavior. xceadmin and
the configs source are resolved relative to the script's own
location, not the caller's CWD.
Verified live: missing-file auto-build, untouched-when-present (same
mtime/SHA), and forced rebuild reproducing byte-identical output.
Wire config build/config push --dry-run onto the click group.
Thin layer only: one XceAdminError dispatch surface plus an OSError
catch (ConnectionRefusedError and friends propagate unwrapped from
xcecommon.wire by design and need formatting here), and a
KeyboardInterrupt handler for a clean "cancelled" message (socket
cleanup already happens via push.py's own try/finally).
push_config()/probe_config(): build/send Request{opcode:upgradeConfig}
and the direct-struct ProbeConfigTagRequest respectively, following
redirects to the primary for the former (ProbeConfigTagReply has no
redirect fields, so probe_config connects directly). Maps controller
status/reply-variant failures to typed results/errors. Tested against
a real local TLS mock controller built from the bundled schema, since
neither opcode has a live controller-side handler today (separately
documented, unrelated to this task).
Unblocked mid-task by bug-1461 (commit e53c086): controller.capnp had
a struct-name collision that miscompiled UpgradeConfigRequest.sha256/
.bytes, making this task's core request impossible to construct via
pycapnp until fixed at the schema root.
Generic, opcode-agnostic redirect-following against RequestReply's
redirectHost/redirectPort. Follows up to 3 hops, displaying each
redirect target; a 4th redirect raises TooManyRedirectsError. A
connection timeout to any hop is interpreted as "no primary
available" and raised as NoPrimaryError naming that hop.
config build pipeline: scan the 7 known YAML filenames, validate
each present file, run cross-file business-rule checks, map to
ClusterConfig, serialize. Missing files become empty sections;
schema-valid-but-unmapped fields and undefined references warn to
stderr instead of failing. write_output() handles -o <path>/-o -
and returns the hex SHA-256 digest (printing it is the CLI's job).
TLS connection + 8-byte native-order length-prefixed framing to the
controller, verified byte-for-byte against request.c's own framing
and myapp.c's client TLS posture (SSL_VERIFY_NONE, no CN check).
Opcode-agnostic: moves raw bytes only, so the same send_request()
serves both OP_upgradeConfig and ProbeConfigTag's direct-struct
variant. Connection always closes on error, including handshake
timeout.
Map validated YAML section dicts into ClusterConfig capnp field
values via a declarative SECTION_MAP driving two generic mapper
shapes (list/scalar). Preserves YAML source order, sources no
timestamp/hostname/username, and is proven byte-deterministic
against both synthetic and real src/configs/*.yaml fixtures.
Load controller.capnp once per process via a cached loader that
resolves identically under editable and pinned installs. Missing or
corrupted schema content converts a pycapnp uncaught-C++-abort into a
clean SchemaNotFoundError via isolated subprocess pre-validation.
Validate operator-authored YAML against the 7 bundled JSON schemas,
enforce reserved-name and duplicate-name rejection, and flag undefined
group/resource cross-references as non-blocking warnings.
Add python3 + pycapnp/pyyaml/jsonschema/click/pytest/ruff (+pip) to the
nix dev shell, wire a new `xceadmin` case into xce-build.sh (schema
sync from src/schema/ and src/configs/schema/, pip install by profile,
pytest), package-data config so bundled schemas survive a non-editable
install, xce-build SKILL.md docs, and ruff format/check hooks matching
this repo's stdin-JSON hook convention (not brief.md's env-var sketch,
which would have silently no-op'd).
Verified live end-to-end inside `nix develop ./src` for both the
default (editable) and release (pinned) profiles; found and fixed 3
real bugs along the way (missing pip, PEP 668 externally-managed-
environment, missing package-data for release-profile bundling) —
logged as bug-1458/1459/1460.
Requirements: 5.5, 8.1, 8.2, 8.6
Editable-installable xceadmin package with a real click-group entry
point (zero subcommands yet) and xcecommon as an independently
importable standalone package.
Requirements: 5.4
