Implements Phase 1 of .kiro/specs/external-license-resources/ (design.md):
externally-reported license pools pushed by a dedicated daemon through the
existing resval pipeline, with EXTERNAL-aware admission arithmetic.
Wire & controller:
- controller.capnp HostRole.extres (@5, appended); xce_join_req accepts it
- extres.c: HostRole_extres join table (endpoint+name, no workermgr slot),
leave cleanup; join.c/cluster.c dispatch
- opupd.c do_update_resvals: extres branch runs the resource hooks and
update_resval_list(c, NULL, ...) — host=="" entries route through
update_val_res' SHARED branch into c->sharedinst; per-entry -EPERM for
non-shared targets is logged, never fails the list
- worker.c update_val_res: NULL worker only writes SHARED resources
- cluster.c cluster_build_resources: rebuilt on resourceApplyDef (bug-1854)
— EEXIST-tolerant so definitions added by an activation register on a
RUNNING controller (bug-1973), and RES_APPLY_CHANGED updates definition
fields (valtype/ordtype/flags/interval) in place so a corrected config
heals a stale registration (bug-1974); seed_sharedinst_from_config skips
EXTERNAL pools (daemon-reported free counts only; seeding would fabricate
capacity while the daemon is down)
Scheduler:
- core_events.c RESVALS_DELTA: NULL/"" host marks a SHARED value and lands
in core->sharedInst via dup_val (bug-1585 ownership rule), mirroring
snapshot_load's host=="" precedent — previously WARN-skipped as unknown
worker
- resource.c unsatisfy_: EXTERNAL skips the inuse term
(F >= uncomit + wanted; F is already net of our checkouts)
xceesd (src/esd, new component):
- script collectors polled per source, debounce on unchanged values,
fail-open stale policy; pushes host=="" shared resvals over xcecli
- reconnect gap fix (bug-1975): with a stable value the debounce pushed
nothing and reconnects only fire inside a push — after a controller
restart the daemon sat on a dead connection forever; the main loop now
forces a full re-push when xcecli_is_connection_broken() (bug-1760
predicate) latches
Config/mapping:
- resources.yaml: extlic sample (location: [external])
- mapping.py: RESOURCE_FLAG_EXTERNAL (0x10) + location:["external"] mapping;
xceadmin tests
Tests (TDD, red->green per fix): test_extres_resvals.c (join+resval
pipeline), test_config_resource_reconcile.c +2 (register past EEXIST /
hot-apply changed definition), test_core_events.c +131 lines (shared DELTA
landing), mapping tests. Suites: controller 1358/0, scheduler 100/0,
esd 7/0, lib 1/0; check-no-fork covers src/esd.
Live t1 E2E: config gen=15 push -> xceesd push extlic=4 lands in
sharedinst -> rusage[extlic=3] job RUN; second job PENDs at free=1;
release re-admits it; controller restart -> forced re-push restores the
value and admission resumes. Full precommit job verification passed.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Breakpoint 2 of the admin-plane gaps (YAML flag/resourcemap wiring), plus
the queue payload ownership fix that the t1 controller segfault of
2026-10-23 12:41:13 traced to.
1. YAML resourcemap wiring (admin-plane-gaps 1.10):
- schema: Resource.distributions @8 / hostBased @9 uncommented --
resourcemap had no wire destination at all before this.
- resobj.c: resobj_init's #if 0 stub replaced with a real deep copy
(distributions + hbased_defined); resobj_destroy frees them. Deep
copy is mandatory: the source is myconf->cluster_config, which
keeps owning and later frees its tree.
- controller: reconcile_config_resources + seed_sharedinst_from_config
(cluster.c) seed c->sharedinst for SHARED|GLOBAL resources whose
distribution carries a value (the license-pool shape), from both
config_activate_committed paths, ahead of the conf4sched push.
- xceadmin: mapping.py maps resourcemap -> distributions (only the
location:[all]+value shape sets SHARED|GLOBAL), build.py renders
shared=true/distributions; golden expected.bin regenerated.
2. xcesched_push_configs resval double free (t1 12:39:52 abort): the
seeded push freed the resval entries AND the backing array via
free_name_val_list, then xlist_fini freed the same array again.
Detach xreslist before the FREEUP/xlist_fini block (same ownership
class bug-1506 fixed in sched_worker_change). Latent since inception:
every pre-1.10 push ran with nresvals==0 and never armed it.
3. queue payload deep-copy (t1 12:41:13 encode_Threshold_list segfault):
queue_init (create) MOVED the definition's users/hosts/thresholds/
window payloads out, and queue_refresh_def MOVED them out again --
but reconcile_config_queues runs MORE THAN ONCE over the same
cluster_config object (boot: pre-election activation AND phase2's
build_queues), so the second pass refreshed live queues from the
drained definition: thresholds became the poisoned (count>0,
array=NULL) pair, the generated encode_Threshold_list has no NULL
guard, and the next queue/worker push died on the NULL base.
Both paths now deep-copy (queue_init and queue_refresh_def); the
definition keeps its payloads, (NULL/0) genuinely means "no list in
this config", and reconcile is idempotent. New dup_group_config
helper in utils.c (the generated code has no GroupConfig dup).
Mirrors resobj_init's distributions deep-copy and worker.c
apply_join_state's dup pattern.
4. UT: test_reconcile_thresholds_survive_second_pass drives a triple
reconcile of a threshold-carrying queue and asserts definition
intactness plus a real encode_QueueConf -> wire -> decode round
trip; new test_config_resource_reconcile.c covers seed/deep-copy/
non-seed classes and idempotent re-seed.
Suites: controller_unit_test 1348/1348, sched_push 4, sched_worker_change 3,
apiserver_push 2, host_runusage 11, xcelib 759, xcealloc_oom 11,
99 scheduler binaries, all worker + harness suites, xceadmin pytest 375.
t1 E2E: boot double-reconcile clean, scheduler rejoin + full conf4sched
push applied, recovered crash-era PEND job reached DONE, 30-job saturation
flow (24 RUN + 6 PEND -> DONE), live gen-12 config activation with
re-push to the connected scheduler -- no crash, all four daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
- bug-1683 (test_wire known-good capture): the client closed its TLS
socket with the server's TLS 1.3 NewSessionTickets unread, so the kernel
sent RST and the server lost the unread payload record; the server
swallowed the OSError. The client now waits for the capture to finish,
handler errors are recorded and asserted, and a 50 ms "tickets-pending"
variant makes the RST race deterministic.
- bug-1485 (live HA push, versionTag=0): the CLI's 0 was an error reply
(-110): after a late-joiner election the primary's own-term no-op only
commits ~6 s later via heartbeat catch-up, so a push in that window
misses the majority barrier. The fixture now waits until the latest
own-term no-op has committed (wait_for_primary_write_ready); versionTag
asserts print the CLI's stdout/stderr. The controller latency itself is
bug-1913.
- bug-1725 (activation line missing): "Config upgraded" and the
activation line are logged after the reply; both tests now poll for all
three commit lines (shared assert_primary_committed).
xceadmin suite 373 passed; live push tests looped (95 runs incl. heavy
CPU load, 0 failures from these mechanisms).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Adds an if/then JSON Schema (draft 2020-12) conditional constraint to
limits.schema.json's Limit definition: when percent is true, value
must be a number in [0.0, 1.0]. This is the class of authoring mistake
bug-1831/bug-1832 fixed manually (night_mem_pct: value: 50 instead of
0.5) -- now caught automatically at real `xceadmin config build` time
(the actual cluster.bin generation path deploy-ha.sh uses), not merely
by a unit test.
Empirically verified against the real jsonschema library this repo
uses: a violation surfaces with the exact same file/json_pointer/
constraint contract validate_yaml()'s existing ValidationError already
provides, so zero changes were needed to its error-reporting code.
While implementing this, the new constraint caught an identical
pre-existing defect in xceadmin's own checked-in "full" golden test
fixture (fixtures/full/configs/limits.yaml, night_mem_pct: value: 50)
-- fixed the same way, and regenerated the checked-in expected.bin
golden artifact to match the corrected input.
Added 3 tests to test_validation.py: reject an out-of-range percent
value, accept an in-range one, and confirm a non-percent value is
never range-checked regardless of magnitude.
Build gotcha discovered and documented (cerebrum.md): this repo has
two copies of every bundled schema -- src/configs/schema/ (canonical)
and src/xceadmin/src/xcecommon/schema_files/json/ (a gitignored local
mirror meson-build.sh always overwrites from the canonical copy on
every build). Only the canonical copy needed a source-controlled edit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xceadmin's --controller option (config push, and scheduler record
start/stop/status) accepted only a single host:port. Passing every
configured controller as a comma-separated list -- the natural way to
query an HA cluster without knowing which node is alive/primary --
mis-parsed the whole joined string as one bogus hostname and failed
with a DNS-resolution error.
Adds xceadmin.discovery.resolve_controllers() (comma-split candidate
parsing) and follow_redirects_any() (tries each candidate in order,
skipping ones that are unreachable at the network level -- connection
refused, DNS failure, or a discovery timeout -- mirroring xcecli/
xce_test's own client-side master-rotation failover). A candidate that
IS reachable but returns a genuine protocol-level failure is not
retried against a different node. push.py and scheduler.py's own
duplicated single-host _split_controller helpers are removed in favor
of this shared implementation.
Also fixes a related gap bug-1821's own live verification surfaced:
right after a primary is killed, a standby doesn't start a new
election until its own heartbeat timeout elapses (~10s), so it keeps
redirecting every candidate to the now-dead old primary -- exhausting
the whole candidate list and raising NoPrimaryError, which
_send_admin_request()'s existing transient-retry loop (bug-1713/1715)
never caught. That loop now retries on NoPrimaryError too, using its
own (longer) budget sized for the heartbeat-timeout-plus-election
window rather than the RPC-level "already electing" budget the
original retry logic was tuned for.
TDD RED->GREEN, 304/304 tests passing. Live-verified end to end: a
comma-list correctly skips a dead first candidate and reaches a live
one on a healthy cluster, and correctly rides out a real primary kill
and re-election (retry loop visibly cycling ~15s) to reach the newly-
elected primary rather than failing immediately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Add workerResyncWindowMs/workerResyncTimeoutMs to xce.schema.json
(minimum:0, matching the optional-scalar-field precedent -- these
aren't `required` with a floor like heartbeatInterval, so minimum:1
would be wrong) and to mapping.py's YAML->capnp field table. Regenerate
both golden config-build fixtures (task 1.1 grew ClusterConfig's wire
layout by 2 fields; full/expected.bin and minimal/expected.bin each
grew by exactly 8 bytes). full's fixture YAML now sets real values
(900000ms/15000ms); minimal deliberately left untouched (no xce.yaml)
per its own locked-down "hosts+queues only" boundary test -- its
expected.bin still regenerates to reflect the new default-0 fields.
Two new tests assert specific decoded values, not just "build succeeds".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
test_live_config_push_against_standby_returns_standby_rejection_not_redirect
locked in the OLD controller behavior (a standby flatly rejects
OP_upgradeConfig with no redirect fields). That gap is now closed:
myreq_handler's gate-rejection path calls fill_redirect_reply() on
every -XCE_ERR_STANDBY rejection (bug-1700), and handle_ctrlhello() now
actually populates primary_host/primary_port from the static --peers
list (bug-1701) -- a standby pushing config now redirects to the
primary and the push succeeds. Renamed to
test_live_config_push_against_standby_redirects_and_succeeds and
rewritten to assert the new behavior.
_assert_primary_committed() changed from a single-shot log read to a
poll loop (up to 5s) for the "config activated from committed
marker..." line -- accounts for the documented async gap between the
majority-ack-gated client reply and the primary's own local WAL-apply-
pump activation, a real race the old single-shot read could hit.
meson-build.sh: wire src/tools and each build profile's controller
subdir into the isolated repo-mirror the E2E tests run against, and
export XCEADMIN_BIN explicitly, so these live tests actually find real
binaries instead of always skipping.
Verified: xceadmin suite 288/288 (incl. both tests in this file);
controller suite 1220/1222 (only the pre-existing bug-1561 flake).
Updated admin-config-tool design.md and roadmap.md to mark the
dispatch-gate and redirect-wiring gaps RESOLVED, matching this test's
new assertions and the live evidence.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Once bug-1700/1701's server-side redirect fix landed, xceadmin scheduler
record status/start/stop against a standby correctly redirects to the
primary -- but the redirected request can still transiently fail if that
node is mid-(re-)election (-10013 E_XCE_NOT_READY, expected/normal right
after a redirect on a freshly-booted or just-failed-over cluster) or if
the scheduler connection was mid-reconnect (schedadminreplymsg{ok=false,
error="scheduler disconnected"}, a different, non-numeric signaling path
via bug-1693's disconnect-detection mechanism).
bug-1713: _send_admin_request() wraps the request/redirect/decode cycle
in a bounded retry loop, re-running follow_redirects() FROM SCRATCH each
attempt (not a bare resend to the same host:port) so a later attempt can
discover a newly-settled or different primary.
bug-1715: extends the same retry loop to the second, differently-signaled
transient family -- checks admin_reply.error against a module-level
frozenset of known-transient reason strings.
Live-verified on a real 3-node cluster.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
_live_e2e_common.py's CONTROLLER_CANDIDATES/XCEADMIN_CANDIDATES only ever
resolved to the three default profile-dir names (xce.build, build.release,
build.dbg), with no way to point the live E2E suite at a binary built to a
custom/explicit build dir. Adds XCE_CONTROLLER_BIN/XCEADMIN_BIN env-var
overrides via a new resolve_binary_candidates() helper -- when set, the
override becomes the sole candidate; otherwise the existing three-name
fallback list is unchanged. Mirrors the ${XCE_CONTROLLER_BIN:-default}
convention already used by clean_failover_preservation.sh/run-ha-soak.sh.
Skip messages in both live E2E test files now mention the override.
TDD: 4 new unit tests in test_live_e2e_common.py (no override, empty
override, override-becomes-sole-candidate, and an end-to-end reload proving
the real module constants respect their env vars). Full xceadmin suite
(288 tests) confirmed unaffected: 287 pass, 1 pre-existing unrelated
failure (a stale redirect-gap assertion, unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds DIRTY_FINGERPRINT = "" to _build_info.py's committed fallback stub
and cli.py's separate ImportError fallback class. New
_compose_version_display() applies the strip-and-append rule via a
genuine anchored removesuffix("-dirty"), mirroring the C/Go
implementations (tasks 3.1/4.4/4.5).
Updates one pre-existing test whose hard-coded literal-GIT_COMMIT
assertion no longer holds once a fingerprint is present, to compute its
expected value via the production function instead -- independently
assessed by review as a legitimate strengthening, not a weakened check.
4 new dedicated unit tests independently guard the transform itself with
hard-coded literal assertions, including an anchored-suffix-not-blind-
substring regression case.
Verified independently by review (including a second, differently-
constructed mutation test) across both install paths via a full real
/xce-build xceadmin cycle: editable install shows "unknown, unknown" with
no import error; wheel built from the dirty tree shows <hash>-<fingerprint>,
never <hash>-dirty-<fingerprint>. 267 passed/3 skipped.
This completes all xceadmin work (tasks 5.1-5.2) for this spec.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Commits a fallback src/xceadmin/src/xceadmin/_build_info.py (GIT_COMMIT/
BUILD_TIME both "unknown") for install paths that never run meson-build.sh
(e.g. `pip install -e .`), and wires click.version_option() on cli.py's
top-level group. cli.py imports _build_info defensively (try/except
ImportError with an inline fallback) and composes the version string from
importlib.metadata.version("xceadmin") plus both build-info fields.
Verified independently by review: editable install shows "unknown" for
both fields with no import error; the meson-built wheel shows real,
matching commit id + build time; full xceadmin suite 261 passed/3 skipped
(258 baseline + 3 new tests); ruff clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three related enhancements to make /xce-deploy-ha's output a
self-contained, directly-usable cluster for manual testing:
- xceadmin now builds an installable wheel (meson-build.sh) with
correct dependencies (click/pycapnp/PyYAML/jsonschema, verified
against actual imports -- pyproject.toml was missing several).
- deploy-ha.sh resolves the deploy environment's own bash
($BASH_BIN, baked into every generated script's shebang -- this
environment's deploy-ha.sh had drifted to #!/bin/sh while still
using bash-only syntax) and generates scripts/start-worker.sh /
scripts/start-scheduler.sh sharing the controllers' --peers, plus
a standalone xceadmin install (wheel extracted directly, resolved
$PYTHONPATH baked into a wrapper script -- nix devShell packages
are $PYTHONPATH entries, not merged into python3's own
site-packages, so a --system-site-packages venv can't see them).
- bug-1566: xceadmin's --controller option had no relationship to a
deployment's actual peers (hardcoded localhost:9991 default) --
every command required an explicit --controller or silently tried
the wrong port. Fixed via a $XCEADMIN_CONTROLLER envvar fallback
(click precedence: explicit flag > envvar > bare default),
exported by deploy-ha.sh's generated scripts/xceadmin wrapper.
Fixed/validated via TDD + live reproduction against a real deployed
cluster in a fully stripped environment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds `xceadmin config dump CLUSTER_BIN_PATH`, the reverse of `config
build` -- decodes a serialized ClusterConfig back to human-readable text
so a built config can be inspected without a throwaway pycapnp script.
Motivated directly by how bug-1556 (hosts.yaml's max_slot: -1 auto-detect
sentinel silently becoming mxj=0) was actually found: by hand-decoding a
live cluster.bin. The dump renders the same sentinel as
"max_slot=-1 (auto-detect from CPU count)", not the raw uint32.
Also adds parse_size_or_number() to xcecommon.mapping so a queue/host's
sched_stop threshold on a byte-valued resource (maxmem/mem/tmp/maxswap/
maxtmp) can be written as a human-friendly size string ("500MB", "2GB")
instead of a raw byte count; a plain number still passes through
unchanged for non-byte resources like load averages. hosts.schema.json
and queues.schema.json's sched_stop schema gains a matching SizeOrNumber
type. dump_cluster_config() humanizes the same byte-valued thresholds
back on the way out (_format_bytes_human).
New ClusterBinFileError covers every local cluster.bin read/decode
failure mode without letting a raw OSError/capnp.KjException escape the
CLI. RESOURCE_VALTYPE/RESOURCE_ORDTYPE/RESOURCE_FLAG_* made public in
mapping.py so dump_cluster_config can reuse the same int<->name tables
build_cluster_config already uses, keeping both directions in sync by
construction.
256 xceadmin tests pass (incl. new test_config_dump.py and the
parse_size_or_number/SizeOrNumber cases in test_mapping.py/test_cli.py),
verified via `xce-build.sh xceadmin`'s full pinned out-of-tree install
+ pytest run before this commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Simplification per review feedback on e3b3ffa: a generic schema-driven
unit field touched the capnp wire schema (2 structs), C structs
(xce_resource_t/xce_resval_t), resobj.c's deep-copy/destroy, opget.c's
reply-build lookup, resources.yaml/resources.schema.json, and mapping.py
-- real cross-cutting cost for a mechanism only one resource (maxmem)
actually needs today.
Reverted all of that. The controller CLI (test.c) now just knows
"maxmem" is bytes-valued via a small explicit is_bytes_valued_resource()
name list, and adaptively picks the best-fit human unit at display time
(format_bytes_human, unchanged) -- same observable behavior (resval/
workerlist render maxmem as e.g. "30.90 GB"), zero wire schema surface.
Regenerated the xceadmin golden fixture back to its pre-unit-field size
(5304 bytes, exactly matching the size right after the mxj-only fix,
confirming a clean, complete removal).
Verified: 1095/1095 controller_unit_test (3 runs; one hit the
pre-existing flaky segfault already logged as bug-1553, unrelated),
228/228 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three bugs found while diagnosing a live controller log, all in the
worker resource-value path:
1. Every host relying on the documented hosts.yaml auto-detect default
(no max_slot, meaning "use CPU count") got 0 scheduling slots,
permanently blocking the scheduler from placing jobs on it. mxj is
UInt32 on the wire and can't carry the schema's -1 sentinel; mapping.py
worked around this by leaving it unset, which defaults to 0 -- and 0 is
never < 0, so apply_join_state()'s auto-detect fallback never fired.
Fixed by encoding the sentinel as 0xFFFFFFFF, which the controller's
`int mxj` field reinterprets back to -1 via its existing implicit
uint32_t->int conversion. No C-side or wire schema change needed for
this part; the fallback itself was already correct but had zero test
coverage for a negative mxj, now added.
2. update_resvals()/update_resval_list() logged the failure reason via
PERROR, which reads process-global errno -- but update_resval()'s
failure paths return raw negative codes directly and never touch
errno, so the controller log filled with misleading "update_resval:
Success" / "update_resval: Resource temporarily unavailable" lines
instead of the real reason. Fixed to decode the actual rc via
xce_strerror_r and name the failing resource.
3. maxmem is canonically stored/wired as raw bytes but had no unit
metadata, so resval/workerlist printed it as a bare number that read
as garbage (e.g. workerlist's fixed-width columns visually smeared a
multi-GB byte count across the MaxCPU/MaxMem headers). Added an
optional `unit` field to the Resource/ResourceVal capnp structs (and
xce_resource_t/xce_resval_t/resources.yaml) so resval and workerlist
can render a human-readable value (e.g. "30.90 GB") while the stored
value stays raw bytes -- display-only, every other resource unaffected.
Regenerated the xceadmin golden fixture (fixtures/full/expected.bin)
twice: once for the mxj sentinel encoding change, once for the new
Resource.unit field growing every resource's wire size by one pointer
slot. Both diffs verified byte-for-byte isolated to exactly the expected
change before regenerating.
Verified: 1095/1095 controller_unit_test (stable across repeated runs),
229/229 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Two related findings logged to .wolf/buglog.json but deliberately not
fixed here (out of scope per discussion): a pre-existing flaky segfault
in the combined controller_unit_test binary, and maxswap/maxtmp being
hardcoded stub values on the worker side.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- probe_config() now wraps probe in Request envelope matching push_config()
- Controller OPCode_probeConfigTag handler replies through xce_reply()
with redirectHost/redirectPort populated
- follow_redirects works identically for both push and probe flows
Add real CliRunner-through-real-mock-TLS-server coverage for scheduler
record start/stop/status (incl. standby->primary redirect handoff) and
dump, closing the gap task 10.5 left open (which only monkeypatched the
CLI's function bindings). 227/227 xceadmin tests green. Closes Group 10.
Add scheduler command group with nested record sub-group (start/stop/status)
and sibling dump command, delegating to the client functions from 10.3/10.4
with no business logic in the CLI layer. 220/220 xceadmin tests green.
Adds dump() to scheduler.py -- a pure local .replay file decoder, zero
network I/O. Key finding: pycapnp's from_bytes() does not validate a
message against the schema type you ask it to decode as (trial-decoding
SnapshotEntry bytes as ReplayEvent raises nothing, returns plausible
garbage), ruling out trial-decode-and-catch for message-type
discrimination. Instead reads each message's raw segment-table/
root-pointer framing to extract its (dataWordCount, pointerCount) shape,
computed live from the schema, and matches against the four replay
message types' distinct shapes before decoding. AC 20's stream-ordering
rules (msg 1 = ReplayHeader, msg 2 = SnapshotEntry) are enforced
structurally, not inferred.
Adversarial review reproduced a real (if forward-looking) gap: an actual
modified schema with a shape collision caused an uncaught raw capnp
exception to escape dump(). Closed with a pairwise-distinctness
assertion right after the four shapes are computed, raising a typed
ReplayFileError before any message is decoded if a future schema change
ever makes two shapes collide -- restoring dump()'s own docstring
guarantee unconditionally. New regression test simulates the exact
collision the review found.
201 passed, 3 skipped (up from 188 pre-task).
Adds record_start()/record_stop()/record_status() to new
src/xceadmin/src/xceadmin/scheduler.py, following push_config()'s exact
pipeline shape (load_schema -> follow_redirects -> decode_reply). Deliberately
diverges from push_config()'s "return failure as data" convention because
this task's own Observable text requires raising a typed error on an error
reply, not just connection/decode failure -- a new SchedAdminError
(xcecommon/errors.py) carries the full reply state (error/recording/file/
event_count/started_at) when SchedAdminReply.ok is false.
test_scheduler.py mirrors test_config_push_mock.py's real mock-TLS-server
technique, covering all three actions, redirect-following (standby-to-
primary), and the error-reply-raises-SchedAdminError path across the
error codes reference table.
188 passed, 3 skipped via the real xceadmin test runner (meson compile
-C xce.build xceadmin).
bug-1485 (flaky test_live_config_push_against_primary_reaches_success_path,
~24/25 fail rate under pytest, primary process going <defunct> after
promotion) could never be diagnosed because controller stderr was
DEVNULL'd -- no ASAN report or signal was ever captured from a real
failure.
Redirects each node's stderr to deploy_dir/logs/<name>.stderr.log and adds
an XCE_KEEP_DEPLOY_DIR=1 debug toggle to preserve the deploy dir across
fixture cleanup for inspection. Ran the live test 40 times in this
environment (25 idle + 15 under concurrent build contention): 40/40
passed, 0/120 captured stderr logs non-empty -- could not reproduce.
Downgraded to low-priority/environment-flake, left open; the capture
infrastructure stays in place for the next real failure.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXkenK8rqE8mdS1YKjsq78
Standby-redirect half is done and correct: OP_upgradeConfig has no
server-side redirect wiring today, matches design.md's own documented
"Out of Boundary" gap, and this test locks in the real current
rejection behavior.
Primary-push success-path half is unreliable (~24/25 live fails in one
investigation round, see bug-1485) -- the elected primary was observed
as a <defunct> zombie shortly after promotion, but a dedicated debug
pass could not reproduce it in 26+ clean runs on an idle host and
captured no crash evidence either way. tasks.md marks 6.3 _Blocked:_
pending re-verification on an idle host with real stderr capture.
Deploys a real xce_controller (no HA peers) and runs real xceadmin
config build/push over actual TLS+capnp -- no mock server -- regression
locking task 6.1's envelope fix. Skips cleanly when no controller binary
is available. Surfaced a real, out-of-scope finding: single-node pushes
always report ackedCount==0/exit 1 even on full server-side success,
since there are no standbys to ACK.
config push crashed on every real controller with a capnp KjException --
xce_reply() wraps every reply in an outer Request{body.replymsg} envelope,
but discovery.py/push.py decoded raw wire bytes directly as a bare
RequestReply. Adds a shared decode_reply() unwrap point used by both call
sites, and fixes the same wrong-shape assumption baked into task 5.6's
and test_cli.py's mock-server fixtures.
Cross-reference warnings ("references undefined entry") were genuine
gaps in the example configs, not validator bugs: src/configs/hosts.yaml
had a typo'd resource ref (mg -> mem) and referenced a nonexistent
host3/hostgroup pair; queues.yaml referenced hgroup1/hgroup2/ugroup1
instead of the real hg1/ug1 names; resources.yaml was missing the
r1m/r15s/tmp built-in worker load-index resources
(worker/src/resource_sync.c's init_builtin_load_indices()). Added the
missing host3 host, hg2 hostgroup, ug2 usergroup, and three resource
entries; fixed the typo'd refs.
"schema-valid but has no ClusterConfig destination" warnings for
resources.yaml's type/order/consumable/release fields were a real gap
in xcecommon/mapping.py's _build_resource, not an inherent limitation
-- the encoding (XCE_VAL_*/xce_order_t/XCE_RESF_CONSUMABLE/
XCE_RESF_RELEASE) is documented in include/xce/types.h and actively
read by the scheduler; the old docstring's claim that it was
undocumented was wrong. Only resourcemap remains genuinely unmapped
(its capnp destination fields are commented out).
Updated test_mapping.py/test_config_build.py for the new mapping
behavior and regenerated the full golden fixture's expected.bin
(legitimate representation-drift from the mapping fix, not a bug).
deploy-ha.sh no longer takes a <data-dir> argument or --build-config
flag. Every deploy now generates a fresh self-signed TLS cert/key
(openssl, RSA 2048, 10y validity) directly into the deploy dir's
certs/, and builds cluster.bin directly into controller/ctrl-1/conf/
via xceadmin -- no intermediate staging directory, nothing preserved
across runs. run-ha-soak.sh (/xce-test-ha) updated to match: dropped
its own XCE_HA_DATADIR-based precondition entirely. Updated 4 callers
(run-ha-soak.sh, test-heartbeat-tolerance.sh, test-commit-durability.sh,
both xce-deploy-ha/xce-test-ha SKILL.md docs).
Separately: xcecommon.wire.connect() was loading a client-side TLS
cert, diverging from the real reference client -- netio.c's
create_ssl_context() presents no certificate at all when acting as a
client (cert_file/key_file both NULL, bare TLS_client_method()).
Fixed connect() to match, and since nothing else consumed cert_dir
(verify_mode=CERT_NONE throughout, no CA check either), removed it
end-to-end: discovery.follow_redirects(), push.push_config()/
probe_config(), and the --cert-dir CLI flag.
Server-side cert loading (mock TLS test servers) is unaffected --
moved into a shared session-scoped tls_certs fixture (new
tests/conftest.py, openssl-generated) replacing 3 duplicated
REPO_HA_DATA_DIR/skipif blocks. Net result: those tests no longer skip
under a hermetic nix build sandbox (which never had ha/data/) --
verified 165/165 passed, 0 skipped, via both meson compile and a full
nix build --option sandbox false .#xcecore_debug rebuild (previously
128 passed / 37 skipped there).
ha/data/{xce.crt,xce.key} removed from git tracking. Requirements
2.3/2.4 and 8.3/8.4/8.5 marked superseded in requirements.md with
dated correction notes.
Adds end-to-end propagation tests through push_config's real
composition with xceadmin.discovery (not just discovery.py's own
unit tests): a 4-mock-server redirect chain proving TooManyRedirectsError
surfaces correctly, and a monkeypatched-timeout test proving
NoPrimaryError does too, without waiting out a real 30s socket
timeout. Also locks in and documents a real wire-contract limitation:
ProbeConfigTagReply carries no rejection signal, so dry-run's
accepted=False path is unreachable today by design, not an oversight.
This completes all 18 sub-tasks in tasks.md.
tests/fixtures/{minimal,full}/configs/*.yaml + checked-in expected.bin
golden files, added to test_config_build.py. Catches representation
drift (bytes changing even when decoded content stays correct) that
the existing structural/decode-based assertions can't detect.
5.1 (mapping) and 5.2 (wire framing) are already fully satisfied by
task 2.3's test_mapping.py and task 2.4's test_wire.py -- no new
tests, would have been pure duplication.
5.3: test_schema_bundling.py cross-checks both bundling mechanisms
(capnp schema, JSON validation schemas) resolve consistently under
the active install, and found a real gap: a corrupted bundled JSON
schema crashed validate_yaml() with a raw json.JSONDecodeError
instead of a clean ValidationError. Fixed in xcecommon/validation.py.
Wire config build/config push --dry-run onto the click group.
Thin layer only: one XceAdminError dispatch surface plus an OSError
catch (ConnectionRefusedError and friends propagate unwrapped from
xcecommon.wire by design and need formatting here), and a
KeyboardInterrupt handler for a clean "cancelled" message (socket
cleanup already happens via push.py's own try/finally).
push_config()/probe_config(): build/send Request{opcode:upgradeConfig}
and the direct-struct ProbeConfigTagRequest respectively, following
redirects to the primary for the former (ProbeConfigTagReply has no
redirect fields, so probe_config connects directly). Maps controller
status/reply-variant failures to typed results/errors. Tested against
a real local TLS mock controller built from the bundled schema, since
neither opcode has a live controller-side handler today (separately
documented, unrelated to this task).
Unblocked mid-task by bug-1461 (commit e53c086): controller.capnp had
a struct-name collision that miscompiled UpgradeConfigRequest.sha256/
.bytes, making this task's core request impossible to construct via
pycapnp until fixed at the schema root.
Generic, opcode-agnostic redirect-following against RequestReply's
redirectHost/redirectPort. Follows up to 3 hops, displaying each
redirect target; a 4th redirect raises TooManyRedirectsError. A
connection timeout to any hop is interpreted as "no primary
available" and raised as NoPrimaryError naming that hop.
config build pipeline: scan the 7 known YAML filenames, validate
each present file, run cross-file business-rule checks, map to
ClusterConfig, serialize. Missing files become empty sections;
schema-valid-but-unmapped fields and undefined references warn to
stderr instead of failing. write_output() handles -o <path>/-o -
and returns the hex SHA-256 digest (printing it is the CLI's job).
TLS connection + 8-byte native-order length-prefixed framing to the
controller, verified byte-for-byte against request.c's own framing
and myapp.c's client TLS posture (SSL_VERIFY_NONE, no CN check).
Opcode-agnostic: moves raw bytes only, so the same send_request()
serves both OP_upgradeConfig and ProbeConfigTag's direct-struct
variant. Connection always closes on error, including handshake
timeout.
Map validated YAML section dicts into ClusterConfig capnp field
values via a declarative SECTION_MAP driving two generic mapper
shapes (list/scalar). Preserves YAML source order, sources no
timestamp/hostname/username, and is proven byte-deterministic
against both synthetic and real src/configs/*.yaml fixtures.
Load controller.capnp once per process via a cached loader that
resolves identically under editable and pinned installs. Missing or
corrupted schema content converts a pycapnp uncaught-C++-abort into a
clean SchemaNotFoundError via isolated subprocess pre-validation.
Validate operator-authored YAML against the 7 bundled JSON schemas,
enforce reserved-name and duplicate-name rejection, and flag undefined
group/resource cross-references as non-blocking warnings.
Editable-installable xceadmin package with a real click-group entry
point (zero subcommands yet) and xcecommon as an independently
importable standalone package.
Requirements: 5.4
