Load controller.capnp once per process via a cached loader that
resolves identically under editable and pinned installs. Missing or
corrupted schema content converts a pycapnp uncaught-C++-abort into a
clean SchemaNotFoundError via isolated subprocess pre-validation.
Validate operator-authored YAML against the 7 bundled JSON schemas,
enforce reserved-name and duplicate-name rejection, and flag undefined
group/resource cross-references as non-blocking warnings.
Add python3 + pycapnp/pyyaml/jsonschema/click/pytest/ruff (+pip) to the
nix dev shell, wire a new `xceadmin` case into xce-build.sh (schema
sync from src/schema/ and src/configs/schema/, pip install by profile,
pytest), package-data config so bundled schemas survive a non-editable
install, xce-build SKILL.md docs, and ruff format/check hooks matching
this repo's stdin-JSON hook convention (not brief.md's env-var sketch,
which would have silently no-op'd).
Verified live end-to-end inside `nix develop ./src` for both the
default (editable) and release (pinned) profiles; found and fixed 3
real bugs along the way (missing pip, PEP 668 externally-managed-
environment, missing package-data for release-profile bundling) —
logged as bug-1458/1459/1460.
Requirements: 5.5, 8.1, 8.2, 8.6
Editable-installable xceadmin package with a real click-group entry
point (zero subcommands yet) and xcecommon as an independently
importable standalone package.
Requirements: 5.4
kiro-validate-impl's feature-level requirements-coverage pass found two
low-severity gaps: Requirement 4.1 (discard removed-field values without
error) had no citing task, though it was already satisfied structurally
by task 1.1's direct removal + capnp's default decode behavior -- cited
now on task 5.1, where it's live-exercised. Requirement 3.3's citing task
(5.2) didn't document how worker-liveness was verified -- added a
clarifying bullet recording the actual (indirect, code-path-untouched)
verification method.
No functional changes.
Wires all 11 task-6.1 CLI-sourced myconf_t fields into
init_controller_tunables() via the codebase's existing 0-sentinel fallback
idiom ((myconf->x > 0) ? myconf->x : <original hardcoded constant>);
omitting all flags reproduces today's exact hardcoded values.
Resolves both gaps flagged in task 6.1's review: resmgr_inst_delta bridges
to the existing tunables.resmgr_inst field by meaning (confirmed via
zarray_init's delta parameter), and xstrings_delta is deliberately left
unwired with an operator-facing WARN rather than adding a dead field --
the underlying xstring_table_init/kv_params_t has no delta/growth-step
concept at all, unlike the kvmgr-backed jobmgr/workermgr/qmgr/resmgr.
This completes clusterconfig-schema-split -- all required tasks (1.1-5.3)
and optional Requirement 6 tasks (6.1-6.2) are done.
Adds 11 new CLI flags (--jobmgr-nhash/-delta, --workermgr-nhash/-delta,
--qmgr-nhash/-delta, --resmgr-nhash/-delta/-inst-delta,
--xstrings-nhash/-delta) mirroring the existing --vcpus OPT_*+struct
arguments+myconf_t pattern. Zero behavior change until task 6.2 wires
these into init_controller_tunables().
Review flagged two naming/coverage gaps for task 6.2 to resolve:
xstrings_delta has no corresponding cluster_t.tunables field yet, and
resmgr_inst_delta maps to tunables' existing resmgr_inst by name only.
Live-verified all three cluster_open_listener code paths not exercised by
the 3-node soak: no --peers binds hardcoded default 9990; a single
self-referencing --peers entry binds its custom port instead; --peers
given but local name absent still refuses to start with -EINVAL, unchanged.
Independently reproduced by the reviewer. This closes the actual
regression risk this spec's design was built around.
Review also found (bug-1457, logged, not fixed here) that pre-spec
single-node startup was already broken with -EINVAL before this spec
began, due to an unrelated myapp_run listener-open/config-load ordering
issue -- task 2.1's rewrite incidentally repaired it. research.md updated
to record this against the spec's original preservation premise.
Deployed a fresh 3-node cluster seeded from the task-5.1-regenerated
ha/data/cluster.bin: election, --peers listener binding, and config-push
replication all behave identically to pre-spec (byte-identical config
loaded on all 3 nodes, verified via md5sum). Independently reproduced by
the reviewer on separate ports. Verification-only, no source changes.
Re-encodes the checked-in cluster.bin fixture against the renumbered
ClusterConfig schema (task 1.1), preserving all 13 retained fields'
semantic values byte-for-byte (verified via capnp convert round-trip and
independently reproduced in review) and dropping the 8 removed fields'
now-discarded values. deploy-ha.sh's verbatim-copy usage needs no changes.
Without this, every future /xce-deploy-ha and /xce-test-ha run would seed
ctrl-1 from a blob whose retained fields silently decode to wrong bytes
under the new schema.
Fresh asan-profile build (xce.build/, /xce-build controller equivalent):
zero compile errors, meson controller suite Ok:3/Fail:0, controller_unit_test
976/976 pass, no AddressSanitizer/LeakSanitizer output. Independently
reproduced by the reviewer. Verification-only, no source changes.
Drops the two cfg.port = ...; struct-literal assignments in
test_commit_barrier_rollout.c and test_config_repl.c (removed field, task
1.2). Repo-wide sweep found no other genuine cluster_config_t literal
referencing a removed field.
Full clean build now succeeds (153/153 targets) and controller_unit_test
passes 976/976 with zero new failures.
Removes SET_DEFAULT_IF_UNSET(c->port, 0, 9990) from fill_default_cluster_config()
in myconf.c and config_repl.c, now dead since cluster_config_t.port was removed
in task 1.2 and cluster_open_listener stopped reading it in task 2.1. All of
src/controller/{cluster,myconf,config_repl}.c now compile clean.
Deletes the dead candidates/ncandidates KV-mirror loop from
cluster_build_config; init_controller_tunables() left untouched (it only
ever read hardcoded constants). cluster.c now compiles clean.
These capnp structs were used only by the jobmgr/workermgr/qmgr/xstrings
and resmgr fields already removed in task 1.1, and were deliberately left
in place at the time since nothing required their removal. Task 1.2 then
deleted their corresponding hand-written C typedefs, breaking the
capnp-generated bindings' $C.mapname references. Closes that gap (new
Requirement 2.5).
Replaces the cluster_config->port fallback (dead after task 1.2) with:
--peers candidate match (unchanged, including single self-referencing
entries for a custom single-node port) -> hardcoded XCE_CONTROLLER_DEFAULT_PORT
(9990) when --peers was never passed at all. The --peers-given-but-local-
name-absent case still returns -EINVAL and refuses to start, unchanged.
Also folds in task 1.3's discovery (HashZArrayConfig/ResmgrConfig capnp
structs left orphaned by tasks 1.1/1.2, breaking codegen) into
requirements.md (new Req 2.5)/design.md/research.md/tasks.md, and fixes
a design.md citation error (validate_peer_count -> xce_parse_peers) found
during task 2.1's review.
Deletes port, ncandidates, candidates/master_candidate_t**, vcpus, jobmgr,
workermgr, qmgr, resmgr, xstrings members from cluster_config_t (and their
now-orphaned supporting typedefs) to match the schema shrunk in task 1.1.
Other .c files still referencing these members intentionally do not
compile yet — that's tasks 2.1/2.2/2.3/3.
Removes port, candidates, vcpus, jobmgr, workermgr, qmgr, resmgr, xstrings
from the ClusterConfig capnp struct plus the now-unused MasterCandidate
struct, per the config-design.md shared-vs-local classification rule.
Cap'n Proto forbids ordinal gaps, so the 13 retained fields are renumbered
to a contiguous @0..@12 (accepted pre-1.0 wire-compat break — see
research.md's "Design Revision 2"). requirements.md/design.md updated to
match: Req 3.1 drops field-number invariance, Req 3.2/4.1 state a pre-spec
cluster.bin is not expected to decode correctly, and a new Req 5.3 covers
regenerating the checked-in ha/data/cluster.bin fixture in a later task.
Also folds in this session's live design revision: Requirement 1.7's port
replacement mechanism is the existing --peers self-reference + a hardcoded
default (no new --port CLI flag), confirmed via interactive review with
the spec owner.
bug-1453: remove unused client-side CA-certificate plumbing. C-side:
drop ca_cert_path from xce_cluster_config_t, do_connect()'s ca_cert
param, --ca-cert flags on worker/scheduler, and controller's cert
population (TLS is server-side only, SSL_VERIFY_NONE everywhere on
the client side already, verified zero SSL_CTX_load_verify_locations
calls in the C tree). Go-side: delete xcecli's self-signed cert
TOFU-pinning flow (cmd/auth/cert/*), CACert field, SetRootCAs/
loadRootCAs, and every call site.
bug-1454: fix xcecli_ctx_t leak in scheduler.c sched_init() — the
unconditional initial ctx_init was never freed before the HA
candidate loop re-initialized it on the first iteration.
bug-1455: sched_init()'s memset(sched, 0, ...) was silently wiping
sched->cluster_cfg that main.c had populated from --peers just before
calling sched_init(), so --peers was always ignored and the scheduler
fell back to the hardcoded localhost:9990 single-host default.
Restructured sched_init() to take cluster_cfg as an explicit parameter
(assigned after the memset) instead of relying on caller-side
ordering with a separate sched_set_cluster_cfg() setter, which is now
removed.
bug-1456: revert an unsafe stack-local xce_client_config_t introduced
while fixing bug-1455 — xcecli_ctx_init() stores that pointer as
ctx->conf and client_failover.c dereferences it for the ctx's entire
life (runtime reconnect), so a per-loop-iteration stack copy would
dangle the moment sched_init() returned. Back to mutating the
caller-owned, process-lifetime config in place.
Verified: full ASAN build (all components), meson test suite 9/9 OK,
manual scheduler testsuite 15/15, Go build/vet/test all pass.
buglog.json: bug-1452 marked FIXED with root cause and fix writeup.
cerebrum.md: Do-Not-Repeat entry for the commit_engine_fini/persist_stop
ordering contract, plus the flag-noted latent duplicate in
test_demotion_convergence.c.
memory.md/anatomy.md/session state: routine session bookkeeping.
test_persist_truncate.c's three continuation tests called
commit_engine_fini(&ce) before persist_stop(p), backwards from
commit_engine_fini's own documented contract (persist_stop must run
first). With the fsync_worker flusher thread still alive, it could be
mid-flight in commit_on_durable_advance -> commit_advance_locked
(no lock held) while commit_engine_fini concurrently destroyed
ce->lock/ce->cond and nulled ce->p -- a TOCTOU race where the guard
check reads ce->p as valid but a later re-read observes NULL, SEGV at
exactly offsetof(persist_t, committed_lsn) = 0x38.
This was misdiagnosed as caused by g_drive_async_demotion/catchup_offload
(bug-1450/1451): flipping those flags only perturbed thread-scheduling
timing enough to trigger the always-latent race more reliably. The same
crash signature was independently observed on 2026-07-07, before those
flags existed, proving they were never causal.
Fix: call persist_stop(p) before commit_engine_fini(&ce) in all three
tests. teardown_persist's own internal persist_stop becomes a safe
no-op on the second call. No production code changed.
Verified: full ASAN controller_unit_test suite, 976/976 tests, 6 runs
(1 via meson + 5 standalone direct binary runs), zero ASAN reports.
test_demotion_convergence.c has the same commit_engine_fini-before-
persist_stop ordering (line 233/235) against a real persist_start'd
fixture -- not fixed here, flagged for a follow-up sweep.
- ha-async-io.md: document the same-batch endpoint lifetime hazard
(bug-1447) and its ref-whole-batch-before-dispatch fix pattern
- xce-audit-invariants SKILL.md: add object-lifetime-within-a-dispatch-batch
as a 6th audit dimension, so a future pass checks other batch loops
(vtaskmgr drain, peer-list iteration) for the same shape
- product.md: replace the stale "HA consistency family is the current
focus" framing with the actual 2026-07-18 status (umbrella complete,
9-spec validate-impl batch closed, job-array-single-wal-record shipped)
- roadmap.md: correct 5 entries that were stale relative to actual spec
status (ha-consistency-invariants/ha-wal-replication/libxce-failover
pulled out of "In Flight", ha-term-safety and raft-drive-async marked
implemented+GO, raft-sim-conformance and job-array-single-wal-record
given closure annotations) -- struck through, not deleted, per this
file's own additive-update convention
- testing.md: document that meson's aggregate Fail:0 can mask Unity
sub-group failures -- always check the standalone binary's own summary
- src/controller/CLAUDE.md: fix a stale line-citation for the votedFor
atomicity invariant (had drifted onto unrelated post-election code)
/xce-audit-invariants controller found two live violations of the async
I/O thread-role contract: ha_demotion_convergence_fn (ha_convergence.c)
ran WAL truncation + tlog_exit()+execv() directly on the delaymgr thread,
and ha_catchup_continuation_fn (config_repl.c) ran unbounded WAL-streaming
to a standby directly on delaymgr -- the same single thread that drives
heartbeats and election timeouts cluster-wide, so either blocks liveness
for the whole cluster and risks a spurious election.
A prior vtask-pool offload attempt (2026-07-10, bug-1429) deadlocked
controller_unit_test because c->ha.pool is never initialized by the
lightweight unit-test fixtures these functions run under, and was fully
reverted. This fix instead uses the codebase's established slow-lane
pattern (the same mechanism already used for snapshot-send, config-push,
and peer connect) -- a 4th and 5th instance, not a new mechanism.
New g_drive_async_demotion_offload / g_drive_async_catchup_offload flags
ship default OFF, matching the original rollout discipline of the three
reference offloads (shipped OFF, flipped ON later after a dedicated
compatibility audit). Flipping ON was tried during validation and
reproduced two distinct hazards against pre-existing fixtures that never
call slowlane_init: the already-known hang, and a new ASan SEGV in
commit_advance_locked (logged as bug-1452, not root-caused, deferred to
the future flip-to-ON audit). Both flags verified OFF and the full
976-test suite clean, then the committed state re-verified clean against
a live 10-round chaotic and 10-round sequential HA soak (zero job loss,
zero divergence, both PASS).
job-array-single-wal-record shipped end-to-end, ha-consistency-invariants
umbrella closed out, Unity test-harness suite-wide fix, and today's Group 1
kiro-validate-impl batch (9 specs) which surfaced and fixed a real
same-batch endpoint UAF crash (bug-1447) plus a harness timing race
(bug-1449).
Raft-drive-async's reconnect-offload burst (Task 2.1/2.2) exposed a
latent race in io_worker's epoll dispatch loop: ha.c's
on_standby_connected() evicts a stale duplicate peer channel via
iomgr_unmonitor()+endpoint_unref(), but epoll_ctl(DEL) only stops
FUTURE epoll_wait() calls -- it does not invalidate an event for that
same endpoint already sitting in the CURRENT batch's events[] array.
A later event in the same batch then dereferences the freed
endpoint_t/SSL*, producing the ASAN SEGV observed deep inside
OpenSSL's ssl3_read_bytes during raft-drive-async's round-9 soak.
The original "concurrent SSL_do_handshake from two io_worker threads"
hypothesis was wrong and is retracted -- iomgr_start() creates exactly
one io_worker thread per iomgr_t.
Fix: extract io_worker_dispatch_batch(), which takes endpoint_ref() on
every endpoint in the batch before dispatching any of them and
endpoint_unref()s each only after the whole batch completes -- the
same lock-free ref/unref bracket already used for bug-1298 and
INVOKE_MSG_HANDLER, just never applied to io_worker's own loop. Pure
atomics, no locks, no blocking added to the hot path.
Adds test_netio_batch_dispatch_uaf.c, which drives the real dispatch
loop with a synthetic epoll_event[] batch (TDD RED confirmed via ASAN
heap-use-after-free before the fix, GREEN after). Verified further
with two independent clean 10-round live sequential HA soaks (33 and
36 jobs, zero loss) and full ASAN suite runs (xcelib_test 294/0,
controller_unit_test 972/0). raft-drive-async's kiro-validate-impl
re-run flips NO-GO to GO.
run-ha-soak.sh's sequential mode waited for full 3-of-3 reconvergence
between rounds 2+ but had no equivalent wait before round 1's first
kill. A slow cold-start election could let the kill land before
config-push reached both standbys, leaving them at config tag=0 --
ha_on_primary_ctrl_fd_rdhup's voter-only guard then correctly refuses
to let an unconfigured standby self-elect, producing a false "no
primary elected" round-1 failure. Adds node_config_tag/
cluster_config_converged/wait_for_initial_config_convergence,
mirroring the existing reconvergence-wait's polling style, and gates
sequential mode's round 1 on it.
libxce-failover, raft-sim-conformance, ha-commit-durability,
ha-wal-replication, ha-heartbeat-committed-propagation, ha-prevote,
ha-elect-ctx-lifecycle, and ha-entity-snapshots all had 100% of tasks
checked but had never run a formal /kiro-validate-impl closure pass
(spec.json stuck at phase "tasks-generated", some with stale
ready_for_implementation flags). Each now has a fresh validate_impl
GO decision backed by an ASAN build, full standalone test-suite run,
and a live 10-round sequential HA soak with zero committed-job loss.
Records the controller_unit_test nested UNITY_BEGIN/UNITY_END fix
(commit a81f2a2) in cerebrum.md for future sessions, plus routine
anatomy/memory/token-ledger session bookkeeping.
Only 21 of 86 test_*.c group functions called their own nested
UNITY_BEGIN()/UNITY_END(). Each nested call reset Unity's global
NumberOfTests/TestFailures counters, so controller_test.c's final
`return UNITY_END();` -- the value meson test actually gates on --
only ever reflected whichever small window of tests ran since the
LAST nested reset, silently discarding pass/fail evidence for
everything before it (measured: 16 of 972 total tests in one run).
The same nested-BEGIN calls also caused PASS/FAIL banners to show
the wrong source file for tests in files that don't reset it
themselves (Unity.TestFile just carries over from whichever group
last called UNITY_BEGIN()).
Fix: replace every nested UNITY_BEGIN() with UnitySetTestFile(__FILE__)
(sets file attribution only, no counter reset) and drop the matching
UNITY_END(); add the same UnitySetTestFile(__FILE__) as the first
statement to the other group functions that had neither. The 2
genuinely standalone binaries (test_job_array_sched_push.c,
test_job_array_apiserver_push.c -- each has its own int main(), not
registered via controller_test.c's TEST() macro) are untouched.
Verified: single consolidated summary now reads
"972 Tests 0 Failures 0 Ignored OK", exactly matching the
ground-truth RUN_TEST call-site count across all 83 registered
group files (976 total minus 4 belonging to the 2 standalone
binaries). Every PASS/FAIL banner now names its own true source
file. Full xce-build.sh all regression pass clean (Ok: 9, Fail: 0).
No production code touched -- test-harness plumbing only.
All tasks (1.1-6.4) were already [x] with no prior validate_impl
record. Fixed stale delta.impl_status (still said task 6.4 blocked on
sandbox overlayfs despite tasks.md recording it VERIFIED live since
2026-07-02).
Full suite 9/9 green, 36/36 requirement acceptance criteria mapped,
cross-task wire contract verified field-for-field end to end (schema
-> typemap -> handler -> dispatch -> libxce client -> CLI), two live
smoke runs (3-round sequential HA soak + targeted 3-node deploy
exercising Req 10/11 directly) both PASS. One transient test failure
during the run traced to a concurrent second session competing for
the same build directory, not a regression -- reproduced clean 3x in
isolation.
All 19 tasks were already [x] with no prior validate_impl record.
Full suite 9/9 green, 46/46 requirement acceptance criteria mapped,
fresh 6-round live chaotic-multi-kill soak PASS (zero committed
divergence/job loss), src/lib/raft protocol-ownership boundary
respected across all 35 tagged commits.
Three known_limitations recorded, all non-blocking: two are
design.md describing an originally-planned architecture that shipped
differently after reviewer-approved safety proofs (commit-source
switch stayed on the durable-majority barrier + own-term WAL_NOOP
gate rather than switching to raft_commit_index; AppendEntries send
side still goes through repl_broadcast, not raft_send_append_entries)
-- behavior is correct, docs just weren't updated to match. Third is
a stale TODO comment in ha.c whose own body already says IMPLEMENTED.
raft_cb_become_leader()'s eligibility gate read
persist_apply_resource_fenced() before this process's own first
cluster_init_phase2()/persist_recover()/replay_wal() call ever ran --
so a node whose first-ever replay hit a resource-exhaustion apply
failure could still become CTRL_PRIMARY, because the fence that
replay would engage didn't exist yet when the gate checked it. Every
LATER promotion evaluation was already correctly protected; this was
a narrow gap specific to a process's single very-first election.
Fix: re-check persist_apply_resource_fenced() immediately after
cluster_init_phase2() completes on first boot, refusing promotion
(same WARN + ha_arm_promotion_fence_timer() + return pattern as the
existing refusal branch) if that first replay just engaged the fence.
Confirmed ha_elect_evaluate_and_signal (the other election path) is
dead in production for the promotion outcome -- every live call site
of handle_request_vote_reply passes vote_granted=0 since bug-828
redirected real grant recording to the record-only, non-signaling
ha_record_vote_reply -- so raft_cb_become_leader is the sole live
promotion path and this fix's scope is complete.
TDD: new test test_first_boot_promotion_evaluates_own_first_replay_fence
wires ha_phase2_hook to actually call persist_recover(), reproducing
production ordering exactly. RED-confirmed against the unfixed code
at the precise predicted assertion; GREEN-confirmed after the fix.
Full controller_unit_test regression (bundles every HA-adjacent
testsuite/test_*.c) green, 0 new failures.
Originally documented (not fixed) in job-array-single-wal-record
task 7.2's Implementation Notes; logged as bug-1445 and fixed here,
outside that spec's own task list per its persist.c-only boundary.
Root cause and reproduction path were already documented inline in
tasks.md task 7.2's Implementation Notes but never had a standalone
buglog entry. Cross-referenced from spec.json's known_limitations.
Not fixed -- suggested shape captured in the bug entry: a small,
single-call-site ha.c ordering fix (move the first promotion-gate
evaluation to run after the first replay_wal(), or have that one gate
call treat "not yet replayed" as non-promotable), not a new Kiro spec.
spec.json: phase -> implementation-complete, new GO decision (2026-07-17)
covering the full feature plus phase 7's residual-gap closure; prior
decision preserved as prior_decision; bug-1443 known_limitations entry
retired (closed by task 7.1) and replaced with the one limitation that
is still genuinely open (task 7.2's first-boot promotion-ordering gap).
tasks.md: task 3.1's implementation note updated from "open, unowned
gap" to "closed by task 7.1" -- it predated that fix and had gone
stale.
Validation evidence: full C suite 9/9 green (incl. libxce test post
rtti-fix), Go suite green (88.1% backend coverage), clean TBD/secrets
grep, live array-soak smoke PASS (O(1) replay time confirmed at
2-element vs 10,000-element scale), cross-task integration verified at
every seam phase 7 touched, 10/10 requirement groups covered, zero
design drift, zero blocked tasks.
Upstream librtti repin (see prior commit) resolves the SIGABRT this
note warned future tasks to expect and ignore -- updated so it no
longer reads as a live gotcha.
librtti master 7212ba1 -> 53cf11e (revCount 20->21): makeInstance() was
not freeing its calloc'd allocation when a type's .init() returned
negative, causing a 240-byte LeakSanitizer leak (SIGABRT under
abort_on_error=1) on any job_init() failure path -- most visibly
libxce test's test_job_init_propagates_makeJobsub_failure.
Verified fixed: fresh rebuild resolves the new rev, full libxce test
suite green, and test_jobj_hardening passes clean under
ASAN_OPTIONS=detect_leaks=1:abort_on_error=1 with no leak stack.
