bug-1463: commit 94f50ce6 (bug-1453 CA-cert-plumbing removal) missed
GetJobOutput's dead tls.Config/loadRootCAs block in request.go, plus
peek.go's SetRootCAs call and serverInfo.CACert field access (the
field no longer exists on APIServerInfo) -- broke the xcecli build.
Removed all three dangling references. Also enabled NewAPIClient's
previously-commented-out InsecureSkipVerify TLS config, since no
xcecli method set any TLS config at all -- any https:// apiserver URL
would have failed cert verification, not just GetJobOutput. Matches
bug-1453's documented design (server-side TLS only, no peer
verification, mTLS deferred as P3 hardening).
Documented both the api-server (http+https) and controller
(server-side-cert-only, no client certs) TLS postures in
.kiro/steering/security.md, replacing a stale line claiming TLS is
handled at the network layer, plus a follow-up note for the planned
HAProxy TLS-termination reverse proxy once its cert type and
apiserver's own listener fate are decided.
xcesched_push_configs() sent two hardcoded limit1/limits2 test
literals to the scheduler (behind a #if 0/FIXME "we use builtin limit
currently"), ignoring whatever was actually loaded from cluster_config.
Migrate those literals into limits.yaml as real entries and wire
conf4sched->nlimits/limits from c->myconf->cluster_config directly.
cluster_init_phase2 also downgrades its "cluster_config not loaded"
ERROR to INFO — a standby legitimately reaching phase2 before the
primary's config-push has delivered bytes yet is normal cold-boot
behavior, not a failure; it already matches load_primary_config's
INFO-level framing for the same condition.
bug-1462: raft_cb_become_leader baselined committed_config_tag to the
majority-safe floor across voters, correct for WAL LSN safety but wrong
for config tags — a leader's own actively-loaded config isn't
speculative like an unacknowledged WAL entry. On a fresh deploy every
peer reports tag=0 except the new primary, so the floor stuck at 0
forever and late-joining standbys never received a config push.
ha_promotion_config_tag_floor() clamps to max(safe_tag, self_tag),
mirroring bug-1060's WAL-truncation clamp pattern. Validated with
20-round sequential (63 jobs, zero loss) and 20-round chaotic
overlapping-kill (zero committed divergence, zero committed job loss)
HA failover soaks; unit suite 977/977 green.
deploy-ha.sh no longer takes a <data-dir> argument or --build-config
flag. Every deploy now generates a fresh self-signed TLS cert/key
(openssl, RSA 2048, 10y validity) directly into the deploy dir's
certs/, and builds cluster.bin directly into controller/ctrl-1/conf/
via xceadmin -- no intermediate staging directory, nothing preserved
across runs. run-ha-soak.sh (/xce-test-ha) updated to match: dropped
its own XCE_HA_DATADIR-based precondition entirely. Updated 4 callers
(run-ha-soak.sh, test-heartbeat-tolerance.sh, test-commit-durability.sh,
both xce-deploy-ha/xce-test-ha SKILL.md docs).
Separately: xcecommon.wire.connect() was loading a client-side TLS
cert, diverging from the real reference client -- netio.c's
create_ssl_context() presents no certificate at all when acting as a
client (cert_file/key_file both NULL, bare TLS_client_method()).
Fixed connect() to match, and since nothing else consumed cert_dir
(verify_mode=CERT_NONE throughout, no CA check either), removed it
end-to-end: discovery.follow_redirects(), push.push_config()/
probe_config(), and the --cert-dir CLI flag.
Server-side cert loading (mock TLS test servers) is unaffected --
moved into a shared session-scoped tls_certs fixture (new
tests/conftest.py, openssl-generated) replacing 3 duplicated
REPO_HA_DATA_DIR/skipif blocks. Net result: those tests no longer skip
under a hermetic nix build sandbox (which never had ha/data/) --
verified 165/165 passed, 0 skipped, via both meson compile and a full
nix build --option sandbox false .#xcecore_debug rebuild (previously
128 passed / 37 skipped there).
ha/data/{xce.crt,xce.key} removed from git tracking. Requirements
2.3/2.4 and 8.3/8.4/8.5 marked superseded in requirements.md with
dated correction notes.
Both meson-build.sh (venv creation) and meson-install.sh (wrapper
shebang) resolved python3 independently at run time (bare `python3 -m
venv` / `command -v python3`) -- correct as long as invoked inside
nix develop/build, but nothing prevented a stray non-nix system
python3 earlier on PATH from shadowing it.
meson.build now resolves python3 once via import('python').
find_installation('python3') -- same mechanism as find_program('go')/
capnp/packcc elsewhere in this build, printed at `meson setup` time
as "Program python3 found: YES (/nix/store/...)" -- and threads that
exact resolved path into both scripts as an explicit argument instead
of a fresh lookup.
Verified: meson setup prints the pinned python3, xceadmin target
rebuilds clean (165/165 passed), and the built venv's own python3
resolves through to that exact nix store path.
deploy-ha.sh's --build-config path (task 4.2) pointed at
src/xceadmin/.venv/bin/xceadmin, broken by today's meson/nix rewiring
(xceadmin no longer builds a venv there). Fixed to search the meson
build dir (xce.build/build.release/build.dbg), a nix-installed
result/bin, then PATH.
/xce-deploy-ha's SKILL.md never mentioned xceadmin or --build-config
(predates task 4.2) — documented both.
/xce-test-ha's run-ha-soak.sh hard-required a pre-existing cluster.bin
under ha/data/, exit 2 if missing, with no xceadmin path at all. Now
always builds cluster.bin fresh from src/configs/*.yaml via xceadmin
before every soak run, same resolution order as deploy-ha.sh.
ha/data/cluster.bin is no longer a checked-in fixture: removed from
git tracking and gitignored. It was stale relative to xceadmin's
output anyway (different hash from a different, older generation
path) -- the xceadmin-built one matches the known-good golden-fixture
hash from admin-config-tool task 5.4.
xceadmin's custom_target only built+tested (stamp-file output, no
install:true) — nix build .#xcecore succeeded but shipped nothing in
result/bin, unlike every other component.
Add meson.add_install_script('meson-install.sh'): copies the
xceadmin/xcecommon package dirs (not the whole build-dir venv, whose
pip-generated shebang hardcodes a build path that stops existing after
the build) into $out/lib/xceadmin-site, then writes a wrapper at
$out/bin/xceadmin whose shebang and PYTHONPATH are resolved against
the install script's own nix build environment — real /nix/store
paths for python3 and each dependency (pycapnp, pyyaml, jsonschema,
click), so nix's reference scanner keeps them alive in the closure.
No venv activation needed at runtime.
Verified: meson install --destdir and nix build --option sandbox
false .#xcecore both produce a working result/bin/xceadmin, runnable
standalone with a stripped-down PATH-only environment.
Reopens the 2026-07-21 design decision to keep xceadmin outside meson.
Standard onboarding consistency (README 3.3 `nix build .#xcecore` and
3.4 `meson compile` working for xceadmin like every other component)
outweighs the earlier concern about meson's single-artifact model not
fitting a multi-file pip install.
- src/xceadmin/meson.build: custom_target() wrapping install+test,
stamp-file output, build_always_stale (matches this repo's existing
clean-rebuild-every-invocation convention).
- src/xceadmin/meson-build.sh: out-of-tree pinned install + pytest;
stages a repo-mirror/ symlink tree so tests' fixed-relative-path
assumptions (src/configs/, src/schema/, ha/data/) still resolve from
a copied location, without touching their deliberate fail-loud
behavior.
- src/meson.build: subdir('xceadmin') under component != 'xcecli'.
- src/tools/xce-build.sh: xceadmin case now delegates to
`meson compile -C "$BUILD_DIR" xceadmin` instead of its own
pip/venv implementation, like every other component.
Verified both build paths: meson compile (165/165 passed) and
nix build --option sandbox false .#xcecore_debug (128 passed / 37
skipped — cert-dependent tests correctly skip under nix's
git-tracked-only hermetic source). AppImage bundling (README 3.5)
is explicitly deferred to future work, not part of this change.
design.md, research.md (new), brief.md and requirements.md (updated
during spec authoring: Req 1.6 resolved/superseded by bundling, Req
5.5 added, Req 4.3/4.4 extended for limits.yaml) had been sitting
uncommitted since before implementation began -- the entire 18-task
implementation was built against these documents without them ever
landing in git.
src/configs/*.yaml + schema/*.schema.json: the example config
directory and bundled JSON schemas most of the test suite reads as
"real repo fixtures" (test_config_build.py, test_mapping.py,
test_validation.py, etc.) were likewise never committed -- those
tests would silently skip (skipif guards) rather than run for anyone
else checking out this branch.
Adds end-to-end propagation tests through push_config's real
composition with xceadmin.discovery (not just discovery.py's own
unit tests): a 4-mock-server redirect chain proving TooManyRedirectsError
surfaces correctly, and a monkeypatched-timeout test proving
NoPrimaryError does too, without waiting out a real 30s socket
timeout. Also locks in and documents a real wire-contract limitation:
ProbeConfigTagReply carries no rejection signal, so dry-run's
accepted=False path is unreachable today by design, not an oversight.
This completes all 18 sub-tasks in tasks.md.
tests/fixtures/{minimal,full}/configs/*.yaml + checked-in expected.bin
golden files, added to test_config_build.py. Catches representation
drift (bytes changing even when decoded content stays correct) that
the existing structural/decode-based assertions can't detect.
5.1 (mapping) and 5.2 (wire framing) are already fully satisfied by
task 2.3's test_mapping.py and task 2.4's test_wire.py -- no new
tests, would have been pure duplication.
5.3: test_schema_bundling.py cross-checks both bundling mechanisms
(capnp schema, JSON validation schemas) resolve consistently under
the active install, and found a real gap: a corrupted bundled JSON
schema crashed validate_yaml() with a raw json.JSONDecodeError
instead of a clean ValidationError. Fixed in xcecommon/validation.py.
Add --build-config (parsed anywhere in the arg list) that rebuilds
<data-dir>/cluster.bin from configs/ via the venv's xceadmin before
deploying; a missing cluster.bin auto-builds the same way even
without the flag. An existing cluster.bin with no flag passed is
left byte-for-byte untouched, matching prior behavior. xceadmin and
the configs source are resolved relative to the script's own
location, not the caller's CWD.
Verified live: missing-file auto-build, untouched-when-present (same
mtime/SHA), and forced rebuild reproducing byte-identical output.
Wire config build/config push --dry-run onto the click group.
Thin layer only: one XceAdminError dispatch surface plus an OSError
catch (ConnectionRefusedError and friends propagate unwrapped from
xcecommon.wire by design and need formatting here), and a
KeyboardInterrupt handler for a clean "cancelled" message (socket
cleanup already happens via push.py's own try/finally).
push_config()/probe_config(): build/send Request{opcode:upgradeConfig}
and the direct-struct ProbeConfigTagRequest respectively, following
redirects to the primary for the former (ProbeConfigTagReply has no
redirect fields, so probe_config connects directly). Maps controller
status/reply-variant failures to typed results/errors. Tested against
a real local TLS mock controller built from the bundled schema, since
neither opcode has a live controller-side handler today (separately
documented, unrelated to this task).
Unblocked mid-task by bug-1461 (commit e53c086): controller.capnp had
a struct-name collision that miscompiled UpgradeConfigRequest.sha256/
.bytes, making this task's core request impossible to construct via
pycapnp until fixed at the schema root.
controller.capnp declared `struct Data $C.mapname("xce_data_t")` (an
OPCode_get query-result tagged union) at file scope, which shadows
Cap'n Proto's builtin Data blob type for every later `: Data` field in
the same file. This silently miscompiled 7 fields across 6 structs as
a struct-pointer type instead of a raw byte blob: UpgradeConfigRequest
.sha256/.bytes, ReplConfigReply.sha256/.bytes, ReplConfigPushRequest
.sha256/.bytes, ReplEnvelope.payload, SnapshotTransfer's chunk
payload, WalCatchupEntry.payload/.batchPayloads, and CtrlHelloRequest
.key. Production C code had already routed around it for years via
hand-rolled raw capn_ptr manipulation (see request.c's own comments);
nothing before now needed the generated codec/accessors to actually
work for these fields.
Renamed the local struct to XceData (capnp-level name only; the
$C.mapname("xce_data_t") C typedef is unchanged) and explicitly
repointed getdata/pushmsg at XceData by name, since those two fields
correctly want the local tagged union. Updated every derived
generated-identifier reference across 19 hand-written files spanning
controller, lib, scheduler, and the Go apiserver/xcecli CGo bindings.
Fixed 5 call sites in lib/request.c that had been typed against the
old workaround alias but now legitimately receive the correct
capn_data type (a pure type-name swap -- Data_ptr/XceData_ptr and
capn_data share the identical {capn_ptr p} layout).
Verified via full rebuild+test: lib (1/1), controller (3/3, incl. the
job-array push regression tests), scheduler (15/15), Go apiserver +
xcecli (all packages green), and a direct pycapnp round-trip proving
UpgradeConfigRequest.sha256/.bytes now encode/decode correctly.
Bug: bug-1461 (.wolf/buglog.json). Unblocks admin-config-tool task 3.3.
Generic, opcode-agnostic redirect-following against RequestReply's
redirectHost/redirectPort. Follows up to 3 hops, displaying each
redirect target; a 4th redirect raises TooManyRedirectsError. A
connection timeout to any hop is interpreted as "no primary
available" and raised as NoPrimaryError naming that hop.
config build pipeline: scan the 7 known YAML filenames, validate
each present file, run cross-file business-rule checks, map to
ClusterConfig, serialize. Missing files become empty sections;
schema-valid-but-unmapped fields and undefined references warn to
stderr instead of failing. write_output() handles -o <path>/-o -
and returns the hex SHA-256 digest (printing it is the CLI's job).
TLS connection + 8-byte native-order length-prefixed framing to the
controller, verified byte-for-byte against request.c's own framing
and myapp.c's client TLS posture (SSL_VERIFY_NONE, no CN check).
Opcode-agnostic: moves raw bytes only, so the same send_request()
serves both OP_upgradeConfig and ProbeConfigTag's direct-struct
variant. Connection always closes on error, including handshake
timeout.
Map validated YAML section dicts into ClusterConfig capnp field
values via a declarative SECTION_MAP driving two generic mapper
shapes (list/scalar). Preserves YAML source order, sources no
timestamp/hostname/username, and is proven byte-deterministic
against both synthetic and real src/configs/*.yaml fixtures.
Load controller.capnp once per process via a cached loader that
resolves identically under editable and pinned installs. Missing or
corrupted schema content converts a pycapnp uncaught-C++-abort into a
clean SchemaNotFoundError via isolated subprocess pre-validation.
Validate operator-authored YAML against the 7 bundled JSON schemas,
enforce reserved-name and duplicate-name rejection, and flag undefined
group/resource cross-references as non-blocking warnings.
Add python3 + pycapnp/pyyaml/jsonschema/click/pytest/ruff (+pip) to the
nix dev shell, wire a new `xceadmin` case into xce-build.sh (schema
sync from src/schema/ and src/configs/schema/, pip install by profile,
pytest), package-data config so bundled schemas survive a non-editable
install, xce-build SKILL.md docs, and ruff format/check hooks matching
this repo's stdin-JSON hook convention (not brief.md's env-var sketch,
which would have silently no-op'd).
Verified live end-to-end inside `nix develop ./src` for both the
default (editable) and release (pinned) profiles; found and fixed 3
real bugs along the way (missing pip, PEP 668 externally-managed-
environment, missing package-data for release-profile bundling) —
logged as bug-1458/1459/1460.
Requirements: 5.5, 8.1, 8.2, 8.6
Editable-installable xceadmin package with a real click-group entry
point (zero subcommands yet) and xcecommon as an independently
importable standalone package.
Requirements: 5.4
Wires all 11 task-6.1 CLI-sourced myconf_t fields into
init_controller_tunables() via the codebase's existing 0-sentinel fallback
idiom ((myconf->x > 0) ? myconf->x : <original hardcoded constant>);
omitting all flags reproduces today's exact hardcoded values.
Resolves both gaps flagged in task 6.1's review: resmgr_inst_delta bridges
to the existing tunables.resmgr_inst field by meaning (confirmed via
zarray_init's delta parameter), and xstrings_delta is deliberately left
unwired with an operator-facing WARN rather than adding a dead field --
the underlying xstring_table_init/kv_params_t has no delta/growth-step
concept at all, unlike the kvmgr-backed jobmgr/workermgr/qmgr/resmgr.
This completes clusterconfig-schema-split -- all required tasks (1.1-5.3)
and optional Requirement 6 tasks (6.1-6.2) are done.
Adds 11 new CLI flags (--jobmgr-nhash/-delta, --workermgr-nhash/-delta,
--qmgr-nhash/-delta, --resmgr-nhash/-delta/-inst-delta,
--xstrings-nhash/-delta) mirroring the existing --vcpus OPT_*+struct
arguments+myconf_t pattern. Zero behavior change until task 6.2 wires
these into init_controller_tunables().
Review flagged two naming/coverage gaps for task 6.2 to resolve:
xstrings_delta has no corresponding cluster_t.tunables field yet, and
resmgr_inst_delta maps to tunables' existing resmgr_inst by name only.
Drops the two cfg.port = ...; struct-literal assignments in
test_commit_barrier_rollout.c and test_config_repl.c (removed field, task
1.2). Repo-wide sweep found no other genuine cluster_config_t literal
referencing a removed field.
Full clean build now succeeds (153/153 targets) and controller_unit_test
passes 976/976 with zero new failures.
Removes SET_DEFAULT_IF_UNSET(c->port, 0, 9990) from fill_default_cluster_config()
in myconf.c and config_repl.c, now dead since cluster_config_t.port was removed
in task 1.2 and cluster_open_listener stopped reading it in task 2.1. All of
src/controller/{cluster,myconf,config_repl}.c now compile clean.
Deletes the dead candidates/ncandidates KV-mirror loop from
cluster_build_config; init_controller_tunables() left untouched (it only
ever read hardcoded constants). cluster.c now compiles clean.
These capnp structs were used only by the jobmgr/workermgr/qmgr/xstrings
and resmgr fields already removed in task 1.1, and were deliberately left
in place at the time since nothing required their removal. Task 1.2 then
deleted their corresponding hand-written C typedefs, breaking the
capnp-generated bindings' $C.mapname references. Closes that gap (new
Requirement 2.5).
Replaces the cluster_config->port fallback (dead after task 1.2) with:
--peers candidate match (unchanged, including single self-referencing
entries for a custom single-node port) -> hardcoded XCE_CONTROLLER_DEFAULT_PORT
(9990) when --peers was never passed at all. The --peers-given-but-local-
name-absent case still returns -EINVAL and refuses to start, unchanged.
Also folds in task 1.3's discovery (HashZArrayConfig/ResmgrConfig capnp
structs left orphaned by tasks 1.1/1.2, breaking codegen) into
requirements.md (new Req 2.5)/design.md/research.md/tasks.md, and fixes
a design.md citation error (validate_peer_count -> xce_parse_peers) found
during task 2.1's review.
Deletes port, ncandidates, candidates/master_candidate_t**, vcpus, jobmgr,
workermgr, qmgr, resmgr, xstrings members from cluster_config_t (and their
now-orphaned supporting typedefs) to match the schema shrunk in task 1.1.
Other .c files still referencing these members intentionally do not
compile yet — that's tasks 2.1/2.2/2.3/3.
Removes port, candidates, vcpus, jobmgr, workermgr, qmgr, resmgr, xstrings
from the ClusterConfig capnp struct plus the now-unused MasterCandidate
struct, per the config-design.md shared-vs-local classification rule.
Cap'n Proto forbids ordinal gaps, so the 13 retained fields are renumbered
to a contiguous @0..@12 (accepted pre-1.0 wire-compat break — see
research.md's "Design Revision 2"). requirements.md/design.md updated to
match: Req 3.1 drops field-number invariance, Req 3.2/4.1 state a pre-spec
cluster.bin is not expected to decode correctly, and a new Req 5.3 covers
regenerating the checked-in ha/data/cluster.bin fixture in a later task.
Also folds in this session's live design revision: Requirement 1.7's port
replacement mechanism is the existing --peers self-reference + a hardcoded
default (no new --port CLI flag), confirmed via interactive review with
the spec owner.
bug-1453: remove unused client-side CA-certificate plumbing. C-side:
drop ca_cert_path from xce_cluster_config_t, do_connect()'s ca_cert
param, --ca-cert flags on worker/scheduler, and controller's cert
population (TLS is server-side only, SSL_VERIFY_NONE everywhere on
the client side already, verified zero SSL_CTX_load_verify_locations
calls in the C tree). Go-side: delete xcecli's self-signed cert
TOFU-pinning flow (cmd/auth/cert/*), CACert field, SetRootCAs/
loadRootCAs, and every call site.
bug-1454: fix xcecli_ctx_t leak in scheduler.c sched_init() — the
unconditional initial ctx_init was never freed before the HA
candidate loop re-initialized it on the first iteration.
bug-1455: sched_init()'s memset(sched, 0, ...) was silently wiping
sched->cluster_cfg that main.c had populated from --peers just before
calling sched_init(), so --peers was always ignored and the scheduler
fell back to the hardcoded localhost:9990 single-host default.
Restructured sched_init() to take cluster_cfg as an explicit parameter
(assigned after the memset) instead of relying on caller-side
ordering with a separate sched_set_cluster_cfg() setter, which is now
removed.
bug-1456: revert an unsafe stack-local xce_client_config_t introduced
while fixing bug-1455 — xcecli_ctx_init() stores that pointer as
ctx->conf and client_failover.c dereferences it for the ctx's entire
life (runtime reconnect), so a per-loop-iteration stack copy would
dangle the moment sched_init() returned. Back to mutating the
caller-owned, process-lifetime config in place.
Verified: full ASAN build (all components), meson test suite 9/9 OK,
manual scheduler testsuite 15/15, Go build/vet/test all pass.
test_persist_truncate.c's three continuation tests called
commit_engine_fini(&ce) before persist_stop(p), backwards from
commit_engine_fini's own documented contract (persist_stop must run
first). With the fsync_worker flusher thread still alive, it could be
mid-flight in commit_on_durable_advance -> commit_advance_locked
(no lock held) while commit_engine_fini concurrently destroyed
ce->lock/ce->cond and nulled ce->p -- a TOCTOU race where the guard
check reads ce->p as valid but a later re-read observes NULL, SEGV at
exactly offsetof(persist_t, committed_lsn) = 0x38.
This was misdiagnosed as caused by g_drive_async_demotion/catchup_offload
(bug-1450/1451): flipping those flags only perturbed thread-scheduling
timing enough to trigger the always-latent race more reliably. The same
crash signature was independently observed on 2026-07-07, before those
flags existed, proving they were never causal.
Fix: call persist_stop(p) before commit_engine_fini(&ce) in all three
tests. teardown_persist's own internal persist_stop becomes a safe
no-op on the second call. No production code changed.
Verified: full ASAN controller_unit_test suite, 976/976 tests, 6 runs
(1 via meson + 5 standalone direct binary runs), zero ASAN reports.
test_demotion_convergence.c has the same commit_engine_fini-before-
persist_stop ordering (line 233/235) against a real persist_start'd
fixture -- not fixed here, flagged for a follow-up sweep.
- ha-async-io.md: document the same-batch endpoint lifetime hazard
(bug-1447) and its ref-whole-batch-before-dispatch fix pattern
- xce-audit-invariants SKILL.md: add object-lifetime-within-a-dispatch-batch
as a 6th audit dimension, so a future pass checks other batch loops
(vtaskmgr drain, peer-list iteration) for the same shape
- product.md: replace the stale "HA consistency family is the current
focus" framing with the actual 2026-07-18 status (umbrella complete,
9-spec validate-impl batch closed, job-array-single-wal-record shipped)
- roadmap.md: correct 5 entries that were stale relative to actual spec
status (ha-consistency-invariants/ha-wal-replication/libxce-failover
pulled out of "In Flight", ha-term-safety and raft-drive-async marked
implemented+GO, raft-sim-conformance and job-array-single-wal-record
given closure annotations) -- struck through, not deleted, per this
file's own additive-update convention
- testing.md: document that meson's aggregate Fail:0 can mask Unity
sub-group failures -- always check the standalone binary's own summary
- src/controller/CLAUDE.md: fix a stale line-citation for the votedFor
atomicity invariant (had drifted onto unrelated post-election code)
/xce-audit-invariants controller found two live violations of the async
I/O thread-role contract: ha_demotion_convergence_fn (ha_convergence.c)
ran WAL truncation + tlog_exit()+execv() directly on the delaymgr thread,
and ha_catchup_continuation_fn (config_repl.c) ran unbounded WAL-streaming
to a standby directly on delaymgr -- the same single thread that drives
heartbeats and election timeouts cluster-wide, so either blocks liveness
for the whole cluster and risks a spurious election.
A prior vtask-pool offload attempt (2026-07-10, bug-1429) deadlocked
controller_unit_test because c->ha.pool is never initialized by the
lightweight unit-test fixtures these functions run under, and was fully
reverted. This fix instead uses the codebase's established slow-lane
pattern (the same mechanism already used for snapshot-send, config-push,
and peer connect) -- a 4th and 5th instance, not a new mechanism.
New g_drive_async_demotion_offload / g_drive_async_catchup_offload flags
ship default OFF, matching the original rollout discipline of the three
reference offloads (shipped OFF, flipped ON later after a dedicated
compatibility audit). Flipping ON was tried during validation and
reproduced two distinct hazards against pre-existing fixtures that never
call slowlane_init: the already-known hang, and a new ASan SEGV in
commit_advance_locked (logged as bug-1452, not root-caused, deferred to
the future flip-to-ON audit). Both flags verified OFF and the full
976-test suite clean, then the committed state re-verified clean against
a live 10-round chaotic and 10-round sequential HA soak (zero job loss,
zero divergence, both PASS).
Raft-drive-async's reconnect-offload burst (Task 2.1/2.2) exposed a
latent race in io_worker's epoll dispatch loop: ha.c's
on_standby_connected() evicts a stale duplicate peer channel via
iomgr_unmonitor()+endpoint_unref(), but epoll_ctl(DEL) only stops
FUTURE epoll_wait() calls -- it does not invalidate an event for that
same endpoint already sitting in the CURRENT batch's events[] array.
A later event in the same batch then dereferences the freed
endpoint_t/SSL*, producing the ASAN SEGV observed deep inside
OpenSSL's ssl3_read_bytes during raft-drive-async's round-9 soak.
The original "concurrent SSL_do_handshake from two io_worker threads"
hypothesis was wrong and is retracted -- iomgr_start() creates exactly
one io_worker thread per iomgr_t.
Fix: extract io_worker_dispatch_batch(), which takes endpoint_ref() on
every endpoint in the batch before dispatching any of them and
endpoint_unref()s each only after the whole batch completes -- the
same lock-free ref/unref bracket already used for bug-1298 and
INVOKE_MSG_HANDLER, just never applied to io_worker's own loop. Pure
atomics, no locks, no blocking added to the hot path.
Adds test_netio_batch_dispatch_uaf.c, which drives the real dispatch
loop with a synthetic epoll_event[] batch (TDD RED confirmed via ASAN
heap-use-after-free before the fix, GREEN after). Verified further
with two independent clean 10-round live sequential HA soaks (33 and
36 jobs, zero loss) and full ASAN suite runs (xcelib_test 294/0,
controller_unit_test 972/0). raft-drive-async's kiro-validate-impl
re-run flips NO-GO to GO.
Only 21 of 86 test_*.c group functions called their own nested
UNITY_BEGIN()/UNITY_END(). Each nested call reset Unity's global
NumberOfTests/TestFailures counters, so controller_test.c's final
`return UNITY_END();` -- the value meson test actually gates on --
only ever reflected whichever small window of tests ran since the
LAST nested reset, silently discarding pass/fail evidence for
everything before it (measured: 16 of 972 total tests in one run).
The same nested-BEGIN calls also caused PASS/FAIL banners to show
the wrong source file for tests in files that don't reset it
themselves (Unity.TestFile just carries over from whichever group
last called UNITY_BEGIN()).
Fix: replace every nested UNITY_BEGIN() with UnitySetTestFile(__FILE__)
(sets file attribution only, no counter reset) and drop the matching
UNITY_END(); add the same UnitySetTestFile(__FILE__) as the first
statement to the other group functions that had neither. The 2
genuinely standalone binaries (test_job_array_sched_push.c,
test_job_array_apiserver_push.c -- each has its own int main(), not
registered via controller_test.c's TEST() macro) are untouched.
Verified: single consolidated summary now reads
"972 Tests 0 Failures 0 Ignored OK", exactly matching the
ground-truth RUN_TEST call-site count across all 83 registered
group files (976 total minus 4 belonging to the 2 standalone
binaries). Every PASS/FAIL banner now names its own true source
file. Full xce-build.sh all regression pass clean (Ok: 9, Fail: 0).
No production code touched -- test-harness plumbing only.
raft_cb_become_leader()'s eligibility gate read
persist_apply_resource_fenced() before this process's own first
cluster_init_phase2()/persist_recover()/replay_wal() call ever ran --
so a node whose first-ever replay hit a resource-exhaustion apply
failure could still become CTRL_PRIMARY, because the fence that
replay would engage didn't exist yet when the gate checked it. Every
LATER promotion evaluation was already correctly protected; this was
a narrow gap specific to a process's single very-first election.
Fix: re-check persist_apply_resource_fenced() immediately after
cluster_init_phase2() completes on first boot, refusing promotion
(same WARN + ha_arm_promotion_fence_timer() + return pattern as the
existing refusal branch) if that first replay just engaged the fence.
Confirmed ha_elect_evaluate_and_signal (the other election path) is
dead in production for the promotion outcome -- every live call site
of handle_request_vote_reply passes vote_granted=0 since bug-828
redirected real grant recording to the record-only, non-signaling
ha_record_vote_reply -- so raft_cb_become_leader is the sole live
promotion path and this fix's scope is complete.
TDD: new test test_first_boot_promotion_evaluates_own_first_replay_fence
wires ha_phase2_hook to actually call persist_recover(), reproducing
production ordering exactly. RED-confirmed against the unfixed code
at the precise predicted assertion; GREEN-confirmed after the fix.
Full controller_unit_test regression (bundles every HA-adjacent
testsuite/test_*.c) green, 0 new failures.
Originally documented (not fixed) in job-array-single-wal-record
task 7.2's Implementation Notes; logged as bug-1445 and fixed here,
outside that spec's own task list per its persist.c-only boundary.
librtti master 7212ba1 -> 53cf11e (revCount 20->21): makeInstance() was
not freeing its calloc'd allocation when a type's .init() returned
negative, causing a 240-byte LeakSanitizer leak (SIGABRT under
abort_on_error=1) on any job_init() failure path -- most visibly
libxce test's test_job_init_propagates_makeJobsub_failure.
Verified fixed: fresh rebuild resolves the new rev, full libxce test
suite green, and test_jobj_hardening passes clean under
ASAN_OPTIONS=detect_leaks=1:abort_on_error=1 with no leak stack.
sched_handle_dataset()'s Data_data_jobArray case never freed
dataset->rows (the decoded xce_job_submit_t*) after sched_on_job_array()
consumed it -- free_Dataset() is an empty no-op for every table type, so
nothing else owned this call. One bounded leak per delivered array push.
Free it via the existing free_JobSubmit_ptr() teardown on both the
success and error path, mirroring the sibling list-shaped cases'
ownership-transfer convention.
Rewrote test_job_array_expand.c's init_job_array() fixture to
heap-allocate + strdup its fields (matching the real wire-decode shape)
since freeing the old stack/string-literal fixture crashed. Redelivery
test now builds a fresh descriptor per delivery, matching how a real
redelivery is always a freshly-decoded wire message.
Closes task 7.3 (job-array-single-wal-record), the last of the three
post-GO residual gaps reopened 2026-07-16.
replay_wal() now halts logical apply and engages persist_mark_apply_resource_fenced()
on a negative apply_committed_entry() return during boot/restart replay, mirroring
persist_apply_pump_run()'s existing live-path halt-and-fence behavior (task 4.1).
Physical scan bookkeeping (current_lsn, CRC chain) stays unconditional per entry,
consistent with persist_recover()'s own established current_lsn/last_applied_lsn
divergence precedent.
Went through one remediation round: round 1 exercised the wrong ha.c gate (periodic
re-alert timer); round 2 corrected the test to drive the real election-time gate
(raft_cb_become_leader_with_cluster_for_test()), with teeth independently confirmed
via a temporary fence-clear RED/GREEN cycle. Round-2 reviewer APPROVED.
Documents a real, NOT-fixed residual gap found during this task: a node's very
first-ever promotion evaluation in a fresh process runs before that node's own
first persist_recover()/replay_wal() call, so this fence is invisible to that one
promotion check (ha.c:1303-1408 gate runs before ha.c:1427-1439's first-boot
recovery). Every promotion evaluation after a process's first one is protected.
Out of this task's persist.c-only boundary; flagged for a dedicated ha.c follow-up.
restore_job_array()'s bulk-insert pass could fail partway through
(key_add() on realloc growth exhaustion), leaving an already-inserted
prefix permanently visible in c->jobmgr but excluded from
out_new_jobs -- so those elements existed and were individually
addressable but never reached HOOK_DETAIL_JOB_NEW, never scheduled or
pushed, not on this attempt or any later successful retry. A literal
violation of Requirements 2.3 and 9.4, tracked as bug-1443.
Adds kvmgr_reserve()/xzarray_reserve(), new purely-additive primitives
in shared kvmgr.c/zarray.c (following the existing kvmgr_capture_keys()
template -- bare extern function, no RTTI vtable slot, caller-holds-
write-lock contract), that pre-grow c->jobmgr's backing store before
the bulk-insert pass begins, closing the dominant N-scaling failure
trigger structurally.
Found a genuine off-by-one in the existing, unmodified
xzarray_ensure_items(): its strict less-than sufficiency check means
reserving exactly array_size and then performing array_size real
stores still triggers one more growth cycle on the last store.
Reserving array_size + 1 is the exact minimal correct compensation,
independently re-derived and confirmed by review with concrete
arithmetic.
For the now-much-narrower residual case (kvtab_add()'s own small
per-entry allocation, or a content collision -- both non-N-scaling,
honestly documented as still theoretically possible): restore_job_array()
now populates out_new_jobs with whatever prefix was genuinely inserted
before a failure, and apply_committed_entry() fires HOOK_DETAIL_JOB_NEW
for that non-empty partial prefix even when the overall return is
negative, while still propagating the negative rc so the LSN correctly
halts and the Live Apply Resource Fence still engages for the
remainder. A genuinely-inserted element is never left un-pushed again.
Task 7.1 of job-array-single-wal-record (Requirements 2.3, 9.4, 9.6).
Extends the HA soak harness with a new --array-soak mode: small (N=2)
and large (N=10000) array submission sub-rounds, each with one clean
kill/failover, asserting one WAL record per submission regardless of
array size, per-element addressability (status/cancel/completion)
spread across low/mid/high indices post-failover, and a real
millisecond-resolution replay-time bound comparing large-array vs
small-array kill-to-addressable latency.
waldump.c gains a WAL_JOB_ARRAY_SUBMIT case in wal_type_name() so the
scenario can grep an exact type name for its live WAL-record-count
delta check.
Went through one remediation round: the replay-time measurement
initially used whole-second resolution (collapsed to 0s/n-a on a real
run) and never asserted on the comparison, giving zero actual evidence
for the O(1)-replay-time requirement. Fixed with $EPOCHREALTIME
(millisecond resolution -- this sandbox's BusyBox date silently
mangles the more common `date +%s%3N` idiom) and a real bound
assertion that hard-fails the run on a genuine O(N) regression,
independently verified to have real teeth.
Live-verified 3 times with genuinely discriminating numbers each time;
final run: small=210ms, large=163ms -- large-array replay was not
slower, live evidence of the feature's O(1) claim.
Task 6.1 of job-array-single-wal-record (Requirements 4.3, 5.2, 5.3,
8.2, 8.3). Depends on and reuses task 6.0a's own soak verification.
replay_wal() (persist.c ~3902) tallied *new_job_submits_out only for
hdr.type == WAL_JOB_SUBMIT. WAL_JOB_ARRAY_SUBMIT entries were
dispatched and correctly materialized N real jobmgr entries via
apply_committed_entry()'s array case, but never counted --
persist_verify_apply_correctness() then saw real jobmgr growth exceed
the undercounted expectation and declared APPLY_DIVERGENT, engaging a
PERMANENT, non-self-healing fence (distinct from task 4.2's
self-healing Live Apply Resource Fence).
Live-reproduced during task 6.1's soak attempt on 2 of 3 nodes in a
single small-array (N=2) sub-round via two different trigger paths.
Deterministic, not size-dependent -- the first array submission to a
live cluster would fence any node that restarts or hits an
election-path recovery fallback before its next checkpoint, shrinking
the promotable pool. A pre-existing comment at persist.c ~3863-3874
(predating this spec) had already flagged this exact code region as
needing re-verification before tasks 4.1/6.1+ went live; this is that
re-verification, and it failed.
Fix adds a WAL_JOB_ARRAY_SUBMIT branch tallying apply_rc (the genuine
per-record element count apply_committed_entry()'s array case already
returns) when apply_rc > 0, mirroring the existing dedup-safe pattern
for WAL_JOB_SUBMIT. Traced the full return-value space of
restore_job_array()/apply_committed_entry()'s array case to confirm
every tallied value is provably equal to real jobmgr growth -- the fix
cannot overcount and so cannot mask a genuine divergence.
Verified via a from-scratch live 3-node HA soak re-run (--array-soak),
independently by both implementer and reviewer: zero APPLY_DIVERGENT
across all three original trigger paths, one-WAL-record/one-commit-
barrier confirmed for N=2 and N=10000, per-element post-failover
status/cancel/completion confirmed.
Task 6.0a of job-array-single-wal-record (Requirements 3.1, 3.2, 4.1,
9.4). Inserted mid-run after live soak testing surfaced the bug; user
confirmed sequencing (fix now, blocking task 6.1+).
