A restarted standby could get permanently stuck unable to catch up,
with the primary re-streaming the identical doomed AppendEntries range
every heartbeat with zero progress -- discovered via a live 3-node
config-upgrade-snapshot soak. Liveness-only; no safety invariant
(Election Safety, Leader Completeness, State Machine Safety, Commit
Durability) was ever violated.
Three-layer root cause, all now closed:
1. ha_stream_wal_to_standby (config_repl.c) used to skip advancing its
prev_streamed_lsn/prev_streamed_term tracker across WAL_CHECKPOINT
entries (a local-only, never-replicated marker that still consumes
a real LSN slot). Now advances across them too, while still never
shipping a catchup entry for the checkpoint itself.
2. ha_apply_term_entry_via_library (ha.c) now JIT-resyncs
ha->raft.log_len from persist_durable_floor() before consuming an
AE, to correct for persist_write_checkpoint_marker advancing
current_lsn without going through raft_log_append/record_entry_term
(pre-existing gap, bug-1466, not itself fixed here).
3. The actual root cause, found via real instrumentation after layers
1-2 alone still left a regression test red: record_entry_term's
array-growth realloc zero-filled new slots, silently reclassifying
a checkpoint's deliberately-never-recorded gap slot from "absent"
to "present, term=0/WAL_LEGACY_TERM" the instant any later LSN got
recorded -- a time-dependent read where a permanent fact was
assumed. Fixed with a TERM_INDEX_UNSET (UINT64_MAX) sentinel that
growth now fills gap slots with instead of zero, so entry_term_locked
can distinguish "never explicitly written" from "explicitly
recorded as legacy term 0" regardless of later array growth.
The same determinism fix exposed a second, independent instance of the
identical wedge via the snapshot-install catch-up path instead of
WAL-streaming catch-up: persist_reconcile_checkpoint_term recorded a
wire checkpoint_term of 0 as a raw, explicit write (bypassing the
growth-fill sentinel entirely), so a snapshot-installed standby read
the checkpoint LSN back as WAL_LEGACY_TERM(1) forever while the leader
now correctly read its own checkpoint LSN as absent(0) -- the same
permanent logOk mismatch, a different code path. Fixed by substituting
TERM_INDEX_UNSET for a wire value of exactly 0 before recording; a
wire 0 is unambiguous (persist_entry_term's own contract never returns
raw 0 for a present entry, so 0 can only mean genuine absence).
Independently reviewed across three rounds: sentinel mechanism
verified via real RED->GREEN revert-and-rebuild; a documentation
claim that config_repl.c/ha.c were "comment-only" corrected; the
snapshot-path gap independently traced, confirmed real via its own
RED->GREEN cycle, and fixed; two stale doc comments (persist.h's
public contract, ha.c's call-site rationale) still describing the
disproven pre-fix legacy-0 convention corrected. Suite: 1028 -> 1035
tests, 0 failures.
repl_add_standby()'s "find first empty slot" scan treated any ep==NULL
slot as available, with no awareness that a disconnected slot can still
hold an unexpired bug-1147 WAL-retention guard (last_disconnected_lsn,
set by repl_remove_standby, meant to be cleared only by
repl_expire_disconnect_slots once snap_interval_s genuinely elapses).
The scan could pick a guard-holding slot over a genuinely virgin one
purely by lower index, then unconditionally memset it -- destroying the
guard before trim_wal_segments/repl_trim_lsn could ever honor it.
Confirmed via 2/2 reproducible live 3-node soak runs (config-upgrade-
snapshot task 8.9's new --config-upgrade-wal-trim mode): a killed
standby's still-needed WAL segment got trimmed out from under it.
Now a two-pass scan: prefer a genuinely virgin slot; only fall back to
reusing a guard-holding slot (oldest last_disconnect_time first, to
discard the least remaining protection) when no virgin slot exists --
logged via WARN() so the discarded guard is operator-visible rather
than silent. Defense-in-depth at the slot-allocation layer, independent
of whatever triggers a given repl_add_standby call (a separate,
not-fully-root-caused ha.c reconnect-sweep defect is the specific
trigger seen in the soak, out of this fix's scope).
Independently reviewed across two rounds: round 1 flagged the
exhausted-fallback path silently discarding a guard with zero logging;
remediated with a WARN() matching this file's existing repl_broadcast
convention, empirically confirmed to fire only on the fallback path
with correct content. Suite: 1028 -> 1030 tests, 0 failures (net of
this fix; final combined total with bug-1472 landing concurrently:
1033 tests, 0 failures).
worker_from_capnp()'s idempotency guard returned -EEXIST unconditionally
whenever the target worker already existed, discarding the decoded WAL
payload entirely. Correct for a duplicate WAL_WORKER_CTRL/JOIN replay,
but wrong for HOOK_DETAIL_WORKER_UPDATE (hostctrl -c/-o admin close/open):
the target existing is the normal case there, yet the status change was
silently dropped on every member other than the one that processed the
RPC directly -- a permanent, unconditional, cluster-wide replication gap
for host admin-status, confirmed via byte-precise checkpoint divergence
on a live 3-node soak.
Now decodes the entry's hook_detail before applying the existing-worker
guard: JOIN keeps its no-op short-circuit; UPDATE replaces (not ORs,
since HOST_OPEN must clear closed bits) the existing worker's
XCE_WORKER_ADMIN_MASK bits with the decoded payload's, in place, under
the worker's own write lock. HOOK_DETAIL_WORKER_DEAD is deliberately
left unfixed -- its replication would need fields (resvals) that never
round-trip through the Worker capnp wire schema at all, and this
codebase already treats liveness state as locally-derived rather than
WAL-replicated elsewhere -- filed separately as bug-1472-remainder.
Independently reviewed: RED->GREEN cycle re-reproduced from scratch,
replace-vs-OR semantics and lock discipline verified directly, DEAD
deferral corroborated against an independent design-invariant comment
this fix didn't originally cite. Suite: 1030 -> 1033 tests, 0 failures.
Add regression test proving Req 4.4/7.2: a second WAL_CONFIG_MARKER
committing while the first's forced checkpoint is still in flight
(simulated via the checkpoint_in_progress CAS across both markers'
commit+apply) cleanly supersedes it, with the settled state reflecting
only the newer marker. Verified via a field-aware byte comparison
(post-checkpoint_lsn) against a non-racing control fixture, plus
mid-race assertions that materialization is still incomplete at that
point (cfgref_present false, fence set) despite activation already
tracking the newer config. Test-only; production mechanism (already
implemented) unmodified. Suite: 1027 -> 1028 tests, 0 failures.
Prove Requirements 5.2/5.4/5.5/6.3 via deterministic fault injection at
the two critical upgrade windows, entirely at the unit level using the
XCE_TESTING hooks earlier tasks built for exactly this purpose -- no
live soak needed.
test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization
extends the existing fault-hook seam test: after simulating a crash
before the checkpoint write durably completes, it boots a genuinely
fresh persist_t/cluster_t against the same on-disk WAL via
persist_recover() (a real boot-replay simulation, not a same-process
re-call) and confirms materialization resumes and completes correctly.
test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain
proves a promoted member's materialization completes identically to a
non-promoted control member's, by draining each through its actual
production entry point and comparing checkpoints byte-for-byte -- a
regression guard against any future role-conditional branch creeping
into the marker-apply path.
Build the live 3-node chaotic-multi-kill config-upgrade soak scenario
(run-ha-soak.sh --config-upgrade) required by Requirements 4.1, 4.3, 6.1,
6.4. This needed genuinely new client-side infrastructure: a hand-rolled
capnp send path for OP_upgradeConfig (the auto-generated encoder is a
no-op for its Data fields, worked around the same way
xce_replconfigpush_req_raw already does), plus cmd_hostctrl/cmd_upgradeconfig
CLI commands.
Building and running this live surfaced two real production bugs, both
fixed here:
- bug-1468: an unrelated prior commit inverted OPCode_upgradeConfig's
primary/standby dispatch guard in cluster.c, making config upgrades
unreachable from any network client.
- bug-1469: apply_committed_entry's WAL_CONFIG_MARKER case never called
config_activate_committed(), so the node that originates a config
upgrade never refreshed its own in-memory state as part of committing
it. Killing that node before some later, unrelated event happened to
refresh it left its checkpoint permanently stale -- the first live,
empirical reproduction of bug-857's originally hypothesized failure
mode. Fixed by activating before the marker-triggered checkpoint write,
since checkpoint content is built from live runtime tables that only
reflect the reload after activation runs.
A separate, narrower checkpoint divergence (bug-1471, not config-related)
remains open and still blocks a clean soak PASS -- task 8.6 stays
in-progress until it's resolved.
Prove Requirement 6.2 with a real, code-backed mechanism instead of the
originally hypothesized lost_and_found re-homing, which turned out to be
aspirational and never implemented (confirmed by exhaustive grep at both
the pre-fix baseline commit and HEAD — bug-857 itself was flagged in
buglog.json as never empirically reproduced).
test_bug857_regression.c demonstrates both halves of the requirement
entirely within controller_unit_test, no live soak or old-commit build
needed: schedule_checkpoint's pre-existing WAL-threshold trigger is
decoupled from the config-marker boundary and produces genuinely
divergent per-member checkpoint content, while the marker-triggered
checkpoint this spec added converges members regardless of differing
per-member processing pace.
design.md's Requirements Traceability table and Testing Strategy section
are corrected to match; the redirection reasoning is recorded in
tasks.md's Implementation Notes for the benefit of tasks 8.6-8.9, which
still need live multi-process HA-soak infrastructure for Requirement 6.1
that this task deliberately did not build.
Build configcompare_paths(), an offline sibling to snapdiff/walcompare that
recomputes each member's active-config sha256 fresh from its own on-disk
workdir/name/conf bytes and reports identical vs. divergent (Req 4.2).
Wire it as a cmd_configcompare CLI subcommand in test.c, matching its
siblings, after review caught it shipped as an unreachable library function
in round 1.
Extend snapverify to match the current on-disk checkpoint format: bump
SNAP_FORMAT_VERSION to 2, replace the stale raw queue-frame decode with the
capnp length-prefixed walk, add host-frame and config-bytes-section
structural walks, and decode (not just presence-check) the header's
cfgref tag/sha. Without this fix snapverify rejected every checkpoint
written by the current codebase.
do_upgrade_config_with_bytes drops step 9 (wait-for-checkpoint,
restart_requested=1, cbuf_put stop signal) -- materialization now
happens entirely via task 5's apply_committed_entry WAL_CONFIG_MARKER
case on every member's own apply, no restart required. Confirmed
do_upgrade_config_with_bytes was the sole production setter of
restart_requested; cluster.c's read of it is now permanently
false-for-this-trigger, which is the intended behavior.
Adds xce_upgrade_marker_committed_hook (#ifdef XCE_TESTING, mirrors
xce_snapshot_chunk_hook's nullable-function-pointer shape), firing
right after config_commit_change succeeds -- a deterministic seam for
later leadership-change fault injection (task 8.7) at exactly the
marker-commit-to-every-member-installed window. NULL by default, true
no-op in production.
Task 7 of 19.
Requirements: 1.2, 1.3, 1.4, 2.2, 5.4, 6.3
Wires task 6.1's upgrade_materialization_verify/upgrade_materialization_fenced
into ha_dispatch_gate's CTRL_PRIMARY branch (sequential block alongside,
not merged into, the existing config_serve_fenced check -- same
replication/read-only opcode allowlist, refuses state-changing ops
with -XCE_ERR_NOT_READY while fenced) and the promotion gate
(raft_cb_become_leader refuses to become CTRL_PRIMARY while fenced,
introduces no new WAL/log-completeness eligibility mechanism, never
self-terminates). Adds a 4-function self-triggering retry trampoline
(upgrade_materialization_retry_fn/_vtask_entry/_delaymgr_trampoline/
schedule_upgrade_materialization_retry) mirroring
config_activate_retry_fn's exact shape, hooked into
ha_cb_commit_advanced_stub alongside the existing config-byte retry.
Fixes a real bug caught on review: the fence had no production path
that could ever engage it from its zero-initialized default -- every
setter call site was gated on the fence already being set (closed
loop), making the new gate checks permanently dead code. Fixed with
two unconditional upgrade_materialization_verify() call sites:
persist.c's apply_committed_entry WAL_CONFIG_MARKER case (right after
persist_checkpoint_marker_triggered returns, regardless of its rc) and
myapp.c's boot sequence (right after config_activate_committed, for
Req 5.2 crash/restart resume). New test drives the fence through the
real production trigger (a forced -EBUSY checkpoint miss on a real
marker commit+apply), not a direct verify() call, proving the
end-to-end path actually works. Task 6.2 of 19.
Requirements: 3.1, 3.2, 3.3, 3.4, 4.3, 5.5
Adds a new, independent fence (ha_state_t.upgrade_materialization_fence)
distinct from the existing config-byte serve-fence: that fence tracks
whether the locally-held config bytes hash-match the committed marker;
this one tracks whether THIS MEMBER'S OWN upgrade-snapshot
materialization (task 5's marker-triggered checkpoint) has actually
completed for the current committed marker, with the live on-disk
config bytes still hash-verifying against it right now.
upgrade_materialization_verify(c) mirrors config_activate_committed's
shape: reads the current committed marker, checks whether
persist.cfgref_present/cfgref_tag/cfgref_sha already reflect it (stamped
by every checkpoint via scan_latest_committed_config_ref -- proof this
member's own materialization ran for this exact marker), and separately
re-reads the LIVE on-disk config bytes fresh on every call (never a
cached/embedded copy) to hash-verify them too. Both must hold for the
fence to clear. Logs loud only on the 0->1 transition, never
exits/aborts, always fail-loud + fence + stay alive + retriable.
upgrade_materialization_fenced(c) is the trivial atomic-read accessor.
Not yet wired into the dispatch gate, promotion gate, or an automatic
retry trigger -- that's task 6.2. Task 6.1 of 19.
Requirements: 1.8, 3.2, 3.4
Adds an explicit case WAL_CONFIG_MARKER to apply_committed_entry,
replacing its no-op treatment, that forces a checkpoint at exactly the
marker's own (term, lsn) via two new helpers:
persist_checkpoint_write_locked(p, checkpoint_lsn) -- the checkpoint
core (pin lsn, scan config ref, write_snapshot, trim, rotate) taking
an explicit LSN and never acquiring apply_pump_lock itself
persist_checkpoint_marker_triggered(p, lsn) -- shares the existing
checkpoint_in_progress CAS guard, calls the above with the marker's
own lsn
This fixes two hazards design.md's "call persist_checkpoint() directly"
text didn't account for: (1) apply_committed_entry's live-pump caller
(persist_apply_pump_run, used by both primary and standby online-apply)
already holds apply_pump_lock across the dispatch call -- a direct
persist_checkpoint() call would self-deadlock on that mutex; (2)
last_applied_lsn/replay position only advance AFTER apply_committed_entry
returns at both call sites, so pinning checkpoint_lsn from
last_applied_lsn inside the marker case would label the checkpoint with
the previous entry's LSN, not the marker's own -- fixed by threading the
marker's own lsn parameter through explicitly. A -EBUSY outcome (a
concurrent periodic checkpoint mid-flight) is logged at TRACE and treated
as expected/self-resolving, never retried. The new case has no effect on
entity interpretation/re-homing (Req 1.8).
New test_upgrade_snapshot_apply.c drives WAL_CONFIG_MARKER through the
REAL production call paths (persist_apply_pump_run and boot replay via
persist_recover), not a bare apply_committed_entry() call, so the fix is
actually proven against the lock-holding context rather than assumed.
Also fixes a genuine regression in test_config_repl.c surfaced by
WAL_CONFIG_MARKER apply having a real side effect for the first time
(bug-1466): an fsync-worker-thread race (t42_wait_applied polling helper,
mirrors test_apply_pump.c's tap_wait_applied) and one test call site
trusting an unreliable raft-derived LSN. Surfaces (but does not fix, out
of this spec's boundary) a pre-existing ha-term-safety/raft-log-replication
gap: persist_write_checkpoint_marker bypasses ha_primary_log_append,
breaking its documented current_lsn==log_len+1 invariant once a
checkpoint interleaves with primary appends -- log_len feeds real
RequestVote freshness comparisons, not advisory-only as first assumed;
severity corrected in bug-1466. Task 5 of 19.
Requirements: 1.1, 1.5, 1.8, 4.4, 5.2, 5.3
load_snapshot_entities' queue-frame decode migrates from the old raw
{name_len,name,status} layout to {payload_len, capnp QueueAdminState
bytes}, still calling the unmodified restore_queue_state. Adds a new
host-frame section decode after queues, dispatching to
restore_host_state (worker.c, task 3) with the same bounds-checking
and restore-before-advance discipline as the existing job/queue loops.
Un-ignores test_snapent_load_snapshot_restores_entities_before_advancing_position
(blocked since task 2.2) -- now passes for real. Also fixes a genuine
regression in test_snapshot_recv_out_of_order_chunks_assemble_correctly
(test_config_repl.c): its synthetic checkpoint-body filler landed on
the new host_count field's offset, tripping the cluster==NULL guard --
pinned to a well-formed host_count=0, matching that test's existing
fmt/job_count/queue_count precedent. Task 4 of 19.
Requirements: 1.10
Adds restore_host_state(cluster_t*, const char*, uint32_t admin_status)
to worker.c: OR's admin bits into an existing worker's status on hit;
on miss, creates a minimal XCE_WORKER_UNAVAIL placeholder via
makeWorker (mirroring the lost_and_found sentinel host precedent),
with a valid non-NULL hostattrs_t so the host's later real join routes
through rejoin_worker (not init_worker) and preserves the restored
admin status. Idempotent under retry (find first, create only if
missing). Standalone -- not yet wired into load_snapshot (task 4).
Requirements: 1.10, 1.11, 5.8
write_snapshot now appends a flat [config_bytes_len][raw bytes]
section after the host frames, reading the locally-held active config
straight off disk (read_active_config_bytes, persist.c) rather than
calling into config_repl.c -- mirrors scan_latest_committed_config_ref's
existing precedent of avoiding a persist.c -> config_repl.c
function-call dependency. Populated on every checkpoint, not only
marker-triggered ones (Req 2.4). No local config yet (early boot)
degrades to an explicit empty section rather than failing the
checkpoint; genuine corruption still fails loud. Task 2.3 of 19.
Requirements: 2.1, 2.3, 2.4, 5.1
collect_queue_frames now encodes each queue as a QueueAdminState capnp
frame (capnp_encode_queue_admin_state, mirroring task 2.1's host-frame
encoder) instead of the raw {name_len,name,status} layout;
write_snapshot's queue trailer writes [payload_len][capnp bytes] per
queue. restore_queue_state and load_snapshot's queue decode are
untouched (task 4's boundary).
Temporarily TEST_IGNORE_MESSAGE's
test_snapent_load_snapshot_restores_entities_before_advancing_position
(m_queues=2 real write+load round trip hits the still-raw loader) --
tracked in tasks.md Implementation Notes, must be re-enabled by task 4.
Matches existing codebase precedent (bug-847) for this pattern. Task
2.2 of 19.
Requirements: 1.9, 5.7
Adds collect_host_frames + capnp_encode_host_admin_state, walking
c->workermgr the same way collect_job_frames/collect_queue_frames do.
Every host currently in the table gets one HostAdminState frame
(status masked to XCE_WORKER_ADMIN_MASK only, no dynamic/liveness
fields), written into write_snapshot after the queue trailer. Task 2.1
of 19.
Requirements: 1.6, 1.7, 1.9
Adds independently-extensible Cap'n Proto frames for the upgrade
snapshot's host/queue admin-state sections (frame-only, not reusing
Worker/QueueConf which bundle dynamic state) and bumps
SNAPSHOT_FORMAT_VERSION 1->2 so a checkpoint written under the prior
raw queue-frame layout is fail-loud rejected rather than
misinterpreted. Task 1 of 19.
Requirements: 1.9, 5.7
xcesched_push_configs() sent two hardcoded limit1/limits2 test
literals to the scheduler (behind a #if 0/FIXME "we use builtin limit
currently"), ignoring whatever was actually loaded from cluster_config.
Migrate those literals into limits.yaml as real entries and wire
conf4sched->nlimits/limits from c->myconf->cluster_config directly.
cluster_init_phase2 also downgrades its "cluster_config not loaded"
ERROR to INFO — a standby legitimately reaching phase2 before the
primary's config-push has delivered bytes yet is normal cold-boot
behavior, not a failure; it already matches load_primary_config's
INFO-level framing for the same condition.
bug-1462: raft_cb_become_leader baselined committed_config_tag to the
majority-safe floor across voters, correct for WAL LSN safety but wrong
for config tags — a leader's own actively-loaded config isn't
speculative like an unacknowledged WAL entry. On a fresh deploy every
peer reports tag=0 except the new primary, so the floor stuck at 0
forever and late-joining standbys never received a config push.
ha_promotion_config_tag_floor() clamps to max(safe_tag, self_tag),
mirroring bug-1060's WAL-truncation clamp pattern. Validated with
20-round sequential (63 jobs, zero loss) and 20-round chaotic
overlapping-kill (zero committed divergence, zero committed job loss)
HA failover soaks; unit suite 977/977 green.
controller.capnp declared `struct Data $C.mapname("xce_data_t")` (an
OPCode_get query-result tagged union) at file scope, which shadows
Cap'n Proto's builtin Data blob type for every later `: Data` field in
the same file. This silently miscompiled 7 fields across 6 structs as
a struct-pointer type instead of a raw byte blob: UpgradeConfigRequest
.sha256/.bytes, ReplConfigReply.sha256/.bytes, ReplConfigPushRequest
.sha256/.bytes, ReplEnvelope.payload, SnapshotTransfer's chunk
payload, WalCatchupEntry.payload/.batchPayloads, and CtrlHelloRequest
.key. Production C code had already routed around it for years via
hand-rolled raw capn_ptr manipulation (see request.c's own comments);
nothing before now needed the generated codec/accessors to actually
work for these fields.
Renamed the local struct to XceData (capnp-level name only; the
$C.mapname("xce_data_t") C typedef is unchanged) and explicitly
repointed getdata/pushmsg at XceData by name, since those two fields
correctly want the local tagged union. Updated every derived
generated-identifier reference across 19 hand-written files spanning
controller, lib, scheduler, and the Go apiserver/xcecli CGo bindings.
Fixed 5 call sites in lib/request.c that had been typed against the
old workaround alias but now legitimately receive the correct
capn_data type (a pure type-name swap -- Data_ptr/XceData_ptr and
capn_data share the identical {capn_ptr p} layout).
Verified via full rebuild+test: lib (1/1), controller (3/3, incl. the
job-array push regression tests), scheduler (15/15), Go apiserver +
xcecli (all packages green), and a direct pycapnp round-trip proving
UpgradeConfigRequest.sha256/.bytes now encode/decode correctly.
Bug: bug-1461 (.wolf/buglog.json). Unblocks admin-config-tool task 3.3.
Wires all 11 task-6.1 CLI-sourced myconf_t fields into
init_controller_tunables() via the codebase's existing 0-sentinel fallback
idiom ((myconf->x > 0) ? myconf->x : <original hardcoded constant>);
omitting all flags reproduces today's exact hardcoded values.
Resolves both gaps flagged in task 6.1's review: resmgr_inst_delta bridges
to the existing tunables.resmgr_inst field by meaning (confirmed via
zarray_init's delta parameter), and xstrings_delta is deliberately left
unwired with an operator-facing WARN rather than adding a dead field --
the underlying xstring_table_init/kv_params_t has no delta/growth-step
concept at all, unlike the kvmgr-backed jobmgr/workermgr/qmgr/resmgr.
This completes clusterconfig-schema-split -- all required tasks (1.1-5.3)
and optional Requirement 6 tasks (6.1-6.2) are done.
Adds 11 new CLI flags (--jobmgr-nhash/-delta, --workermgr-nhash/-delta,
--qmgr-nhash/-delta, --resmgr-nhash/-delta/-inst-delta,
--xstrings-nhash/-delta) mirroring the existing --vcpus OPT_*+struct
arguments+myconf_t pattern. Zero behavior change until task 6.2 wires
these into init_controller_tunables().
Review flagged two naming/coverage gaps for task 6.2 to resolve:
xstrings_delta has no corresponding cluster_t.tunables field yet, and
resmgr_inst_delta maps to tunables' existing resmgr_inst by name only.
Drops the two cfg.port = ...; struct-literal assignments in
test_commit_barrier_rollout.c and test_config_repl.c (removed field, task
1.2). Repo-wide sweep found no other genuine cluster_config_t literal
referencing a removed field.
Full clean build now succeeds (153/153 targets) and controller_unit_test
passes 976/976 with zero new failures.
Removes SET_DEFAULT_IF_UNSET(c->port, 0, 9990) from fill_default_cluster_config()
in myconf.c and config_repl.c, now dead since cluster_config_t.port was removed
in task 1.2 and cluster_open_listener stopped reading it in task 2.1. All of
src/controller/{cluster,myconf,config_repl}.c now compile clean.
Deletes the dead candidates/ncandidates KV-mirror loop from
cluster_build_config; init_controller_tunables() left untouched (it only
ever read hardcoded constants). cluster.c now compiles clean.
Replaces the cluster_config->port fallback (dead after task 1.2) with:
--peers candidate match (unchanged, including single self-referencing
entries for a custom single-node port) -> hardcoded XCE_CONTROLLER_DEFAULT_PORT
(9990) when --peers was never passed at all. The --peers-given-but-local-
name-absent case still returns -EINVAL and refuses to start, unchanged.
Also folds in task 1.3's discovery (HashZArrayConfig/ResmgrConfig capnp
structs left orphaned by tasks 1.1/1.2, breaking codegen) into
requirements.md (new Req 2.5)/design.md/research.md/tasks.md, and fixes
a design.md citation error (validate_peer_count -> xce_parse_peers) found
during task 2.1's review.
bug-1453: remove unused client-side CA-certificate plumbing. C-side:
drop ca_cert_path from xce_cluster_config_t, do_connect()'s ca_cert
param, --ca-cert flags on worker/scheduler, and controller's cert
population (TLS is server-side only, SSL_VERIFY_NONE everywhere on
the client side already, verified zero SSL_CTX_load_verify_locations
calls in the C tree). Go-side: delete xcecli's self-signed cert
TOFU-pinning flow (cmd/auth/cert/*), CACert field, SetRootCAs/
loadRootCAs, and every call site.
bug-1454: fix xcecli_ctx_t leak in scheduler.c sched_init() — the
unconditional initial ctx_init was never freed before the HA
candidate loop re-initialized it on the first iteration.
bug-1455: sched_init()'s memset(sched, 0, ...) was silently wiping
sched->cluster_cfg that main.c had populated from --peers just before
calling sched_init(), so --peers was always ignored and the scheduler
fell back to the hardcoded localhost:9990 single-host default.
Restructured sched_init() to take cluster_cfg as an explicit parameter
(assigned after the memset) instead of relying on caller-side
ordering with a separate sched_set_cluster_cfg() setter, which is now
removed.
bug-1456: revert an unsafe stack-local xce_client_config_t introduced
while fixing bug-1455 — xcecli_ctx_init() stores that pointer as
ctx->conf and client_failover.c dereferences it for the ctx's entire
life (runtime reconnect), so a per-loop-iteration stack copy would
dangle the moment sched_init() returned. Back to mutating the
caller-owned, process-lifetime config in place.
Verified: full ASAN build (all components), meson test suite 9/9 OK,
manual scheduler testsuite 15/15, Go build/vet/test all pass.
test_persist_truncate.c's three continuation tests called
commit_engine_fini(&ce) before persist_stop(p), backwards from
commit_engine_fini's own documented contract (persist_stop must run
first). With the fsync_worker flusher thread still alive, it could be
mid-flight in commit_on_durable_advance -> commit_advance_locked
(no lock held) while commit_engine_fini concurrently destroyed
ce->lock/ce->cond and nulled ce->p -- a TOCTOU race where the guard
check reads ce->p as valid but a later re-read observes NULL, SEGV at
exactly offsetof(persist_t, committed_lsn) = 0x38.
This was misdiagnosed as caused by g_drive_async_demotion/catchup_offload
(bug-1450/1451): flipping those flags only perturbed thread-scheduling
timing enough to trigger the always-latent race more reliably. The same
crash signature was independently observed on 2026-07-07, before those
flags existed, proving they were never causal.
Fix: call persist_stop(p) before commit_engine_fini(&ce) in all three
tests. teardown_persist's own internal persist_stop becomes a safe
no-op on the second call. No production code changed.
Verified: full ASAN controller_unit_test suite, 976/976 tests, 6 runs
(1 via meson + 5 standalone direct binary runs), zero ASAN reports.
test_demotion_convergence.c has the same commit_engine_fini-before-
persist_stop ordering (line 233/235) against a real persist_start'd
fixture -- not fixed here, flagged for a follow-up sweep.
- ha-async-io.md: document the same-batch endpoint lifetime hazard
(bug-1447) and its ref-whole-batch-before-dispatch fix pattern
- xce-audit-invariants SKILL.md: add object-lifetime-within-a-dispatch-batch
as a 6th audit dimension, so a future pass checks other batch loops
(vtaskmgr drain, peer-list iteration) for the same shape
- product.md: replace the stale "HA consistency family is the current
focus" framing with the actual 2026-07-18 status (umbrella complete,
9-spec validate-impl batch closed, job-array-single-wal-record shipped)
- roadmap.md: correct 5 entries that were stale relative to actual spec
status (ha-consistency-invariants/ha-wal-replication/libxce-failover
pulled out of "In Flight", ha-term-safety and raft-drive-async marked
implemented+GO, raft-sim-conformance and job-array-single-wal-record
given closure annotations) -- struck through, not deleted, per this
file's own additive-update convention
- testing.md: document that meson's aggregate Fail:0 can mask Unity
sub-group failures -- always check the standalone binary's own summary
- src/controller/CLAUDE.md: fix a stale line-citation for the votedFor
atomicity invariant (had drifted onto unrelated post-election code)
/xce-audit-invariants controller found two live violations of the async
I/O thread-role contract: ha_demotion_convergence_fn (ha_convergence.c)
ran WAL truncation + tlog_exit()+execv() directly on the delaymgr thread,
and ha_catchup_continuation_fn (config_repl.c) ran unbounded WAL-streaming
to a standby directly on delaymgr -- the same single thread that drives
heartbeats and election timeouts cluster-wide, so either blocks liveness
for the whole cluster and risks a spurious election.
A prior vtask-pool offload attempt (2026-07-10, bug-1429) deadlocked
controller_unit_test because c->ha.pool is never initialized by the
lightweight unit-test fixtures these functions run under, and was fully
reverted. This fix instead uses the codebase's established slow-lane
pattern (the same mechanism already used for snapshot-send, config-push,
and peer connect) -- a 4th and 5th instance, not a new mechanism.
New g_drive_async_demotion_offload / g_drive_async_catchup_offload flags
ship default OFF, matching the original rollout discipline of the three
reference offloads (shipped OFF, flipped ON later after a dedicated
compatibility audit). Flipping ON was tried during validation and
reproduced two distinct hazards against pre-existing fixtures that never
call slowlane_init: the already-known hang, and a new ASan SEGV in
commit_advance_locked (logged as bug-1452, not root-caused, deferred to
the future flip-to-ON audit). Both flags verified OFF and the full
976-test suite clean, then the committed state re-verified clean against
a live 10-round chaotic and 10-round sequential HA soak (zero job loss,
zero divergence, both PASS).
Only 21 of 86 test_*.c group functions called their own nested
UNITY_BEGIN()/UNITY_END(). Each nested call reset Unity's global
NumberOfTests/TestFailures counters, so controller_test.c's final
`return UNITY_END();` -- the value meson test actually gates on --
only ever reflected whichever small window of tests ran since the
LAST nested reset, silently discarding pass/fail evidence for
everything before it (measured: 16 of 972 total tests in one run).
The same nested-BEGIN calls also caused PASS/FAIL banners to show
the wrong source file for tests in files that don't reset it
themselves (Unity.TestFile just carries over from whichever group
last called UNITY_BEGIN()).
Fix: replace every nested UNITY_BEGIN() with UnitySetTestFile(__FILE__)
(sets file attribution only, no counter reset) and drop the matching
UNITY_END(); add the same UnitySetTestFile(__FILE__) as the first
statement to the other group functions that had neither. The 2
genuinely standalone binaries (test_job_array_sched_push.c,
test_job_array_apiserver_push.c -- each has its own int main(), not
registered via controller_test.c's TEST() macro) are untouched.
Verified: single consolidated summary now reads
"972 Tests 0 Failures 0 Ignored OK", exactly matching the
ground-truth RUN_TEST call-site count across all 83 registered
group files (976 total minus 4 belonging to the 2 standalone
binaries). Every PASS/FAIL banner now names its own true source
file. Full xce-build.sh all regression pass clean (Ok: 9, Fail: 0).
No production code touched -- test-harness plumbing only.
raft_cb_become_leader()'s eligibility gate read
persist_apply_resource_fenced() before this process's own first
cluster_init_phase2()/persist_recover()/replay_wal() call ever ran --
so a node whose first-ever replay hit a resource-exhaustion apply
failure could still become CTRL_PRIMARY, because the fence that
replay would engage didn't exist yet when the gate checked it. Every
LATER promotion evaluation was already correctly protected; this was
a narrow gap specific to a process's single very-first election.
Fix: re-check persist_apply_resource_fenced() immediately after
cluster_init_phase2() completes on first boot, refusing promotion
(same WARN + ha_arm_promotion_fence_timer() + return pattern as the
existing refusal branch) if that first replay just engaged the fence.
Confirmed ha_elect_evaluate_and_signal (the other election path) is
dead in production for the promotion outcome -- every live call site
of handle_request_vote_reply passes vote_granted=0 since bug-828
redirected real grant recording to the record-only, non-signaling
ha_record_vote_reply -- so raft_cb_become_leader is the sole live
promotion path and this fix's scope is complete.
TDD: new test test_first_boot_promotion_evaluates_own_first_replay_fence
wires ha_phase2_hook to actually call persist_recover(), reproducing
production ordering exactly. RED-confirmed against the unfixed code
at the precise predicted assertion; GREEN-confirmed after the fix.
Full controller_unit_test regression (bundles every HA-adjacent
testsuite/test_*.c) green, 0 new failures.
Originally documented (not fixed) in job-array-single-wal-record
task 7.2's Implementation Notes; logged as bug-1445 and fixed here,
outside that spec's own task list per its persist.c-only boundary.
replay_wal() now halts logical apply and engages persist_mark_apply_resource_fenced()
on a negative apply_committed_entry() return during boot/restart replay, mirroring
persist_apply_pump_run()'s existing live-path halt-and-fence behavior (task 4.1).
Physical scan bookkeeping (current_lsn, CRC chain) stays unconditional per entry,
consistent with persist_recover()'s own established current_lsn/last_applied_lsn
divergence precedent.
Went through one remediation round: round 1 exercised the wrong ha.c gate (periodic
re-alert timer); round 2 corrected the test to drive the real election-time gate
(raft_cb_become_leader_with_cluster_for_test()), with teeth independently confirmed
via a temporary fence-clear RED/GREEN cycle. Round-2 reviewer APPROVED.
Documents a real, NOT-fixed residual gap found during this task: a node's very
first-ever promotion evaluation in a fresh process runs before that node's own
first persist_recover()/replay_wal() call, so this fence is invisible to that one
promotion check (ha.c:1303-1408 gate runs before ha.c:1427-1439's first-boot
recovery). Every promotion evaluation after a process's first one is protected.
Out of this task's persist.c-only boundary; flagged for a dedicated ha.c follow-up.
restore_job_array()'s bulk-insert pass could fail partway through
(key_add() on realloc growth exhaustion), leaving an already-inserted
prefix permanently visible in c->jobmgr but excluded from
out_new_jobs -- so those elements existed and were individually
addressable but never reached HOOK_DETAIL_JOB_NEW, never scheduled or
pushed, not on this attempt or any later successful retry. A literal
violation of Requirements 2.3 and 9.4, tracked as bug-1443.
Adds kvmgr_reserve()/xzarray_reserve(), new purely-additive primitives
in shared kvmgr.c/zarray.c (following the existing kvmgr_capture_keys()
template -- bare extern function, no RTTI vtable slot, caller-holds-
write-lock contract), that pre-grow c->jobmgr's backing store before
the bulk-insert pass begins, closing the dominant N-scaling failure
trigger structurally.
Found a genuine off-by-one in the existing, unmodified
xzarray_ensure_items(): its strict less-than sufficiency check means
reserving exactly array_size and then performing array_size real
stores still triggers one more growth cycle on the last store.
Reserving array_size + 1 is the exact minimal correct compensation,
independently re-derived and confirmed by review with concrete
arithmetic.
For the now-much-narrower residual case (kvtab_add()'s own small
per-entry allocation, or a content collision -- both non-N-scaling,
honestly documented as still theoretically possible): restore_job_array()
now populates out_new_jobs with whatever prefix was genuinely inserted
before a failure, and apply_committed_entry() fires HOOK_DETAIL_JOB_NEW
for that non-empty partial prefix even when the overall return is
negative, while still propagating the negative rc so the LSN correctly
halts and the Live Apply Resource Fence still engages for the
remainder. A genuinely-inserted element is never left un-pushed again.
Task 7.1 of job-array-single-wal-record (Requirements 2.3, 9.4, 9.6).
Extends the HA soak harness with a new --array-soak mode: small (N=2)
and large (N=10000) array submission sub-rounds, each with one clean
kill/failover, asserting one WAL record per submission regardless of
array size, per-element addressability (status/cancel/completion)
spread across low/mid/high indices post-failover, and a real
millisecond-resolution replay-time bound comparing large-array vs
small-array kill-to-addressable latency.
waldump.c gains a WAL_JOB_ARRAY_SUBMIT case in wal_type_name() so the
scenario can grep an exact type name for its live WAL-record-count
delta check.
Went through one remediation round: the replay-time measurement
initially used whole-second resolution (collapsed to 0s/n-a on a real
run) and never asserted on the comparison, giving zero actual evidence
for the O(1)-replay-time requirement. Fixed with $EPOCHREALTIME
(millisecond resolution -- this sandbox's BusyBox date silently
mangles the more common `date +%s%3N` idiom) and a real bound
assertion that hard-fails the run on a genuine O(N) regression,
independently verified to have real teeth.
Live-verified 3 times with genuinely discriminating numbers each time;
final run: small=210ms, large=163ms -- large-array replay was not
slower, live evidence of the feature's O(1) claim.
Task 6.1 of job-array-single-wal-record (Requirements 4.3, 5.2, 5.3,
8.2, 8.3). Depends on and reuses task 6.0a's own soak verification.
replay_wal() (persist.c ~3902) tallied *new_job_submits_out only for
hdr.type == WAL_JOB_SUBMIT. WAL_JOB_ARRAY_SUBMIT entries were
dispatched and correctly materialized N real jobmgr entries via
apply_committed_entry()'s array case, but never counted --
persist_verify_apply_correctness() then saw real jobmgr growth exceed
the undercounted expectation and declared APPLY_DIVERGENT, engaging a
PERMANENT, non-self-healing fence (distinct from task 4.2's
self-healing Live Apply Resource Fence).
Live-reproduced during task 6.1's soak attempt on 2 of 3 nodes in a
single small-array (N=2) sub-round via two different trigger paths.
Deterministic, not size-dependent -- the first array submission to a
live cluster would fence any node that restarts or hits an
election-path recovery fallback before its next checkpoint, shrinking
the promotable pool. A pre-existing comment at persist.c ~3863-3874
(predating this spec) had already flagged this exact code region as
needing re-verification before tasks 4.1/6.1+ went live; this is that
re-verification, and it failed.
Fix adds a WAL_JOB_ARRAY_SUBMIT branch tallying apply_rc (the genuine
per-record element count apply_committed_entry()'s array case already
returns) when apply_rc > 0, mirroring the existing dedup-safe pattern
for WAL_JOB_SUBMIT. Traced the full return-value space of
restore_job_array()/apply_committed_entry()'s array case to confirm
every tallied value is provably equal to real jobmgr growth -- the fix
cannot overcount and so cannot mask a genuine divergence.
Verified via a from-scratch live 3-node HA soak re-run (--array-soak),
independently by both implementer and reviewer: zero APPLY_DIVERGENT
across all three original trigger paths, one-WAL-record/one-commit-
barrier confirmed for N=2 and N=10000, per-element post-failover
status/cancel/completion confirmed.
Task 6.0a of job-array-single-wal-record (Requirements 3.1, 3.2, 4.1,
9.4). Inserted mid-run after live soak testing surfaced the bug; user
confirmed sequencing (fix now, blocking task 6.1+).
Adds apiserver_push_jobarray() (apiserver.c), mirroring task 5.1's
xcesched_push_jobarray() exactly: same (base jobid, array_size, one
spec) Data.jobArray derivation, same direct xce_request()/OPCode_push
transport. apiserver_job_new_fun() branches on l->count > 1; the
existing single-job path (apiserver_push_joblist()) is byte-for-byte
unchanged.
Additionally, and unlike the scheduler (single target), replicates
apiserver_push_joblist()'s existing per-connected-server fan-out loop
(N_APISERVER_MAX iteration, same lock discipline) -- verified against
2 simulated connected servers, each receiving exactly one push.
New standalone apiserver_push_unit_test binary (same link-collision
rationale as task 5.1's sched_push_unit_test), correctly omitting
egress.c since apiserver_push_joblist() has no WAL-commit filter.
Task 5.2 of job-array-single-wal-record (Requirements 6.2, 6.4).
Adds xcesched_push_jobarray() (sched.c), encoding one compact
(base jobid, array_size, one spec) Data.jobArray descriptor -- reusing
the JobSubmit capnp struct already added to the Data union in task 1.2
-- instead of N individually-encoded job structs. sched_job_new_fun()
branches on l->count > 1 to call it; the existing single-job path
(l->count == 1 -> xcesched_push_joblist()) is byte-for-byte unchanged.
Sends via direct xce_request()/OPCode_push/Request_body_pushmsg rather
than the existing xce_push_data_req() helper, which hard-rejects any
payload outside jobs/workers/resources/resvals and cannot carry a
single non-list descriptor. design.md corrected to record this
transport deviation for task 5.2 to follow.
New standalone sched_push_unit_test binary (registered in
meson.build): sched.c/dataex.c/egress.c were never linked into
controller_unit_test, and two existing test files there ship local
fakes of on_scheduler_join()/notify()/collect_confvars() that would
collide with the real definitions at link time.
Task 5.1 of job-array-single-wal-record (Requirements 6.1, 6.3, 6.4).
New test_job_array_fence.c exercises the full Requirement 9.4-9.6
engage-refuse-clear-recover sequence through real production seams:
forced array-expansion failure -> fence engages -> promotion refused
at a real ha.c gate site -> zero partial elements visible -> a second,
independent node's progress is unaffected -> clearing the failure and
retrying self-heals the fence and re-opens promotion eligibility.
Deliberately drives this through task 3.1's construction-phase
(scratch-build) seam rather than the bulk-insert-phase seam: the
bulk-insert seam has a known, separately-tested residual gap (an
already-inserted element prefix can survive a mid-pass key_add()
failure with no key_del to undo it), so asserting "no partial set"
against it would misrepresent an already-documented open gap as
resolved. That gap remains open and unowned (see tasks.md
Implementation Notes) -- Phase 4 delivers the generic halt/fence/gate
machinery for it, not the partial-visibility guarantee itself.
Task 4.4 of job-array-single-wal-record (Requirements 9.4, 9.5, 9.6).
This completes Phase 4 (Live Apply Resource Fence).
Threads !persist_apply_resource_fenced(&c->persist) as a required
conjunct into both existing promotion-eligibility call sites in ha.c:
the periodic fence-release check (promotion_fence_timeout_fn) and the
election-time check (raft_cb_become_leader). A node with the fence
engaged is refused promotion at both points; once it self-clears
(task 4.2), the node is eligible again with no other action.
Task 4.3 of job-array-single-wal-record (Requirements 9.4, 9.5).
Adds a new atomic LSN field on persist_t (distinct from the existing
boot-only unrecoverable-fence flag) plus three primitives:
persist_apply_resource_fenced(), persist_mark_apply_resource_fenced(),
persist_clear_apply_resource_fence(). Wired into
persist_apply_pump_run(): engages on task 4.1's halt condition, clears
unconditionally on the walk's next successful advance — provably
equivalent to "clears only when the same stuck LSN's retry succeeds"
since the walk is strictly sequential and never advances past a
failed position.
Known residual gap, out of this task's boundary (documented in
tasks.md Implementation Notes): replay_wal() during boot/restart
replay does not yet share this protection, so a node crashing while
fenced can silently re-pass the same failure on restart. Flagged for
a follow-up task, not fixed here.
Task 4.2 of job-array-single-wal-record (Requirements 9.4, 9.5).
persist_apply_pump_run() no longer discards apply_committed_entry()'s
return value. On a negative rc for any WAL type (not just the new
array type), the walk halts at that LSN — no last_applied_lsn advance,
no pending-apply hand-off — logs a loud error, and retries from the
same position on the next kick, mirroring the existing
apply_pump_read_entry failure branch's shape.
Investigated whether this newly turns the pre-existing job-fact
-ENOENT path into a permanent-wedge hazard: it does not, since
c->jobmgr never deletes entries and facts are only ever applied after
their causally-prior submit already succeeded, so a live -ENOENT there
can only mean genuine divergence — halting is the correct behavior.
Task 4.1 of job-array-single-wal-record (Requirement 9.4).
Adds 2 targeted tests after confirming tasks 3.1/3.2/3.3 already
cover most of this task's observable criteria:
- N-way expansion swept across sizes {2,3,100,1000} with exact-
endpoint non-leakage checks (existing coverage only tested size 5)
- Requirement 5.2 (individually addressable exactly as a per-element
job) exercised through the real production command dispatch
(job_ctrl(), not the raw key_find() every existing test used) --
confirms an array-expanded element can be cancelled by identity,
siblings stay untouched, and an out-of-range jobid is rejected
Idempotent no-op, forced-failure-then-retry, and partial-reapply
were already fully covered by 3.1/3.2 -- no new tests needed.
Requirement 9.4's node-fencing behavior is confirmed out of this
task's boundary (Live Apply Resource Fence, task 4.x); restore_job_
array()'s own atomicity contract that the fence depends on is
already proven. No production code changed -- test-only.
Task 3.4 of job-array-single-wal-record.
Adds case WAL_JOB_ARRAY_SUBMIT to apply_wal_entry() (standby
catch-up dispatch), calling restore_job_array() hook-free, matching
the existing WAL_JOB_SUBMIT case's shape exactly.
Review found apply_wal_entry() has zero production call sites -- a
2026-07-10 refactor (ha_standby_advance_committed_and_apply) moved
standby entity materialization onto the same apply_committed_entry()
pump the primary uses; the code comment claiming otherwise predated
that refactor and was never updated. Requirement 3.1/3.2/4.1 are
fully delivered by task 3.2 alone. This task's wiring is kept as
defensive parity with WAL_JOB_SUBMIT's existing, equally-dead case
in the same switch, not as an independent delivery path -- design.md
and tasks.md corrected accordingly.
Task 3.3 of job-array-single-wal-record.
Adds case WAL_JOB_ARRAY_SUBMIT to apply_committed_entry() (the live
apply pump / boot replay dispatch seam): calls restore_job_array()
(task 3.1) instead of restore_job(), and on success fires
HOOK_DETAIL_JOB_NEW exactly once carrying every newly created
element in restore_job_array()'s own out_new_jobs list -- never once
per element. No hook fires on an idempotent no-op (-EEXIST) or any
genuine construction failure, honestly propagating task 3.1's
documented bulk-insert-phase residual gap rather than papering over
it. apply_wal_entry() (standby catch-up) is untouched -- task 3.3.
Task 3.2 of job-array-single-wal-record.
Adds the shared N-way array expansion primitive: decodes a committed
WAL_JOB_ARRAY_SUBMIT payload, scratch-builds every element via a
throwaway kv-table (never touching c->jobmgr during construction),
then makes the whole set visible in one bulk pass under a single
write-lock acquisition -- no window where a concurrent reader can
observe a partially-constructed array. Idempotent per index (skips
already-existing jobids; whole-record no-op if all exist). Includes
an XCE_TESTING-guarded fault-injection seam. Not yet wired into any
live dispatch path (tasks 3.2/3.3).
Two review rounds: round 1 found the bulk-insert pass silently
claimed success even when key_add() itself could fail on allocation
exhaustion (c->jobmgr grows via realloc, no reservation primitive
exists anywhere in kv.c/kvmgr.c/zarray.c to pre-check this). Fixed
by checking key_add()'s return and reporting failure honestly
instead of a false success. The underlying table has no key_del, so
an already-inserted prefix on this specific failure mode can remain
genuinely visible while the call reports failure -- documented as a
residual gap in design.md and tasks.md for tasks 3.2/4.x to account
for (that prefix is never pushed to scheduler/API server, even on a
later successful retry, since it's never included in out_new_jobs).
Task 3.1 of job-array-single-wal-record.
