Merge the scheduler-join NULL guard from the remote js branch: if the
controller's cluster config is not loaded yet (cold boot / config not
pushed yet), xcesched_push_configs would dereference a NULL cluster_config
and crash the controller (ASan SEGV on scheduler join). Skip the push; the
scheduler receives config on a later join once config is available.
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 3 (B1 dual-state + B2): a job never lingers as RUN when its worker
is lost.
Dual-state B1 (user-approved): RDHUP (worker TCP dropped = confirmed dead)
marks its RUN jobs EXIT immediately (mark_worker_jobs_exit). Heartbeat
timeout (unknown) keeps them UNKWN+TRANSIENT awaiting reconnect
reconciliation; the TRANSIENT TTL GC is the backstop. handle_worker_leave
now calls mark_worker_jobs_exit (was mark_worker_jobs_unknown in Phase 2).
B2 (worker side): resend_job_status now checks the recovered process is
alive (kill(pid,0)) before re-reporting RUN — a dead process is reported
finished so the controller converges to EXIT instead of a stale RUN. And
resend_job_status runs on EVERY rejoin (checking_connection), not just
startup, so a reconnecting worker's jobs are reconciled (clears UNKWN via
the existing notify_jobstart claim path).
Design: docs/Job-Finish-Robustness-设计文档.md §4.4 (dual-state decision).
Unit test: worker RDHUP marks jobs EXIT (not UNKWN).
Phase 2 of the job-finish robustness design (Part B, B1+B3).
B1 (worker liveness): a worker that stops reporting (heartbeat timeout,
already detected by check_worker_timeout_) or disconnects now has its RUN
jobs marked JOB_STAT_UNKWN|JOB_STAT_TRANSIENT (transient_since set, durable
WAL_JOB_UPDATE) instead of lingering as RUN forever. Marking happens
AFTER the worker lock is released (B5: avoids worker->jobmgr inversion
with the scheduler's jobmgr->worker order).
B3 (restart): persist_recover marks all recovered RUN jobs UNKWN+TRANSIENT
in-memory after replay — a controller restart leaves worker connectivity
unknown. In-memory only (no WAL append) because persist_recover may run
on a future standby that must never append.
Unit tests (red->green): worker-loss marks its RUN jobs unknown, other-host
jobs untouched, restart marks all RUN jobs. test_replay_apply_seam updated:
replay still drives facts to RUN, then B3 adds UNKWN+TRANSIENT.
A worker-reported finish may arrive before the job exists: job
construction is deferred to apply_committed_entry (the submit's WAL
entry), leaving a commit->apply window. on_job_finish dropped such
finishes ('can not find job'), so a fast job that finished in that
window stayed RUN forever (e.g. noexec-test/newcfg-test/verify-fix).
Add an in-memory unknown-finish registry: on_job_finish registers the
finish when the job isn't materialized, and apply_committed_entry's
WAL_JOB_SUBMIT case drains it right after restore_job — appending the
durable WAL_JOB_FINISH and mutating the job to DONE/EXIT immediately so
the scheduler never dispatches a job that already finished.
Design: docs/Job-Finish-Robustness-设计文档.md (Part A of A+B; Part B
worker-liveness + reconnect reconciliation deferred to later phases).
Unit tests (red->green): registry register/take, apply-once-job-
materializes (DONE), exit-code maps to EXIT, drain no-op without
registration.
Test fixtures create WAL/snapshot scratch dirs via mkdtemp
(/tmp/xce_e2e_ckpt_*, /tmp/xce_ucaf61_*, /tmp/xce_ts_*) but their
teardown only frees the persist_t — the dir is never removed. persist_init
preallocates a 64MB WAL segment per init, so a full controller_unit_test
run leaves ~28G in /tmp in under a minute, filling the root filesystem.
Remove only xce_* DIRECTORIES at suite start (stale leftovers) and via
atexit (this run). Files are untouched: the user's /tmp/xce-mcp binary
and the suite's own /tmp/xce_test.log survive the -type d filter.
bug-1506 (controller): sched_worker_change pointed w->host.resvals_to_sched
at resvalist.data, then xlist_fini freed it — a concurrent reader (the
WORKER_JOIN hook encoding the worker's resvals) hit an ASAN
heap-use-after-free that crashed the controller and failed every job. Take
ownership of the resval data via MOVE_PTR (mirrors collect_resvals_workers);
the prior array is intentionally leaked (freeing it races the same readers).
bug-1507 (worker): the apiserver's bug-1500 execuser fix sent the submit
user ("admin") as execuser, but that user does not exist in the worker
host's /etc/passwd. systemd failed the worker unit with EXIT_USER (217),
so every job died immediately after submit. job_start_local_task now
checks the execuser with getpwnam() and falls back to "nobody" when the
user does not exist.
Verified with a full job lifecycle (submit -> RUN as nobody -> DONE) on
the clean build: all 4 daemons healthy, zero ASAN.
Co-Authored-By: Claude <noreply@anthropic.com>
The RUSAGE_UPDATE job hook fires with (cluster_t*, xce_job_rusages_t*) as
its varargs — the apiserver and scheduler subscribers parse exactly that
contract — but persist_job_hook_fun's default case treated the second arg
as a job_t* and read j->job.state at offset 56. A xce_job_rusages_t is 56
bytes and shares only jobid@0 with xce_job_t (which is why the jobid read
coincidentally worked), so the state read landed one byte past the struct:
ASAN heap-buffer-overflow (READ of size 4) in thread T7 whenever a job runs
and the worker pushes rusage updates. This was the recurring controller
crash (term 3/4/5) blocking job-PENDING verification.
Add a dedicated HOOK_DETAIL_JOB_RUSAGE_UPDATE case that reads jobid from
the xce_job_rusages_t* and bakes the job's current state from a jobmgr
key_find on the vararg cluster_t* (race-free: the sole caller
update_job_rusages() holds that job's write lock while firing the
AFTER_EVENT hook; re-locking the job inside the hook would deadlock).
Regression test: test_job_hook_rusage_writes_wal_entry now passes a real
56-byte xce_job_rusages_t instead of a full job_t (which masked the
overflow) — ASAN stack-buffer-overflow under the old code, PASS under the
fix. Full controller unit suite: 458 tests, 0 failures.
Co-Authored-By: Claude <noreply@anthropic.com>
Two pre-existing failures (2026-05-13, commit 491c255) surfaced by the
controller unit-test suite (1070 Tests 2 Failures), both in the config
version storage path:
1. conf_trim_old_versions sorted cluster.bin.* versions by
second-granularity st_mtime. Versions saved within the same second
(rapid replication, or the trim test's 15ms-spaced saves) tie, glibc's
stable mergesort keeps creation order, and the newest (current-pointer)
entry lands at the protected keep_n index — so nothing is pruned and
keep_n+1 files remain. Store st_mtim (struct timespec) and compare
tv_sec then tv_nsec.
2. cr_setup() built its tmpdir path from (pid, time(NULL)) at second
granularity; a Unity assertion failure longjmps past cr_teardown(),
leaking the dir, and the next test (same pid+second) silently inherited
the leftover conf/current — cascading a second failure
(test_conf_get_saved_tag_no_config read a real tag instead of 0). Add a
per-invocation counter to the path so no two tests share a dir.
Verified: nix develop ./src --command bash src/tools/xce-build.sh all →
meson 9/9 Ok 0 Fail, controller 1070 Tests 0 Failures, xceadmin 172
passed, frontend Go all ok.
Co-Authored-By: Claude <noreply@anthropic.com>
wal_type_name() never had a case for WAL_CONFIG_MARKER (type 50) -- every
config-change marker entry printed as UNKNOWN instead of a real, defined
type. Added the case plus payload decode (tag/sha256) matching config_repl.c's
config_marker_encode/_decode byte layout, hand-decoded here rather than
linking config_repl.c into xce_test's link graph.
Gap detection also had no way to tell "committed prefix missing!" (real
loss) apart from a normal checkpoint-triggered WAL trim (persist.c's
trim_wal_segments, floor-guarded so it never removes a segment a standby
still needs) -- every long-running node's WAL, once checkpointed, would
false-alarm on every inspection. Cross-checks the sibling data/snap/
directory (snapshot LSN is literally in the filename, snap-<hex>.capnp)
and downgrades the message when a snapshot covers the gap.
Adds waldump_stats_t.leading_gap_explained and 3 new tests plus one
assertion on the existing gap test.
CLAUDE.md's TLA+ Action-to-Code Mapping table gains corrected/new rows:
production vs. dormant AppendEntries send mechanisms, WAL catch-up and
config-push reply integrity, and vote/PreVote failure routing across
all failure timings -- each citation independently re-verified against
live source. The old AppendEntries row's citations had already drifted
onto unrelated code before this task; corrected, not just appended to.
Protocol Verification Checklist gains 3 new items for the guarantees
this feature closes.
ha-term-safety/spec.json's known_limitations entry for the dormant
send-path drift gets an additive RESOLVED pointer back to this spec's
disposition, so a future reader lands on the answer instead of
re-discovering the drift.
Requirement 1 (systematic conformance audit deliverable) complete.
Adds test_batch_payload_persist_failure_propagates_to_whole_batch,
proving persist_submit_job_payloads_async resolves every payload in a
3-entry batch with the same first-failure rc when the first payload's
append fails, rather than letting later payloads independently
succeed (Req 5.3).
Fault injection: chmod the WAL leaf directory to 0500 to force
open_new_segment's EACCES (surfaced as -EIO), restored the instant the
first failure is observed -- a transient fault, not a permanent one,
so the test genuinely distinguishes propagation from independent
per-payload failure. No existing fault hook was reachable from this
code path; this is a real, previously-uncovered gap.
1066 -> 1067 tests, 0 failures. Task 5 (regression-lock tests for
already-shipped/already-conformant behavior) now complete -- Core
phase of raft-adapter-conformance finished (tasks 1-5).
Adds test_5_1_wal_catchup_reject_sends_nack_reply, driving a genuine
(term,lsn) conflict against a real persisted WAL entry through
ha_apply_term_entry_via_library and asserting
ha_send_wal_ack_reply_ex(success=0) fires via ha_wal_ack_send_hook.
Note: test_config_repl.c's test_bug1479_ae_reject_sends_nack_not_silence
already covers the same call site via a different (forward-gap)
stimulus, added in the original bug-1479 commit. design.md's own File
Structure Plan explicitly calls for this new test in this file
regardless, so this is disclosed, design-sanctioned defense-in-depth,
not accidental duplication -- both tests independently confirmed to
fail together when the reply call is removed.
1065 -> 1066 tests, 0 failures.
New test drives repl_broadcast and ha_stream_wal_to_standby through
real production fixtures (reused from tasks 2.3/6.2), asserts both
actually fired (non-vacuous), and asserts
ha_send_append_entries_stub_call_count stays 0 throughout -- the
Requirement 3.3 non-regression guard. RED-verified: manually invoking
ha_cb_send_append_entries_for_test inside the test reliably trips the
assertion; reverted for GREEN.
1064 -> 1065 tests, 0 failures. Task 4 (dormant sender documentation +
guard) now complete.
ha_cb_send_append_entries_stub's doc-comment now plainly states it is
reachable only via raft_send_append_entries (zero production callers,
grep-confirmed), and names repl_broadcast/ha_stream_wal_to_standby as
the actual production send/retry/backoff mechanism (Req 3.3, amended
to permit this documentation-only resolution over retire-or-wire-in).
Adds an XCE_TESTING-only ha_send_append_entries_stub_call_count
counter, mirroring the ha_wal_ack_send_hook idiom, as the guard against
this ever silently regaining a production caller. Task 4.2 will drive
the real production path and assert it stays zero.
1063 -> 1064 tests, 0 failures. Zero behavior change to any production
send path.
Adds pre_vote_ep_err_handler_for_test (XCE_TESTING-guarded) and two
tests: PreVote post-connect failure reaches pre_vote_reply_dispatch's
tally (Req 4.3), and a round mixing that failure mode with a delivered
reply reaches its quorum decision via the fast all-replied path
without the full round timeout (Req 4.5). Real-vote coverage (Req 4.2)
was already proven by task 3.1's own test, cited not duplicated.
pre_vote_reply_dispatch has no mock hook seam, so both tests assert
directly on raft_state_t's tally fields, mirroring test_ha_prevote.c's
existing verification technique.
1061 -> 1063 tests, 0 failures.
Splits vote_ep_err_handler into a real-vote handler and a new
pre_vote_ep_err_handler, each registered at its existing (already
separate) connection-setup call site. Extracts the identical
CTRL_PRIMARY-role cleanup into a shared vote_ep_handle_promoted_disconnect
helper, behavior unchanged.
Each handler's election-phase branch now routes a post-connect failure
(RDHUP/SSL-error/remote-close after connect, before any reply) through
the same production tally the connect-time failure branches already
use (vote_reply_dispatch / pre_vote_reply_dispatch, bug-1423/1424's
fix), instead of the legacy handle_request_vote_reply that never
updated the tally. Closes the one failure window bug-1423 didn't cover.
1060 -> 1061 tests, 0 failures. PreVote-specific test coverage is
task 3.2's scope (pre_vote_reply_dispatch has no hook seam analogous
to xce_vote_reply_hook; verified only by code symmetry + full-suite
regression here).
Adds the remaining 3 of 4 handle_repl_config_push reject-path tests
(malformed request, missing bytes/sha256, conf_save_config failure via
a deterministic ENOTDIR fixture), the EAGAIN-still-accepts test, and
term-adoption-on-accept (task 2.2's own test only covered the reject
case). Requirement 2.4 (unknown-outcome handling) is already covered
by the pre-existing test_wait_all_with_timeout_slow_threads, cited
rather than duplicated.
1055 -> 1060 tests, 0 failures.
myreq_handler's ReplConfigPushReply branch now routes cack->term
through raft_on_higher_term_observed (same entry point the
heartbeat-reply and WAL-NACK vectors use), regardless of accept/reject,
without altering the existing accept/reject branching.
Adds myreq_handler_for_test (XCE_TESTING-only) to drive the real
dispatcher in tests, following the peer_ctrl_msg_handler_for_test
precedent. Requires cluster.c in controller_unit_test_src plus the
established link-satisfying fake stubs for its unrelated switch-case
dependencies (same pattern already used by test_worker_join.c and
test_commit_barrier_reply.c).
handle_repl_config_push now sends a ReplConfigPushReply{accepted,term}
from every exit path instead of silently dropping on the four reject
conditions (malformed request, missing bytes/sha256, sha256 mismatch,
conf_save_config failure). Term is sourced live from c->ha.term.
-EAGAIN activation-defer still replies accepted=1, unchanged.
Adds an XCE_TESTING-only send-hook mirroring ha_wal_ack_send_hook so
tests can observe the reply payload without a live endpoint.
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:
1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
library's nextIndex backoff and immediately repl_start_catchup's the
standby at the corrected position, instead of the 16-strike silent
eviction with no catch-up driver after it. Correct from the start, but
live soak still failed identically: nothing ever produced the NACK this
code was waiting for.
2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
silently dropping the reject. ha_send_wal_ack_reply had hardcoded
success=1 unconditionally -- the only WalAckReply sender in the
controller -- making (1)'s NACK-routing correct but unreachable dead
code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
the NACK without a live endpoint. Live-confirmed the NACK now reaches
the primary -- but the scenario still failed for a third reason.
3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
catch-up range was hard-bounded at committed_lsn, which can never advance
in this exact scenario because advancing it requires an ACK from the very
standby that's stuck catching up (2-of-3 alive, this standby is the only
path to a new majority) -- a self-referential deadlock. Widens the
STREAM_WAL ceiling (and the convergence check gating it) to the leader's
own log tail, matching raft.tla's real AppendEntries semantics
(leaderCommit is advisory, never a send-cap), while reusing bug-1478's
exact guard so this doesn't reopen that trap (never widen onto a trailing
local-only checkpoint's own unstreamable LSN).
Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A node that takes myapp_run's skip_election passive path (bug-849 --
"leader already known after the startup stagger") makes exactly ONE
cluster_init_phase2 attempt. All 5 call sites (myapp.c's skip_election
block, raft_cb_become_leader, raft_cb_become_follower,
ha_elect_evaluate_and_signal's two branches) latched ha->phase2_done = 1
unconditionally after calling cluster_init_phase2, discarding its return
value. If that one attempt raced config availability and failed, the node
was permanently stranded: commit_engine_init never ran, so
apply_committed_entry never applied any committed WAL entry for the rest
of the process's life -- and for a skip_election node specifically,
nothing else would ever retry it, since the other 4 call sites only fire
via that node's own raft-library state transitions, which never happen
for a passive standby whose primary keeps heartbeating normally.
Extracted a shared, idempotent retry (ha_phase2_retry_fn, mirrors
upgrade_materialization_retry_fn's shape) plus a delaymgr-scheduled,
self-rescheduling trigger (ha_schedule_phase2_retry) -- self-rescheduling
because, unlike config_activate_retry/upgrade_materialization_retry which
are re-triggered externally by recurring commit-advance events, nothing
else recurs for a node stuck on the passive path. Wired into all 5 call
sites; raft_cb_become_leader additionally refuses promotion on failure
(mirroring the sibling upgrade_materialization_fenced refusal already in
that function -- a promoted primary with an unwired commit engine is far
worse than a delayed promotion).
TDD: 4 new tests, RED confirmed by temporarily reverting the rc-gating.
GREEN: full suite 1044 tests, 0 failures. Live-verified via
run-ha-soak.sh --config-upgrade (the scenario that found this bug): the
retry fires for real (4 failed attempts, then success) and the soak's
checkpoint-identity/config-identity/host-admin-status assertions -- the
things bug-1477 blocked -- all pass.
Two independent call sites duplicated target = max(committed_lsn,
last_written): ha.c's heartbeat/stateReport dispatch (repl_ack_decode_and_
dispatch) and config_repl.c's catchup_derive_target (used by the ACK-driven
catch-up continuation path). Once config-upgrade-snapshot's marker-triggered
checkpoint advances last_written one LSN past an already-committed
committed_lsn (the checkpoint's own consumed, never-replicated LSN slot),
the unconditional max() permanently wedges every standby's catch-up target
at an unreachable value -- the primary re-streams an empty range every
heartbeat forever.
Extracted the shared, pure ha_compute_repl_target_lsn(committed_lsn,
last_written) (ha.h/ha.c), which only falls back to last_written on genuine
committed_lsn==0 bootstrap, and pointed both call sites at it so they can
never disagree again.
Fixing the second call site exposed three pre-existing tests in
test_convergence_verification.c that hardcoded the old buggy formula's
output as their expected target; updated their fixture values (committed_lsn
4->9, matching appended_lsn) to decouple them from the now-fixed edge case,
which already has dedicated pure-function coverage in test_ha_role.c.
GREEN: full controller_unit_test suite = 1040 tests, 0 failures, clean
across multiple runs. Live-verified via run-ha-soak.sh
--config-upgrade-wal-trim (18 jobs, zero job loss, ctrl-1 reconverged 3-of-3
using the preserved floor segment) -- unblocks config-upgrade-snapshot
task 8.9.
myconf->lock (mylock_init/fini wiring) had landed but load_primary_config()
itself was left in its RED-phase reproduction state (unguarded, decoding
directly into the shared myconf->cluster_config field), so the fix's own
regression test failed on resume: thread B raced through while thread A was
parked mid-call.
Wraps the whole free-decode-publish sequence in myconf->lock (acquired first,
released on every return path) and decodes into a local cluster_config_t*
first, publishing into myconf->cluster_config only after full decode and
validation succeed -- closing the UAF/double-free window a live 3-node ASan
soak caught (two concurrent handle_repl_config_push deliveries for the same
standby racing two load_primary_config calls).
Verified 4/4 clean full-suite runs (1036 tests, 0 failures) including the
specific race test.
A restarted standby could get permanently stuck unable to catch up,
with the primary re-streaming the identical doomed AppendEntries range
every heartbeat with zero progress -- discovered via a live 3-node
config-upgrade-snapshot soak. Liveness-only; no safety invariant
(Election Safety, Leader Completeness, State Machine Safety, Commit
Durability) was ever violated.
Three-layer root cause, all now closed:
1. ha_stream_wal_to_standby (config_repl.c) used to skip advancing its
prev_streamed_lsn/prev_streamed_term tracker across WAL_CHECKPOINT
entries (a local-only, never-replicated marker that still consumes
a real LSN slot). Now advances across them too, while still never
shipping a catchup entry for the checkpoint itself.
2. ha_apply_term_entry_via_library (ha.c) now JIT-resyncs
ha->raft.log_len from persist_durable_floor() before consuming an
AE, to correct for persist_write_checkpoint_marker advancing
current_lsn without going through raft_log_append/record_entry_term
(pre-existing gap, bug-1466, not itself fixed here).
3. The actual root cause, found via real instrumentation after layers
1-2 alone still left a regression test red: record_entry_term's
array-growth realloc zero-filled new slots, silently reclassifying
a checkpoint's deliberately-never-recorded gap slot from "absent"
to "present, term=0/WAL_LEGACY_TERM" the instant any later LSN got
recorded -- a time-dependent read where a permanent fact was
assumed. Fixed with a TERM_INDEX_UNSET (UINT64_MAX) sentinel that
growth now fills gap slots with instead of zero, so entry_term_locked
can distinguish "never explicitly written" from "explicitly
recorded as legacy term 0" regardless of later array growth.
The same determinism fix exposed a second, independent instance of the
identical wedge via the snapshot-install catch-up path instead of
WAL-streaming catch-up: persist_reconcile_checkpoint_term recorded a
wire checkpoint_term of 0 as a raw, explicit write (bypassing the
growth-fill sentinel entirely), so a snapshot-installed standby read
the checkpoint LSN back as WAL_LEGACY_TERM(1) forever while the leader
now correctly read its own checkpoint LSN as absent(0) -- the same
permanent logOk mismatch, a different code path. Fixed by substituting
TERM_INDEX_UNSET for a wire value of exactly 0 before recording; a
wire 0 is unambiguous (persist_entry_term's own contract never returns
raw 0 for a present entry, so 0 can only mean genuine absence).
Independently reviewed across three rounds: sentinel mechanism
verified via real RED->GREEN revert-and-rebuild; a documentation
claim that config_repl.c/ha.c were "comment-only" corrected; the
snapshot-path gap independently traced, confirmed real via its own
RED->GREEN cycle, and fixed; two stale doc comments (persist.h's
public contract, ha.c's call-site rationale) still describing the
disproven pre-fix legacy-0 convention corrected. Suite: 1028 -> 1035
tests, 0 failures.
repl_add_standby()'s "find first empty slot" scan treated any ep==NULL
slot as available, with no awareness that a disconnected slot can still
hold an unexpired bug-1147 WAL-retention guard (last_disconnected_lsn,
set by repl_remove_standby, meant to be cleared only by
repl_expire_disconnect_slots once snap_interval_s genuinely elapses).
The scan could pick a guard-holding slot over a genuinely virgin one
purely by lower index, then unconditionally memset it -- destroying the
guard before trim_wal_segments/repl_trim_lsn could ever honor it.
Confirmed via 2/2 reproducible live 3-node soak runs (config-upgrade-
snapshot task 8.9's new --config-upgrade-wal-trim mode): a killed
standby's still-needed WAL segment got trimmed out from under it.
Now a two-pass scan: prefer a genuinely virgin slot; only fall back to
reusing a guard-holding slot (oldest last_disconnect_time first, to
discard the least remaining protection) when no virgin slot exists --
logged via WARN() so the discarded guard is operator-visible rather
than silent. Defense-in-depth at the slot-allocation layer, independent
of whatever triggers a given repl_add_standby call (a separate,
not-fully-root-caused ha.c reconnect-sweep defect is the specific
trigger seen in the soak, out of this fix's scope).
Independently reviewed across two rounds: round 1 flagged the
exhausted-fallback path silently discarding a guard with zero logging;
remediated with a WARN() matching this file's existing repl_broadcast
convention, empirically confirmed to fire only on the fallback path
with correct content. Suite: 1028 -> 1030 tests, 0 failures (net of
this fix; final combined total with bug-1472 landing concurrently:
1033 tests, 0 failures).
worker_from_capnp()'s idempotency guard returned -EEXIST unconditionally
whenever the target worker already existed, discarding the decoded WAL
payload entirely. Correct for a duplicate WAL_WORKER_CTRL/JOIN replay,
but wrong for HOOK_DETAIL_WORKER_UPDATE (hostctrl -c/-o admin close/open):
the target existing is the normal case there, yet the status change was
silently dropped on every member other than the one that processed the
RPC directly -- a permanent, unconditional, cluster-wide replication gap
for host admin-status, confirmed via byte-precise checkpoint divergence
on a live 3-node soak.
Now decodes the entry's hook_detail before applying the existing-worker
guard: JOIN keeps its no-op short-circuit; UPDATE replaces (not ORs,
since HOST_OPEN must clear closed bits) the existing worker's
XCE_WORKER_ADMIN_MASK bits with the decoded payload's, in place, under
the worker's own write lock. HOOK_DETAIL_WORKER_DEAD is deliberately
left unfixed -- its replication would need fields (resvals) that never
round-trip through the Worker capnp wire schema at all, and this
codebase already treats liveness state as locally-derived rather than
WAL-replicated elsewhere -- filed separately as bug-1472-remainder.
Independently reviewed: RED->GREEN cycle re-reproduced from scratch,
replace-vs-OR semantics and lock discipline verified directly, DEAD
deferral corroborated against an independent design-invariant comment
this fix didn't originally cite. Suite: 1030 -> 1033 tests, 0 failures.
Add regression test proving Req 4.4/7.2: a second WAL_CONFIG_MARKER
committing while the first's forced checkpoint is still in flight
(simulated via the checkpoint_in_progress CAS across both markers'
commit+apply) cleanly supersedes it, with the settled state reflecting
only the newer marker. Verified via a field-aware byte comparison
(post-checkpoint_lsn) against a non-racing control fixture, plus
mid-race assertions that materialization is still incomplete at that
point (cfgref_present false, fence set) despite activation already
tracking the newer config. Test-only; production mechanism (already
implemented) unmodified. Suite: 1027 -> 1028 tests, 0 failures.
Prove Requirements 5.2/5.4/5.5/6.3 via deterministic fault injection at
the two critical upgrade windows, entirely at the unit level using the
XCE_TESTING hooks earlier tasks built for exactly this purpose -- no
live soak needed.
test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization
extends the existing fault-hook seam test: after simulating a crash
before the checkpoint write durably completes, it boots a genuinely
fresh persist_t/cluster_t against the same on-disk WAL via
persist_recover() (a real boot-replay simulation, not a same-process
re-call) and confirms materialization resumes and completes correctly.
test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain
proves a promoted member's materialization completes identically to a
non-promoted control member's, by draining each through its actual
production entry point and comparing checkpoints byte-for-byte -- a
regression guard against any future role-conditional branch creeping
into the marker-apply path.
Build the live 3-node chaotic-multi-kill config-upgrade soak scenario
(run-ha-soak.sh --config-upgrade) required by Requirements 4.1, 4.3, 6.1,
6.4. This needed genuinely new client-side infrastructure: a hand-rolled
capnp send path for OP_upgradeConfig (the auto-generated encoder is a
no-op for its Data fields, worked around the same way
xce_replconfigpush_req_raw already does), plus cmd_hostctrl/cmd_upgradeconfig
CLI commands.
Building and running this live surfaced two real production bugs, both
fixed here:
- bug-1468: an unrelated prior commit inverted OPCode_upgradeConfig's
primary/standby dispatch guard in cluster.c, making config upgrades
unreachable from any network client.
- bug-1469: apply_committed_entry's WAL_CONFIG_MARKER case never called
config_activate_committed(), so the node that originates a config
upgrade never refreshed its own in-memory state as part of committing
it. Killing that node before some later, unrelated event happened to
refresh it left its checkpoint permanently stale -- the first live,
empirical reproduction of bug-857's originally hypothesized failure
mode. Fixed by activating before the marker-triggered checkpoint write,
since checkpoint content is built from live runtime tables that only
reflect the reload after activation runs.
A separate, narrower checkpoint divergence (bug-1471, not config-related)
remains open and still blocks a clean soak PASS -- task 8.6 stays
in-progress until it's resolved.
Prove Requirement 6.2 with a real, code-backed mechanism instead of the
originally hypothesized lost_and_found re-homing, which turned out to be
aspirational and never implemented (confirmed by exhaustive grep at both
the pre-fix baseline commit and HEAD — bug-857 itself was flagged in
buglog.json as never empirically reproduced).
test_bug857_regression.c demonstrates both halves of the requirement
entirely within controller_unit_test, no live soak or old-commit build
needed: schedule_checkpoint's pre-existing WAL-threshold trigger is
decoupled from the config-marker boundary and produces genuinely
divergent per-member checkpoint content, while the marker-triggered
checkpoint this spec added converges members regardless of differing
per-member processing pace.
design.md's Requirements Traceability table and Testing Strategy section
are corrected to match; the redirection reasoning is recorded in
tasks.md's Implementation Notes for the benefit of tasks 8.6-8.9, which
still need live multi-process HA-soak infrastructure for Requirement 6.1
that this task deliberately did not build.
Build configcompare_paths(), an offline sibling to snapdiff/walcompare that
recomputes each member's active-config sha256 fresh from its own on-disk
workdir/name/conf bytes and reports identical vs. divergent (Req 4.2).
Wire it as a cmd_configcompare CLI subcommand in test.c, matching its
siblings, after review caught it shipped as an unreachable library function
in round 1.
Extend snapverify to match the current on-disk checkpoint format: bump
SNAP_FORMAT_VERSION to 2, replace the stale raw queue-frame decode with the
capnp length-prefixed walk, add host-frame and config-bytes-section
structural walks, and decode (not just presence-check) the header's
cfgref tag/sha. Without this fix snapverify rejected every checkpoint
written by the current codebase.
do_upgrade_config_with_bytes drops step 9 (wait-for-checkpoint,
restart_requested=1, cbuf_put stop signal) -- materialization now
happens entirely via task 5's apply_committed_entry WAL_CONFIG_MARKER
case on every member's own apply, no restart required. Confirmed
do_upgrade_config_with_bytes was the sole production setter of
restart_requested; cluster.c's read of it is now permanently
false-for-this-trigger, which is the intended behavior.
Adds xce_upgrade_marker_committed_hook (#ifdef XCE_TESTING, mirrors
xce_snapshot_chunk_hook's nullable-function-pointer shape), firing
right after config_commit_change succeeds -- a deterministic seam for
later leadership-change fault injection (task 8.7) at exactly the
marker-commit-to-every-member-installed window. NULL by default, true
no-op in production.
Task 7 of 19.
Requirements: 1.2, 1.3, 1.4, 2.2, 5.4, 6.3
Wires task 6.1's upgrade_materialization_verify/upgrade_materialization_fenced
into ha_dispatch_gate's CTRL_PRIMARY branch (sequential block alongside,
not merged into, the existing config_serve_fenced check -- same
replication/read-only opcode allowlist, refuses state-changing ops
with -XCE_ERR_NOT_READY while fenced) and the promotion gate
(raft_cb_become_leader refuses to become CTRL_PRIMARY while fenced,
introduces no new WAL/log-completeness eligibility mechanism, never
self-terminates). Adds a 4-function self-triggering retry trampoline
(upgrade_materialization_retry_fn/_vtask_entry/_delaymgr_trampoline/
schedule_upgrade_materialization_retry) mirroring
config_activate_retry_fn's exact shape, hooked into
ha_cb_commit_advanced_stub alongside the existing config-byte retry.
Fixes a real bug caught on review: the fence had no production path
that could ever engage it from its zero-initialized default -- every
setter call site was gated on the fence already being set (closed
loop), making the new gate checks permanently dead code. Fixed with
two unconditional upgrade_materialization_verify() call sites:
persist.c's apply_committed_entry WAL_CONFIG_MARKER case (right after
persist_checkpoint_marker_triggered returns, regardless of its rc) and
myapp.c's boot sequence (right after config_activate_committed, for
Req 5.2 crash/restart resume). New test drives the fence through the
real production trigger (a forced -EBUSY checkpoint miss on a real
marker commit+apply), not a direct verify() call, proving the
end-to-end path actually works. Task 6.2 of 19.
Requirements: 3.1, 3.2, 3.3, 3.4, 4.3, 5.5
Adds a new, independent fence (ha_state_t.upgrade_materialization_fence)
distinct from the existing config-byte serve-fence: that fence tracks
whether the locally-held config bytes hash-match the committed marker;
this one tracks whether THIS MEMBER'S OWN upgrade-snapshot
materialization (task 5's marker-triggered checkpoint) has actually
completed for the current committed marker, with the live on-disk
config bytes still hash-verifying against it right now.
upgrade_materialization_verify(c) mirrors config_activate_committed's
shape: reads the current committed marker, checks whether
persist.cfgref_present/cfgref_tag/cfgref_sha already reflect it (stamped
by every checkpoint via scan_latest_committed_config_ref -- proof this
member's own materialization ran for this exact marker), and separately
re-reads the LIVE on-disk config bytes fresh on every call (never a
cached/embedded copy) to hash-verify them too. Both must hold for the
fence to clear. Logs loud only on the 0->1 transition, never
exits/aborts, always fail-loud + fence + stay alive + retriable.
upgrade_materialization_fenced(c) is the trivial atomic-read accessor.
Not yet wired into the dispatch gate, promotion gate, or an automatic
retry trigger -- that's task 6.2. Task 6.1 of 19.
Requirements: 1.8, 3.2, 3.4
Adds an explicit case WAL_CONFIG_MARKER to apply_committed_entry,
replacing its no-op treatment, that forces a checkpoint at exactly the
marker's own (term, lsn) via two new helpers:
persist_checkpoint_write_locked(p, checkpoint_lsn) -- the checkpoint
core (pin lsn, scan config ref, write_snapshot, trim, rotate) taking
an explicit LSN and never acquiring apply_pump_lock itself
persist_checkpoint_marker_triggered(p, lsn) -- shares the existing
checkpoint_in_progress CAS guard, calls the above with the marker's
own lsn
This fixes two hazards design.md's "call persist_checkpoint() directly"
text didn't account for: (1) apply_committed_entry's live-pump caller
(persist_apply_pump_run, used by both primary and standby online-apply)
already holds apply_pump_lock across the dispatch call -- a direct
persist_checkpoint() call would self-deadlock on that mutex; (2)
last_applied_lsn/replay position only advance AFTER apply_committed_entry
returns at both call sites, so pinning checkpoint_lsn from
last_applied_lsn inside the marker case would label the checkpoint with
the previous entry's LSN, not the marker's own -- fixed by threading the
marker's own lsn parameter through explicitly. A -EBUSY outcome (a
concurrent periodic checkpoint mid-flight) is logged at TRACE and treated
as expected/self-resolving, never retried. The new case has no effect on
entity interpretation/re-homing (Req 1.8).
New test_upgrade_snapshot_apply.c drives WAL_CONFIG_MARKER through the
REAL production call paths (persist_apply_pump_run and boot replay via
persist_recover), not a bare apply_committed_entry() call, so the fix is
actually proven against the lock-holding context rather than assumed.
Also fixes a genuine regression in test_config_repl.c surfaced by
WAL_CONFIG_MARKER apply having a real side effect for the first time
(bug-1466): an fsync-worker-thread race (t42_wait_applied polling helper,
mirrors test_apply_pump.c's tap_wait_applied) and one test call site
trusting an unreliable raft-derived LSN. Surfaces (but does not fix, out
of this spec's boundary) a pre-existing ha-term-safety/raft-log-replication
gap: persist_write_checkpoint_marker bypasses ha_primary_log_append,
breaking its documented current_lsn==log_len+1 invariant once a
checkpoint interleaves with primary appends -- log_len feeds real
RequestVote freshness comparisons, not advisory-only as first assumed;
severity corrected in bug-1466. Task 5 of 19.
Requirements: 1.1, 1.5, 1.8, 4.4, 5.2, 5.3
load_snapshot_entities' queue-frame decode migrates from the old raw
{name_len,name,status} layout to {payload_len, capnp QueueAdminState
bytes}, still calling the unmodified restore_queue_state. Adds a new
host-frame section decode after queues, dispatching to
restore_host_state (worker.c, task 3) with the same bounds-checking
and restore-before-advance discipline as the existing job/queue loops.
Un-ignores test_snapent_load_snapshot_restores_entities_before_advancing_position
(blocked since task 2.2) -- now passes for real. Also fixes a genuine
regression in test_snapshot_recv_out_of_order_chunks_assemble_correctly
(test_config_repl.c): its synthetic checkpoint-body filler landed on
the new host_count field's offset, tripping the cluster==NULL guard --
pinned to a well-formed host_count=0, matching that test's existing
fmt/job_count/queue_count precedent. Task 4 of 19.
Requirements: 1.10
Adds restore_host_state(cluster_t*, const char*, uint32_t admin_status)
to worker.c: OR's admin bits into an existing worker's status on hit;
on miss, creates a minimal XCE_WORKER_UNAVAIL placeholder via
makeWorker (mirroring the lost_and_found sentinel host precedent),
with a valid non-NULL hostattrs_t so the host's later real join routes
through rejoin_worker (not init_worker) and preserves the restored
admin status. Idempotent under retry (find first, create only if
missing). Standalone -- not yet wired into load_snapshot (task 4).
Requirements: 1.10, 1.11, 5.8
write_snapshot now appends a flat [config_bytes_len][raw bytes]
section after the host frames, reading the locally-held active config
straight off disk (read_active_config_bytes, persist.c) rather than
calling into config_repl.c -- mirrors scan_latest_committed_config_ref's
existing precedent of avoiding a persist.c -> config_repl.c
function-call dependency. Populated on every checkpoint, not only
marker-triggered ones (Req 2.4). No local config yet (early boot)
degrades to an explicit empty section rather than failing the
checkpoint; genuine corruption still fails loud. Task 2.3 of 19.
Requirements: 2.1, 2.3, 2.4, 5.1
collect_queue_frames now encodes each queue as a QueueAdminState capnp
frame (capnp_encode_queue_admin_state, mirroring task 2.1's host-frame
encoder) instead of the raw {name_len,name,status} layout;
write_snapshot's queue trailer writes [payload_len][capnp bytes] per
queue. restore_queue_state and load_snapshot's queue decode are
untouched (task 4's boundary).
Temporarily TEST_IGNORE_MESSAGE's
test_snapent_load_snapshot_restores_entities_before_advancing_position
(m_queues=2 real write+load round trip hits the still-raw loader) --
tracked in tasks.md Implementation Notes, must be re-enabled by task 4.
Matches existing codebase precedent (bug-847) for this pattern. Task
2.2 of 19.
Requirements: 1.9, 5.7
Adds collect_host_frames + capnp_encode_host_admin_state, walking
c->workermgr the same way collect_job_frames/collect_queue_frames do.
Every host currently in the table gets one HostAdminState frame
(status masked to XCE_WORKER_ADMIN_MASK only, no dynamic/liveness
fields), written into write_snapshot after the queue trailer. Task 2.1
of 19.
Requirements: 1.6, 1.7, 1.9
Adds independently-extensible Cap'n Proto frames for the upgrade
snapshot's host/queue admin-state sections (frame-only, not reusing
Worker/QueueConf which bundle dynamic state) and bumps
SNAPSHOT_FORMAT_VERSION 1->2 so a checkpoint written under the prior
raw queue-frame layout is fail-loud rejected rather than
misinterpreted. Task 1 of 19.
Requirements: 1.9, 5.7
xcesched_push_configs() sent two hardcoded limit1/limits2 test
literals to the scheduler (behind a #if 0/FIXME "we use builtin limit
currently"), ignoring whatever was actually loaded from cluster_config.
Migrate those literals into limits.yaml as real entries and wire
conf4sched->nlimits/limits from c->myconf->cluster_config directly.
cluster_init_phase2 also downgrades its "cluster_config not loaded"
ERROR to INFO — a standby legitimately reaching phase2 before the
primary's config-push has delivered bytes yet is normal cold-boot
behavior, not a failure; it already matches load_primary_config's
INFO-level framing for the same condition.
bug-1462: raft_cb_become_leader baselined committed_config_tag to the
majority-safe floor across voters, correct for WAL LSN safety but wrong
for config tags — a leader's own actively-loaded config isn't
speculative like an unacknowledged WAL entry. On a fresh deploy every
peer reports tag=0 except the new primary, so the floor stuck at 0
forever and late-joining standbys never received a config push.
ha_promotion_config_tag_floor() clamps to max(safe_tag, self_tag),
mirroring bug-1060's WAL-truncation clamp pattern. Validated with
20-round sequential (63 jobs, zero loss) and 20-round chaotic
overlapping-kill (zero committed divergence, zero committed job loss)
HA failover soaks; unit suite 977/977 green.
controller.capnp declared `struct Data $C.mapname("xce_data_t")` (an
OPCode_get query-result tagged union) at file scope, which shadows
Cap'n Proto's builtin Data blob type for every later `: Data` field in
the same file. This silently miscompiled 7 fields across 6 structs as
a struct-pointer type instead of a raw byte blob: UpgradeConfigRequest
.sha256/.bytes, ReplConfigReply.sha256/.bytes, ReplConfigPushRequest
.sha256/.bytes, ReplEnvelope.payload, SnapshotTransfer's chunk
payload, WalCatchupEntry.payload/.batchPayloads, and CtrlHelloRequest
.key. Production C code had already routed around it for years via
hand-rolled raw capn_ptr manipulation (see request.c's own comments);
nothing before now needed the generated codec/accessors to actually
work for these fields.
Renamed the local struct to XceData (capnp-level name only; the
$C.mapname("xce_data_t") C typedef is unchanged) and explicitly
repointed getdata/pushmsg at XceData by name, since those two fields
correctly want the local tagged union. Updated every derived
generated-identifier reference across 19 hand-written files spanning
controller, lib, scheduler, and the Go apiserver/xcecli CGo bindings.
Fixed 5 call sites in lib/request.c that had been typed against the
old workaround alias but now legitimately receive the correct
capn_data type (a pure type-name swap -- Data_ptr/XceData_ptr and
capn_data share the identical {capn_ptr p} layout).
Verified via full rebuild+test: lib (1/1), controller (3/3, incl. the
job-array push regression tests), scheduler (15/15), Go apiserver +
xcecli (all packages green), and a direct pycapnp round-trip proving
UpgradeConfigRequest.sha256/.bytes now encode/decode correctly.
Bug: bug-1461 (.wolf/buglog.json). Unblocks admin-config-tool task 3.3.
Wires all 11 task-6.1 CLI-sourced myconf_t fields into
init_controller_tunables() via the codebase's existing 0-sentinel fallback
idiom ((myconf->x > 0) ? myconf->x : <original hardcoded constant>);
omitting all flags reproduces today's exact hardcoded values.
Resolves both gaps flagged in task 6.1's review: resmgr_inst_delta bridges
to the existing tunables.resmgr_inst field by meaning (confirmed via
zarray_init's delta parameter), and xstrings_delta is deliberately left
unwired with an operator-facing WARN rather than adding a dead field --
the underlying xstring_table_init/kv_params_t has no delta/growth-step
concept at all, unlike the kvmgr-backed jobmgr/workermgr/qmgr/resmgr.
This completes clusterconfig-schema-split -- all required tasks (1.1-5.3)
and optional Requirement 6 tasks (6.1-6.2) are done.
Adds 11 new CLI flags (--jobmgr-nhash/-delta, --workermgr-nhash/-delta,
--qmgr-nhash/-delta, --resmgr-nhash/-delta/-inst-delta,
--xstrings-nhash/-delta) mirroring the existing --vcpus OPT_*+struct
arguments+myconf_t pattern. Zero behavior change until task 6.2 wires
these into init_controller_tunables().
Review flagged two naming/coverage gaps for task 6.2 to resolve:
xstrings_delta has no corresponding cluster_t.tunables field yet, and
resmgr_inst_delta maps to tunables' existing resmgr_inst by name only.
Drops the two cfg.port = ...; struct-literal assignments in
test_commit_barrier_rollout.c and test_config_repl.c (removed field, task
1.2). Repo-wide sweep found no other genuine cluster_config_t literal
referencing a removed field.
Full clean build now succeeds (153/153 targets) and controller_unit_test
passes 976/976 with zero new failures.
