Rewires main.c's live scheduling loop onto the new core-driven
architecture: xce_sched_driver_translate_pending_events() (task 5.1)
-> xce_core_begin_session() -> plugin_begin_session()+sched_account_jobs()
-> xce_core_signal_go() -> decision-drain via the registered
xce_core_decision_cb_t -> plugin_end_session() -> xce_core_end_session(),
all in the new exported xce_sched_driver_run_normal_mode_round()
(driver.c), replacing the old inline sched_update()+sched_begin_session()
+sched_go()+switch+sched_end_session() sequence.
xce_core_go() (task 4.4) is split into xce_core_begin_session()/
xce_core_signal_go() -- a pure decomposition (xce_core_go() itself is
now defined in terms of both, task 4.4's approved test suite passes
unmodified) -- specifically so the driver can interleave
plugin_begin_session()/sched_account_jobs() in the exact safety-critical
window task 4.4 deliberately excluded from pure-core scope: after the
resval reset, strictly before the picker thread is signaled. This is
empirically proven by test_driver_session_ordering.c via real
cross-thread happens-before assertions, not just documented.
xce_sched_driver_on_decision() is the new registered callback:
DISPATCH/PEND forward to the existing sched_publish_dispatch()/
sched_publish_pend_reason(); PREEMPT fails loudly (-ENOSYS, logged)
since no xcecli_preempt() client API exists and nothing in the live
allocator produces one yet (scheduler-preemption's territory).
Adds a --sim CLI flag (parse-only, loud not-yet-implemented failure,
since SimDriver is Phase 3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds driver.h/driver.c: xce_sched_driver_translate_pending_events()
drains core->pend_updates via the existing dataq-swap pattern (same
mechanism sched_update() uses), translates all 10 xce_sched_update_
request_t op types into the matching xce_core_event_t, and pushes the
batch through xce_core_push_events(). Scoped to xce_sched_core_t*
alone -- pend_updates already lives on core since task 4.1, and this
task is translation-only (no decision publishing, no mode-selection,
no recording -- all later work). handler.c (the enqueue side) is
untouched. First task in Group 5 (Core: Normal-Mode Driver).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds test_core_full_cycle.c: drives xce_core_push_events() -> xce_core_go()
-> decision drain -> xce_core_end_session() end-to-end against a real
picker pthread (running the real xce_job_picker_iface_t hooks on a
test-local stub picker) and a main-thread stand-in allocator, since
sched_allocator() is xce_scheduler_t*-coupled and out of pure-core
scope. Proves both preconditions (core_go reentrancy, push_events
mid-session) reject against a genuinely active session and recover
once it ends. Closes out group 4 (Foundation: Scheduler Core
Extraction) -- tasks 4.1-4.6 all complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds xce_core_now() (new core_clock.c, kept separate from core.c so
test_core_isolation.c's wall-clock scan still enforces zero direct
clock_gettime()/time() calls elsewhere in the core): the two-anchor
plus speed formula (virtual_clock_base + (elapsed * speed_num) /
speed_den), covering all four modes (normal, discrete/real-time/fast-
slow simulation) via four new atomic fields on xce_sched_core_t,
defaulted to normal-mode (1/1 speed) in xce_sched_core_init().
requirements.md's literal tlog_reg_format_func(fn, core_ptr) call
shape doesn't exist in the real vendored tlog library (registration
takes only the callback; userptr is per-call-site, not bindable at
registration) -- resolved with a module-level active-core pointer read
by a new core-scoped format function, not wired into mylog_init()/
main.c yet. xce_core_go()/xce_core_end_session() (task 4.4) now stamp
session_start/compute session_duration_ns via xce_core_now(), with
their existing session-active/reentrancy behavior untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds xce_core_go()/xce_core_end_session() (new core_session.c):
xce_core_go() is non-blocking, performs begin-session state reset
(select-eval-ctx, resval resets), signals the picker thread via wbuf,
and returns -EINVAL on a reentrant call while a session is already
active (proven a true no-op, not just an early return). Draining
core->decisions and invoking the driver's on_decision callback is the
driver's own responsibility between core_go() and core_end_session()
(confirmed driver-owned, not core-owned, per the Driver <-> Core
protocol). xce_core_set_decision_cb() registers the callback + driver
context as a separate setter, keeping xce_sched_core_init()'s live
sched_init() call site untouched. Adds XCE_DECISION_PREEMPT to
decision.h (jobid/victims[]/reason) alongside the existing DISPATCH/
PEND types. Additive only — sched_go/sched_begin_session/
sched_end_session/sched_allocator/main.c remain untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Defines xce_core_event_type_t/xce_core_event_t (9 event kinds: config,
job, worker, resource, resval full/delta, job state change, job update,
queue update, job rusage) in core.h. Implements xce_core_push_events()
in a new core_events.c (split out of core.c to stay under the 800-line
cap): rejects -EBUSY mid-session, otherwise applies each event's core-
state mutation and fires the corresponding picker hook synchronously,
exactly once per event, before returning. CONFIG_PUSH decomposes the
pushed config's embedded queues/resources/resvals via shared helpers
(also reused by QUEUE_UPDATE's upsert path), mirroring the existing
sched_init_config()/sched_init_resources()/sched_init_sharedInst()/
sched_init_queues(). Narrows sched_resolve_job_picker() to take
xce_sched_core_t* (pure type change, zero behavior change). Purely
additive — not wired into the live scheduling path yet (task 5.x).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Hardcodes jbuf's cbuf capacity to the literal 1 in xce_sched_core_init()
(no parameter path, no local variable to drift), backed by a regression
test. Audits every kvmgr iteration reachable from the scheduling loop
(jobs/queues/users/projects): only jobs is ever iterated, via xzarray's
deterministic ascending-slot walk with no process-random seeding —
findings recorded durably as a comment above xce_sched_core_init().
core->workers iteration-order stability is flagged as an open follow-up
(scheduler-allocation's territory), not this task's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Splits xce_scheduler_t along the controller-I/O seam: xce_sched_core_t
(new core.h/core.c) now holds all scheduling state (jobs/queues/workers/
users/projects/resource managers/plugins/three-thread-cbuf pipeline),
while xce_scheduler_t shrinks to {core; apictx; cluster_cfg}. Proven via
an nm-based link-time isolation test (zero netio/xcecli/idgen/delaymgr
symbols reachable from core.c) plus a reproducible wall-clock-call guard.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Closes out group 2 (Core: Builtin FCFS Picker). Confirms builtin_fcfs's
ordering matches legacy fcfs_elect_job (byte-identical comparators, same
sched_queue_is_ready/sched_job_is_ready gates, provably equivalent jobid
sequence), confirms tasks 1.1-2.5 left every pre-existing test file
unmodified, and confirms resreq immutability, pend-reason factory-only
allocation, and select_eval_ctx_t memoization all hold across every file
this spec has touched so far. Adds a durable regression lock for the
pend-reason factory invariant (no direct calloc(pendreason_t) outside
make_pendreason()).
Task 2.4 already made sched_job_picker() delegate entirely to the picker
interface with zero direct queue/job traversal, satisfying Req 1 AC 1/AC 7
as a byproduct. This adds a source-text regression test locking that
property in place: asserts sched_job_picker()'s body contains none of the
forbidden direct-traversal symbols and does contain the expected
picker-interface call sites.
Registers builtin_fcfs at scheduler startup and makes it the default
job_pick_plugin, replacing the hardcoded "xce_sched_fcfs" literal in
sched_job_picker() with a fail-loud lookup through sched_resolve_job_picker().
The picker thread now branches legacy job_elect_fn plugins (delegated
unchanged, preserving their self-contained wbuf-driven loop) from native
xce_job_picker_iface_t pickers (driven by a new per-session loop), and the
allocator now calls on_dispatch/on_pend for every picker as Req 1 AC 4
requires. Closes task 2.3's flagged picker/allocator concurrency race with a
mutex on fcfs_plugin_data_t, and adds cancel points to next_job().
Adds begin_session/next_job/end_session/on_dispatch/on_pend, driving a
dual-cursor walk (queue then job) over builtin_fcfs's private ordq_t
state. next_job() returns candidates in (queue priority desc, job
priority desc, queue_time asc) order and NULL once exhausted;
on_dispatch removes permanently, on_pend is a true no-op.
Review flagged an unsynchronized cross-thread access risk between
next_job() (picker thread) and on_dispatch/on_pend (allocator thread)
once wired live in task 2.4 — recorded in tasks.md Implementation Notes
for that task to close before registering builtin_fcfs as the default.
Requirements: 3.2, 3.4, 3.6
Adds on_queue_update, on_job_push, on_job_state_change, and on_job_update
to builtin_fcfs, driving the picker's private ordq_t candidate sets
directly (no core/sched queries), matching fcfs.c's existing ordering
semantics on the new finer-grained hook interface.
Requirements: 3.2
New src/scheduler/src/plugins/builtin_fcfs.c implements
builtin_fcfs_init()/fini() and the private fcfs_plugin_data_t (ordq_t
queues, per-queue ordq_t pendJobs) that all later builtin_fcfs hooks
build on. fcfs.c (today's live job_elect_fn-based picker) is untouched
-- still production, until task 2.4 flips the registration.
picker == xce_sched_plugin_t* calling convention (private state stored
in plugin->data) confirmed consistent with task 1.2's legacy wrapper.
Requirements: 3.1 (context), 3.3
plugin_register() now adapts an old-style job_elect_fn-only plugin to
xce_job_picker_iface_t via legacy_picker_wrapper_iface (next_job()
forwards to job_elect_fn; other hooks are safe no-ops covered elsewhere
in the existing lifecycle), logging a one-time deprecation warning. A
plugin that already sets picker_iface directly skips the wrapper and
the warning. Reuses the existing plugin_register()/key_find() mechanism
-- no parallel registration system.
Requirements: 1.6, 2.5
Add xce_job_picker_iface_t to plugin.h with all 13 hooks across
lifecycle/state-change/session/post-allocation/optional categories,
matching design.md's literal struct. Documents per-category thread
assignment, next_job() cancel-safety, and begin_session() session-reset
requirements. Forward-declares xce_sched_core_t (built in a later task).
Requirements: 1.2, 1.5, 2.6
- Host admin-state (open/closed/locked) now in scope for the upgrade
snapshot; host and queue admin-state frames migrate to independently-
extensible Cap'n Proto structs (one format-version bump covers both)
- bug-1418 blocker cleared: fixed 2026-07-09 via raft-apply-decoupling,
independently re-verified 2026-07-23 with a clean 10-round chaotic soak
- New Design Invariants section (6 standing rules) and Requirement 7
(regression coverage for host-placeholder/rejoin, overlapping upgrades,
format-mismatch rejection, WAL-trim-under-chaos)
Spec reached GO with 15/15 tasks complete (see prior task-confirmation
commits); these doc artifacts were generated during the design/research
phases but never separately staged.
rshen/Letmein was a live, checked-in cluster_admin credential in
auth.yaml. Renamed to a non-identifying username with a clearly-marked
placeholder password.
docs: add full ops deployment guide (src/docs/deployment.md)
Covers building and running all four daemons plus xceadmin, based on
direct source verification (CLI flags, config file shapes, TLS posture)
and a live single-node deploy. Documents real production gaps (no
process supervision, plaintext non-controller traffic, exposed dev
creds) rather than treating the repo as a finished runbook.
Standby-redirect coverage is done and correct (redirect wiring for
OP_upgradeConfig genuinely doesn't exist server-side yet, matches
design.md's own documented gap). Primary-push success-path test is
flaky (bug-1485) -- a dedicated debug round could not reproduce the
crash in 26+ clean runs, so root cause is unconfirmed. Test files left
uncommitted in the worktree pending re-verification on an idle host.
Deploys a real xce_controller (no HA peers) and runs real xceadmin
config build/push over actual TLS+capnp -- no mock server -- regression
locking task 6.1's envelope fix. Skips cleanly when no controller binary
is available. Surfaced a real, out-of-scope finding: single-node pushes
always report ackedCount==0/exit 1 even on full server-side success,
since there are no standbys to ACK.
config push crashed on every real controller with a capnp KjException --
xce_reply() wraps every reply in an outer Request{body.replymsg} envelope,
but discovery.py/push.py decoded raw wire bytes directly as a bare
RequestReply. Adds a shared decode_reply() unwrap point used by both call
sites, and fixes the same wrong-shape assumption baked into task 5.6's
and test_cli.py's mock-server fixtures.
Found a clean, reusable fault-injection technique: vote/PreVote
endpoints are TLS, and endpoint_connect() succeeds at the TCP level
independent of the handshake, so an accept-then-close impostor
listener on a peer's port deterministically causes an async
SSL_do_handshake failure post-connect -- landing in
vote_ep_err_handler/pre_vote_ep_err_handler (task 3.1's fix target),
never the already-fixed bug-1423 connect-refused path.
Independently reproduced by the reviewer from scratch: both a
RequestVote round and a PreVote round resolved via the fast
all-replied-no-quorum path in single-digit milliseconds, vs the
5000ms configured election timeout (confirmed firing later as a
no-op). Requirements 4.2, 4.3, 4.5 proven live, both vote kinds.
All 15 raft-adapter-conformance sub-tasks now complete. 1067 tests,
0 failures, 8 commits' worth of production fixes (config-push reply
integrity, vote/PreVote failure routing, dormant-sender
documentation), plus 6 regression-lock/confirmation/live-soak tasks.
Two residual gaps logged as new bugs during the sweep (bug-1483,
bug-1484), both pre-existing and out of this spec's scope.
Live 3-node cluster soak confirmed handle_repl_config_push's SHA-256
mismatch reject branch fires in production: 3/3 injected mismatched
pushes against a live standby produced a decoded
ReplConfigPushReply{accepted=false, term=<live raft term>, status=0},
seq-correlated, corroborated by the standby's local WARN log. Cluster
stayed healthy throughout (no crash/step-down/fence).
One rejection round: the real elected primary's own push has no
TOCTOU window to inject a mismatch without new relay code (confirmed
by reading config_commit_change/push_config_to_standbys), so the soak
plays the sender/primary role directly against a standby instead --
remediated from log-inference-only to actual decoded wire-reply
verification per reviewer's cheap-fix path.
No production/test code changed -- pure live observation.
Verified 190 tests across 8 files cover Requirements 2.4, 3.1-3.4,
4.1, 4.4, 7.1-7.3, 7.5, 8.1-8.4. 1067 tests, 0 failures -- unchanged,
zero-diff verification task.
Task 7.2's sweep surfaced a real, pre-existing gap (bug-1421, not
this feature's fault): no test anywhere drives the real
repl_ack_decode_and_dispatch path and asserts term adoption fires for
the heartbeat-reply or WAL-NACK term-discovery vectors -- only the
config-push vector this feature added (task 2.2) is actually tested.
Logged as bug-1484; outside this spec's task list to fix.
Verified all 17 cited tests in test_commit_barrier_rollout.c,
test_commit_barrier_reply.c, and test_term_safety_adapter.c exist and
pass, covering Requirements 5.1, 5.2, 5.4, 6.1-6.4. 1067 tests, 0
failures -- unchanged, as expected for a zero-diff verification task.
CLAUDE.md's TLA+ Action-to-Code Mapping table gains corrected/new rows:
production vs. dormant AppendEntries send mechanisms, WAL catch-up and
config-push reply integrity, and vote/PreVote failure routing across
all failure timings -- each citation independently re-verified against
live source. The old AppendEntries row's citations had already drifted
onto unrelated code before this task; corrected, not just appended to.
Protocol Verification Checklist gains 3 new items for the guarantees
this feature closes.
ha-term-safety/spec.json's known_limitations entry for the dormant
send-path drift gets an additive RESOLVED pointer back to this spec's
disposition, so a future reader lands on the answer instead of
re-discovering the drift.
Requirement 1 (systematic conformance audit deliverable) complete.
Adds test_batch_payload_persist_failure_propagates_to_whole_batch,
proving persist_submit_job_payloads_async resolves every payload in a
3-entry batch with the same first-failure rc when the first payload's
append fails, rather than letting later payloads independently
succeed (Req 5.3).
Fault injection: chmod the WAL leaf directory to 0500 to force
open_new_segment's EACCES (surfaced as -EIO), restored the instant the
first failure is observed -- a transient fault, not a permanent one,
so the test genuinely distinguishes propagation from independent
per-payload failure. No existing fault hook was reachable from this
code path; this is a real, previously-uncovered gap.
1066 -> 1067 tests, 0 failures. Task 5 (regression-lock tests for
already-shipped/already-conformant behavior) now complete -- Core
phase of raft-adapter-conformance finished (tasks 1-5).
Adds test_5_1_wal_catchup_reject_sends_nack_reply, driving a genuine
(term,lsn) conflict against a real persisted WAL entry through
ha_apply_term_entry_via_library and asserting
ha_send_wal_ack_reply_ex(success=0) fires via ha_wal_ack_send_hook.
Note: test_config_repl.c's test_bug1479_ae_reject_sends_nack_not_silence
already covers the same call site via a different (forward-gap)
stimulus, added in the original bug-1479 commit. design.md's own File
Structure Plan explicitly calls for this new test in this file
regardless, so this is disclosed, design-sanctioned defense-in-depth,
not accidental duplication -- both tests independently confirmed to
fail together when the reply call is removed.
1065 -> 1066 tests, 0 failures.
New test drives repl_broadcast and ha_stream_wal_to_standby through
real production fixtures (reused from tasks 2.3/6.2), asserts both
actually fired (non-vacuous), and asserts
ha_send_append_entries_stub_call_count stays 0 throughout -- the
Requirement 3.3 non-regression guard. RED-verified: manually invoking
ha_cb_send_append_entries_for_test inside the test reliably trips the
assertion; reverted for GREEN.
1064 -> 1065 tests, 0 failures. Task 4 (dormant sender documentation +
guard) now complete.
ha_cb_send_append_entries_stub's doc-comment now plainly states it is
reachable only via raft_send_append_entries (zero production callers,
grep-confirmed), and names repl_broadcast/ha_stream_wal_to_standby as
the actual production send/retry/backoff mechanism (Req 3.3, amended
to permit this documentation-only resolution over retire-or-wire-in).
Adds an XCE_TESTING-only ha_send_append_entries_stub_call_count
counter, mirroring the ha_wal_ack_send_hook idiom, as the guard against
this ever silently regaining a production caller. Task 4.2 will drive
the real production path and assert it stays zero.
1063 -> 1064 tests, 0 failures. Zero behavior change to any production
send path.
Adds pre_vote_ep_err_handler_for_test (XCE_TESTING-guarded) and two
tests: PreVote post-connect failure reaches pre_vote_reply_dispatch's
tally (Req 4.3), and a round mixing that failure mode with a delivered
reply reaches its quorum decision via the fast all-replied path
without the full round timeout (Req 4.5). Real-vote coverage (Req 4.2)
was already proven by task 3.1's own test, cited not duplicated.
pre_vote_reply_dispatch has no mock hook seam, so both tests assert
directly on raft_state_t's tally fields, mirroring test_ha_prevote.c's
existing verification technique.
1061 -> 1063 tests, 0 failures.
Splits vote_ep_err_handler into a real-vote handler and a new
pre_vote_ep_err_handler, each registered at its existing (already
separate) connection-setup call site. Extracts the identical
CTRL_PRIMARY-role cleanup into a shared vote_ep_handle_promoted_disconnect
helper, behavior unchanged.
Each handler's election-phase branch now routes a post-connect failure
(RDHUP/SSL-error/remote-close after connect, before any reply) through
the same production tally the connect-time failure branches already
use (vote_reply_dispatch / pre_vote_reply_dispatch, bug-1423/1424's
fix), instead of the legacy handle_request_vote_reply that never
updated the tally. Closes the one failure window bug-1423 didn't cover.
1060 -> 1061 tests, 0 failures. PreVote-specific test coverage is
task 3.2's scope (pre_vote_reply_dispatch has no hook seam analogous
to xce_vote_reply_hook; verified only by code symmetry + full-suite
regression here).
Adds the remaining 3 of 4 handle_repl_config_push reject-path tests
(malformed request, missing bytes/sha256, conf_save_config failure via
a deterministic ENOTDIR fixture), the EAGAIN-still-accepts test, and
term-adoption-on-accept (task 2.2's own test only covered the reject
case). Requirement 2.4 (unknown-outcome handling) is already covered
by the pre-existing test_wait_all_with_timeout_slow_threads, cited
rather than duplicated.
1055 -> 1060 tests, 0 failures.
myreq_handler's ReplConfigPushReply branch now routes cack->term
through raft_on_higher_term_observed (same entry point the
heartbeat-reply and WAL-NACK vectors use), regardless of accept/reject,
without altering the existing accept/reject branching.
Adds myreq_handler_for_test (XCE_TESTING-only) to drive the real
dispatcher in tests, following the peer_ctrl_msg_handler_for_test
precedent. Requires cluster.c in controller_unit_test_src plus the
established link-satisfying fake stubs for its unrelated switch-case
dependencies (same pattern already used by test_worker_join.c and
test_commit_barrier_reply.c).
handle_repl_config_push now sends a ReplConfigPushReply{accepted,term}
from every exit path instead of silently dropping on the four reject
conditions (malformed request, missing bytes/sha256, sha256 mismatch,
conf_save_config failure). Term is sourced live from c->ha.term.
-EAGAIN activation-defer still replies accepted=1, unchanged.
Adds an XCE_TESTING-only send-hook mirroring ha_wal_ack_send_hook so
tests can observe the reply payload without a live endpoint.
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:
1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
library's nextIndex backoff and immediately repl_start_catchup's the
standby at the corrected position, instead of the 16-strike silent
eviction with no catch-up driver after it. Correct from the start, but
live soak still failed identically: nothing ever produced the NACK this
code was waiting for.
2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
silently dropping the reject. ha_send_wal_ack_reply had hardcoded
success=1 unconditionally -- the only WalAckReply sender in the
controller -- making (1)'s NACK-routing correct but unreachable dead
code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
the NACK without a live endpoint. Live-confirmed the NACK now reaches
the primary -- but the scenario still failed for a third reason.
3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
catch-up range was hard-bounded at committed_lsn, which can never advance
in this exact scenario because advancing it requires an ACK from the very
standby that's stuck catching up (2-of-3 alive, this standby is the only
path to a new majority) -- a self-referential deadlock. Widens the
STREAM_WAL ceiling (and the convergence check gating it) to the leader's
own log tail, matching raft.tla's real AppendEntries semantics
(leaderCommit is advisory, never a send-cap), while reusing bug-1478's
exact guard so this doesn't reopen that trap (never widen onto a trailing
local-only checkpoint's own unstreamable LSN).
Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
admin-config-tool's spec.json blocked_on and tasks.md both still described
the OP_upgradeConfig dispatch-gate defect as an open blocker. It was fixed
2026-07-25 (bug-1468, commit 959b0cda) by config-upgrade-snapshot task 8.6,
which independently hit the same wall building a live OP_upgradeConfig
sender -- the fix's own comment in cluster.c already cross-references this
spec. The controller-side path has since been exercised live and
successfully many times via config-upgrade-snapshot's --config-upgrade
soak. All 23 admin-config-tool tasks are already [x]; this was pure
tracking-metadata debt, not a reopened implementation gap.
Also appends a cross-spec gap analysis (admin-config-tool x
config-upgrade-snapshot x clusterconfig-schema-split) to all three specs'
research.md, confirming clean schema integration between the latter two
and flagging the one remaining real gap: a new commit-timeout issue in
config-upgrade-snapshot task 8.6's force-elect step, surfaced now that
bug-1477 is fixed, not yet root-caused.
Two independent call sites duplicated target = max(committed_lsn,
last_written): ha.c's heartbeat/stateReport dispatch (repl_ack_decode_and_
dispatch) and config_repl.c's catchup_derive_target (used by the ACK-driven
catch-up continuation path). Once config-upgrade-snapshot's marker-triggered
checkpoint advances last_written one LSN past an already-committed
committed_lsn (the checkpoint's own consumed, never-replicated LSN slot),
the unconditional max() permanently wedges every standby's catch-up target
at an unreachable value -- the primary re-streams an empty range every
heartbeat forever.
Extracted the shared, pure ha_compute_repl_target_lsn(committed_lsn,
last_written) (ha.h/ha.c), which only falls back to last_written on genuine
committed_lsn==0 bootstrap, and pointed both call sites at it so they can
never disagree again.
Fixing the second call site exposed three pre-existing tests in
test_convergence_verification.c that hardcoded the old buggy formula's
output as their expected target; updated their fixture values (committed_lsn
4->9, matching appended_lsn) to decouple them from the now-fixed edge case,
which already has dedicated pure-function coverage in test_ha_role.c.
GREEN: full controller_unit_test suite = 1040 tests, 0 failures, clean
across multiple runs. Live-verified via run-ha-soak.sh
--config-upgrade-wal-trim (18 jobs, zero job loss, ctrl-1 reconverged 3-of-3
using the preserved floor segment) -- unblocks config-upgrade-snapshot
task 8.9.
bug-1472 (the prior blocker, worker-control replication) is fixed and
verified (7547a8f1). Rebuilding and re-running --config-upgrade on top of
that fix surfaced a new, unrelated defect: a node that takes the
skip_election fast path in myapp_run can permanently lose commit_engine_init
if its one cluster_init_phase2() attempt races config availability, since
none of ha.c's four retry call sites re-attempt for a node that never
contests an election. That member then never fires the marker-triggered
checkpoint (task 5), leaving it permanently unable to catch up past that
LSN (WAL_CHECKPOINT is local-only, non-replicated by design) -- a distinct,
permanent wedge from bug-1475's already-fixed transient race.
Root-caused via a dedicated debug pass (log/grep/git-blame evidence, logged
as bug-1477). Out of config-upgrade-snapshot's boundary -- same pattern as
bug-1471/1472/1474 -- so task 8.6 stays blocked until bug-1477 is fixed
upstream in HA bootstrap/election.
Add run_config_upgrade_wal_trim_round() / --config-upgrade-wal-trim to
run-ha-soak.sh: forces real WAL segment rotation via small
--wal-segment-bytes, kills a non-primary node, drives a live config
upgrade to fire the marker-triggered checkpoint's forced trim, then
inspects survivors' on-disk WAL directory before the killed node
rejoins to directly verify the mid-catch-up retention guard
(repl_trim_lsn/trim_wal_segments, bug-1147) preserves what it still
needs.
Two independent live 3-node runs both reproduce a real, pre-existing,
out-of-spec-boundary defect (bug-1474): a heartbeat reconnect-sweep
race reuses a just-killed standby's freed replicator slot and clobbers
its retention guard before trim ever runs. Root-caused via repl.c/ha.c
and confirmed via git blame to predate this spec and be reachable
under the old periodic-checkpoint trigger identically -- not caused by
anything config-upgrade-snapshot changed. Task 8.9 stays blocked per
tasks.md until bug-1474 is fixed upstream; infra is done and correctly
proves the violation.
Independently reviewed: round design verified faithful to the
unit-level fixture, bug-1474 diagnosis independently re-derived from
source and raw logs (not trusted from the implementer's report).
Add regression test proving Req 4.4/7.2: a second WAL_CONFIG_MARKER
committing while the first's forced checkpoint is still in flight
(simulated via the checkpoint_in_progress CAS across both markers'
commit+apply) cleanly supersedes it, with the settled state reflecting
only the newer marker. Verified via a field-aware byte comparison
(post-checkpoint_lsn) against a non-racing control fixture, plus
mid-race assertions that materialization is still incomplete at that
point (cfgref_present false, fence set) despite activation already
tracking the newer config. Test-only; production mechanism (already
implemented) unmodified. Suite: 1027 -> 1028 tests, 0 failures.
Prove Requirements 5.2/5.4/5.5/6.3 via deterministic fault injection at
the two critical upgrade windows, entirely at the unit level using the
XCE_TESTING hooks earlier tasks built for exactly this purpose -- no
live soak needed.
test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization
extends the existing fault-hook seam test: after simulating a crash
before the checkpoint write durably completes, it boots a genuinely
fresh persist_t/cluster_t against the same on-disk WAL via
persist_recover() (a real boot-replay simulation, not a same-process
re-call) and confirms materialization resumes and completes correctly.
test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain
proves a promoted member's materialization completes identically to a
non-promoted control member's, by draining each through its actual
production entry point and comparing checkpoints byte-for-byte -- a
regression guard against any future role-conditional branch creeping
into the marker-apply path.
Root-caused bug-1471's single-byte checkpoint divergence to a genuine,
pre-existing bug in WAL_WORKER_CTRL apply (worker_from_capnp treats
"-EEXIST" as a benign no-op for every hook detail, which is correct for
JOIN replay but silently discards UPDATE/DEAD field changes since the
target already existing is the normal case there). This predates
config-upgrade-snapshot by ~2.5 months and is unrelated to config
upgrades -- filed as bug-1472, genuinely outside this spec's boundary.
Task 8.6's own infrastructure and its two in-scope bug fixes (bug-1468,
bug-1469) are already committed and reviewed; only the Requirement 6.4
host-admin-status-survival assertion is blocked on bug-1472, per the
project's upstream-ownership handling rather than working around it here.
Build the live 3-node chaotic-multi-kill config-upgrade soak scenario
(run-ha-soak.sh --config-upgrade) required by Requirements 4.1, 4.3, 6.1,
6.4. This needed genuinely new client-side infrastructure: a hand-rolled
capnp send path for OP_upgradeConfig (the auto-generated encoder is a
no-op for its Data fields, worked around the same way
xce_replconfigpush_req_raw already does), plus cmd_hostctrl/cmd_upgradeconfig
CLI commands.
Building and running this live surfaced two real production bugs, both
fixed here:
- bug-1468: an unrelated prior commit inverted OPCode_upgradeConfig's
primary/standby dispatch guard in cluster.c, making config upgrades
unreachable from any network client.
- bug-1469: apply_committed_entry's WAL_CONFIG_MARKER case never called
config_activate_committed(), so the node that originates a config
upgrade never refreshed its own in-memory state as part of committing
it. Killing that node before some later, unrelated event happened to
refresh it left its checkpoint permanently stale -- the first live,
empirical reproduction of bug-857's originally hypothesized failure
mode. Fixed by activating before the marker-triggered checkpoint write,
since checkpoint content is built from live runtime tables that only
reflect the reload after activation runs.
A separate, narrower checkpoint divergence (bug-1471, not config-related)
remains open and still blocks a clean soak PASS -- task 8.6 stays
in-progress until it's resolved.
Prove Requirement 6.2 with a real, code-backed mechanism instead of the
originally hypothesized lost_and_found re-homing, which turned out to be
aspirational and never implemented (confirmed by exhaustive grep at both
the pre-fix baseline commit and HEAD — bug-857 itself was flagged in
buglog.json as never empirically reproduced).
test_bug857_regression.c demonstrates both halves of the requirement
entirely within controller_unit_test, no live soak or old-commit build
needed: schedule_checkpoint's pre-existing WAL-threshold trigger is
decoupled from the config-marker boundary and produces genuinely
divergent per-member checkpoint content, while the marker-triggered
checkpoint this spec added converges members regardless of differing
per-member processing pace.
design.md's Requirements Traceability table and Testing Strategy section
are corrected to match; the redirection reasoning is recorded in
tasks.md's Implementation Notes for the benefit of tasks 8.6-8.9, which
still need live multi-process HA-soak infrastructure for Requirement 6.1
that this task deliberately did not build.
Build configcompare_paths(), an offline sibling to snapdiff/walcompare that
recomputes each member's active-config sha256 fresh from its own on-disk
workdir/name/conf bytes and reports identical vs. divergent (Req 4.2).
Wire it as a cmd_configcompare CLI subcommand in test.c, matching its
siblings, after review caught it shipped as an unreachable library function
in round 1.
Extend snapverify to match the current on-disk checkpoint format: bump
SNAP_FORMAT_VERSION to 2, replace the stale raw queue-frame decode with the
capnp length-prefixed walk, add host-frame and config-bytes-section
structural walks, and decode (not just presence-check) the header's
cfgref tag/sha. Without this fix snapverify rejected every checkpoint
written by the current codebase.
