Adds xce_sched_core_save_snapshot()/xce_sched_core_load_snapshot(),
split across snapshot.c/snapshot_wire.c/snapshot_load.c (800-line cap)
along real save/load/shared-helper boundaries. Serializes the full
core state (config, jobs, queues, workers, resources, resvals, users,
projects, prng_seed, internal_state) into/out of sched_snapshot_data_t
(task 7.3's committed C mirror of the capnp SnapshotData struct) as
plain in-memory structures -- no wire encoding here, that's a later
task. After load, rebuilds the picker's private state by re-invoking
its state-change hooks (queue updates, then job pushes) against the
restored core state, matching xce_core_push_events()'s own hook-firing
order; picker-private state itself is never stored in the snapshot.
Adds a new prng_seed field to xce_sched_core_t -- requirements.md/
design.md both presuppose it but no prior task added it. Seeded to 0,
no PRNG consumer wired yet (deliberately out of scope).
Cross-core behavioral-equivalence proven directly: two independent
cores, each with its own registered picker, save on one and load into
a freshly-initialized other, both driven through an identical real
session, decision sequences asserted identical in order.
Found and fixed two real bugs along the way: a missing prng_seed
restore in this task's own new code, and a genuine guard bug in
already-committed core.c (sched_get_user()/sched_get_project()
treating key_find()'s expected -ENOENT as fatal, silently breaking
get-or-create for any new name) that also affects the live production
path (make_sched_job(), JOBS_PUSH) -- a narrowly-scoped two-line fix,
independently traced to the primitive level and confirmed correct by
review.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Audit of all ~19 new types/enums from tasks 7.1-7.4 found one genuine
gap: ReplayEvent's own envelope union (11 members across 7.1+7.2) was
never actually constructed/encoded in any test, despite its individual
payload types each having their own round-trip test. Adds 3 tests
(jobStateChange/jobUpdate/endSession variants) proving the union
mechanism itself round-trips correctly.
No new schema/typemap changes -- audit-only. Decided against creating
design.md's originally-named test_replay_format.c: tasks 7.1-7.4
independently, consistently (4-for-4) chose the existing
test_e2e_capnp.c instead. Deliberately did not add Request.body/
RequestReply.data union-wiring tests for OPCode_schedAdmin, matching
universal codebase precedent (opcode-dispatch union members are always
tested via functional dispatch tests, never bare schema round-trips,
and the real dispatch logic doesn't exist until task 10.1).
Closes out Group 7 (Foundation: Replay Wire Format) entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds SchedAdminAction (recordStart/recordStop/recordStatus),
SchedAdminRequest (action/recordFile), SchedAdminReply (ok/error/
recording/file/eventCount/startedAt) to controller.capnp, matching
requirements.md Req 5 AC 10 verbatim. Adds schedAdmin@42 to OPCode,
schedadminmsg@31 to Request.body, schedadminreplymsg@19 to
RequestReply.data -- the exact next-available ordinals, verified live
before implementation and re-verified byte-identical against every
existing value/member.
Deliberately schema-only, per this task's own scope -- the controller
dispatch logic (cluster.c's case OPCode_schedAdmin:) is task 10.1's
job. cluster.c, src/frontend/, src/xceadmin/, src/scheduler/ all
untouched.
Round-trip tests cover all 3 SchedAdminAction values and both
ok=true/ok=false SchedAdminReply cases, including an asymmetric
ok=false/recording=true combination that rules out a Bool-field
bit-packing mixup at the wire level.
Closes out Group 7 (Foundation: Replay Wire Format) entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds DispatchDecision (wraps JobAlloc), PendDecision (wraps
JobPendReason), PreemptVictim, PreemptDecision to controller.capnp.
Replaces task 7.1's empty SnapshotData placeholder wholesale with the
real 11-field struct (config/jobs/queues/workers/resources/resvals/
users/projects/prngSeed/internalState/internalVersion), in both the
capnp schema and its typemap.h C mirror. Extends ReplayDecision's
ts-only placeholder with a purely additive body union (dispatch@1/
pend@2/preempt@3).
Deviates from design.md's literal SchedUser/SchedProject mapnames
(xce_sched_user_t/xce_sched_project_t): those names are already real
C structs in sched/types.h, which shares a translation unit with
typemap.h in core_events.c/schedobj.c -- reusing them would be a
genuine duplicate-typedef compile error, not a style choice. Used
fresh mapnames (sched_snapshot_user_t/sched_snapshot_project_t)
instead, field lists unchanged, documented in both files.
Found and fixed a real bug mid-implementation: a doc comment's
literal */ substring prematurely closed a C block comment, corrupting
sched_snapshot_data_t and cascading into ~20 unrelated-looking compile
errors (bug-1493).
Five deep round-trip tests added to test_e2e_capnp.c. Full project
build and controller_unit_test (1082/1082) verified clean, confirmed
independently three times.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds JobStateChangeEvent (jobid/oldState/newState/exitCode) and
JobUpdateEvent (jobid/changedFields/job: Job) to controller.capnp,
matching design.md's spec verbatim. JobUpdateEvent.job reuses the
existing full 12-field Job/xce_job_t record, verified distinct from
the narrower SchedJob struct sharing the same mapname.
Extends ReplayEvent.body's union (task 7.1) with jobStateChange@10/
jobUpdate@11 -- the only two valid next ordinals, since capnp requires
strictly sequential ordinals with no gaps across a struct and task
7.1 deliberately stopped at @9. Existing @0-@9 fields and every other
struct in the schema are byte-identical.
Deep round-trip tests in test_e2e_capnp.c exercise every field with
non-default values, including the 3-level nested Job substructs
(submitData/allocs/pidinfo/pendreason). JOB_F_* bitmask bit values
deliberately left undefined -- a later task's job, not this
schema-only one's.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds ReplayCluster/ReplayHeader/SnapshotEntry/ReplayEvent/
SessionSummary/ReplaySession to controller.capnp, additively, first
task of Phase 3 (Foundation: Replay Wire Format).
Requirements.md's AC 20 capnp sketch isn't self-contained -- ReplayEvent's
union references JobStateChangeEvent/JobUpdateEvent (task 7.2's job),
SnapshotEntry.data/ReplayDecision's union reference SnapshotData/
DispatchDecision/PendDecision/PreemptDecision (task 7.3's job), none
of which exist yet. Resolved by verifying capnp's real constraints
against the toolchain directly: SnapshotData is an empty {} placeholder
(task 7.3 replaces its mapname'd C struct wholesale when adding the
real ~11 fields), ReplayDecision holds only {ts} (task 7.3 appends a
body union at new ordinals), and ReplayEvent's union holds only the 9
members with pre-existing backing types (Conf4Sched/SchedJobData/
WorkerData/ResourceData/ResValList/QueueConf/SchedJobRusages/Void),
deferring jobStateChange/jobUpdate to task 7.2 as new ordinals --
capnp requires strictly sequential ordinals with no holes, so this
split is compiler-mandated, not stylistic.
Round-trip test added to test_e2e_capnp.c. Full project build (all
binaries sharing this schema) and 1075/1075 controller_unit_test
verified clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Rewires main.c's live scheduling loop onto the new core-driven
architecture: xce_sched_driver_translate_pending_events() (task 5.1)
-> xce_core_begin_session() -> plugin_begin_session()+sched_account_jobs()
-> xce_core_signal_go() -> decision-drain via the registered
xce_core_decision_cb_t -> plugin_end_session() -> xce_core_end_session(),
all in the new exported xce_sched_driver_run_normal_mode_round()
(driver.c), replacing the old inline sched_update()+sched_begin_session()
+sched_go()+switch+sched_end_session() sequence.
xce_core_go() (task 4.4) is split into xce_core_begin_session()/
xce_core_signal_go() -- a pure decomposition (xce_core_go() itself is
now defined in terms of both, task 4.4's approved test suite passes
unmodified) -- specifically so the driver can interleave
plugin_begin_session()/sched_account_jobs() in the exact safety-critical
window task 4.4 deliberately excluded from pure-core scope: after the
resval reset, strictly before the picker thread is signaled. This is
empirically proven by test_driver_session_ordering.c via real
cross-thread happens-before assertions, not just documented.
xce_sched_driver_on_decision() is the new registered callback:
DISPATCH/PEND forward to the existing sched_publish_dispatch()/
sched_publish_pend_reason(); PREEMPT fails loudly (-ENOSYS, logged)
since no xcecli_preempt() client API exists and nothing in the live
allocator produces one yet (scheduler-preemption's territory).
Adds a --sim CLI flag (parse-only, loud not-yet-implemented failure,
since SimDriver is Phase 3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds driver.h/driver.c: xce_sched_driver_translate_pending_events()
drains core->pend_updates via the existing dataq-swap pattern (same
mechanism sched_update() uses), translates all 10 xce_sched_update_
request_t op types into the matching xce_core_event_t, and pushes the
batch through xce_core_push_events(). Scoped to xce_sched_core_t*
alone -- pend_updates already lives on core since task 4.1, and this
task is translation-only (no decision publishing, no mode-selection,
no recording -- all later work). handler.c (the enqueue side) is
untouched. First task in Group 5 (Core: Normal-Mode Driver).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds test_core_full_cycle.c: drives xce_core_push_events() -> xce_core_go()
-> decision drain -> xce_core_end_session() end-to-end against a real
picker pthread (running the real xce_job_picker_iface_t hooks on a
test-local stub picker) and a main-thread stand-in allocator, since
sched_allocator() is xce_scheduler_t*-coupled and out of pure-core
scope. Proves both preconditions (core_go reentrancy, push_events
mid-session) reject against a genuinely active session and recover
once it ends. Closes out group 4 (Foundation: Scheduler Core
Extraction) -- tasks 4.1-4.6 all complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds xce_core_now() (new core_clock.c, kept separate from core.c so
test_core_isolation.c's wall-clock scan still enforces zero direct
clock_gettime()/time() calls elsewhere in the core): the two-anchor
plus speed formula (virtual_clock_base + (elapsed * speed_num) /
speed_den), covering all four modes (normal, discrete/real-time/fast-
slow simulation) via four new atomic fields on xce_sched_core_t,
defaulted to normal-mode (1/1 speed) in xce_sched_core_init().
requirements.md's literal tlog_reg_format_func(fn, core_ptr) call
shape doesn't exist in the real vendored tlog library (registration
takes only the callback; userptr is per-call-site, not bindable at
registration) -- resolved with a module-level active-core pointer read
by a new core-scoped format function, not wired into mylog_init()/
main.c yet. xce_core_go()/xce_core_end_session() (task 4.4) now stamp
session_start/compute session_duration_ns via xce_core_now(), with
their existing session-active/reentrancy behavior untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds xce_core_go()/xce_core_end_session() (new core_session.c):
xce_core_go() is non-blocking, performs begin-session state reset
(select-eval-ctx, resval resets), signals the picker thread via wbuf,
and returns -EINVAL on a reentrant call while a session is already
active (proven a true no-op, not just an early return). Draining
core->decisions and invoking the driver's on_decision callback is the
driver's own responsibility between core_go() and core_end_session()
(confirmed driver-owned, not core-owned, per the Driver <-> Core
protocol). xce_core_set_decision_cb() registers the callback + driver
context as a separate setter, keeping xce_sched_core_init()'s live
sched_init() call site untouched. Adds XCE_DECISION_PREEMPT to
decision.h (jobid/victims[]/reason) alongside the existing DISPATCH/
PEND types. Additive only — sched_go/sched_begin_session/
sched_end_session/sched_allocator/main.c remain untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Defines xce_core_event_type_t/xce_core_event_t (9 event kinds: config,
job, worker, resource, resval full/delta, job state change, job update,
queue update, job rusage) in core.h. Implements xce_core_push_events()
in a new core_events.c (split out of core.c to stay under the 800-line
cap): rejects -EBUSY mid-session, otherwise applies each event's core-
state mutation and fires the corresponding picker hook synchronously,
exactly once per event, before returning. CONFIG_PUSH decomposes the
pushed config's embedded queues/resources/resvals via shared helpers
(also reused by QUEUE_UPDATE's upsert path), mirroring the existing
sched_init_config()/sched_init_resources()/sched_init_sharedInst()/
sched_init_queues(). Narrows sched_resolve_job_picker() to take
xce_sched_core_t* (pure type change, zero behavior change). Purely
additive — not wired into the live scheduling path yet (task 5.x).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Hardcodes jbuf's cbuf capacity to the literal 1 in xce_sched_core_init()
(no parameter path, no local variable to drift), backed by a regression
test. Audits every kvmgr iteration reachable from the scheduling loop
(jobs/queues/users/projects): only jobs is ever iterated, via xzarray's
deterministic ascending-slot walk with no process-random seeding —
findings recorded durably as a comment above xce_sched_core_init().
core->workers iteration-order stability is flagged as an open follow-up
(scheduler-allocation's territory), not this task's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Splits xce_scheduler_t along the controller-I/O seam: xce_sched_core_t
(new core.h/core.c) now holds all scheduling state (jobs/queues/workers/
users/projects/resource managers/plugins/three-thread-cbuf pipeline),
while xce_scheduler_t shrinks to {core; apictx; cluster_cfg}. Proven via
an nm-based link-time isolation test (zero netio/xcecli/idgen/delaymgr
symbols reachable from core.c) plus a reproducible wall-clock-call guard.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Closes out group 2 (Core: Builtin FCFS Picker). Confirms builtin_fcfs's
ordering matches legacy fcfs_elect_job (byte-identical comparators, same
sched_queue_is_ready/sched_job_is_ready gates, provably equivalent jobid
sequence), confirms tasks 1.1-2.5 left every pre-existing test file
unmodified, and confirms resreq immutability, pend-reason factory-only
allocation, and select_eval_ctx_t memoization all hold across every file
this spec has touched so far. Adds a durable regression lock for the
pend-reason factory invariant (no direct calloc(pendreason_t) outside
make_pendreason()).
Task 2.4 already made sched_job_picker() delegate entirely to the picker
interface with zero direct queue/job traversal, satisfying Req 1 AC 1/AC 7
as a byproduct. This adds a source-text regression test locking that
property in place: asserts sched_job_picker()'s body contains none of the
forbidden direct-traversal symbols and does contain the expected
picker-interface call sites.
Registers builtin_fcfs at scheduler startup and makes it the default
job_pick_plugin, replacing the hardcoded "xce_sched_fcfs" literal in
sched_job_picker() with a fail-loud lookup through sched_resolve_job_picker().
The picker thread now branches legacy job_elect_fn plugins (delegated
unchanged, preserving their self-contained wbuf-driven loop) from native
xce_job_picker_iface_t pickers (driven by a new per-session loop), and the
allocator now calls on_dispatch/on_pend for every picker as Req 1 AC 4
requires. Closes task 2.3's flagged picker/allocator concurrency race with a
mutex on fcfs_plugin_data_t, and adds cancel points to next_job().
Adds begin_session/next_job/end_session/on_dispatch/on_pend, driving a
dual-cursor walk (queue then job) over builtin_fcfs's private ordq_t
state. next_job() returns candidates in (queue priority desc, job
priority desc, queue_time asc) order and NULL once exhausted;
on_dispatch removes permanently, on_pend is a true no-op.
Review flagged an unsynchronized cross-thread access risk between
next_job() (picker thread) and on_dispatch/on_pend (allocator thread)
once wired live in task 2.4 — recorded in tasks.md Implementation Notes
for that task to close before registering builtin_fcfs as the default.
Requirements: 3.2, 3.4, 3.6
Adds on_queue_update, on_job_push, on_job_state_change, and on_job_update
to builtin_fcfs, driving the picker's private ordq_t candidate sets
directly (no core/sched queries), matching fcfs.c's existing ordering
semantics on the new finer-grained hook interface.
Requirements: 3.2
New src/scheduler/src/plugins/builtin_fcfs.c implements
builtin_fcfs_init()/fini() and the private fcfs_plugin_data_t (ordq_t
queues, per-queue ordq_t pendJobs) that all later builtin_fcfs hooks
build on. fcfs.c (today's live job_elect_fn-based picker) is untouched
-- still production, until task 2.4 flips the registration.
picker == xce_sched_plugin_t* calling convention (private state stored
in plugin->data) confirmed consistent with task 1.2's legacy wrapper.
Requirements: 3.1 (context), 3.3
plugin_register() now adapts an old-style job_elect_fn-only plugin to
xce_job_picker_iface_t via legacy_picker_wrapper_iface (next_job()
forwards to job_elect_fn; other hooks are safe no-ops covered elsewhere
in the existing lifecycle), logging a one-time deprecation warning. A
plugin that already sets picker_iface directly skips the wrapper and
the warning. Reuses the existing plugin_register()/key_find() mechanism
-- no parallel registration system.
Requirements: 1.6, 2.5
Add xce_job_picker_iface_t to plugin.h with all 13 hooks across
lifecycle/state-change/session/post-allocation/optional categories,
matching design.md's literal struct. Documents per-category thread
assignment, next_job() cancel-safety, and begin_session() session-reset
requirements. Forward-declares xce_sched_core_t (built in a later task).
Requirements: 1.2, 1.5, 2.6
bug-1485 (flaky test_live_config_push_against_primary_reaches_success_path,
~24/25 fail rate under pytest, primary process going <defunct> after
promotion) could never be diagnosed because controller stderr was
DEVNULL'd -- no ASAN report or signal was ever captured from a real
failure.
Redirects each node's stderr to deploy_dir/logs/<name>.stderr.log and adds
an XCE_KEEP_DEPLOY_DIR=1 debug toggle to preserve the deploy dir across
fixture cleanup for inspection. Ran the live test 40 times in this
environment (25 idle + 15 under concurrent build contention): 40/40
passed, 0/120 captured stderr logs non-empty -- could not reproduce.
Downgraded to low-priority/environment-flake, left open; the capture
infrastructure stays in place for the next real failure.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXkenK8rqE8mdS1YKjsq78
bug-1421 added two term-discovery fallback vectors so a stale-term primary
steps down instead of retrying AppendEntries forever: heartbeat-reply
state-report term adoption and WAL-NACK reply_term adoption, both inside
repl_ack_decode_and_dispatch. Neither had a test driving the real
production dispatch path and asserting raft_on_higher_term_observed
actually fires.
Adds two tests mirroring test_config_repl.c's existing config-push term
test, driving peer_ctrl_msg_handler -> repl_ack_decode_and_dispatch via
wire-shape capnp packets. Both pass immediately -- the underlying bug-1421
code was already correct, just unverified end-to-end -- closing
raft-adapter-conformance's Requirement 7.5 unverified claim.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXkenK8rqE8mdS1YKjsq78
ha_arm_heartbeat_timeout_vtask published a calloc'd liveness_poll_ctx_t
into s_liveness_ctx unconditionally but only ever freed it via
liveness_poll_fn's cancelled-check path, which never runs if
delay_task_ms was never called (c==NULL, delaymgr not running) or itself
failed. Both no-schedule cases now free the ctx immediately, establishing
the invariant that s_liveness_ctx is non-NULL only when a delaymgr task
actually holds a live reference to it.
Regression test drives a real LOST PreVote round without stubbing
ha_prevote_rearm_liveness_hook (every other LOST-path test in the file
stubs it specifically to avoid this code path) and asserts the ctx isn't
left dangling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXkenK8rqE8mdS1YKjsq78
wal_type_name() never had a case for WAL_CONFIG_MARKER (type 50) -- every
config-change marker entry printed as UNKNOWN instead of a real, defined
type. Added the case plus payload decode (tag/sha256) matching config_repl.c's
config_marker_encode/_decode byte layout, hand-decoded here rather than
linking config_repl.c into xce_test's link graph.
Gap detection also had no way to tell "committed prefix missing!" (real
loss) apart from a normal checkpoint-triggered WAL trim (persist.c's
trim_wal_segments, floor-guarded so it never removes a segment a standby
still needs) -- every long-running node's WAL, once checkpointed, would
false-alarm on every inspection. Cross-checks the sibling data/snap/
directory (snapshot LSN is literally in the filename, snap-<hex>.capnp)
and downgrades the message when a snapshot covers the gap.
Adds waldump_stats_t.leading_gap_explained and 3 new tests plus one
assertion on the existing gap test.
trim_wal_segments + rotate_snapshots run on every checkpoint --
operators don't need to manually purge data/. Documents the
snap_keep_n=2 default and that it's a cluster-config field, not a CLI
flag.
--system-site-packages means pip skips vendoring click/pycapnp/pyyaml/
jsonschema since nix already provides them -- deliberate, to keep
pycapnp's ABI-sensitive build pinned to the exact capnproto C library
nix built it against. Activating the venv outside nix develop (or the
AppImage) hits ModuleNotFoundError immediately; not a packaging bug.
Standby-redirect half is done and correct: OP_upgradeConfig has no
server-side redirect wiring today, matches design.md's own documented
"Out of Boundary" gap, and this test locks in the real current
rejection behavior.
Primary-push success-path half is unreliable (~24/25 live fails in one
investigation round, see bug-1485) -- the elected primary was observed
as a <defunct> zombie shortly after promotion, but a dedicated debug
pass could not reproduce it in 26+ clean runs on an idle host and
captured no crash evidence either way. tasks.md marks 6.3 _Blocked:_
pending re-verification on an idle host with real stderr capture.
rshen/Letmein was a live, checked-in cluster_admin credential in
auth.yaml. Renamed to a non-identifying username with a clearly-marked
placeholder password.
docs: add full ops deployment guide (src/docs/deployment.md)
Covers building and running all four daemons plus xceadmin, based on
direct source verification (CLI flags, config file shapes, TLS posture)
and a live single-node deploy. Documents real production gaps (no
process supervision, plaintext non-controller traffic, exposed dev
creds) rather than treating the repo as a finished runbook.
Deploys a real xce_controller (no HA peers) and runs real xceadmin
config build/push over actual TLS+capnp -- no mock server -- regression
locking task 6.1's envelope fix. Skips cleanly when no controller binary
is available. Surfaced a real, out-of-scope finding: single-node pushes
always report ackedCount==0/exit 1 even on full server-side success,
since there are no standbys to ACK.
config push crashed on every real controller with a capnp KjException --
xce_reply() wraps every reply in an outer Request{body.replymsg} envelope,
but discovery.py/push.py decoded raw wire bytes directly as a bare
RequestReply. Adds a shared decode_reply() unwrap point used by both call
sites, and fixes the same wrong-shape assumption baked into task 5.6's
and test_cli.py's mock-server fixtures.
CLAUDE.md's TLA+ Action-to-Code Mapping table gains corrected/new rows:
production vs. dormant AppendEntries send mechanisms, WAL catch-up and
config-push reply integrity, and vote/PreVote failure routing across
all failure timings -- each citation independently re-verified against
live source. The old AppendEntries row's citations had already drifted
onto unrelated code before this task; corrected, not just appended to.
Protocol Verification Checklist gains 3 new items for the guarantees
this feature closes.
ha-term-safety/spec.json's known_limitations entry for the dormant
send-path drift gets an additive RESOLVED pointer back to this spec's
disposition, so a future reader lands on the answer instead of
re-discovering the drift.
Requirement 1 (systematic conformance audit deliverable) complete.
Adds test_batch_payload_persist_failure_propagates_to_whole_batch,
proving persist_submit_job_payloads_async resolves every payload in a
3-entry batch with the same first-failure rc when the first payload's
append fails, rather than letting later payloads independently
succeed (Req 5.3).
Fault injection: chmod the WAL leaf directory to 0500 to force
open_new_segment's EACCES (surfaced as -EIO), restored the instant the
first failure is observed -- a transient fault, not a permanent one,
so the test genuinely distinguishes propagation from independent
per-payload failure. No existing fault hook was reachable from this
code path; this is a real, previously-uncovered gap.
1066 -> 1067 tests, 0 failures. Task 5 (regression-lock tests for
already-shipped/already-conformant behavior) now complete -- Core
phase of raft-adapter-conformance finished (tasks 1-5).
Adds test_5_1_wal_catchup_reject_sends_nack_reply, driving a genuine
(term,lsn) conflict against a real persisted WAL entry through
ha_apply_term_entry_via_library and asserting
ha_send_wal_ack_reply_ex(success=0) fires via ha_wal_ack_send_hook.
Note: test_config_repl.c's test_bug1479_ae_reject_sends_nack_not_silence
already covers the same call site via a different (forward-gap)
stimulus, added in the original bug-1479 commit. design.md's own File
Structure Plan explicitly calls for this new test in this file
regardless, so this is disclosed, design-sanctioned defense-in-depth,
not accidental duplication -- both tests independently confirmed to
fail together when the reply call is removed.
1065 -> 1066 tests, 0 failures.
New test drives repl_broadcast and ha_stream_wal_to_standby through
real production fixtures (reused from tasks 2.3/6.2), asserts both
actually fired (non-vacuous), and asserts
ha_send_append_entries_stub_call_count stays 0 throughout -- the
Requirement 3.3 non-regression guard. RED-verified: manually invoking
ha_cb_send_append_entries_for_test inside the test reliably trips the
assertion; reverted for GREEN.
1064 -> 1065 tests, 0 failures. Task 4 (dormant sender documentation +
guard) now complete.
ha_cb_send_append_entries_stub's doc-comment now plainly states it is
reachable only via raft_send_append_entries (zero production callers,
grep-confirmed), and names repl_broadcast/ha_stream_wal_to_standby as
the actual production send/retry/backoff mechanism (Req 3.3, amended
to permit this documentation-only resolution over retire-or-wire-in).
Adds an XCE_TESTING-only ha_send_append_entries_stub_call_count
counter, mirroring the ha_wal_ack_send_hook idiom, as the guard against
this ever silently regaining a production caller. Task 4.2 will drive
the real production path and assert it stays zero.
1063 -> 1064 tests, 0 failures. Zero behavior change to any production
send path.
Adds pre_vote_ep_err_handler_for_test (XCE_TESTING-guarded) and two
tests: PreVote post-connect failure reaches pre_vote_reply_dispatch's
tally (Req 4.3), and a round mixing that failure mode with a delivered
reply reaches its quorum decision via the fast all-replied path
without the full round timeout (Req 4.5). Real-vote coverage (Req 4.2)
was already proven by task 3.1's own test, cited not duplicated.
pre_vote_reply_dispatch has no mock hook seam, so both tests assert
directly on raft_state_t's tally fields, mirroring test_ha_prevote.c's
existing verification technique.
1061 -> 1063 tests, 0 failures.
Splits vote_ep_err_handler into a real-vote handler and a new
pre_vote_ep_err_handler, each registered at its existing (already
separate) connection-setup call site. Extracts the identical
CTRL_PRIMARY-role cleanup into a shared vote_ep_handle_promoted_disconnect
helper, behavior unchanged.
Each handler's election-phase branch now routes a post-connect failure
(RDHUP/SSL-error/remote-close after connect, before any reply) through
the same production tally the connect-time failure branches already
use (vote_reply_dispatch / pre_vote_reply_dispatch, bug-1423/1424's
fix), instead of the legacy handle_request_vote_reply that never
updated the tally. Closes the one failure window bug-1423 didn't cover.
1060 -> 1061 tests, 0 failures. PreVote-specific test coverage is
task 3.2's scope (pre_vote_reply_dispatch has no hook seam analogous
to xce_vote_reply_hook; verified only by code symmetry + full-suite
regression here).
Adds the remaining 3 of 4 handle_repl_config_push reject-path tests
(malformed request, missing bytes/sha256, conf_save_config failure via
a deterministic ENOTDIR fixture), the EAGAIN-still-accepts test, and
term-adoption-on-accept (task 2.2's own test only covered the reject
case). Requirement 2.4 (unknown-outcome handling) is already covered
by the pre-existing test_wait_all_with_timeout_slow_threads, cited
rather than duplicated.
1055 -> 1060 tests, 0 failures.
myreq_handler's ReplConfigPushReply branch now routes cack->term
through raft_on_higher_term_observed (same entry point the
heartbeat-reply and WAL-NACK vectors use), regardless of accept/reject,
without altering the existing accept/reject branching.
Adds myreq_handler_for_test (XCE_TESTING-only) to drive the real
dispatcher in tests, following the peer_ctrl_msg_handler_for_test
precedent. Requires cluster.c in controller_unit_test_src plus the
established link-satisfying fake stubs for its unrelated switch-case
dependencies (same pattern already used by test_worker_join.c and
test_commit_barrier_reply.c).
handle_repl_config_push now sends a ReplConfigPushReply{accepted,term}
from every exit path instead of silently dropping on the four reject
conditions (malformed request, missing bytes/sha256, sha256 mismatch,
conf_save_config failure). Term is sourced live from c->ha.term.
-EAGAIN activation-defer still replies accepted=1, unchanged.
Adds an XCE_TESTING-only send-hook mirroring ha_wal_ack_send_hook so
tests can observe the reply payload without a live endpoint.
Additive term @2 field on ReplConfigPushReply so a configuration-push
reply can carry the replying member's current term for term-discovery
(Requirement 7.4). Companion repl_config_push_reply_t struct field
added in typemap.h per the existing $C.mapname convention.
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:
1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
library's nextIndex backoff and immediately repl_start_catchup's the
standby at the corrected position, instead of the 16-strike silent
eviction with no catch-up driver after it. Correct from the start, but
live soak still failed identically: nothing ever produced the NACK this
code was waiting for.
2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
silently dropping the reject. ha_send_wal_ack_reply had hardcoded
success=1 unconditionally -- the only WalAckReply sender in the
controller -- making (1)'s NACK-routing correct but unreachable dead
code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
the NACK without a live endpoint. Live-confirmed the NACK now reaches
the primary -- but the scenario still failed for a third reason.
3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
catch-up range was hard-bounded at committed_lsn, which can never advance
in this exact scenario because advancing it requires an ACK from the very
standby that's stuck catching up (2-of-3 alive, this standby is the only
path to a new majority) -- a self-referential deadlock. Widens the
STREAM_WAL ceiling (and the convergence check gating it) to the leader's
own log tail, matching raft.tla's real AppendEntries semantics
(leaderCommit is advisory, never a send-cap), while reusing bug-1478's
exact guard so this doesn't reopen that trap (never widen onto a trailing
local-only checkpoint's own unstreamable LSN).
Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A node that takes myapp_run's skip_election passive path (bug-849 --
"leader already known after the startup stagger") makes exactly ONE
cluster_init_phase2 attempt. All 5 call sites (myapp.c's skip_election
block, raft_cb_become_leader, raft_cb_become_follower,
ha_elect_evaluate_and_signal's two branches) latched ha->phase2_done = 1
unconditionally after calling cluster_init_phase2, discarding its return
value. If that one attempt raced config availability and failed, the node
was permanently stranded: commit_engine_init never ran, so
apply_committed_entry never applied any committed WAL entry for the rest
of the process's life -- and for a skip_election node specifically,
nothing else would ever retry it, since the other 4 call sites only fire
via that node's own raft-library state transitions, which never happen
for a passive standby whose primary keeps heartbeating normally.
Extracted a shared, idempotent retry (ha_phase2_retry_fn, mirrors
upgrade_materialization_retry_fn's shape) plus a delaymgr-scheduled,
self-rescheduling trigger (ha_schedule_phase2_retry) -- self-rescheduling
because, unlike config_activate_retry/upgrade_materialization_retry which
are re-triggered externally by recurring commit-advance events, nothing
else recurs for a node stuck on the passive path. Wired into all 5 call
sites; raft_cb_become_leader additionally refuses promotion on failure
(mirroring the sibling upgrade_materialization_fenced refusal already in
that function -- a promoted primary with an unwired commit engine is far
worse than a delayed promotion).
TDD: 4 new tests, RED confirmed by temporarily reverting the rc-gating.
GREEN: full suite 1044 tests, 0 failures. Live-verified via
run-ha-soak.sh --config-upgrade (the scenario that found this bug): the
retry fires for real (4 failed attempts, then success) and the soak's
checkpoint-identity/config-identity/host-admin-status assertions -- the
things bug-1477 blocked -- all pass.
Two independent call sites duplicated target = max(committed_lsn,
last_written): ha.c's heartbeat/stateReport dispatch (repl_ack_decode_and_
dispatch) and config_repl.c's catchup_derive_target (used by the ACK-driven
catch-up continuation path). Once config-upgrade-snapshot's marker-triggered
checkpoint advances last_written one LSN past an already-committed
committed_lsn (the checkpoint's own consumed, never-replicated LSN slot),
the unconditional max() permanently wedges every standby's catch-up target
at an unreachable value -- the primary re-streams an empty range every
heartbeat forever.
Extracted the shared, pure ha_compute_repl_target_lsn(committed_lsn,
last_written) (ha.h/ha.c), which only falls back to last_written on genuine
committed_lsn==0 bootstrap, and pointed both call sites at it so they can
never disagree again.
Fixing the second call site exposed three pre-existing tests in
test_convergence_verification.c that hardcoded the old buggy formula's
output as their expected target; updated their fixture values (committed_lsn
4->9, matching appended_lsn) to decouple them from the now-fixed edge case,
which already has dedicated pure-function coverage in test_ha_role.c.
GREEN: full controller_unit_test suite = 1040 tests, 0 failures, clean
across multiple runs. Live-verified via run-ha-soak.sh
--config-upgrade-wal-trim (18 jobs, zero job loss, ctrl-1 reconverged 3-of-3
using the preserved floor segment) -- unblocks config-upgrade-snapshot
task 8.9.
myconf->lock (mylock_init/fini wiring) had landed but load_primary_config()
itself was left in its RED-phase reproduction state (unguarded, decoding
directly into the shared myconf->cluster_config field), so the fix's own
regression test failed on resume: thread B raced through while thread A was
parked mid-call.
Wraps the whole free-decode-publish sequence in myconf->lock (acquired first,
released on every return path) and decodes into a local cluster_config_t*
first, publishing into myconf->cluster_config only after full decode and
validation succeed -- closing the UAF/double-free window a live 3-node ASan
soak caught (two concurrent handle_repl_config_push deliveries for the same
standby racing two load_primary_config calls).
Verified 4/4 clean full-suite runs (1036 tests, 0 failures) including the
specific race test.
A restarted standby could get permanently stuck unable to catch up,
with the primary re-streaming the identical doomed AppendEntries range
every heartbeat with zero progress -- discovered via a live 3-node
config-upgrade-snapshot soak. Liveness-only; no safety invariant
(Election Safety, Leader Completeness, State Machine Safety, Commit
Durability) was ever violated.
Three-layer root cause, all now closed:
1. ha_stream_wal_to_standby (config_repl.c) used to skip advancing its
prev_streamed_lsn/prev_streamed_term tracker across WAL_CHECKPOINT
entries (a local-only, never-replicated marker that still consumes
a real LSN slot). Now advances across them too, while still never
shipping a catchup entry for the checkpoint itself.
2. ha_apply_term_entry_via_library (ha.c) now JIT-resyncs
ha->raft.log_len from persist_durable_floor() before consuming an
AE, to correct for persist_write_checkpoint_marker advancing
current_lsn without going through raft_log_append/record_entry_term
(pre-existing gap, bug-1466, not itself fixed here).
3. The actual root cause, found via real instrumentation after layers
1-2 alone still left a regression test red: record_entry_term's
array-growth realloc zero-filled new slots, silently reclassifying
a checkpoint's deliberately-never-recorded gap slot from "absent"
to "present, term=0/WAL_LEGACY_TERM" the instant any later LSN got
recorded -- a time-dependent read where a permanent fact was
assumed. Fixed with a TERM_INDEX_UNSET (UINT64_MAX) sentinel that
growth now fills gap slots with instead of zero, so entry_term_locked
can distinguish "never explicitly written" from "explicitly
recorded as legacy term 0" regardless of later array growth.
The same determinism fix exposed a second, independent instance of the
identical wedge via the snapshot-install catch-up path instead of
WAL-streaming catch-up: persist_reconcile_checkpoint_term recorded a
wire checkpoint_term of 0 as a raw, explicit write (bypassing the
growth-fill sentinel entirely), so a snapshot-installed standby read
the checkpoint LSN back as WAL_LEGACY_TERM(1) forever while the leader
now correctly read its own checkpoint LSN as absent(0) -- the same
permanent logOk mismatch, a different code path. Fixed by substituting
TERM_INDEX_UNSET for a wire value of exactly 0 before recording; a
wire 0 is unambiguous (persist_entry_term's own contract never returns
raw 0 for a present entry, so 0 can only mean genuine absence).
Independently reviewed across three rounds: sentinel mechanism
verified via real RED->GREEN revert-and-rebuild; a documentation
claim that config_repl.c/ha.c were "comment-only" corrected; the
snapshot-path gap independently traced, confirmed real via its own
RED->GREEN cycle, and fixed; two stale doc comments (persist.h's
public contract, ha.c's call-site rationale) still describing the
disproven pre-fix legacy-0 convention corrected. Suite: 1028 -> 1035
tests, 0 failures.
