Extends bug-1588's job.c fix to the remaining raw-%lu jobid logging sites:
8 worker daemon files (xcecd_worker.c, controller_connector.c, uv_ipc.c,
xcecd_worker_utils.c, xcecd_dbus_manager.c, xcecd.c, main.c, xcecd_cgroups.c)
and src/controller/testsuite/test.c (xce_test, heavily used for controller
testing). 99 call sites converted to fmt_jobid()+%s, matching the existing
convention. Non-jobid %lu values (lsn, client id, byte offset) and raw-numeric
identifier/path construction (checkpoint dir, systemd unit name, XCE_JOBID
env var) are deliberately left untouched.
Verified via /xce-build worker and controller (asan, --incremental): both
compile clean with zero warnings tied to the change; worker suite 4/4 OK,
controller suite 4/5 OK (the one failure is the pre-existing bug-1561
overlayfs fdatasync flake, unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
controller/main.c and scheduler/src/main.c now compose argp_program_version
via a genuinely separate buffer/call to xce_compose_version_ex() (reading
STR(VERSION_DIRTY_FINGERPRINT), matching the existing commit_id/build_time
macro pattern). The startup-log composition call site -- same function,
same arguments, same buffer -- is left completely unchanged in both files.
controller/testsuite/test.c (xce_test, confirmed to have no startup log)
gets a direct in-place swap. All three files carry explicit comments
marking the --version/startup-log divergence as deliberate.
Verified independently by review with real builds and live daemon starts:
--version shows <commit>-<fingerprint> for all three binaries, matching
compile_commands.json; live startup logs for both daemons show the
unchanged old <commit>-dirty format with the fingerprint never present.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Migrates argp_program_version in controller/main.c, scheduler/main.c,
and xce_test's test.c from the compile-time MK_VERSION/STR construction
to xce_compose_version(), computed into a static buffer and assigned
before argp_parse() runs (argp_program_version is a file-scope pointer,
not a constant expression it can be initialized from at compile time).
Each daemon's startup log now carries the identical version string
(never recomputed) on its first INFO line -- scheduler's BOTH
mutually-exclusive branches (--peers and no-peers), not just the one
literally named in the task text, since "when the daemon starts" is
unconditional.
Live-verified: --version and startup-log output are byte-identical for
both daemons across all branches. No remaining MK_VERSION callers
repo-wide.
Task 3.2/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
apply_committed_entry()'s shared WAL_JOB_START/FINISH/STOP/RESUME/RUSAGE/
CKPT/PREASON dispatch case only ever called the generic, state-only
apply_job_update_transition(j, state) -- start_time/finish_time/run_time
were never touched, so `xcecli joblist` always showed StartTime=0 for
RUN/DONE/EXIT jobs and FinishTime=0 for DONE jobs (EXIT jobs got a
finish_time only via the separate, richer WAL_JOB_FAIL dispatch path).
Fix is additive: stamp time(NULL) keyed on wal_type right after
apply_job_update_transition() succeeds, still under the same
lock(j, RTTI_LOCK_WRITE) -- WAL_JOB_START sets start_time, WAL_JOB_FINISH
sets finish_time and derives run_time. No change to
apply_job_update_transition()'s signature or the dispatch structure.
Add test_ace_job_start_sets_start_time and
test_ace_job_finish_sets_finish_time_and_run_time to
test_apply_committed_entry.c, following the existing tace_seed_job
fixture pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The schedAdmin dispatch case blocks its vtask handler thread for up to 5s
waiting on a cross-process reply while holding c->sched.lock -- bounded
blocking on a vtask thread, which ha-async-io.md permits only under an
admission reservation. The real risk isn't just the wait itself: since
c->sched.lock serializes concurrent requests one at a time, a burst of N
concurrent schedAdmin calls could already occupy N vtask threads for up to
N*5s each before this fix.
Now acquires vtaskmgr_blocking_acquire(&c->vtaskmgr) before ever locking
c->sched.lock, releasing on every exit path -- the same shared admission
budget commit.c's commit-wait barrier already gates on (wired via
commit_set_pool). Admission-exhausted requests fast-reject with
{ok=false, error="server busy, try again"} rather than queuing. New test:
test_schedadmin_fast_rejects_when_admission_exhausted, using a poison
c->sched.ep sentinel to prove the fast-reject happens before any
lock/forward attempt.
fmt_jobid() renders the raw idgen()-packed jobid in the same
cluster-apiserver-timestamp-sequence form already used everywhere else
this codebase logs/displays a jobid (scheduler.c/handler.c, worker's
process_manager.c, xcecli, waldump). job.c's own TRACE lines were the one
place still printing the raw 64-bit packed integer.
sched_worker_change() aliased w->host.resvals_to_sched to a local xlist_t's
backing array, pushed it to the scheduler, then freed that array via
xlist_fini() without resetting the alias - leaving a dangling pointer for
the rest of the worker's life. On disconnect (handle_worker_leave ->
mark_worker_unavail -> persist_worker_hook_fun -> capnp_encode_worker ->
encode_Worker -> encode_NameValue_list), this dereferenced freed memory and
crashed the controller - but only once a scheduler had ever connected
(on_scheduler_join gates the hook registration), which made it easy to miss
in isolated worker-only testing.
The correct reset already existed in the function but was dead-code'd
behind #if 0 /*FIXME*/ - apparently from a prior fix attempt that got
wrapped instead of landed. Fix mirrors the proven-correct pattern already
used by the sibling apiserver_worker_change(): null the alias before
xlist_fini() frees the backing array, rather than double-freeing via
free_name_val_list() first.
Adds test_sched_worker_change.c, a regression test driving the real
production hook-dispatch path (xce_run_hooks -> sched_worker_hook_fun ->
sched_worker_dead -> sched_worker_change) and the real encode_Worker()
encoder the original crash traversed - confirmed to reproduce the exact
ASAN heap-use-after-free when the fix is reverted, and pass clean when
restored, closing the gap that let this regress silently once already.
Found while live-testing the resource-value-verification feature (deploying
a scheduler + injecting worker resource values via xce_test, then
disconnecting) - unrelated to that feature's own changes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cmd_resupdate() now issues a follow-up xcecli_resval() query per submitted
(resource, worker) pair after the update RPC, since update_resval_list()'s
reply can't carry per-item status. Reports accepted/rejected per name, with
a bounded retry (5x100ms) against propagation races.
Live-confirmed: `resupdate -n mem -v 5 -n not-a-real-resource -v 1` reports
mem accepted and not-a-real-resource rejected in the same invocation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
set_resval() now delegates type resolution to resval_typeresolve.c; unknown
resource names default to numeric, overridable via a new -t <numeric|string>
flag. cmd_resupdate()'s getopt loop defers per-pair finalization so a
trailing -t still applies to its pair, while preserving the exact original
duplicate-value rejection behavior (round-2 fix after review caught a
last-write-wins regression in that refactor).
Live-confirmed: resupdate -n r15m -v 1.5 (previously silently dropped) now
succeeds and is visible via resval.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Move set_resval()'s resource-name lookup table into a new, dependency-light
resval_typeresolve.c/.h module so it's independently unit-testable (test.c
can't be white-box included into Unity). Adds RED tests for the not-yet-
implemented numeric-fallback/type-override behavior (task 2.1).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
xce_scheduler -i -s <replay>: "info worker <name>" showed
total=0.00 available=0.00 for every resource, even though the live
cluster (xce_test resval/workerlist) reported real non-zero values
for the same worker.
Two independent root causes, both live-traced end-to-end against a
real deployed cluster (value-level TRACE at every hop: controller
push, scheduler apply, snapshot save, snapshot load/decode):
1. src/scheduler/src/core_events.c: XCE_CORE_EVENT_WORKERS_PUSH
called sched_add_worker() -- a plain key_add(), add-only --
unconditionally on every push. A worker connecting triggers
multiple WORKERS_PUSH events in quick succession (an early push
before the controller finishes writing real capacity values,
carrying stale/zero data, followed moments later by a push
carrying the real values). The FIRST push always won the race;
every push after it for the same worker name silently failed
-EEXIST and its resource values were discarded -- not just at
join, for the rest of the process's lifetime (unlike jobs, there
was no "update an existing worker" event path at all). Fixed via
a new core_upsert_worker() (mirrors the existing
core_upsert_queue() add-or-update precedent): looks up the
worker by name first, updates status+resvals in place if found,
else builds+adds a new one as before.
2. src/scheduler/src/repl_info.c: repl_info_worker_one()/
repl_info_resource_one() checked `xce_val_t.which ==
XCE_VAL_NUMERIC` -- but XCE_VAL_NUMERIC (xce/types.h, value 1)
is a resource-DEFINITION value-type constant
(xce_resource_t.valtype's own enum), a completely different enum
from xce_val_t.which's real domain, Value_u_which
(controller.capnp.h, Value_u_fvalue=5). Every producer in this
codebase sets xce_val_t.which to Value_u_fvalue, never
XCE_VAL_NUMERIC, so the comparison was always false -- every
genuinely-numeric resval displayed as 0.00 unconditionally,
independent of fix (1). Corrected all 4 occurrences to
Value_u_fvalue.
Both fixed via TDD (RED test before each fix, GREEN after):
test_core_events.c's test_workers_push_twice_updates_existing_
worker_in_place (two pushes, same name, different values -- both
must apply) and test_repl_breakpoints_inspection.c's
test_info_worker_reports_resource_values (a real numeric resval
must display, not fall back to 0.00).
Live-validated by reproducing the user's exact original scenario
end-to-end (deploy a real cluster, record, `xce_scheduler -i -s
<replay>`, `info worker <name>`) -- now shows slots=22.00,
maxmem=33182089216.00 (30.9GB), mem=17151664128.00 (16GB), matching
the user's own live xce_test resval/workerlist numbers exactly.
Full scheduler suite green (41/41, excluding pre-existing unrelated
bug-1560); controller suite green (1102/1104, the same pre-existing
bug-1561 pair, confirmed unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
persist_flush()'s synchronous fdatasync path advanced durable_lsn on
success but never called on_durable_advance() -- only the async
fsync_worker did. This gap was previously masked by bug-1563's own
busy-spin (a fast accidental repoll covered for the missing
notification); fixing that spin unmasked it, causing
test_durable_majority_commit_end_to_end to fail deterministically
(3/3 runs).
Fixed by firing on_durable_advance() from persist_flush() too,
whenever it genuinely advances durable_lsn. Fixed via TDD (RED test
before the fix, GREEN after) and re-validated with a fresh 40-round
sequential + chaotic HA soak (both pass cleanly, zero fdatasync
errors, zero divergence/loss).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bug-1562: ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c) returned
without re-arming the liveness watchdog when a debounced RDHUP landed after
the in-flight PreVote round had already concluded (e.g. a lone cold-booting
node whose first round loses in milliseconds with no peers up yet). That was
the last timer ever scheduled -- the node froze permanently in CTRL_STANDBY,
so a 3-node cluster with a long inter-node start stagger could never elect a
primary. Root-caused via a live local reproduction matching the reporting
user's exact timing (byte-for-byte match of their hung ctrl-1.log), fixed by
re-arming the watchdog in that branch (mirroring the function's own 5.2
branch), and confirmed end-to-end: the same scenario now reaches CTRL_PRIMARY
and full 3-node config convergence.
bug-1563: fsync_worker's retry loop (persist.c) only backs off when no work
is pending, but flusher_work_pending_locked() can't distinguish "fresh work
just landed" from "still stuck from the last failed fdatasync" -- a
sustained sync failure left durable_lsn permanently behind tip, so the loop
skipped its wait every iteration and spun fdatasync as fast as the CPU/lock
allowed (a 40-round chaotic /xce-test-ha soak hit 300k+ failed attempts on
one node in under a minute). Fixed by waiting out the normal wal_sync_ms
cadence when a flush was pending but made no progress, leaving the
legitimate idle and just-succeeded fast paths untouched.
Both fixed via TDD (RED test before the fix, GREEN after) and live-validated
by re-running 40-round sequential and chaotic HA soaks end-to-end: both pass
cleanly, and the chaotic run's excluded fdatasync-error count dropped from
304,689 to 0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bundles the fixes validated this session across ha.c/persist.c/worker.c:
- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
closed with a recovery-complete gate that closes at process-lifecycle
start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
check mirroring the existing job-count check (count_jobmgr_entries and
count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
applies mark_worker_unavail()'s mutations via a shared
apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
fixed for promoted vote endpoints via a shared votes_idx_for_peer()
index-scale converter.
Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Simplification per review feedback on e3b3ffa: a generic schema-driven
unit field touched the capnp wire schema (2 structs), C structs
(xce_resource_t/xce_resval_t), resobj.c's deep-copy/destroy, opget.c's
reply-build lookup, resources.yaml/resources.schema.json, and mapping.py
-- real cross-cutting cost for a mechanism only one resource (maxmem)
actually needs today.
Reverted all of that. The controller CLI (test.c) now just knows
"maxmem" is bytes-valued via a small explicit is_bytes_valued_resource()
name list, and adaptively picks the best-fit human unit at display time
(format_bytes_human, unchanged) -- same observable behavior (resval/
workerlist render maxmem as e.g. "30.90 GB"), zero wire schema surface.
Regenerated the xceadmin golden fixture back to its pre-unit-field size
(5304 bytes, exactly matching the size right after the mxj-only fix,
confirming a clean, complete removal).
Verified: 1095/1095 controller_unit_test (3 runs; one hit the
pre-existing flaky segfault already logged as bug-1553, unrelated),
228/228 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three bugs found while diagnosing a live controller log, all in the
worker resource-value path:
1. Every host relying on the documented hosts.yaml auto-detect default
(no max_slot, meaning "use CPU count") got 0 scheduling slots,
permanently blocking the scheduler from placing jobs on it. mxj is
UInt32 on the wire and can't carry the schema's -1 sentinel; mapping.py
worked around this by leaving it unset, which defaults to 0 -- and 0 is
never < 0, so apply_join_state()'s auto-detect fallback never fired.
Fixed by encoding the sentinel as 0xFFFFFFFF, which the controller's
`int mxj` field reinterprets back to -1 via its existing implicit
uint32_t->int conversion. No C-side or wire schema change needed for
this part; the fallback itself was already correct but had zero test
coverage for a negative mxj, now added.
2. update_resvals()/update_resval_list() logged the failure reason via
PERROR, which reads process-global errno -- but update_resval()'s
failure paths return raw negative codes directly and never touch
errno, so the controller log filled with misleading "update_resval:
Success" / "update_resval: Resource temporarily unavailable" lines
instead of the real reason. Fixed to decode the actual rc via
xce_strerror_r and name the failing resource.
3. maxmem is canonically stored/wired as raw bytes but had no unit
metadata, so resval/workerlist printed it as a bare number that read
as garbage (e.g. workerlist's fixed-width columns visually smeared a
multi-GB byte count across the MaxCPU/MaxMem headers). Added an
optional `unit` field to the Resource/ResourceVal capnp structs (and
xce_resource_t/xce_resval_t/resources.yaml) so resval and workerlist
can render a human-readable value (e.g. "30.90 GB") while the stored
value stays raw bytes -- display-only, every other resource unaffected.
Regenerated the xceadmin golden fixture (fixtures/full/expected.bin)
twice: once for the mxj sentinel encoding change, once for the new
Resource.unit field growing every resource's wire size by one pointer
slot. Both diffs verified byte-for-byte isolated to exactly the expected
change before regenerating.
Verified: 1095/1095 controller_unit_test (stable across repeated runs),
229/229 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Two related findings logged to .wolf/buglog.json but deliberately not
fixed here (out of scope per discussion): a pre-existing flaky segfault
in the combined controller_unit_test binary, and maxswap/maxtmp being
hardcoded stub values on the worker side.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Add OPCode_probeConfigTag dispatch in cluster.c
- Implement probe_config_tag handler returning config tag via
standard xce_reply() → RequestReply envelope
- Populate redirectHost/redirectPort for follow_redirects support
- Wire probeConfigTag reply through typemap
- Add probeConfigTagReply struct to controller.capnp schema
controller_unit_test crashed deterministically in write_WalCatchupEntry
(test_wal_catchup_append_term_fields_round_trip) -- the same bug class
as bug-1504 (SnapshotData.internalState): this codebase's capnpc-c
backend silently no-ops Data-field encoding in generated encode_XXX()
composite helpers without zero-initializing the caller's local struct
first, leaving an uninitialized capn_ptr written onto the wire.
Confirmed zero production exposure before fixing: xce_wal_batch_encode_
raw()/wal_entry_to_wire() (request.c) never call the generic
encode_WalCatchupEntry() at all -- they memset() their own struct and
call write_WalCatchupEntry() directly with every field hand-set, so
this was 100% test-only exposure (grepped every caller of the generic
encode/decode pair; only the crashing test uses it).
Fix mirrors bug-1504: WalCatchupEntry.payload (@2) and batchPayloads
(@10) switched from Data to List(UInt8) in controller.capnp (new
batchPayloadsLen @11 companion; payload reuses the existing
wirePayloadLen @9), typemap.h's wal_catchup_entry_t field types
updated, and request.c's extract_data_bytes() widened to take a raw
capn_ptr (it already only used .p internally) so its call sites work
regardless of which Data-shaped wrapper the generated accessor
returns now. batchPayloads' single-top-level-blob shape confirmed
this does not reintroduce bug-847's per-element cross-segment
far-pointer loss -- that issue is inherent to per-element data
regardless of Data vs List(UInt8) spelling.
Strengthened test_wal_catchup_append_term_fields_round_trip with real
payload/batchPayloads content assertions -- previously never
exercised, since the test's src fields were left NULL.
Verified: controller suite 3/3 (was 2/3, crashing), scheduler suite
unaffected at 37/37 (unrelated schema area).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the interactive simulation REPL's execution-control slice (Req 5
AC 22): a single-threaded input loop (readline-based) supporting
step [N], continue, quit, and help, pausing before the first replay
event and blocking the driver loop while active.
Review round 1 raised two findings, both resolved here:
- GNU readline (GPL) was linked directly into xce_scheduler, a shipped
production daemon. Switched the whole project to libedit (BSD) for
line editing -- scheduler and controller (xce_test) both now use
editline/readline.h, API-compatible (readline()/add_history()).
- Getting a clean regression run after the libedit swap required a
full clean rebuild, which exposed a genuine pre-existing SIGSEGV
(bug-1504): SnapshotData.internalState was a capnp Data field, and
this codebase's capnpc-c backend silently no-ops Data-field
encoding without zero-initializing the caller's stack struct,
leaving an uninitialized capn_ptr written onto the wire. Fixed by
switching internalState to List(UInt8) (with a new internalStateLen
wire field), matching the working FetchFileReply.content/bytesRead
precedent -- no hand-written accessor code needed.
A second, unrelated instance of the same bug class was found in the
controller's WAL-catchup path (WalCatchupEntry.payload) and
deliberately left unfixed as bug-1505, out of scheduler-core's
boundary; needs its own project-wide Data-field audit.
Scheduler suite: 34/34 clean. RED-phase reproduction recorded in
tasks.md (repl_should_pause() step-budget check disabled -> 2/4
tests fail as expected -> reverted -> 34/34 green).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Closes the two genuine gaps beyond task 10.1's own coverage (standby and
disconnected-scheduler rejection were already tested):
- Forward timeout: adds an XCE_TESTING-only timeout-override seam
(SCHEDADMIN_TIMEOUT_MS) in cluster.c, verified inert in production
builds (-DXCE_TESTING absent from xce_controller's target and every
project-wide arg; the non-test branch is a bare 5000L literal).
test_schedadmin_forward_timeout drives the real 5s (shrunk to 200ms
under test) pthread_cond_timedwait to genuine expiry and asserts the
exact {ok=false, error="request timeout"} shape from AC 10.
- Round-trip coverage: two more outcome-branch tests
(rpc_status_only, business_error) using the same real xce_encode/
xce_decode functions production uses. A literal two-binary
controller<->scheduler integration was judged out of scope (separate
meson targets, incompatible fixtures) since task 9.4's own
test_admin_request.c already white-box-drives the real scheduler
handler rigorously.
Controller suite 3/3 (1093/1093 assertions, up from 1090), scheduler
26/26 unaffected, clang-format clean, zero diff-hygiene noise (git diff
matches git diff -w for cluster.c).
Adds the spec's one and only src/controller/ change: a new
case OPCode_schedAdmin in myreq_handler() (cluster.c) that mirrors the
existing OPCode_upgradeConfig case -- primary-only gate, explicit reply
on every path, stateless pass-through with no WAL entry.
Forwards to the scheduler via a new xce_schedadmin_req() (async send,
mirrors xce_fetchfile_req) and blocks the vtask thread on a condvar with
a 5s deadline while holding c->sched.lock for the round-trip, matching
this spec's own research-phase design decision (parity with
OPCode_upgradeConfig's no-RBAC posture, blocking wait accepted for
low-frequency admin traffic). A new branch in the existing
OPCode_reply case demuxes the scheduler's SchedAdminReply to a
process-global single-slot pending-waiter mechanism, serialized by
c->sched.lock, matching job.c's existing fetchfile-proxy pattern.
Adversarial review independently traced double-free/cleanup, the
timeout/signal race, and reply-opcode demux correctness against source;
found and a follow-up round closed a narrow-window leak via a carefully
lock-ordered unlock->unregister->relock sequence (avoiding an AB-BA
inversion against schedadmin_signal()'s lock acquisition order).
New test_sched_admin_dispatch.c drives the real myreq_handler via the
existing myreq_handler_for_test seam: not-primary rejection (proven via
a poison-pointer sched.ep that would crash if forwarding were ever
attempted), not-connected rejection, and a genuine two-thread round-trip
test exercising the real signal/demux path.
Controller suite 3/3 (1090/1090 assertions), scheduler suite 26/26
unaffected, clang-format clean.
Audit of all ~19 new types/enums from tasks 7.1-7.4 found one genuine
gap: ReplayEvent's own envelope union (11 members across 7.1+7.2) was
never actually constructed/encoded in any test, despite its individual
payload types each having their own round-trip test. Adds 3 tests
(jobStateChange/jobUpdate/endSession variants) proving the union
mechanism itself round-trips correctly.
No new schema/typemap changes -- audit-only. Decided against creating
design.md's originally-named test_replay_format.c: tasks 7.1-7.4
independently, consistently (4-for-4) chose the existing
test_e2e_capnp.c instead. Deliberately did not add Request.body/
RequestReply.data union-wiring tests for OPCode_schedAdmin, matching
universal codebase precedent (opcode-dispatch union members are always
tested via functional dispatch tests, never bare schema round-trips,
and the real dispatch logic doesn't exist until task 10.1).
Closes out Group 7 (Foundation: Replay Wire Format) entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds SchedAdminAction (recordStart/recordStop/recordStatus),
SchedAdminRequest (action/recordFile), SchedAdminReply (ok/error/
recording/file/eventCount/startedAt) to controller.capnp, matching
requirements.md Req 5 AC 10 verbatim. Adds schedAdmin@42 to OPCode,
schedadminmsg@31 to Request.body, schedadminreplymsg@19 to
RequestReply.data -- the exact next-available ordinals, verified live
before implementation and re-verified byte-identical against every
existing value/member.
Deliberately schema-only, per this task's own scope -- the controller
dispatch logic (cluster.c's case OPCode_schedAdmin:) is task 10.1's
job. cluster.c, src/frontend/, src/xceadmin/, src/scheduler/ all
untouched.
Round-trip tests cover all 3 SchedAdminAction values and both
ok=true/ok=false SchedAdminReply cases, including an asymmetric
ok=false/recording=true combination that rules out a Bool-field
bit-packing mixup at the wire level.
Closes out Group 7 (Foundation: Replay Wire Format) entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds DispatchDecision (wraps JobAlloc), PendDecision (wraps
JobPendReason), PreemptVictim, PreemptDecision to controller.capnp.
Replaces task 7.1's empty SnapshotData placeholder wholesale with the
real 11-field struct (config/jobs/queues/workers/resources/resvals/
users/projects/prngSeed/internalState/internalVersion), in both the
capnp schema and its typemap.h C mirror. Extends ReplayDecision's
ts-only placeholder with a purely additive body union (dispatch@1/
pend@2/preempt@3).
Deviates from design.md's literal SchedUser/SchedProject mapnames
(xce_sched_user_t/xce_sched_project_t): those names are already real
C structs in sched/types.h, which shares a translation unit with
typemap.h in core_events.c/schedobj.c -- reusing them would be a
genuine duplicate-typedef compile error, not a style choice. Used
fresh mapnames (sched_snapshot_user_t/sched_snapshot_project_t)
instead, field lists unchanged, documented in both files.
Found and fixed a real bug mid-implementation: a doc comment's
literal */ substring prematurely closed a C block comment, corrupting
sched_snapshot_data_t and cascading into ~20 unrelated-looking compile
errors (bug-1493).
Five deep round-trip tests added to test_e2e_capnp.c. Full project
build and controller_unit_test (1082/1082) verified clean, confirmed
independently three times.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds JobStateChangeEvent (jobid/oldState/newState/exitCode) and
JobUpdateEvent (jobid/changedFields/job: Job) to controller.capnp,
matching design.md's spec verbatim. JobUpdateEvent.job reuses the
existing full 12-field Job/xce_job_t record, verified distinct from
the narrower SchedJob struct sharing the same mapname.
Extends ReplayEvent.body's union (task 7.1) with jobStateChange@10/
jobUpdate@11 -- the only two valid next ordinals, since capnp requires
strictly sequential ordinals with no gaps across a struct and task
7.1 deliberately stopped at @9. Existing @0-@9 fields and every other
struct in the schema are byte-identical.
Deep round-trip tests in test_e2e_capnp.c exercise every field with
non-default values, including the 3-level nested Job substructs
(submitData/allocs/pidinfo/pendreason). JOB_F_* bitmask bit values
deliberately left undefined -- a later task's job, not this
schema-only one's.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds ReplayCluster/ReplayHeader/SnapshotEntry/ReplayEvent/
SessionSummary/ReplaySession to controller.capnp, additively, first
task of Phase 3 (Foundation: Replay Wire Format).
Requirements.md's AC 20 capnp sketch isn't self-contained -- ReplayEvent's
union references JobStateChangeEvent/JobUpdateEvent (task 7.2's job),
SnapshotEntry.data/ReplayDecision's union reference SnapshotData/
DispatchDecision/PendDecision/PreemptDecision (task 7.3's job), none
of which exist yet. Resolved by verifying capnp's real constraints
against the toolchain directly: SnapshotData is an empty {} placeholder
(task 7.3 replaces its mapname'd C struct wholesale when adding the
real ~11 fields), ReplayDecision holds only {ts} (task 7.3 appends a
body union at new ordinals), and ReplayEvent's union holds only the 9
members with pre-existing backing types (Conf4Sched/SchedJobData/
WorkerData/ResourceData/ResValList/QueueConf/SchedJobRusages/Void),
deferring jobStateChange/jobUpdate to task 7.2 as new ordinals --
capnp requires strictly sequential ordinals with no holes, so this
split is compiler-mandated, not stylistic.
Round-trip test added to test_e2e_capnp.c. Full project build (all
binaries sharing this schema) and 1075/1075 controller_unit_test
verified clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
bug-1421 added two term-discovery fallback vectors so a stale-term primary
steps down instead of retrying AppendEntries forever: heartbeat-reply
state-report term adoption and WAL-NACK reply_term adoption, both inside
repl_ack_decode_and_dispatch. Neither had a test driving the real
production dispatch path and asserting raft_on_higher_term_observed
actually fires.
Adds two tests mirroring test_config_repl.c's existing config-push term
test, driving peer_ctrl_msg_handler -> repl_ack_decode_and_dispatch via
wire-shape capnp packets. Both pass immediately -- the underlying bug-1421
code was already correct, just unverified end-to-end -- closing
raft-adapter-conformance's Requirement 7.5 unverified claim.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXkenK8rqE8mdS1YKjsq78
ha_arm_heartbeat_timeout_vtask published a calloc'd liveness_poll_ctx_t
into s_liveness_ctx unconditionally but only ever freed it via
liveness_poll_fn's cancelled-check path, which never runs if
delay_task_ms was never called (c==NULL, delaymgr not running) or itself
failed. Both no-schedule cases now free the ctx immediately, establishing
the invariant that s_liveness_ctx is non-NULL only when a delaymgr task
actually holds a live reference to it.
Regression test drives a real LOST PreVote round without stubbing
ha_prevote_rearm_liveness_hook (every other LOST-path test in the file
stubs it specifically to avoid this code path) and asserts the ctx isn't
left dangling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXkenK8rqE8mdS1YKjsq78
wal_type_name() never had a case for WAL_CONFIG_MARKER (type 50) -- every
config-change marker entry printed as UNKNOWN instead of a real, defined
type. Added the case plus payload decode (tag/sha256) matching config_repl.c's
config_marker_encode/_decode byte layout, hand-decoded here rather than
linking config_repl.c into xce_test's link graph.
Gap detection also had no way to tell "committed prefix missing!" (real
loss) apart from a normal checkpoint-triggered WAL trim (persist.c's
trim_wal_segments, floor-guarded so it never removes a segment a standby
still needs) -- every long-running node's WAL, once checkpointed, would
false-alarm on every inspection. Cross-checks the sibling data/snap/
directory (snapshot LSN is literally in the filename, snap-<hex>.capnp)
and downgrades the message when a snapshot covers the gap.
Adds waldump_stats_t.leading_gap_explained and 3 new tests plus one
assertion on the existing gap test.
CLAUDE.md's TLA+ Action-to-Code Mapping table gains corrected/new rows:
production vs. dormant AppendEntries send mechanisms, WAL catch-up and
config-push reply integrity, and vote/PreVote failure routing across
all failure timings -- each citation independently re-verified against
live source. The old AppendEntries row's citations had already drifted
onto unrelated code before this task; corrected, not just appended to.
Protocol Verification Checklist gains 3 new items for the guarantees
this feature closes.
ha-term-safety/spec.json's known_limitations entry for the dormant
send-path drift gets an additive RESOLVED pointer back to this spec's
disposition, so a future reader lands on the answer instead of
re-discovering the drift.
Requirement 1 (systematic conformance audit deliverable) complete.
Adds test_batch_payload_persist_failure_propagates_to_whole_batch,
proving persist_submit_job_payloads_async resolves every payload in a
3-entry batch with the same first-failure rc when the first payload's
append fails, rather than letting later payloads independently
succeed (Req 5.3).
Fault injection: chmod the WAL leaf directory to 0500 to force
open_new_segment's EACCES (surfaced as -EIO), restored the instant the
first failure is observed -- a transient fault, not a permanent one,
so the test genuinely distinguishes propagation from independent
per-payload failure. No existing fault hook was reachable from this
code path; this is a real, previously-uncovered gap.
1066 -> 1067 tests, 0 failures. Task 5 (regression-lock tests for
already-shipped/already-conformant behavior) now complete -- Core
phase of raft-adapter-conformance finished (tasks 1-5).
Adds test_5_1_wal_catchup_reject_sends_nack_reply, driving a genuine
(term,lsn) conflict against a real persisted WAL entry through
ha_apply_term_entry_via_library and asserting
ha_send_wal_ack_reply_ex(success=0) fires via ha_wal_ack_send_hook.
Note: test_config_repl.c's test_bug1479_ae_reject_sends_nack_not_silence
already covers the same call site via a different (forward-gap)
stimulus, added in the original bug-1479 commit. design.md's own File
Structure Plan explicitly calls for this new test in this file
regardless, so this is disclosed, design-sanctioned defense-in-depth,
not accidental duplication -- both tests independently confirmed to
fail together when the reply call is removed.
1065 -> 1066 tests, 0 failures.
New test drives repl_broadcast and ha_stream_wal_to_standby through
real production fixtures (reused from tasks 2.3/6.2), asserts both
actually fired (non-vacuous), and asserts
ha_send_append_entries_stub_call_count stays 0 throughout -- the
Requirement 3.3 non-regression guard. RED-verified: manually invoking
ha_cb_send_append_entries_for_test inside the test reliably trips the
assertion; reverted for GREEN.
1064 -> 1065 tests, 0 failures. Task 4 (dormant sender documentation +
guard) now complete.
ha_cb_send_append_entries_stub's doc-comment now plainly states it is
reachable only via raft_send_append_entries (zero production callers,
grep-confirmed), and names repl_broadcast/ha_stream_wal_to_standby as
the actual production send/retry/backoff mechanism (Req 3.3, amended
to permit this documentation-only resolution over retire-or-wire-in).
Adds an XCE_TESTING-only ha_send_append_entries_stub_call_count
counter, mirroring the ha_wal_ack_send_hook idiom, as the guard against
this ever silently regaining a production caller. Task 4.2 will drive
the real production path and assert it stays zero.
1063 -> 1064 tests, 0 failures. Zero behavior change to any production
send path.
Adds pre_vote_ep_err_handler_for_test (XCE_TESTING-guarded) and two
tests: PreVote post-connect failure reaches pre_vote_reply_dispatch's
tally (Req 4.3), and a round mixing that failure mode with a delivered
reply reaches its quorum decision via the fast all-replied path
without the full round timeout (Req 4.5). Real-vote coverage (Req 4.2)
was already proven by task 3.1's own test, cited not duplicated.
pre_vote_reply_dispatch has no mock hook seam, so both tests assert
directly on raft_state_t's tally fields, mirroring test_ha_prevote.c's
existing verification technique.
1061 -> 1063 tests, 0 failures.
Splits vote_ep_err_handler into a real-vote handler and a new
pre_vote_ep_err_handler, each registered at its existing (already
separate) connection-setup call site. Extracts the identical
CTRL_PRIMARY-role cleanup into a shared vote_ep_handle_promoted_disconnect
helper, behavior unchanged.
Each handler's election-phase branch now routes a post-connect failure
(RDHUP/SSL-error/remote-close after connect, before any reply) through
the same production tally the connect-time failure branches already
use (vote_reply_dispatch / pre_vote_reply_dispatch, bug-1423/1424's
fix), instead of the legacy handle_request_vote_reply that never
updated the tally. Closes the one failure window bug-1423 didn't cover.
1060 -> 1061 tests, 0 failures. PreVote-specific test coverage is
task 3.2's scope (pre_vote_reply_dispatch has no hook seam analogous
to xce_vote_reply_hook; verified only by code symmetry + full-suite
regression here).
Adds the remaining 3 of 4 handle_repl_config_push reject-path tests
(malformed request, missing bytes/sha256, conf_save_config failure via
a deterministic ENOTDIR fixture), the EAGAIN-still-accepts test, and
term-adoption-on-accept (task 2.2's own test only covered the reject
case). Requirement 2.4 (unknown-outcome handling) is already covered
by the pre-existing test_wait_all_with_timeout_slow_threads, cited
rather than duplicated.
1055 -> 1060 tests, 0 failures.
myreq_handler's ReplConfigPushReply branch now routes cack->term
through raft_on_higher_term_observed (same entry point the
heartbeat-reply and WAL-NACK vectors use), regardless of accept/reject,
without altering the existing accept/reject branching.
Adds myreq_handler_for_test (XCE_TESTING-only) to drive the real
dispatcher in tests, following the peer_ctrl_msg_handler_for_test
precedent. Requires cluster.c in controller_unit_test_src plus the
established link-satisfying fake stubs for its unrelated switch-case
dependencies (same pattern already used by test_worker_join.c and
test_commit_barrier_reply.c).
handle_repl_config_push now sends a ReplConfigPushReply{accepted,term}
from every exit path instead of silently dropping on the four reject
conditions (malformed request, missing bytes/sha256, sha256 mismatch,
conf_save_config failure). Term is sourced live from c->ha.term.
-EAGAIN activation-defer still replies accepted=1, unchanged.
Adds an XCE_TESTING-only send-hook mirroring ha_wal_ack_send_hook so
tests can observe the reply payload without a live endpoint.
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:
1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
library's nextIndex backoff and immediately repl_start_catchup's the
standby at the corrected position, instead of the 16-strike silent
eviction with no catch-up driver after it. Correct from the start, but
live soak still failed identically: nothing ever produced the NACK this
code was waiting for.
2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
silently dropping the reject. ha_send_wal_ack_reply had hardcoded
success=1 unconditionally -- the only WalAckReply sender in the
controller -- making (1)'s NACK-routing correct but unreachable dead
code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
the NACK without a live endpoint. Live-confirmed the NACK now reaches
the primary -- but the scenario still failed for a third reason.
3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
catch-up range was hard-bounded at committed_lsn, which can never advance
in this exact scenario because advancing it requires an ACK from the very
standby that's stuck catching up (2-of-3 alive, this standby is the only
path to a new majority) -- a self-referential deadlock. Widens the
STREAM_WAL ceiling (and the convergence check gating it) to the leader's
own log tail, matching raft.tla's real AppendEntries semantics
(leaderCommit is advisory, never a send-cap), while reusing bug-1478's
exact guard so this doesn't reopen that trap (never widen onto a trailing
local-only checkpoint's own unstreamable LSN).
Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A node that takes myapp_run's skip_election passive path (bug-849 --
"leader already known after the startup stagger") makes exactly ONE
cluster_init_phase2 attempt. All 5 call sites (myapp.c's skip_election
block, raft_cb_become_leader, raft_cb_become_follower,
ha_elect_evaluate_and_signal's two branches) latched ha->phase2_done = 1
unconditionally after calling cluster_init_phase2, discarding its return
value. If that one attempt raced config availability and failed, the node
was permanently stranded: commit_engine_init never ran, so
apply_committed_entry never applied any committed WAL entry for the rest
of the process's life -- and for a skip_election node specifically,
nothing else would ever retry it, since the other 4 call sites only fire
via that node's own raft-library state transitions, which never happen
for a passive standby whose primary keeps heartbeating normally.
Extracted a shared, idempotent retry (ha_phase2_retry_fn, mirrors
upgrade_materialization_retry_fn's shape) plus a delaymgr-scheduled,
self-rescheduling trigger (ha_schedule_phase2_retry) -- self-rescheduling
because, unlike config_activate_retry/upgrade_materialization_retry which
are re-triggered externally by recurring commit-advance events, nothing
else recurs for a node stuck on the passive path. Wired into all 5 call
sites; raft_cb_become_leader additionally refuses promotion on failure
(mirroring the sibling upgrade_materialization_fenced refusal already in
that function -- a promoted primary with an unwired commit engine is far
worse than a delayed promotion).
TDD: 4 new tests, RED confirmed by temporarily reverting the rc-gating.
GREEN: full suite 1044 tests, 0 failures. Live-verified via
run-ha-soak.sh --config-upgrade (the scenario that found this bug): the
retry fires for real (4 failed attempts, then success) and the soak's
checkpoint-identity/config-identity/host-admin-status assertions -- the
things bug-1477 blocked -- all pass.
Two independent call sites duplicated target = max(committed_lsn,
last_written): ha.c's heartbeat/stateReport dispatch (repl_ack_decode_and_
dispatch) and config_repl.c's catchup_derive_target (used by the ACK-driven
catch-up continuation path). Once config-upgrade-snapshot's marker-triggered
checkpoint advances last_written one LSN past an already-committed
committed_lsn (the checkpoint's own consumed, never-replicated LSN slot),
the unconditional max() permanently wedges every standby's catch-up target
at an unreachable value -- the primary re-streams an empty range every
heartbeat forever.
Extracted the shared, pure ha_compute_repl_target_lsn(committed_lsn,
last_written) (ha.h/ha.c), which only falls back to last_written on genuine
committed_lsn==0 bootstrap, and pointed both call sites at it so they can
never disagree again.
Fixing the second call site exposed three pre-existing tests in
test_convergence_verification.c that hardcoded the old buggy formula's
output as their expected target; updated their fixture values (committed_lsn
4->9, matching appended_lsn) to decouple them from the now-fixed edge case,
which already has dedicated pure-function coverage in test_ha_role.c.
GREEN: full controller_unit_test suite = 1040 tests, 0 failures, clean
across multiple runs. Live-verified via run-ha-soak.sh
--config-upgrade-wal-trim (18 jobs, zero job loss, ctrl-1 reconverged 3-of-3
using the preserved floor segment) -- unblocks config-upgrade-snapshot
task 8.9.
myconf->lock (mylock_init/fini wiring) had landed but load_primary_config()
itself was left in its RED-phase reproduction state (unguarded, decoding
directly into the shared myconf->cluster_config field), so the fix's own
regression test failed on resume: thread B raced through while thread A was
parked mid-call.
Wraps the whole free-decode-publish sequence in myconf->lock (acquired first,
released on every return path) and decodes into a local cluster_config_t*
first, publishing into myconf->cluster_config only after full decode and
validation succeed -- closing the UAF/double-free window a live 3-node ASan
soak caught (two concurrent handle_repl_config_push deliveries for the same
standby racing two load_primary_config calls).
Verified 4/4 clean full-suite runs (1036 tests, 0 failures) including the
specific race test.
A restarted standby could get permanently stuck unable to catch up,
with the primary re-streaming the identical doomed AppendEntries range
every heartbeat with zero progress -- discovered via a live 3-node
config-upgrade-snapshot soak. Liveness-only; no safety invariant
(Election Safety, Leader Completeness, State Machine Safety, Commit
Durability) was ever violated.
Three-layer root cause, all now closed:
1. ha_stream_wal_to_standby (config_repl.c) used to skip advancing its
prev_streamed_lsn/prev_streamed_term tracker across WAL_CHECKPOINT
entries (a local-only, never-replicated marker that still consumes
a real LSN slot). Now advances across them too, while still never
shipping a catchup entry for the checkpoint itself.
2. ha_apply_term_entry_via_library (ha.c) now JIT-resyncs
ha->raft.log_len from persist_durable_floor() before consuming an
AE, to correct for persist_write_checkpoint_marker advancing
current_lsn without going through raft_log_append/record_entry_term
(pre-existing gap, bug-1466, not itself fixed here).
3. The actual root cause, found via real instrumentation after layers
1-2 alone still left a regression test red: record_entry_term's
array-growth realloc zero-filled new slots, silently reclassifying
a checkpoint's deliberately-never-recorded gap slot from "absent"
to "present, term=0/WAL_LEGACY_TERM" the instant any later LSN got
recorded -- a time-dependent read where a permanent fact was
assumed. Fixed with a TERM_INDEX_UNSET (UINT64_MAX) sentinel that
growth now fills gap slots with instead of zero, so entry_term_locked
can distinguish "never explicitly written" from "explicitly
recorded as legacy term 0" regardless of later array growth.
The same determinism fix exposed a second, independent instance of the
identical wedge via the snapshot-install catch-up path instead of
WAL-streaming catch-up: persist_reconcile_checkpoint_term recorded a
wire checkpoint_term of 0 as a raw, explicit write (bypassing the
growth-fill sentinel entirely), so a snapshot-installed standby read
the checkpoint LSN back as WAL_LEGACY_TERM(1) forever while the leader
now correctly read its own checkpoint LSN as absent(0) -- the same
permanent logOk mismatch, a different code path. Fixed by substituting
TERM_INDEX_UNSET for a wire value of exactly 0 before recording; a
wire 0 is unambiguous (persist_entry_term's own contract never returns
raw 0 for a present entry, so 0 can only mean genuine absence).
Independently reviewed across three rounds: sentinel mechanism
verified via real RED->GREEN revert-and-rebuild; a documentation
claim that config_repl.c/ha.c were "comment-only" corrected; the
snapshot-path gap independently traced, confirmed real via its own
RED->GREEN cycle, and fixed; two stale doc comments (persist.h's
public contract, ha.c's call-site rationale) still describing the
disproven pre-fix legacy-0 convention corrected. Suite: 1028 -> 1035
tests, 0 failures.
repl_add_standby()'s "find first empty slot" scan treated any ep==NULL
slot as available, with no awareness that a disconnected slot can still
hold an unexpired bug-1147 WAL-retention guard (last_disconnected_lsn,
set by repl_remove_standby, meant to be cleared only by
repl_expire_disconnect_slots once snap_interval_s genuinely elapses).
The scan could pick a guard-holding slot over a genuinely virgin one
purely by lower index, then unconditionally memset it -- destroying the
guard before trim_wal_segments/repl_trim_lsn could ever honor it.
Confirmed via 2/2 reproducible live 3-node soak runs (config-upgrade-
snapshot task 8.9's new --config-upgrade-wal-trim mode): a killed
standby's still-needed WAL segment got trimmed out from under it.
Now a two-pass scan: prefer a genuinely virgin slot; only fall back to
reusing a guard-holding slot (oldest last_disconnect_time first, to
discard the least remaining protection) when no virgin slot exists --
logged via WARN() so the discarded guard is operator-visible rather
than silent. Defense-in-depth at the slot-allocation layer, independent
of whatever triggers a given repl_add_standby call (a separate,
not-fully-root-caused ha.c reconnect-sweep defect is the specific
trigger seen in the soak, out of this fix's scope).
Independently reviewed across two rounds: round 1 flagged the
exhausted-fallback path silently discarding a guard with zero logging;
remediated with a WARN() matching this file's existing repl_broadcast
convention, empirically confirmed to fire only on the fallback path
with correct content. Suite: 1028 -> 1030 tests, 0 failures (net of
this fix; final combined total with bug-1472 landing concurrently:
1033 tests, 0 failures).
worker_from_capnp()'s idempotency guard returned -EEXIST unconditionally
whenever the target worker already existed, discarding the decoded WAL
payload entirely. Correct for a duplicate WAL_WORKER_CTRL/JOIN replay,
but wrong for HOOK_DETAIL_WORKER_UPDATE (hostctrl -c/-o admin close/open):
the target existing is the normal case there, yet the status change was
silently dropped on every member other than the one that processed the
RPC directly -- a permanent, unconditional, cluster-wide replication gap
for host admin-status, confirmed via byte-precise checkpoint divergence
on a live 3-node soak.
Now decodes the entry's hook_detail before applying the existing-worker
guard: JOIN keeps its no-op short-circuit; UPDATE replaces (not ORs,
since HOST_OPEN must clear closed bits) the existing worker's
XCE_WORKER_ADMIN_MASK bits with the decoded payload's, in place, under
the worker's own write lock. HOOK_DETAIL_WORKER_DEAD is deliberately
left unfixed -- its replication would need fields (resvals) that never
round-trip through the Worker capnp wire schema at all, and this
codebase already treats liveness state as locally-derived rather than
WAL-replicated elsewhere -- filed separately as bug-1472-remainder.
Independently reviewed: RED->GREEN cycle re-reproduced from scratch,
replace-vs-OR semantics and lock discipline verified directly, DEAD
deferral corroborated against an independent design-invariant comment
this fix didn't originally cite. Suite: 1030 -> 1033 tests, 0 failures.
Add regression test proving Req 4.4/7.2: a second WAL_CONFIG_MARKER
committing while the first's forced checkpoint is still in flight
(simulated via the checkpoint_in_progress CAS across both markers'
commit+apply) cleanly supersedes it, with the settled state reflecting
only the newer marker. Verified via a field-aware byte comparison
(post-checkpoint_lsn) against a non-racing control fixture, plus
mid-race assertions that materialization is still incomplete at that
point (cfgref_present false, fence set) despite activation already
tracking the newer config. Test-only; production mechanism (already
implemented) unmodified. Suite: 1027 -> 1028 tests, 0 failures.
Prove Requirements 5.2/5.4/5.5/6.3 via deterministic fault injection at
the two critical upgrade windows, entirely at the unit level using the
XCE_TESTING hooks earlier tasks built for exactly this purpose -- no
live soak needed.
test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization
extends the existing fault-hook seam test: after simulating a crash
before the checkpoint write durably completes, it boots a genuinely
fresh persist_t/cluster_t against the same on-disk WAL via
persist_recover() (a real boot-replay simulation, not a same-process
re-call) and confirms materialization resumes and completes correctly.
test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain
proves a promoted member's materialization completes identically to a
non-promoted control member's, by draining each through its actual
production entry point and comparing checkpoints byte-for-byte -- a
regression guard against any future role-conditional branch creeping
into the marker-apply path.
Build the live 3-node chaotic-multi-kill config-upgrade soak scenario
(run-ha-soak.sh --config-upgrade) required by Requirements 4.1, 4.3, 6.1,
6.4. This needed genuinely new client-side infrastructure: a hand-rolled
capnp send path for OP_upgradeConfig (the auto-generated encoder is a
no-op for its Data fields, worked around the same way
xce_replconfigpush_req_raw already does), plus cmd_hostctrl/cmd_upgradeconfig
CLI commands.
Building and running this live surfaced two real production bugs, both
fixed here:
- bug-1468: an unrelated prior commit inverted OPCode_upgradeConfig's
primary/standby dispatch guard in cluster.c, making config upgrades
unreachable from any network client.
- bug-1469: apply_committed_entry's WAL_CONFIG_MARKER case never called
config_activate_committed(), so the node that originates a config
upgrade never refreshed its own in-memory state as part of committing
it. Killing that node before some later, unrelated event happened to
refresh it left its checkpoint permanently stale -- the first live,
empirical reproduction of bug-857's originally hypothesized failure
mode. Fixed by activating before the marker-triggered checkpoint write,
since checkpoint content is built from live runtime tables that only
reflect the reload after activation runs.
A separate, narrower checkpoint divergence (bug-1471, not config-related)
remains open and still blocks a clean soak PASS -- task 8.6 stays
in-progress until it's resolved.
