After an election where the vote connections are not promoted, both
standbys register later via on_standby_connected; the own-term no-op was
appended before any standby existed, and catch-up waited for heartbeat-
driven state reports (config push one 3 s cycle, WAL stream the next), so
the no-op committed ~6 s late and every client write in that window
failed -110.
ha_send_catchup_probe() sends the ordinary heartbeat request (same fields
as primary_heartbeat_fn; encode + enqueue only; primary only) at
registration and once after the standby's first accepted config-push ACK,
so the decision loop's next action follows at ~RTT. The decision loop,
its one-action-per-report pacing, commit/majority rules and the dispatch
gate are unchanged; WAL streaming still runs only on the vtask. The
one-shot arm is cleared on disconnect and on every become_leader /
become_follower.
Tests: test_late_joiner_catchup_kick.c (6). Controller suite 1285/0.
Live: catch-up starts 0-6 ms after a late registration (was up to ~6 s);
/xce-test-ha sequential 20 rounds, chaotic 6 rounds and sequential 8
rounds under 40-spinner load all PASS with zero job loss.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
handle_jobsubmit classifies (first_jobid, array_size) before the field
checks, the before-event hook and any registry access: a tagged array
starting at index 0 is answered with the new E_XCE_ARRAY_START0 and an
ERROR log asking for the client to be upgraded (7.1, wins per 7.9);
size 0, tags 2/3, untagged multi-element submissions and arrays whose
last element exceeds 32767 (checked without addition) get -EINVAL.
Nothing is stored for a rejected submission. This also closes a crash:
a size-0 submission used to reach payload building and overflow calloc.
Existing array tests move to tagged block|1 first IDs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A promoted vote endpoint carries a peer index relative to the votes
array (which leaves out the primary); a late-joiner's heartbeat endpoint
carries the absolute index. repl_ack_decode_and_dispatch recorded state
reports under the raw index, so with ctrl-1 primary, ctrl-3's vote
endpoint (relative 1) and a rejoined ctrl-2's heartbeat endpoint
(absolute 1) shared standby_reported_*[1]. Each handler then acted on
the other's position over its own endpoint: the snapshot meant for the
empty, rejoining ctrl-2 went to the caught-up ctrl-3, and ctrl-2 never
caught up.
Resolve the absolute index with ha_repl_ack_abs_peer_idx (as the
WAL-ack branch already did) for the state report and for
ha_walack_on_ack, and size catchup_inflight by MAX_HA_CANDIDATES so
every absolute index fits.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- bug-1872 (worker): "Failed to find any disk statistics in
/proc/diskstats" ERROR every 15 s on an NVMe-only host. The whole-disk
filter guessed from the name and skipped anything ending in a digit or
containing 'p' -- every nvmeXnY, mmcblkN, even sdp -- so io was never
reported. diskstats.h now counts a device only when
/sys/block/<name>/device exists (real disks; not partitions, loop, dm,
md, zram), and a host with no disk warns once.
- bug-1873 (worker): "Failed to get worker pid" WARN on job start. MainPID
reads 0 until systemd has forked the new transient unit; xcecd.c already
retries, so the miss is DEBUG and the final failure names the unit.
- bug-1874 (scheduler): deprecation WARN for the legacy xce_sched_fcfs on
every start, though builtin_fcfs is the picker. Registration still wraps
a job_elect_fn plugin but no longer warns; selecting one warns once.
- bug-1875 (apiserver): "Not running as root -- PAM authentication will not
work" with the default provider config_file; now only for provider pam.
- bug-1876 (controller): standbys logged "cluster_init_phase2 ... failed
(rc=-22)" WARN every 3 s while waiting for the primary's config push,
a state the function itself calls normal. It returns -EAGAIN for that
wait; callers log it at INFO, other failures still WARN.
- bug-1877 (controller): "WAL/snapshot stored unencrypted" warned without
looking. persist_path_encrypted() checks fscrypt (statx) and dm-crypt
(sysfs dm/uuid CRYPT-, through LVM slaves); the warning stays when the
storage is not shown to be encrypted (e.g. a plain /tmp).
Tests: worker diskstats_test (2 of 4 RED on the old filter with pc13v's
/proc/diskstats), controller test_phase2_cold_boot and
test_persist_encryption, scheduler picker tests, apiserver
TestRootWarning. Full ASAN build: all 113 suites pass (controller 319/0,
harness 141/0).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
build_queues() only ran at boot, so a queue a live config upgrade added
had no runtime object in c->qmgr (and the scheduler/apiserver never heard
of it) until the controller restarted. The checkpoint-ordering half of
bug-1469 was fixed in 959b0cda; this is the remaining half.
- reconcile_config_queues(c, notify): creates every queue in the active
config that has no runtime object, leaves existing ones untouched,
returns the number created.
- config_activate_committed() calls it (with notify) after every
successful activation -- verified-marker and no-marker paths alike.
- persist_notify_queue_updated_no_wal(): fires HOOK_DETAIL_QUEUE_UPDATE
under the t_apply_seam_notifying guard, so the scheduler/apiserver are
told while no WAL_QUEUE_CTRL is appended -- config queues are
config-owned, built on every member from the committed config.
- build_queues() now delegates to it without notify, so boot and
activation are idempotent in either order.
New test_config_queue_reconcile.c (RED before: the added queue was never
created): creates the added queue, leaves the existing object untouched,
notifies once, and a second reconcile/build_queues adds nothing.
Controller suite passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A worker reports every nonzero job exit as a FAIL (xcecd_job_finished ->
xcecd_notify_job_fail(jobid, stage, exit_code)), not a FINISH. The deferred
WAL_JOB_FAIL fact carried only {jobid, STAGE_FLAG|stage} and no apply seam
touched job.exit_info, so exit_info stayed NULL for every failed job, live
and after replay, and HOOK_DETAIL_JOB_FAIL subscribers saw code 0.
- persist_submit_job_fail_fact(): header + int32_t code trailer, used by
record_job_fail_fact(). Old entries (no trailer) decode as before.
- apply_job_fail_exit_code(): records the code in exit_info for pre-exit
stages only; a POST_EXEC code is the script's and leaves exit_info alone.
Kept out of apply_job_fail_transition() so mark_job_fail()'s internal
-errno failures (not replayed with a code) stay live/replay consistent.
- Both apply seams adopt the code. apply_wal_entry()'s WAL_JOB_FAIL case
also stops assigning the wire state 0 for STAGE_FLAG entries and runs the
real transition, as apply_committed_entry() already does.
New test_wal_job_fail_fact.c (5 tests, 3 RED before the fix); the two
test_worker_resync.c assertions that pinned exit_info == NULL now assert the
worker-reported code. Controller suite: 1228 tests, 0 failures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Relocate the 13 test files that lived inside controller_unit_test into
src/harness/testsuite/, add a new Unity runner (harness_test.c, modeled
on controller_test.c) carrying their extern/RUN_TEST entries, and
register the new harness_unit_test executable + suite in
src/harness/meson.build with an explicit build-order dependency on
xce_test (two of its tests spawn that binary over a pty).
Finish clearing the controller side: remove test_test_cli.c/
test_xcetest_process_entry.c from controller_unit_test_src and their
extern/RUN_TEST entries from controller_test.c, and remove the
temporary xce_test_controller_link symlink bridge task 1.1 added (no
longer needed now that both binaries build as siblings under harness/).
Verified: harness suite 132 Tests 0 Failures 0 Ignored; controller suite
1199 Tests 0 Failures 0 Ignored. 1199+132=1331 matches the pre-move
controller_unit_test baseline exactly (cross-checked against buglog
bug-1798/bug-1805).
Task 2.1 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Move xce_test's 15 production sources + headers from src/controller/testsuite/
into a new src/harness/ component with its own meson.build, and strip every
now-dangling controller-side reference (11 controller_unit_test_src entries,
4 standalone test-binary blocks, 11 controller_test.c extern/RUN_TEST lines)
so the controller build is never left broken.
Fixes a broken relative-symlink bridge (xce_test_controller_link) added to
keep two controller-embedded tests (test_test_cli.c, test_xcetest_process_entry.c)
finding the relocated xce_test binary until task 2.1 moves them too.
Discovered and documented in design.md: four controller-domain regression
tests call functions directly from four of xce_test's own files, requiring
controller_unit_test to permanently recompile them from src/harness/ — a
real dependency neither the original discovery nor design audit caught.
controller_unit_test: 1228 Tests 0 Failures 0 Ignored (verified via the
Unity binary's own printed summary line, not just meson's aggregate count).
Task 1.1 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Bring workerlist/joblist/reslist onto the shared flagspec_t library with an
identically-spelled --json flag, wired to the task-2.2 JSON list formatter
via a new pure field-mapping module (listrow_json.c/.h). Default table
output is unchanged; JSON rows carry raw numeric values (flags, timestamps,
memory sizes) rather than their humanized display strings, per Requirement
5.1's "same fields" contract.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Migrates 17 commands (cmd_hello, cmd_jobinfo, cmd_jobrun, cmd_jobsub,
cmd_resval, cmd_jobkill, cmd_jobckpt, cmd_qctrl, cmd_hostctrl,
cmd_runtasks, cmd_setpend, cmd_jobstart, cmd_jobfinish, cmd_jobfail,
cmd_resupdate, cmd_walcompare, join_worker_parse_attrs) from raw
getopt()/hand-rolled parsing to the shared flagspec_t + getopt_long
mechanism, so every one now uniformly rejects an unrecognized or
malformed flag with a usage summary and, where a close match exists,
a suggestion. cmd_walcompare's hand-rolled parser needed a leading '+'
in its optstring to preserve its pre-migration flags-only-before-
positionals behavior. Also fixes bug-1704: cmd_qctrl's uninitialized
action variable, now sentineled and rejected when unset, mirroring
cmd_hostctrl's existing pattern -- this also makes the destructive-
command gate's -i/-c detection (task 3.3) safe to rely on. Every
existing short flag's letter, value format, and default is unchanged
for a valid invocation. 1357/1357 across the full controller suite
(+4), 0 regressions.
Task 3.1 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 2.1, 2.2, 2.3, 2.4, 6.1, 6.2, 6.3, 6.4, 10.1, 10.2, 10.3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
cmd_join_worker's simulated-worker maxmem default was 64 bytes -- a
leftover from when the field was MB-scale, unfixed when maxmem became
bytes-canonical elsewhere. Now 64 * 1024 * 1024. Extracted the
argument-parsing/defaulting logic into join_worker_parse_attrs()
(join_worker_args.c), since it runs entirely before the function's
network call and is a genuine testable seam -- same precedent as
task 2.5's joblist_display.c. Verified live against a real deployed
cluster (workerlist reports MaxMem: 64.00 MB) plus a permanent 5-test
regression suite. maxswap/maxtmp have the identical stale-default bug
(logged as bug-1793) but are deliberately left unfixed here since
USAGE_JOIN_WORKER's help text currently documents their stale values
accurately -- fixing only the code would create a new docs mismatch;
a locked-in sibling-defaults test documents this for the follow-up.
1353/1353 across the full controller suite (+5), 0 regressions.
Task 2.6 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 12.1
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Eight display/text defects in test.c: worker_status_str's "UNKOWN"
typo, cmd_runtasks's "sequentail" typo (display-only, verified inert
against the real launch_type consumer), cmd_reslist's right-justified
columns and unlabeled hex Flags (now matching workerlist/state_raw's
conventions), joblist's raw-epoch timestamps (now via a format_job_time
helper delegating to the same formatter ctrlstatus already uses) and
combined NTasks column split into MinTasks/MaxTasks, job_state_str's
missing raw-state fallback for ambiguous RUN+EXIT (mirroring
print_jobinfo's existing pattern), and getconf's no-op group/config
list stubs (now genuinely implemented from already-fetched data,
matching the three sibling print functions' convention).
worker_status_str/job_state_str/format_job_time are pure functions with
no networking/REPL/global-state dependency, extracted into a new
joblist_display.c/.h module with a permanent 20-test standalone suite --
following this codebase's own resval_typeresolve.c/misc.c precedent for
testing functions that would otherwise be stranded inside untестable
test.c. cmd_reslist/joblist_query/cmd_getconf's table-printing bodies
inline fetch+print and remain untestable without a live cluster.
1348/1348 across the full controller suite (+20), 0 regressions.
Task 2.5 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 11.1, 11.3, 11.4, 11.5, 11.6, 11.7, 11.8
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
listfmt_json_begin/new_row/add_string/add_number/finish wrap cJSON
(already linked via struct2json_dep) into a minimal array/object
lifecycle library with no generic "record" abstraction -- each command's
own row struct maps its fields directly when wired in later (task 3.4).
Proven via a real stdout-capture-then-reparse round trip, including a
733-character field with no truncation, and stderr staying empty.
1306/1306 controller suite passing (+9).
Task 2.2 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 5.1, 5.3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
confirm_destructive() prompts on stderr and blocking-reads a y/n answer
via a single, non-looping fgets() call (not libedit) -- the mechanism
that makes a real Ctrl-C during the prompt return EINTR and be treated
as a decline (Requirement 9.9), same as an explicit "n" or EOF.
confirm_set_skip()/confirm_is_skip_set() add a session-scoped toggle
mirroring cmd_set's existing -e/+e style. Auto-skips when input is not
interactive, via a new xcetest_is_interactive() accessor over the
existing g_interactive global. Destructive-command enumeration and
per-command wiring (Requirement 9.5) is task 3.3, not this task.
Proven with a real fork+SIGINT test. 1297/1297 controller suite
passing (+7).
Task 2.1 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 9.1, 9.2, 9.3, 9.4, 9.6, 9.7, 9.8, 9.9
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
clitest_spawn/send_line/send_partial/send_tab/send_signal/wait_for_text/
wait_for_exit/kill_and_reap launch the real compiled xce_test behind a
genuine pty (so isatty/g_interactive/libedit behave as a real terminal),
with stderr routed to a separate pipe for real stream-separation testing.
All waits are poll()-bounded with an explicit timeout so a future
regression can't hang the whole test suite. This is the shared
infrastructure tasks 2.3, 3.5, and all of Validation phase (4.1-4.4)
will build their black-box scenarios on top of. One smoke test proves
launch/send/capture/exit-code/stream-separation end to end.
1290/1290 controller suite passing (+1).
Task 1.7 of .kiro/specs/xce-test-cli-ux/tasks.md.
Foundation phase (1.1-1.7) now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
argp already provided --help/-?/--version with exit-without-REPL
behavior, but had no -h alias (its default short help flag is -? per
GNU argp convention) despite Requirement 4.1 naming -h explicitly.
Added a hidden -h option triggering the same help path, and extended
the doc string to describe the in-session help command. Proven against
the real compiled binary via subprocess tests, not just argp's own
in-process behavior. 1289/1289 controller suite passing (+4).
Task 1.6 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 4.1, 4.2, 4.3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
xcetest_run_interruptible() runs a blocking call on a detached worker
thread while the caller polls xcetest_interrupted() every 50ms. A single
atomic CAS (RUNNING -> DONE_UNCLAIMED | ABANDONED) decides which side
frees the heap-allocated call block, so an abandoned call's worker
thread never double-frees or leaks once it eventually finishes. Proven
under active ASAN with a real detached thread and a real SIGINT-forced
abandonment. 1282/1282 controller suite passing (+2).
Task 1.4 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 1.1, 1.2, 1.3, 1.4
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Replaces the bare signal(SIGINT, sigbreak) install with sigaction() and
SA_RESTART cleared, so a blocking read reliably returns EINTR instead of
being silently restarted by glibc's BSD signal() semantics. Adds
xcetest_interrupted() as a read-only accessor over the existing break_out
flag for the interruptible-call adapter (tasks 1.4/1.5) to consume.
Proven by a real fork+SIGINT+blocked-read test that first demonstrates
the old bug, then the fix. 1280/1280 controller suite passing (+2).
Task 1.3 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 9.9
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Adds flagspec_t descriptor table, getopt_long option-table builder, and
the shared stderr-routed rejection path that every flag-accepting
command will migrate onto in later tasks. TDD'd via test_flagspec.c
(1273/1273 controller suite passing).
Task 1.1 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 2.1, 2.2, 2.3, 2.4, 3.1
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Add OPCode_jobStateReport dispatch in myreq_handler, reassembling a
report across multiple incoming pages before handing it to task 3.2's
resync_apply_report() exactly once - proven via a test-only call
counter, not just end-state equivalence (a double-apply could pass an
end-state check if idempotent).
Standby-redirect investigation (Req 8.1/8.3): confirmed
ha_dispatch_gate already redirects every rejected opcode uniformly, so
no opcode-specific code was needed for that. OPCode_jobStateQuery has
no controller-side receive path at all (only sent, never received)
so needed no gate change either.
That same investigation found a different, more consequential gap:
fixes bug-1753, ha_dispatch_gate's quorum-loss fact-admission allowlist
(OPCode_notify/update/setpreason/clrpreason) was missing
OPCode_jobStateReport, so a report arriving during a transient
quorum-loss window - this feature's own core failover scenario - would
be rejected outright with no retry (worker sends fire-and-forget).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
New src/controller/resync.c/resync.h: resync_arm_round(cluster_t*,
worker_t*) enumerates every non-terminal job assigned to a host,
populates worker_t.resync (round_token, query job-ID set, deadline),
sets XCE_WORKER_RECONCILING, and schedules a non-blocking delaymgr
one-shot timeout. resync_resolve_round() clears the round; a fired
timeout resolves it too, via resync_round_is_current()'s by-value
round-token check -- delay.c/delay.h expose no true cancel/dequeue
primitive, so a stale timer from an already-superseded round becomes a
safe no-op instead of touching a newer round's live state (verified
safe against both misresolving the wrong round and use-after-free: the
timeout closure never holds a pointer into resync_state_t itself).
5 new tests: query-set correctness, fresh-token-per-arm, the
RECONCILING flag's set/clear lifecycle, and a direct (non-blocking)
timeout-fire proof via an XCE_TESTING-gated seam confirmed compiled out
of the production binary. Full suite: 1232 tests, 0 failures.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add record_job_reconcile_pending_fact(cluster_t*, job_t*) to job.c:
eagerly applies apply_job_update_transition(job, JOB_STAT_PEND) (real,
live mutation, not a deferred peek -- matching dispatch_a_job_'s
established pattern, not record_job_finish_fact's), releases
job->job.allocs on an applied (non-no-op) transition, then durably
appends via the generic persist_submit_job_fact(WAL_JOB_UPDATE). Caller
holds job's lock throughout and is responsible for calling
persist_notify_job_updated_already_persisted() after unlocking -- this
function deliberately does not call it itself.
Closes bug-1631 at this new call site: a RUN job with a real allocation
reconciled to PEND (Requirement 4.6, worker reports it unrecognized)
would otherwise carry a stale allocation into its next dispatch attempt,
exactly the gap bug-1631 documented as dormant until something re-admits
an already-dispatched job to the scheduler's PEND pool -- this feature
is that path. dispatch_a_job_ itself is unchanged (still correct for a
job's first-ever dispatch, the only case reachable there).
3 tests: allocs release + eager-mutation-before-unlock proof, no-op on
already-pending, and the Requirement 4.4 terminal-state-preservation
proof (rejects overwriting JOB_STAT_DONE).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Two independent gaps in misc.c/job.c, both silently zeroing every real
job's resource-usage data:
- update_host_runusage() matched worker rusage report resnames
"cputime"/"mem", but the real worker's collector
(collect_job_rusages(), xcecd_cgroups.c) sends "cpu_time_sec"/
"memory_rss" -- the two never matched, so cpu_time/mem_usage stayed
at their calloc()'d zero for every non-simulated job. Aligned both
sides, including xce_test's own simulated-worker fixture.
- update_job_rusages() had a standing FIXME: xce_job_runusage_t's
job-level max_mem/avg_mem/used_mem/cpu_time were never derived from
the per-host host_runusage[] entries. Added aggregate_job_runusage()
(sum/max/avg across hosts) and wired it in right after the per-host
update.
Added test_host_runusage.c (standalone unit test, mirrors
resval_typeresolve_unit_test's isolation pattern) plus the
controller_unit_test stub aggregate_job_runusage() needs since that
target excludes the real misc.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Adds rcfile_resolve_path() (rcfile.c/rcfile.h): resolves $XDG_CONFIG_HOME/xce_test/rc
when set and non-empty, else $HOME/.config/xce_test/rc, else NULL. Standalone
libc-only module mirroring the configcompare.c pattern; wired into xce_test_src
and controller_unit_test_src, registered in controller_test.c's Unity runner.
6 unit tests in test_rcfile.c cover all XDG/HOME combinations plus a dedicated
env-var save/restore isolation test, addressing the cross-test leak risk found
during design review (controller_unit_test runs 80+ test files in one process).
Adds the xce-test-rc-file spec (requirements/design/tasks/research), including
design fixes from validation: getline()-based line reading (task 1.2, avoids
truncating long rc-file lines) and the env-isolation requirement above.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Last of the 9 confirmed findings from the /xce-audit-invariants controller
pass. checkpoint_task_fn was registered directly as a delaymgr entry and
ran its full blocking body inline: checkpoint marker pwrite+fdatasync,
write_snapshot()'s genuinely unbounded per-entity walk (scales with
cluster state size) held under apply_pump_lock, WAL trim, and snapshot
rotation/unlink -- all on the same thread that drives heartbeats/election
timeouts. Unlike every other delaymgr operation (snapshot send,
config-push, reconnect, catch-up), checkpoint had no offload mechanism
at all -- the one gap left from the raft-drive-async hardening pass.
Design: full offload via the SlowLane (slowlane_submit), default ON --
NOT the vtask pool. persist_apply_pump_run() already holds apply_pump_lock
across its own per-entry walk while running ON the vtask pool
(VTASK_LIVE_BLOCK); putting checkpoint's unbounded, lock-held
write_snapshot() on that same pool risked starving it, exactly what
ha-async-io.md's own thread-role table warns against. The SlowLane is
the actual established pattern for this I/O class (push_config_transport_only,
the snapshot-send path, catchup_lane_fn, demotion_lane_fn all dispatch
there) -- and persist_checkpoint() already runs on the slow lane in
production TODAY via ha_snapshot_do_send() -> snapshot_lane_fn, so
apply_pump_lock's acquisition from a slow-lane thread is proven-safe,
exercised daily, not new surface. Unlike catchup/demotion (which stay
flag-gated OFF for an unrelated, still-not-root-caused hang, bug-1429),
checkpoint's new flag defaults ON given this reuses an already-proven
code path. Inline-fallback + WARN only on a genuine slowlane_submit
failure (not -EAGAIN/-EEXIST, both non-failures) -- never silently drops
a scheduled checkpoint.
New test_persist_checkpoint_offload.c (3 tests). RED-phase forced the
pre-fix always-inline path: the suite never printed a single result and
had to be killed, proving the test genuinely detects the violation, not
a tautology. GREEN: all 3 pass. Full suite: 1184 tests, 2 failures --
both the pre-existing, unrelated bug-1561 overlayfs sandbox flake
(exact signature match). No live HA soak was run for this specific fix
in isolation (explicitly scoped out, not hidden) -- the mandatory
live-soak proof for this whole audit campaign covers it next.
This closes the last of the 9 confirmed findings from the controller
audit (bug-1684, bug-1687/1689/1690/1691/1692, bug-1693, bug-1694).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Violates this project's own recorded Decision Log entry: "No client
reply-wait timeout as a fix for a hung request -- fail loud at the real
transport/source instead of masking with a deadline backstop."
OPCode_schedAdmin (cluster.c) and on_fetchfile (job.c, the worker proxy
-- cluster.c's own comment says schedAdmin "mirrors on_fetchfile's proxy
pattern", confirming one shared design mistake, not two independent
bugs) both blocked on a multi-second pthread_cond_timedwait, replying
"timeout" on expiry. handle_scheduler_leave()/handle_worker_leave() (the
actual disconnect handlers) only nulled the endpoint pointer -- neither
ever signaled the pending waiter, so a mid-wait disconnect was invisible
until the deadline lapsed. The timeout was the ONLY failure-detection
mechanism, exactly what the decision forbids.
Both sites already had a working signal function on their success path
(schedadmin_signal(), fetchfile_signal()) -- just never wired to the
disconnect handlers. Fixed by calling them from
handle_scheduler_leave()/handle_worker_leave() with a distinguishable
-ECONNRESET outcome, so the waiter wakes immediately on disconnect
instead of only on the deadline. Timeouts kept as defensive backstops
for a genuinely wedged peer that never disconnects cleanly either --
this fix is about not relying SOLELY on the timeout, not removing it.
schedadmin_signal_disconnect() must be called BEFORE handle_scheduler_leave()
acquires c->sched.lock -- the dispatch wait holds that same lock for its
entire duration, so signaling after acquiring it would deadlock.
Found by an /xce-audit-invariants controller pass (Decision Log fidelity
dimension). New test_fetchfile_disconnect.c plus extensions to
test_sched_admin_dispatch.c/test_sched_worker_change.c; RED/GREEN
independently verified against real elapsed-time assertions (pre-fix:
5001ms full-timeout wait; post-fix: <1s immediate wake). Full controller
suite: 1181 tests, 2 failures -- both the pre-existing, unrelated
bug-1561 overlayfs sandbox flake, confirmed via a clean meson-driven run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Last of the five field-parity gaps from the original audit (bug-1678,
bug-1687, bug-1689, bug-1690, bug-1691). set_preason()/clear_preason()
set job.pend_reasons (host/resource-based pending-job explanations), but
persist_job_hook_fun's default: arm only ever captured {jobid, state}.
Neither apply seam restored pend_reasons after a restart.
Reuses the ALREADY-EXISTING JobPendReason/HostBasedReason/ResourceBasedReason
capnp structs -- no schema change needed, unlike bug-1690's WAL_JOB_RUSAGE.
Safety-checked against bug-1690's lesson (RTTI/makeString()-backed fields
need merge-not-replace, not free-and-replace): pend_reasons is plain
malloc/strdup-backed throughout (erase_job_preason is FREEUP()-only,
add_host_preason/add_res_preason use strdup()/calloc()), so a
straightforward free-and-replace is safe here, matching bug-1678/1687's
pattern instead.
New wrinkle beyond the four prior fixes: pend_reasons has a genuine CLEAR
operation, and "trailing bytes present or absent" can't distinguish a
legitimate clear from an old-format payload -- added an explicit presence
byte (0=cleared/adopt NULL, 1=capnp bytes follow), verified by two
dedicated "clear survives restart" tests.
11 new tests in test_wal_job_preason_fact.c; RED/GREEN independently
verified (7 genuine pre-fix failures, one test's own wrong premise about
capnp's garbage-byte behavior caught and corrected during RED/GREEN
iteration). Full controller suite: 1178 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake. No ASAN findings.
This closes the field-complete apply parity invariant class the
xce-audit-invariants controller dimension 7 was added for.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Same class as bug-1678/1687/1689/1690: on_job_ckpt() sets job.ckptinfo
(a plain char*, xstring-backed via makeString()) but is purely hook-driven
-- persist_job_hook_fun's default: arm only ever captured {jobid, state}
for WAL_JOB_CKPT. Neither apply seam restored ckptinfo after a restart.
Simplest of the five field-parity fixes: ckptinfo is a bare string, so no
capnp struct is needed -- mirrors WAL_QUEUE_CTRL's existing NUL-terminated-
string-after-header wire shape instead. Adoption is unconditional (not
gated on a state transition), matching bug-1690's precedent: a checkpoint
report never changes job.state. restore_job_ckpt_state() releases the
prior value via dereference() (the correct RTTI verb for an xstring), not
free().
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension) -- the last of five sibling gaps found together
(bug-1678, bug-1687, bug-1689, bug-1690, this one). 7 new tests in
test_wal_job_ckpt_fact.c; RED/GREEN independently verified (5 genuine
pre-fix failures, 2 backward-compat tests correctly passed unchanged).
Full controller suite: 1167 tests, 2 failures -- both the pre-existing,
unrelated bug-1561 overlayfs sandbox flake. No ASAN findings.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Same class as bug-1678/1687/1689: the WAL fact for WAL_JOB_RUSAGE only
carried {jobid, state}, so job.run_usage (client-reported per-host
CPU/mem usage) was never restored on either apply seam after a restart.
Unlike the prior three, xce_job_runusage_t/xce_host_runusage_t had no
existing Cap'n Proto mapping -- a genuine new schema addition
(HostRunusage/JobRunusage in controller.capnp, mirroring JobPendReason's
List(NestedStruct)+$C.maplistcount style), verified against the real
struct definitions in types.h, codegen confirmed via nm on the built
binary rather than assumed.
Two deliberate deviations from the established precedent, found by
reading the real code rather than copying the pattern blindly:
1. Adoption is unconditional, not gated on a state transition -- a
rusage report almost never changes job.state (repeat reports keep
state at RUN), so gating on JOB_TRANSITION_APPLIED like START/FINISH/
DISPATCH do would make the fix dead code for the realistic case.
2. Merge, not free-and-replace. update_host_runusage() builds run_usage
with host_name sourced from makeString() -- an RTTI-managed pointer,
not malloc() -- and the primary's own live apply pump re-applies every
fact it just wrote. A naive free_JobRunusage_ptr()-then-adopt (matching
the prior three fixes exactly) would free() a non-malloc'd pointer on
the very first live rusage report for a new host -- guaranteed heap
corruption. Fixed with merge_job_runusage(): existing hosts get
scalar-only in-place updates (host_name pointer never touched), only
genuinely new hosts adopt their freshly-decoded (safe, malloc'd)
host_name. Proved via a real makeString()-backed test asserting
pointer identity survives, not just value equality. job_destroy()
(src/lib/jobj.c) was found to have this identical latent mismatch,
pre-existing and independent of this fix -- flagged, not fixed here.
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension). 8 new tests in test_wal_job_rusage_fact.c; RED/GREEN
independently verified. Full controller suite: 1160 tests, 2 failures --
both the pre-existing, unrelated bug-1561 overlayfs sandbox flake. No
ASAN/heap-corruption findings. Backward compatibility confirmed for
old-format bare-payload entries.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Same class as bug-1678/bug-1687: record_job_finish_fact() wrote only
{jobid, state} for WAL_JOB_FINISH. Both apply seams either fabricated
finish_time via time(NULL) or touched nothing beyond state.
Worse than "lost on restart": apply_job_finish_transition() -- the only
function that ever set job.exit_info -- is dead code on the real
production apply path. The raft-apply-decoupling redesign routes actual
mutation through the leaner apply_job_update_transition() instead, so
exit_info was never populated at all, restart or not, before this fix.
job_exitinfo_t ({int exitcode;}) has no existing capnp mapping, so this
uses a small native (non-capnp) trailing payload -- wal_jobstate_t header
+ uint64_t finish_time + int32_t exit_code -- mirroring wal_jobstate_t's
own native-struct style rather than inventing a capnp wrapper for one
field. run_time is never carried on the wire, always re-derived as
finish_time - start_time at apply time. Both apply seams updated;
old-format payloads (exactly sizeof(wal_jobstate_t)) decode unchanged
(fabricated finish_time, NULL exit_info) -- full backward compatibility.
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension). 6 new tests in test_wal_job_finish_fact.c; RED/GREEN
independently verified (4 genuine pre-fix failures with real
expected-vs-fabricated mismatches, 2 backward-compat tests correctly
passed both before and after). Full controller suite: 1152 tests, 2
failures -- both the pre-existing, unrelated bug-1561 overlayfs sandbox
flake (confirmed flaky across repeated re-runs of the same binary).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Same class as bug-1678: apply_job_start_transition() sets the real
job->job.pidinfo (worker-reported PIDs/PGIDs/exdata) and start_time, but
the WAL fact for WAL_JOB_START only carried the shared wal_jobstate_t
{jobid, state} header -- both apply seams (apply_committed_entry() for
primary boot replay, apply_wal_entry() for standby catch-up) restored
state only. pidinfo stayed permanently NULL after a restart; start_time
was fabricated with time(NULL) at replay instead of being carried,
drifting from the true value.
Mirrors bug-1678's exact fix shape: capnp_encode_pidinfo()/decode_pidinfo()
(persist.c, using the pre-existing JobPIDInfo capnp struct -- no schema
change needed) serialize job_pidinfo_t into the WAL payload, trailing a
new uint64_t start_time field appended after wal_jobstate_t.
persist_submit_job_start_fact() replaces the old state-only
persist_submit_job_fact() call for this type. Both apply seams updated;
a payload of exactly sizeof(wal_jobstate_t) (any entry written before
this fix) decodes with pidinfo left NULL and start_time fabricated via
time(NULL), exactly matching pre-fix behavior -- full backward
compatibility, verified by every pre-existing WAL_JOB_START test using
the old bare-struct shape.
Also fixed a genuine, unrelated deadlock this change's larger payload
exposed in test_commit_barrier_reply.c: cbrp_count_jobstate_entries_ex()
matched WAL entries by exact payload size (==), so once WAL_JOB_START
legitimately grew it silently undercounted to 0 and a downstream test
spun forever polling an uncounted target. Fixed the match to >= (jobid/
state always live at a fixed leading offset) -- bisected via /proc/*/wchan
futex inspection (no ptrace in this sandbox), confirmed as the actual
cause, not the teardown-ordering red herring first suspected.
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension, added after bug-1678). 6 new tests in
test_wal_job_start_fact.c; RED/GREEN independently verified (4 tests
genuinely failed pre-fix with exact expected-vs-fabricated value
mismatches, 2 backward-compat tests correctly passed both before and
after). Full controller suite: 1146 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
dispatch_a_job_() (tasks 3.1-3.3) now owns notification entirely for
every outcome -- notify_dispatch() on a retryable failure, mark_job_fail()'s
own hook chain on a terminal failure, nothing on success. handle_jobdispatch()'s
loop still unconditionally called notify_dispatch() itself for every job
regardless, producing two conflicting outcome reports for any failed
dispatch. Removed that caller-side call; the loop is now a thin
dispatch-and-free loop with no notification logic of its own.
New test_jobdispatch_no_double_notify.c drives a 3-job batch (success,
retryable, terminal) through handle_jobdispatch() and confirms exactly
one notification fires (from the retryable job only), not four.
Independently reviewed and approved. Confirmed handle_jobdispatch()'s
sole caller never depended on its notification behavior (always returns
0 unconditionally). Flagged that test_commit_barrier_reply.c's
test_jobdispatch_receipt_ack_no_wal_append is now stale for a second,
independent reason (notify count, not just WAL count) -- both left for
task 5.4 per Requirement 5.3.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
The four retryable-classified failure sites in dispatch_a_job_() (worker
lookup, worker null, worker not ready, RPC failure) now resolve instead
of just returning an error code: lock(j, RTTI_LOCK_WRITE) -> genuinely
classify via xce_classify_dispatch_fail() -> increment dispatch_fail_count
-> within XCE_DISPATCH_FAIL_RETRY_BOUND (3) resolves retryable (nothing
further under the lock), past it resolves terminal via mark_job_fail()
exactly like task 3.2's sites -> unlock(j) -> dereference(j) (closing
bug-1627's 4-site reference leak) -> outside the lock, notify_dispatch()
fires only on the retryable outcome. Each site's natural error code is
captured into a stable local before any intervening call, since errno
isn't guaranteed stable once lock()/ERROR()/free_launchinfo() sit between
the original failure and the eventual return.
handle_jobdispatch() is intentionally untouched -- it still unconditionally
calls notify_dispatch() itself for every outcome, so the retryable path
now temporarily double-notifies until task 4.1 removes that caller-side
call. Expected, not a defect.
New test_dispatch_fail_retry.c (4 tests, including a bounded-escalation
test driving 4 consecutive RPC failures on the same job: 3rd retries,
4th resolves EXIT) plus a fixed stale assertion in test_dispatch_commit.c
and additive, read-only notify-capture accessors in
test_commit_barrier_reply.c (the binary's sole notify() stub host).
Independently reviewed and approved, including explicit scrutiny of the
cross-file test accessor addition and the stale-assertion fix -- both
confirmed legitimate, not scope creep or regression-hiding.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
The two terminal-classified failure sites in dispatch_a_job_() (invalid
allocation, queue-exec-info lookup failure) now actually resolve the job
instead of leaving it stuck at PEND forever with a dangling reference to
nowhere. Each site: lock(j, RTTI_LOCK_WRITE) -> genuinely calls
xce_classify_dispatch_fail(site) and branches on the result (not a
hardcoded assumption) -> mark_job_fail() under the lock (stage=0 routes
apply_job_fail_transition() to JOB_STAT_EXIT) -> unlock(j) -> the existing
dereference(j) (kept exactly where it was, not duplicated) -> the existing
return code, unchanged. No notify_dispatch() added -- mark_job_fail()'s
own hook chain is the sole notification for a terminal outcome per
design.md; handle_jobdispatch()'s still-unconditional notify_dispatch()
call is untouched (task 4.1's job).
New test_dispatch_fail_terminal.c proves both sites resolve to
JOB_STAT_EXIT with allocs left NULL, using a canary-reference technique
to prove no reference over-release (job_t's real refcount has no test
accessor). Independently reviewed and approved; applied the reviewer's
one non-blocking fast-follow (arm the canary before the call under test,
not after, so it's an independent proof rather than relying on other
assertions to crash first) and re-verified with a clean incremental build.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
dispatch_a_job_()'s previously-unconditional job.allocs commit is now
gated on xce_startjob_req() actually succeeding (rc >= 0). In that same
successful branch, under a newly-acquired lock(j, RTTI_LOCK_WRITE) (a
real pre-existing locking gap this call site had -- it took no lock on
the job object itself before this), the job transitions to JOB_STAT_RUN,
dispatch_fail_count resets to 0, and a WAL_JOB_DISPATCH fact is appended
for durability. The failure path is otherwise unchanged -- classification
and resolution of the six failure sites is tasks 3.2/3.3's job, not this
one's.
Adds an XCE_TESTING-only dispatch_a_job_test() wrapper (dispatch_a_job_()
is static) and test_dispatch_commit.c, proving: state->RUN, allocs
committed correctly, counter resets, and durability survives a real
persist_recover() replay with no WAL_JOB_START present.
Independently reviewed and approved. Two pre-existing, out-of-boundary
findings surfaced and logged for later tasks: test_commit_barrier_reply.c's
test_jobdispatch_receipt_ack_no_wal_append will need updating by task 5.4
(Req 5.3 anticipates this), and bug-1429's hang was reconfirmed a 3rd time
(unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
Add uint32_t dispatch_fail_count to the job_t wrapper struct (xce/types.h,
alongside submit_lsn, same in-memory-only/reset-on-failover precedent),
per Requirement 4.3. Data model only -- xce_job_t (wire type) untouched,
job.c untouched; increment/reset wiring and the bounded-retry threshold
check are tasks 3.1/3.3.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New wal_entry_type WAL_JOB_DISPATCH=12, reusing wal_jobstate_t verbatim
(_pad=0, absolute-value semantics), wired into both apply_wal_entry()'s
and apply_committed_entry()'s existing job-state fall-through groups per
Requirement 2.4. Restart-replay test writes only WAL_JOB_SUBMIT+
WAL_JOB_DISPATCH to a real on-disk WAL and confirms job.state==RUN after
a fresh persist_recover(), independent of WAL_JOB_START.
Review found apply_wal_entry() has zero production call sites today
(persist_recover() and standby catch-up both route through
apply_committed_entry() post the 2026-07-10 ha_apply_single_entry
refactor) -- design.md's stale characterization corrected; the dual
case-label registration itself is unaffected and correct.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Define xce_dispatch_fail_site_t/xce_dispatch_fail_class_t and
xce_classify_dispatch_fail() in job.c, mapping each of dispatch_a_job_()'s
six post-lookup failure points to retryable or terminal per Requirement
4.1/4.2. Standalone, not yet wired into dispatch_a_job_() (deferred to
tasks 3.2/3.3). Also adds this spec's requirements/design/research/tasks
docs (first commit touching this spec).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sched_worker_change() aliased w->host.resvals_to_sched to a local xlist_t's
backing array, pushed it to the scheduler, then freed that array via
xlist_fini() without resetting the alias - leaving a dangling pointer for
the rest of the worker's life. On disconnect (handle_worker_leave ->
mark_worker_unavail -> persist_worker_hook_fun -> capnp_encode_worker ->
encode_Worker -> encode_NameValue_list), this dereferenced freed memory and
crashed the controller - but only once a scheduler had ever connected
(on_scheduler_join gates the hook registration), which made it easy to miss
in isolated worker-only testing.
The correct reset already existed in the function but was dead-code'd
behind #if 0 /*FIXME*/ - apparently from a prior fix attempt that got
wrapped instead of landed. Fix mirrors the proven-correct pattern already
used by the sibling apiserver_worker_change(): null the alias before
xlist_fini() frees the backing array, rather than double-freeing via
free_name_val_list() first.
Adds test_sched_worker_change.c, a regression test driving the real
production hook-dispatch path (xce_run_hooks -> sched_worker_hook_fun ->
sched_worker_dead -> sched_worker_change) and the real encode_Worker()
encoder the original crash traversed - confirmed to reproduce the exact
ASAN heap-use-after-free when the fix is reverted, and pass clean when
restored, closing the gap that let this regress silently once already.
Found while live-testing the resource-value-verification feature (deploying
a scheduler + injecting worker resource values via xce_test, then
disconnecting) - unrelated to that feature's own changes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Move set_resval()'s resource-name lookup table into a new, dependency-light
resval_typeresolve.c/.h module so it's independently unit-testable (test.c
can't be white-box included into Unity). Adds RED tests for the not-yet-
implemented numeric-fallback/type-override behavior (task 2.1).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds the interactive simulation REPL's execution-control slice (Req 5
AC 22): a single-threaded input loop (readline-based) supporting
step [N], continue, quit, and help, pausing before the first replay
event and blocking the driver loop while active.
Review round 1 raised two findings, both resolved here:
- GNU readline (GPL) was linked directly into xce_scheduler, a shipped
production daemon. Switched the whole project to libedit (BSD) for
line editing -- scheduler and controller (xce_test) both now use
editline/readline.h, API-compatible (readline()/add_history()).
- Getting a clean regression run after the libedit swap required a
full clean rebuild, which exposed a genuine pre-existing SIGSEGV
(bug-1504): SnapshotData.internalState was a capnp Data field, and
this codebase's capnpc-c backend silently no-ops Data-field
encoding without zero-initializing the caller's stack struct,
leaving an uninitialized capn_ptr written onto the wire. Fixed by
switching internalState to List(UInt8) (with a new internalStateLen
wire field), matching the working FetchFileReply.content/bytesRead
precedent -- no hand-written accessor code needed.
A second, unrelated instance of the same bug class was found in the
controller's WAL-catchup path (WalCatchupEntry.payload) and
deliberately left unfixed as bug-1505, out of scheduler-core's
boundary; needs its own project-wide Data-field audit.
Scheduler suite: 34/34 clean. RED-phase reproduction recorded in
tasks.md (repl_should_pause() step-budget check disabled -> 2/4
tests fail as expected -> reverted -> 34/34 green).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the spec's one and only src/controller/ change: a new
case OPCode_schedAdmin in myreq_handler() (cluster.c) that mirrors the
existing OPCode_upgradeConfig case -- primary-only gate, explicit reply
on every path, stateless pass-through with no WAL entry.
Forwards to the scheduler via a new xce_schedadmin_req() (async send,
mirrors xce_fetchfile_req) and blocks the vtask thread on a condvar with
a 5s deadline while holding c->sched.lock for the round-trip, matching
this spec's own research-phase design decision (parity with
OPCode_upgradeConfig's no-RBAC posture, blocking wait accepted for
low-frequency admin traffic). A new branch in the existing
OPCode_reply case demuxes the scheduler's SchedAdminReply to a
process-global single-slot pending-waiter mechanism, serialized by
c->sched.lock, matching job.c's existing fetchfile-proxy pattern.
Adversarial review independently traced double-free/cleanup, the
timeout/signal race, and reply-opcode demux correctness against source;
found and a follow-up round closed a narrow-window leak via a carefully
lock-ordered unlock->unregister->relock sequence (avoiding an AB-BA
inversion against schedadmin_signal()'s lock acquisition order).
New test_sched_admin_dispatch.c drives the real myreq_handler via the
existing myreq_handler_for_test seam: not-primary rejection (proven via
a poison-pointer sched.ep that would crash if forwarding were ever
attempted), not-connected rejection, and a genuine two-thread round-trip
test exercising the real signal/demux path.
Controller suite 3/3 (1090/1090 assertions), scheduler suite 26/26
unaffected, clang-format clean.
bug-1421 added two term-discovery fallback vectors so a stale-term primary
steps down instead of retrying AppendEntries forever: heartbeat-reply
state-report term adoption and WAL-NACK reply_term adoption, both inside
repl_ack_decode_and_dispatch. Neither had a test driving the real
production dispatch path and asserting raft_on_higher_term_observed
actually fires.
Adds two tests mirroring test_config_repl.c's existing config-push term
test, driving peer_ctrl_msg_handler -> repl_ack_decode_and_dispatch via
wire-shape capnp packets. Both pass immediately -- the underlying bug-1421
code was already correct, just unverified end-to-end -- closing
raft-adapter-conformance's Requirement 7.5 unverified claim.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXkenK8rqE8mdS1YKjsq78
myreq_handler's ReplConfigPushReply branch now routes cack->term
through raft_on_higher_term_observed (same entry point the
heartbeat-reply and WAL-NACK vectors use), regardless of accept/reject,
without altering the existing accept/reject branching.
Adds myreq_handler_for_test (XCE_TESTING-only) to drive the real
dispatcher in tests, following the peer_ctrl_msg_handler_for_test
precedent. Requires cluster.c in controller_unit_test_src plus the
established link-satisfying fake stubs for its unrelated switch-case
dependencies (same pattern already used by test_worker_join.c and
test_commit_barrier_reply.c).
Prove Requirement 6.2 with a real, code-backed mechanism instead of the
originally hypothesized lost_and_found re-homing, which turned out to be
aspirational and never implemented (confirmed by exhaustive grep at both
the pre-fix baseline commit and HEAD — bug-857 itself was flagged in
buglog.json as never empirically reproduced).
test_bug857_regression.c demonstrates both halves of the requirement
entirely within controller_unit_test, no live soak or old-commit build
needed: schedule_checkpoint's pre-existing WAL-threshold trigger is
decoupled from the config-marker boundary and produces genuinely
divergent per-member checkpoint content, while the marker-triggered
checkpoint this spec added converges members regardless of differing
per-member processing pace.
design.md's Requirements Traceability table and Testing Strategy section
are corrected to match; the redirection reasoning is recorded in
tasks.md's Implementation Notes for the benefit of tasks 8.6-8.9, which
still need live multi-process HA-soak infrastructure for Requirement 6.1
that this task deliberately did not build.
Build configcompare_paths(), an offline sibling to snapdiff/walcompare that
recomputes each member's active-config sha256 fresh from its own on-disk
workdir/name/conf bytes and reports identical vs. divergent (Req 4.2).
Wire it as a cmd_configcompare CLI subcommand in test.c, matching its
siblings, after review caught it shipped as an unreachable library function
in round 1.
Extend snapverify to match the current on-disk checkpoint format: bump
SNAP_FORMAT_VERSION to 2, replace the stale raw queue-frame decode with the
capnp length-prefixed walk, add host-frame and config-bytes-section
structural walks, and decode (not just presence-check) the header's
cfgref tag/sha. Without this fix snapverify rejected every checkpoint
written by the current codebase.
Adds an explicit case WAL_CONFIG_MARKER to apply_committed_entry,
replacing its no-op treatment, that forces a checkpoint at exactly the
marker's own (term, lsn) via two new helpers:
persist_checkpoint_write_locked(p, checkpoint_lsn) -- the checkpoint
core (pin lsn, scan config ref, write_snapshot, trim, rotate) taking
an explicit LSN and never acquiring apply_pump_lock itself
persist_checkpoint_marker_triggered(p, lsn) -- shares the existing
checkpoint_in_progress CAS guard, calls the above with the marker's
own lsn
This fixes two hazards design.md's "call persist_checkpoint() directly"
text didn't account for: (1) apply_committed_entry's live-pump caller
(persist_apply_pump_run, used by both primary and standby online-apply)
already holds apply_pump_lock across the dispatch call -- a direct
persist_checkpoint() call would self-deadlock on that mutex; (2)
last_applied_lsn/replay position only advance AFTER apply_committed_entry
returns at both call sites, so pinning checkpoint_lsn from
last_applied_lsn inside the marker case would label the checkpoint with
the previous entry's LSN, not the marker's own -- fixed by threading the
marker's own lsn parameter through explicitly. A -EBUSY outcome (a
concurrent periodic checkpoint mid-flight) is logged at TRACE and treated
as expected/self-resolving, never retried. The new case has no effect on
entity interpretation/re-homing (Req 1.8).
New test_upgrade_snapshot_apply.c drives WAL_CONFIG_MARKER through the
REAL production call paths (persist_apply_pump_run and boot replay via
persist_recover), not a bare apply_committed_entry() call, so the fix is
actually proven against the lock-holding context rather than assumed.
Also fixes a genuine regression in test_config_repl.c surfaced by
WAL_CONFIG_MARKER apply having a real side effect for the first time
(bug-1466): an fsync-worker-thread race (t42_wait_applied polling helper,
mirrors test_apply_pump.c's tap_wait_applied) and one test call site
trusting an unreliable raft-derived LSN. Surfaces (but does not fix, out
of this spec's boundary) a pre-existing ha-term-safety/raft-log-replication
gap: persist_write_checkpoint_marker bypasses ha_primary_log_append,
breaking its documented current_lsn==log_len+1 invariant once a
checkpoint interleaves with primary appends -- log_len feeds real
RequestVote freshness comparisons, not advisory-only as first assumed;
severity corrected in bug-1466. Task 5 of 19.
Requirements: 1.1, 1.5, 1.8, 4.4, 5.2, 5.3
Adds restore_host_state(cluster_t*, const char*, uint32_t admin_status)
to worker.c: OR's admin bits into an existing worker's status on hit;
on miss, creates a minimal XCE_WORKER_UNAVAIL placeholder via
makeWorker (mirroring the lost_and_found sentinel host precedent),
with a valid non-NULL hostattrs_t so the host's later real join routes
through rejoin_worker (not init_worker) and preserves the restored
admin status. Idempotent under retry (find first, create only if
missing). Standalone -- not yet wired into load_snapshot (task 4).
Requirements: 1.10, 1.11, 5.8
replay_wal() now halts logical apply and engages persist_mark_apply_resource_fenced()
on a negative apply_committed_entry() return during boot/restart replay, mirroring
persist_apply_pump_run()'s existing live-path halt-and-fence behavior (task 4.1).
Physical scan bookkeeping (current_lsn, CRC chain) stays unconditional per entry,
consistent with persist_recover()'s own established current_lsn/last_applied_lsn
divergence precedent.
Went through one remediation round: round 1 exercised the wrong ha.c gate (periodic
re-alert timer); round 2 corrected the test to drive the real election-time gate
(raft_cb_become_leader_with_cluster_for_test()), with teeth independently confirmed
via a temporary fence-clear RED/GREEN cycle. Round-2 reviewer APPROVED.
Documents a real, NOT-fixed residual gap found during this task: a node's very
first-ever promotion evaluation in a fresh process runs before that node's own
first persist_recover()/replay_wal() call, so this fence is invisible to that one
promotion check (ha.c:1303-1408 gate runs before ha.c:1427-1439's first-boot
recovery). Every promotion evaluation after a process's first one is protected.
Out of this task's persist.c-only boundary; flagged for a dedicated ha.c follow-up.
Adds apiserver_push_jobarray() (apiserver.c), mirroring task 5.1's
xcesched_push_jobarray() exactly: same (base jobid, array_size, one
spec) Data.jobArray derivation, same direct xce_request()/OPCode_push
transport. apiserver_job_new_fun() branches on l->count > 1; the
existing single-job path (apiserver_push_joblist()) is byte-for-byte
unchanged.
Additionally, and unlike the scheduler (single target), replicates
apiserver_push_joblist()'s existing per-connected-server fan-out loop
(N_APISERVER_MAX iteration, same lock discipline) -- verified against
2 simulated connected servers, each receiving exactly one push.
New standalone apiserver_push_unit_test binary (same link-collision
rationale as task 5.1's sched_push_unit_test), correctly omitting
egress.c since apiserver_push_joblist() has no WAL-commit filter.
Task 5.2 of job-array-single-wal-record (Requirements 6.2, 6.4).
