Two root causes, confirmed by direct code reading (previously misattributed
to an overlayfs sandbox fdatasync artifact):
conf_trim_old_versions() sorted config-version files by st_mtime (second
resolution). The test writes 3 versions 15ms apart -- well within one
second, so ties are the expected outcome, not a rare edge case. qsort()
isn't stable, so a tied ordering could push the protected current-pointer
file (skipped on deletion, never compensated) past keep_n, leaving one
extra file on disk ("Expected 2 Was 3"). Fixed by sorting on st_mtim
(nanosecond-resolution struct timespec) instead of st_mtime.
cr_setup() keyed its tmpdir on getpid()+time(NULL) (second resolution),
risking collision with a stale directory left by an earlier interrupted
run (ASAN abort, timeout, SIGKILL -- none reach cr_teardown()'s rm -rf)
reusing the same PID within the same second -- test_conf_get_saved_tag_
no_config then observed that directory as "not actually empty" and read
a leftover tag instead of the expected 0. Fixed with a monotonic counter
guaranteeing a unique directory per cr_setup() call.
TDD: RED reproduced on the first attempt (both tests failed with the
exact historical signature). GREEN confirmed across 16+ repeat runs
after the fix, zero other regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Found while chasing bug-1714's crash (root cause still open -- see that
entry): two related, independently-confirmed defects, not proven to be
bug-1714's own cause but real bugs on their own merits.
xce_core_push_events()'s JOBS_PUSH case never checked sched_add_job()'s
return value -- if key_add() failed (duplicate jobid), the freshly-built
xce_sched_job_t was handed to picker_iface->on_job_push() anyway, even
though it was never registered in core->jobs. An orphaned job the
picker knows about but sched_get_job() can never find again. Now checks
the return, frees the orphan (at core_build_job()'s own established
cleanup granularity -- xce_sched_job_t has no complete destructor
anywhere in this codebase, kill_sched_job() only frees ->rusages, so
this intentionally doesn't invent a more thorough one), and skips the
picker call.
This makes it safe to fix (not the other way around): sched_on_job_array()'s
per-element dedup check (handler.c) has a TOCTOU race -- a job is only
added to core->jobs later, by the driver thread, so a redelivered array
push arriving on the request thread before the driver catches up isn't
caught. Documented as a best-effort optimization, not a correctness
boundary -- the actual authoritative uniqueness check is sched_add_job()'s
key_add(), which the JOBS_PUSH fix above now handles correctly.
TDD: new test reproduces the duplicate-jobid path (RED: on_job_push
fired twice, Expected 1 Was 2), GREEN after the fix. Full scheduler
regression: 54/55, only the known bug-1561-adjacent flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Follow-up to ebe691c3 (HOOK_DETAIL_JOB_START/JOB_EXIT), same gap for job
failures: apply_committed_entry()'s WAL_JOB_FAIL case only ever fired
the generic HOOK_DETAIL_JOB_UPDATE, leaving sched_job_fail_fun (sched.c)
and apiserver_job_fail_fun (apiserver.c) dead -- the apiserver's
OnJobFinished was starved of data for job failures specifically (the
prior fix only closed this for clean finishes).
Fires HOOK_DETAIL_JOB_FAIL only for the WAL_JOB_FAIL_STAGE_FLAG branch,
which already constructs a real job_failinfo_t -- the legacy _pad==0
absolute-overwrite branch has no fail-specific data and must not
fabricate one, mirroring how the WAL_JOB_START fix only fires when
pidinfo was genuinely decoded. Unlike WAL_JOB_START, no gate-widening
was needed: apply_job_fail_transition() always returns
JOB_TRANSITION_APPLIED for a genuine new fail transition (confirmed by
reading it), so the existing result-gated early return already covers
this correctly.
TDD: RED confirmed (fix reverted, test failed "Expected 2 Was 1"),
GREEN after restoring it. Full regression: 1220 tests, only the known
bug-1561 flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
The `worker)` case's meson compile target list omitted xcecd_virtual --
the no-root worker binary deploy-ha.sh/the xce-deploy-ha skill needs
whenever no real systemd manager is present (the common sandbox case).
/xce-build worker alone left it unbuilt, so deploy-ha.sh silently
soft-skipped generating start-worker.sh. Verified: xcecd_virtual now
builds with a fresh mtime, full worker suite still 5/5.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Adds the previously-uncommitted spec.json/requirements.md/design.md/research.md
for scheduler-snapshot-job-allocation (feature now fully implemented across
tasks 1.1-4.5) and the in-progress xce-test-cli-ux spec's requirements.md/
spec.json. Updates brief.md, a security.md note on an unrelated deferred RBAC
persistence gap found during this feature's cross-component survey, and syncs
OpenWolf's anatomy/memory/token-ledger/daemon-log bookkeeping.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Task 1.1: new deep-copy for xce_job_alloc_t composed from existing
make_alloc/make_host_alloc/make_job_alloc, explicitly preserving each
allocation's original from/duration rather than accepting make_alloc's
own time(NULL) stamp. TDD: 6 new unit tests (single-host, multi-host,
multi-resource-per-host, shared-resource-pool, independence-after-free,
from/duration fidelity), all passing under ASAN.
Also logs bug-1723 (pre-existing, unrelated leak in two bug-1722 test
fixtures, confirmed via git-stash A/B rebuild to be byte-for-byte
identical before and after this change) -- documentation only, not fixed
here.
Requirements: 1.1, 1.4
Found via the mandatory live HA chaotic soak for this session's controller
audit campaign -- unrelated to any of the 9 audit fixes (bug-1684/1687/
1689/1690/1691/1692/1693/1694), all in ha.c/persist.c/job.c/cluster.c/
sched.c/worker.c; this is in myapp.c, untouched by any of them.
myapp_run()'s bootstrap "is a leader already known" check had a fallback
`c.ha.raft.term > 0` meant to detect genuine new raft activity processed
during the startup-stagger nap. But ha_restore_term() loads ha->term from
disk BEFORE this check runs, on every boot -- and Raft terms only ever
increase, never reset. So `raft.term > 0` is trivially true on every
restart for the rest of the cluster's on-disk lifetime, once it has ever
held one election, regardless of whether any leader is currently
reachable. Live-reproduced: every node, on every restart after the
cluster's first election, unconditionally skipped ha_start_election_async
forever -- no node could ever re-assert leadership through a real
election again. bug-1477's existing cluster_init_phase2 retry loop kept
firing correctly (confirmed in the reproduction logs) but could never
succeed, since the deeper cause -- no election ever runs to produce a
live primary to push config from -- was never addressed by that fix.
Fixed by snapshotting `term_at_boot` right after ha_restore_term()/
ha_init_raft() seed the term, strictly before the stagger nap, and
comparing `raft.term > term_at_boot` instead of `raft.term > 0` -- only a
term that has genuinely ADVANCED during this boot cycle (real vote/
heartbeat traffic processed) counts as evidence a leader is known. The
decision was extracted into a new, unit-testable myapp_leader_already_known()
(startup_validation.c, since controller_unit_test deliberately excludes
myapp.c/main.c). voted_for_name/vft==cur and primary_ctrl_fd>=0 checks
are unchanged.
7 new tests in test_ha_startup.c; RED/GREEN independently verified.
Rigorous A/B live-soak proof (identical 6-round --chaotic run, fixed vs
git-stashed-back-to-original): baseline showed term frozen for the whole
run and zero election activity on any non-primary node; fixed run showed
term genuinely progressing and ctrl-2 achieving a real, log-verified
CTRL_PRIMARY transition via a won PreVote round -- proof a node other
than the original primary can now take over, which was structurally
impossible before. Full controller suite: 1191 tests, 2 failures -- both
the pre-existing, unrelated bug-1561 overlayfs sandbox flake.
Residual, NOT independently verified yet (flagged honestly, not fixed
here): the fixed run's chaotic verdict showed committed_job_loss=6
(baseline 0) -- traced via waldump to one jobid missing from every node's
WAL, at the cold-boot window before round 1. Resembles the already-fixed
bug-1487 false-positive class (apply-pump-lag racing verify_classify's
expect-job check) but the reported mechanism doesn't perfectly match
(bug-1487's job was present-in-WAL/not-yet-visible; this one is described
as absent from every WAL entirely) -- needs independent re-verification
before being dismissed, tracked as a follow-up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Last of the 9 confirmed findings from the /xce-audit-invariants controller
pass. checkpoint_task_fn was registered directly as a delaymgr entry and
ran its full blocking body inline: checkpoint marker pwrite+fdatasync,
write_snapshot()'s genuinely unbounded per-entity walk (scales with
cluster state size) held under apply_pump_lock, WAL trim, and snapshot
rotation/unlink -- all on the same thread that drives heartbeats/election
timeouts. Unlike every other delaymgr operation (snapshot send,
config-push, reconnect, catch-up), checkpoint had no offload mechanism
at all -- the one gap left from the raft-drive-async hardening pass.
Design: full offload via the SlowLane (slowlane_submit), default ON --
NOT the vtask pool. persist_apply_pump_run() already holds apply_pump_lock
across its own per-entry walk while running ON the vtask pool
(VTASK_LIVE_BLOCK); putting checkpoint's unbounded, lock-held
write_snapshot() on that same pool risked starving it, exactly what
ha-async-io.md's own thread-role table warns against. The SlowLane is
the actual established pattern for this I/O class (push_config_transport_only,
the snapshot-send path, catchup_lane_fn, demotion_lane_fn all dispatch
there) -- and persist_checkpoint() already runs on the slow lane in
production TODAY via ha_snapshot_do_send() -> snapshot_lane_fn, so
apply_pump_lock's acquisition from a slow-lane thread is proven-safe,
exercised daily, not new surface. Unlike catchup/demotion (which stay
flag-gated OFF for an unrelated, still-not-root-caused hang, bug-1429),
checkpoint's new flag defaults ON given this reuses an already-proven
code path. Inline-fallback + WARN only on a genuine slowlane_submit
failure (not -EAGAIN/-EEXIST, both non-failures) -- never silently drops
a scheduled checkpoint.
New test_persist_checkpoint_offload.c (3 tests). RED-phase forced the
pre-fix always-inline path: the suite never printed a single result and
had to be killed, proving the test genuinely detects the violation, not
a tautology. GREEN: all 3 pass. Full suite: 1184 tests, 2 failures --
both the pre-existing, unrelated bug-1561 overlayfs sandbox flake
(exact signature match). No live HA soak was run for this specific fix
in isolation (explicitly scoped out, not hidden) -- the mandatory
live-soak proof for this whole audit campaign covers it next.
This closes the last of the 9 confirmed findings from the controller
audit (bug-1684, bug-1687/1689/1690/1691/1692, bug-1693, bug-1694).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Violates this project's own recorded Decision Log entry: "No client
reply-wait timeout as a fix for a hung request -- fail loud at the real
transport/source instead of masking with a deadline backstop."
OPCode_schedAdmin (cluster.c) and on_fetchfile (job.c, the worker proxy
-- cluster.c's own comment says schedAdmin "mirrors on_fetchfile's proxy
pattern", confirming one shared design mistake, not two independent
bugs) both blocked on a multi-second pthread_cond_timedwait, replying
"timeout" on expiry. handle_scheduler_leave()/handle_worker_leave() (the
actual disconnect handlers) only nulled the endpoint pointer -- neither
ever signaled the pending waiter, so a mid-wait disconnect was invisible
until the deadline lapsed. The timeout was the ONLY failure-detection
mechanism, exactly what the decision forbids.
Both sites already had a working signal function on their success path
(schedadmin_signal(), fetchfile_signal()) -- just never wired to the
disconnect handlers. Fixed by calling them from
handle_scheduler_leave()/handle_worker_leave() with a distinguishable
-ECONNRESET outcome, so the waiter wakes immediately on disconnect
instead of only on the deadline. Timeouts kept as defensive backstops
for a genuinely wedged peer that never disconnects cleanly either --
this fix is about not relying SOLELY on the timeout, not removing it.
schedadmin_signal_disconnect() must be called BEFORE handle_scheduler_leave()
acquires c->sched.lock -- the dispatch wait holds that same lock for its
entire duration, so signaling after acquiring it would deadlock.
Found by an /xce-audit-invariants controller pass (Decision Log fidelity
dimension). New test_fetchfile_disconnect.c plus extensions to
test_sched_admin_dispatch.c/test_sched_worker_change.c; RED/GREEN
independently verified against real elapsed-time assertions (pre-fix:
5001ms full-timeout wait; post-fix: <1s immediate wake). Full controller
suite: 1181 tests, 2 failures -- both the pre-existing, unrelated
bug-1561 overlayfs sandbox flake, confirmed via a clean meson-driven run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Last of the five field-parity gaps from the original audit (bug-1678,
bug-1687, bug-1689, bug-1690, bug-1691). set_preason()/clear_preason()
set job.pend_reasons (host/resource-based pending-job explanations), but
persist_job_hook_fun's default: arm only ever captured {jobid, state}.
Neither apply seam restored pend_reasons after a restart.
Reuses the ALREADY-EXISTING JobPendReason/HostBasedReason/ResourceBasedReason
capnp structs -- no schema change needed, unlike bug-1690's WAL_JOB_RUSAGE.
Safety-checked against bug-1690's lesson (RTTI/makeString()-backed fields
need merge-not-replace, not free-and-replace): pend_reasons is plain
malloc/strdup-backed throughout (erase_job_preason is FREEUP()-only,
add_host_preason/add_res_preason use strdup()/calloc()), so a
straightforward free-and-replace is safe here, matching bug-1678/1687's
pattern instead.
New wrinkle beyond the four prior fixes: pend_reasons has a genuine CLEAR
operation, and "trailing bytes present or absent" can't distinguish a
legitimate clear from an old-format payload -- added an explicit presence
byte (0=cleared/adopt NULL, 1=capnp bytes follow), verified by two
dedicated "clear survives restart" tests.
11 new tests in test_wal_job_preason_fact.c; RED/GREEN independently
verified (7 genuine pre-fix failures, one test's own wrong premise about
capnp's garbage-byte behavior caught and corrected during RED/GREEN
iteration). Full controller suite: 1178 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake. No ASAN findings.
This closes the field-complete apply parity invariant class the
xce-audit-invariants controller dimension 7 was added for.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Same class as bug-1678/1687/1689/1690: on_job_ckpt() sets job.ckptinfo
(a plain char*, xstring-backed via makeString()) but is purely hook-driven
-- persist_job_hook_fun's default: arm only ever captured {jobid, state}
for WAL_JOB_CKPT. Neither apply seam restored ckptinfo after a restart.
Simplest of the five field-parity fixes: ckptinfo is a bare string, so no
capnp struct is needed -- mirrors WAL_QUEUE_CTRL's existing NUL-terminated-
string-after-header wire shape instead. Adoption is unconditional (not
gated on a state transition), matching bug-1690's precedent: a checkpoint
report never changes job.state. restore_job_ckpt_state() releases the
prior value via dereference() (the correct RTTI verb for an xstring), not
free().
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension) -- the last of five sibling gaps found together
(bug-1678, bug-1687, bug-1689, bug-1690, this one). 7 new tests in
test_wal_job_ckpt_fact.c; RED/GREEN independently verified (5 genuine
pre-fix failures, 2 backward-compat tests correctly passed unchanged).
Full controller suite: 1167 tests, 2 failures -- both the pre-existing,
unrelated bug-1561 overlayfs sandbox flake. No ASAN findings.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Same class as bug-1678/1687/1689: the WAL fact for WAL_JOB_RUSAGE only
carried {jobid, state}, so job.run_usage (client-reported per-host
CPU/mem usage) was never restored on either apply seam after a restart.
Unlike the prior three, xce_job_runusage_t/xce_host_runusage_t had no
existing Cap'n Proto mapping -- a genuine new schema addition
(HostRunusage/JobRunusage in controller.capnp, mirroring JobPendReason's
List(NestedStruct)+$C.maplistcount style), verified against the real
struct definitions in types.h, codegen confirmed via nm on the built
binary rather than assumed.
Two deliberate deviations from the established precedent, found by
reading the real code rather than copying the pattern blindly:
1. Adoption is unconditional, not gated on a state transition -- a
rusage report almost never changes job.state (repeat reports keep
state at RUN), so gating on JOB_TRANSITION_APPLIED like START/FINISH/
DISPATCH do would make the fix dead code for the realistic case.
2. Merge, not free-and-replace. update_host_runusage() builds run_usage
with host_name sourced from makeString() -- an RTTI-managed pointer,
not malloc() -- and the primary's own live apply pump re-applies every
fact it just wrote. A naive free_JobRunusage_ptr()-then-adopt (matching
the prior three fixes exactly) would free() a non-malloc'd pointer on
the very first live rusage report for a new host -- guaranteed heap
corruption. Fixed with merge_job_runusage(): existing hosts get
scalar-only in-place updates (host_name pointer never touched), only
genuinely new hosts adopt their freshly-decoded (safe, malloc'd)
host_name. Proved via a real makeString()-backed test asserting
pointer identity survives, not just value equality. job_destroy()
(src/lib/jobj.c) was found to have this identical latent mismatch,
pre-existing and independent of this fix -- flagged, not fixed here.
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension). 8 new tests in test_wal_job_rusage_fact.c; RED/GREEN
independently verified. Full controller suite: 1160 tests, 2 failures --
both the pre-existing, unrelated bug-1561 overlayfs sandbox flake. No
ASAN/heap-corruption findings. Backward compatibility confirmed for
old-format bare-payload entries.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
