The external-license-resources E2 fixture builder crr_build_config
registers five resources and four distributions, writing resources[4],
dists[3] and vals[4]. All four caller test functions declared only
resources[4]/dists[3]/vals[4], so every build_config call wrote past
the caller's stack arrays -- deterministic ASAN stack-buffer-overflow,
the whole controller suite (137 fixtures) red since 2fcacac2.
Resize the four caller arrays to match the builder's sizing
(resources[5]/dists[4]/vals[5]) with a bug-1977 comment on the sizing
contract. No production code touched.
Verified: src/tools/xce-build.sh controller (asan) 6/6 OK, controller
suite 91s green; buglog record .wolf/bugs/bug-1977.md.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Strategy layer (spec .kiro/specs/external-license-resources design.md 3.5/3.6):
- limits.yaml samples: extlic_share (per-project, projA/projB, cap 3.0) and
extlic_qdist (per-queue, normal/night, cap 4.0) on the externally reported
extlic pool -- instances "limit=extlic_share/p=<proj>" and
"limit=extlic_qdist/q=<queue>" under the existing limit vres mechanism.
Multi-daemon redundancy is free (resval last-writer-wins); manual override
REST deliberately out of scope.
- White-box scheduler test (test_limit_project_scope.c): project-scope
match + vres instantiation, per-project instance fan-out with caps,
unknown underlying resource failing clean. 3/3 under ASAN.
Controller fix (the Phase 3 E2E blocker, TDD red->green):
- dispatch_a_job_() rebuilt j->job.allocs from the scheduler's dispatch
payload but migrated only the per-host records -- allocs_shared (the
limit vres charge among them) was dropped, and its payload pointer
leaked. That rebuilt record is what sched_job_update_fun() re-sends to
the scheduler as ground truth (the JOB_UPDATE adoption there rebuilds
job->accounts from it), so sharedInUse never accumulated the vres
charge: unsatisfy_ admitted past every global limit forever.
Fix: MOVE the shared records alongside the host records (test
test_dc_success_migrates_shared_allocs).
t1 E2E (4 licenses, extlic_share projA/projB cap 3.0): 3 projA RUN, 4th
projA PEND, projB unaffected (own per-project instance); killing one
RUN job re-admits the pending projA; external free pool and per-project
shares enforced independently; all daemons healthy throughout.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Implements Phase 1 of .kiro/specs/external-license-resources/ (design.md):
externally-reported license pools pushed by a dedicated daemon through the
existing resval pipeline, with EXTERNAL-aware admission arithmetic.
Wire & controller:
- controller.capnp HostRole.extres (@5, appended); xce_join_req accepts it
- extres.c: HostRole_extres join table (endpoint+name, no workermgr slot),
leave cleanup; join.c/cluster.c dispatch
- opupd.c do_update_resvals: extres branch runs the resource hooks and
update_resval_list(c, NULL, ...) — host=="" entries route through
update_val_res' SHARED branch into c->sharedinst; per-entry -EPERM for
non-shared targets is logged, never fails the list
- worker.c update_val_res: NULL worker only writes SHARED resources
- cluster.c cluster_build_resources: rebuilt on resourceApplyDef (bug-1854)
— EEXIST-tolerant so definitions added by an activation register on a
RUNNING controller (bug-1973), and RES_APPLY_CHANGED updates definition
fields (valtype/ordtype/flags/interval) in place so a corrected config
heals a stale registration (bug-1974); seed_sharedinst_from_config skips
EXTERNAL pools (daemon-reported free counts only; seeding would fabricate
capacity while the daemon is down)
Scheduler:
- core_events.c RESVALS_DELTA: NULL/"" host marks a SHARED value and lands
in core->sharedInst via dup_val (bug-1585 ownership rule), mirroring
snapshot_load's host=="" precedent — previously WARN-skipped as unknown
worker
- resource.c unsatisfy_: EXTERNAL skips the inuse term
(F >= uncomit + wanted; F is already net of our checkouts)
xceesd (src/esd, new component):
- script collectors polled per source, debounce on unchanged values,
fail-open stale policy; pushes host=="" shared resvals over xcecli
- reconnect gap fix (bug-1975): with a stable value the debounce pushed
nothing and reconnects only fire inside a push — after a controller
restart the daemon sat on a dead connection forever; the main loop now
forces a full re-push when xcecli_is_connection_broken() (bug-1760
predicate) latches
Config/mapping:
- resources.yaml: extlic sample (location: [external])
- mapping.py: RESOURCE_FLAG_EXTERNAL (0x10) + location:["external"] mapping;
xceadmin tests
Tests (TDD, red->green per fix): test_extres_resvals.c (join+resval
pipeline), test_config_resource_reconcile.c +2 (register past EEXIST /
hot-apply changed definition), test_core_events.c +131 lines (shared DELTA
landing), mapping tests. Suites: controller 1358/0, scheduler 100/0,
esd 7/0, lib 1/0; check-no-fork covers src/esd.
Live t1 E2E: config gen=15 push -> xceesd push extlic=4 lands in
sharedinst -> rusage[extlic=3] job RUN; second job PENDs at free=1;
release re-admits it; controller restart -> forced re-push restores the
value and admission resumes. Full precommit job verification passed.
Co-Authored-By: Claude Code <noreply@anthropic.com>
rusage collection (merge cb0c7fe9 regression, root cause of 0 counters):
- collect_job_rusages builds the systemd unit name with
make_job_sd_unit_name (four-segment fmt_jobid_ex form) instead of a bare
%lu of the packed uint64, so the D-Bus property lookups hit the real
per-job unit again. E2E on t1: DONE job reports max_mem/avg_mem/cpu_time
non-zero (previously all 0).
XCE_JOBID written identity:
- exdata sets XCE_JOBID via fmt_jobid (four-segment
cluster-apiserver-timestamp-seq) instead of a bare packed decimal;
XCE_JOBINDEX handling unchanged.
- xrun parses XCE_JOBID with parse_jobid (accepts the four-segment form);
a bare decimal no longer round-trips through the written identity.
- E2E on t1: job sees XCE_JOBID=<submit id>; xrun remote-start round-trip
to another host matches the job.
select WARN (silent degradation):
- parse_resreq_primary WARNs when a select references an unregistered
resource ("...references an unregistered resource -- dropping the select
and skipping any rusage[] clause of this primary") instead of failing
silently; scheduler log shows the WARN on t1 and the job schedules
without the constraint.
CLI env handling (unit-tested; subEnvVars is not flag-bound so this is
not reachable from the production CLI):
- getEnvironmentVariables keeps B=b=c (split at first '=') and warns +
drops a bare key or an '=value' fragment instead of silently mis-parsing.
Docs:
- src/docs/xce.md: document the dead-clause semantics (select/rusage
primary drop behavior) matching the WARN.
bug-1972 (test-only UAF): test_job_live_gc asserted job destruction by
reading refcount from the freed job; rewrite the assertion to observe
destruction without touching freed memory. controller_unit_test is green
again (1354 tests).
Regenerate 3 repl_trace_baseline fixtures whose baselines shifted with the
select-WARN logging; replay tests green.
Spec naming: external-license-resources design.md/brief.md fix the
self-referential daemon-name note (old drafts said xceextresd/extresd).
Verified on t1: rusage_burn job (counters non-zero in DONE), echo_env job
(four-segment XCE_JOBID), select[nosuchres] job (WARN + schedules),
xrun t2 remote-start round-trip, CLI env unit tests, xcelib 762 /
controller 1354 / worker exdata 8+11 / sched repl_trace 4+14+4+10+17
tests green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Terminal jobs (DONE/EXIT/PERR) were only filtered from checkpoints and
WAL replay (job_should_retain, finished_retain_s=3600) — the live jobmgr
never dropped them, so joblist and the scheduler's core->jobs view grew
without bound across a controller's uptime (N4).
Three pieces:
1. kvmgr delete path (src/lib/kvmgr.c): xkvmgr_type had .del_key=NULL
and .vclear=NULL, so key_del()/vclear() on a kvmgr only returned
-EINVAL. New xkvmgr_del_key (key_find to locate the slot idx ->
key_del — kvtab's val_killer is NULL so the idx scalar is untouched ->
vclear(data, idx)) and xkvmgr_vclear forwarder, wired into the
typeinfo. A missing key propagates -ENOENT (same contract as
xkvmgr_find).
2. Controller periodic sweep (job.c): job_live_gc_sweep — primary-only
(standbys rebuild their live table from WAL replay; deleting there
diverges), collects !job_should_retain jobs under the jobmgr READ
lock with reference() (check_workers_timeout collect pattern), then
per-job WRITE-locked key_del. TWO dereferences per purge: the
collection reference and the table's own makeJob reference — the
jobmgr's datakiller (job_obj_killer) is a deliberate no-op, so
key_del transfers the table's ownership duty to the caller; one
dereference would leak at refcount 1 forever (the very accumulation
this sweep drains). Registered by cluster_start_job_live_gc with
ha_start_primary_heartbeat's idempotent static-arm pattern +
delaymgr.running guard (60s, runcount=-1).
3. Registration on both promotion tails (ha.c): raft road after
ha_start_primary_heartbeat and the non-raft ha_promote_to_primary.
NOT cluster_start — it is orphaned on the HA myapp_run path.
Safety: apply_committed_entry's jobstate group and WAL_JOB_UPDATE
already treat a key_find miss as a benign skip (bug-1970, which
explicitly anticipates post-purge facts); persist_verify_apply_correctness
counts only within the boot replay window. The TRANSIENT TTL now takes
live effect too (previously checkpoint-exclusion only).
UTs: kvmgr del INT+STRING (miss propagation, sibling intact, freed-slot
reuse, killer invocation), controller test_job_live_gc.c (expired DONE
purged + refcount 0, fresh/RUN retained, idempotent re-run, EXIT/PERR,
TRANSIENT TTL + transient_since==0 guard, standby no-op, NULL-safe,
arming guard). xcelib 761/761, controller 1354/1354.
E2E (t1): promote arms the sweep; the pre-restart DONE array jobs were
purged across five sweeps (33 jobs) as their 3600s windows elapsed;
submit→RUN→END flows and all daemons healthy throughout.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Breakpoint 2 of the admin-plane gaps (YAML flag/resourcemap wiring), plus
the queue payload ownership fix that the t1 controller segfault of
2026-10-23 12:41:13 traced to.
1. YAML resourcemap wiring (admin-plane-gaps 1.10):
- schema: Resource.distributions @8 / hostBased @9 uncommented --
resourcemap had no wire destination at all before this.
- resobj.c: resobj_init's #if 0 stub replaced with a real deep copy
(distributions + hbased_defined); resobj_destroy frees them. Deep
copy is mandatory: the source is myconf->cluster_config, which
keeps owning and later frees its tree.
- controller: reconcile_config_resources + seed_sharedinst_from_config
(cluster.c) seed c->sharedinst for SHARED|GLOBAL resources whose
distribution carries a value (the license-pool shape), from both
config_activate_committed paths, ahead of the conf4sched push.
- xceadmin: mapping.py maps resourcemap -> distributions (only the
location:[all]+value shape sets SHARED|GLOBAL), build.py renders
shared=true/distributions; golden expected.bin regenerated.
2. xcesched_push_configs resval double free (t1 12:39:52 abort): the
seeded push freed the resval entries AND the backing array via
free_name_val_list, then xlist_fini freed the same array again.
Detach xreslist before the FREEUP/xlist_fini block (same ownership
class bug-1506 fixed in sched_worker_change). Latent since inception:
every pre-1.10 push ran with nresvals==0 and never armed it.
3. queue payload deep-copy (t1 12:41:13 encode_Threshold_list segfault):
queue_init (create) MOVED the definition's users/hosts/thresholds/
window payloads out, and queue_refresh_def MOVED them out again --
but reconcile_config_queues runs MORE THAN ONCE over the same
cluster_config object (boot: pre-election activation AND phase2's
build_queues), so the second pass refreshed live queues from the
drained definition: thresholds became the poisoned (count>0,
array=NULL) pair, the generated encode_Threshold_list has no NULL
guard, and the next queue/worker push died on the NULL base.
Both paths now deep-copy (queue_init and queue_refresh_def); the
definition keeps its payloads, (NULL/0) genuinely means "no list in
this config", and reconcile is idempotent. New dup_group_config
helper in utils.c (the generated code has no GroupConfig dup).
Mirrors resobj_init's distributions deep-copy and worker.c
apply_join_state's dup pattern.
4. UT: test_reconcile_thresholds_survive_second_pass drives a triple
reconcile of a threshold-carrying queue and asserts definition
intactness plus a real encode_QueueConf -> wire -> decode round
trip; new test_config_resource_reconcile.c covers seed/deep-copy/
non-seed classes and idempotent re-seed.
Suites: controller_unit_test 1348/1348, sched_push 4, sched_worker_change 3,
apiserver_push 2, host_runusage 11, xcelib 759, xcealloc_oom 11,
99 scheduler binaries, all worker + harness suites, xceadmin pytest 375.
t1 E2E: boot double-reconcile clean, scheduler rejoin + full conf4sched
push applied, recovered crash-era PEND job reached DONE, 30-job saturation
flow (24 RUN + 6 PEND -> DONE), live gen-12 config activation with
re-push to the connected scheduler -- no crash, all four daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Three fixes so a pushed config edit takes effect on the RUNNING cluster
instead of waiting for a scheduler rejoin (admin-plane-gaps 1.1):
1. queue_refresh_def (controller/queue.c): reconcile_config_queues no
longer leaves existing runtime queues untouched -- it refreshes their
CONFIG-owned fields in place (resreq/pre_exec/post_exec interned
swaps; users/hosts/thresholds/windows via free_* + MOVE_PTR, same
ownership as queue_init; scalars copied) and notifies the live
scheduler/apiserver via the same NO-WAL hook a fresh queue gets.
2. config_refresh (controller/cluster.c): cluster_build_config is now
idempotent -- boot creates the confvar entries, activation re-runs
refresh their interned values in place (makeConfig refuses EEXIST,
kvmgr has no del_key), so heartbeat_interval/
inactive_heartbeat_count exports stop being frozen at boot values.
3. activation re-push (controller/config_repl.c): config_activate_
committed now calls cluster_build_config then xcesched_push_configs
(un-static'd) to the connected scheduler, on both the hash-verified
and -ENOENT fallback activation paths. Lock order follows the
established apply_pump_lock -> sched.lock seam; push failure only
WARNs (scheduler keeps its previous config until next join/push).
E2E-found display gap: apiserver's Go handleNotifyData had no
NotifyData_data_schedUpdateQueue case, so the controller's per-queue
notify was silently dropped and `xce queues list` kept the join-time
snapshot. Added the case (OnQueueUpdated) so hot-applied priority and
queue state changes are visible immediately.
Unit tests: reconcile refresh-in-place + notify counts (test_config_
queue_reconcile.c); activation repushes once per activation, not while
deferred (test_config_repl.c capturing fake); OnQueueUpdated in-place
refresh (handler_test.go). Full controller suite green.
E2E on t1/t2/t3: two config pushes (gen 8->9->10) with no restarts --
controller activation + per-queue refresh + scheduler re-push logged,
scheduler applied at the round boundary, `xce queues list` showed the
new priority immediately, and a submit->RUN->DONE job round-tripped on
the hot-edited queue with all four daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Two independent fixes:
1. Worker pre-exec failure (TODO fix). queue_pre_exec()/job_pre_exec()
previously called exit(1) on failure, killing the one-shot worker
before any status report reached the master -- the job then hung with
no user-visible reason and the worker process vanished. Now the
failure path reports the job FAILED via ipc_worker_send_status with
stage JOB_EXEC_STAGE_{Q_PRE,PRE}_EXEC and the command's real exit
code, skips the job body (no post-exec -- the body never ran,
mirroring LSF pre-exec semantics), and takes the same tail as the
VNC/X11 start failures. main.c now carries job_status->stage through
to xcecd_job_finished() so the stage (not the MAIN default) reaches
the controller's fail fact, deciding both the exit-status merge
(apply_job_fail_transition) and whether the exit code is recorded
(apply_job_fail_exit_code). Also fixes the error print: uv_proc_pre_exec
returns a WEXITSTATUS, not a uv errno, so uv_strerror mangled a plain
exit status into "Unknown system error N".
E2E verified on t1: job-level pre-exec `exit 7` -> job EXIT,
worker logs "Job pre-exec command failed with exit code 7", marker
file shows only the pre-exec ran (no post-exec, no job body), and
the exit code reaches the apiserver's OnJobFinished (exit_code=7)
and the committed WAL_JOB_FAIL fact.
2. Snapshot queue no longer in config wedged election forever.
Checkpoint snapshots record every queue in the live qmgr; if a queue
is deleted from queues.yaml between the checkpoint and a restart,
restore_queue_state returns -ENOENT for it ("queue not yet built
from config -- skip" per its own contract), but
load_snapshot_entities treated any non-zero as fatal ->
persist_recover failed -> cluster_init_phase2 retried forever
(bug-1477 never self-terminates) and no primary was ever elected.
The snapshot queue loop now treats -ENOENT/-EINVAL as a benign skip
with a WARN, matching the same documented policy as
apply_committed_entry's and apply_wal_entry's WAL_QUEUE_CTRL cases;
every other non-zero return stays a genuine failure. Doc comments
updated accordingly.
Live-reproduced and verified on t1 (2026-10-22): deleting a test
queue + restarting the controller previously wedged election
permanently; with the fix the controller boots past the stale frame
with a WARN and reaches CTRL_PRIMARY.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Controller suite flake (user-reported, -j parallel runs): the commit
engine's async timeout sweep armed continuation deadlines with the
hardcoded DEFAULT_SYNC_TIMEOUT_MS (500ms) instead of the configured
sync_timeout_ms, while the blocking persist_commit_barrier honors the
config. Fixtures configuring longer or shorter sync timeouts saw the
sweep resolve -ETIMEDOUT at 500ms before their explicit
truncation/second-call steps, flipping "Expected -17/-2 Was 0/-110"
assertions (test_commit_resolve.c:1458/:1616). commit_submit_async and
commit_sweep_worker now both use p->config.sync_timeout_ms with
DEFAULT_SYNC_TIMEOUT_MS only as the zeroed-config fallback (mirroring
persist_commit_barrier's timeout_ms == 0 convention), and the sweep
WARN reports the effective value.
Apiserver flake (found during full-build verification):
deriveArrayJobStats took the array identity from the first element
seen over store.Jobs.List(), but List() walks a Go map whose order is
randomized per call -- when element rows disagree (the pinned test
case), the stats identity flipped run to run (~12% of -count=1000
runs). The identity carrier is now the lowest Unique_JobID (the array
head, element 1), tracked via arrayAgg.carrierUID; other rows
contribute counters only. Counters were already order-independent and
are unchanged. The test now pins the head-carries-identity contract
explicitly.
Also fixes a latent parse-error trap in the test: a string literal
ending a line (C-style implicit concatenation) triggers Go's automatic
semicolon insertion -- the next line's argument list parses as
"missing ',' before newline". String concatenation across lines must
keep the '+' at the end of the line.
Verified: controller suite 1342/0 (three runs incl. -j 7 parallel
stress); sweep WARNs now match fixture configs (300ms fixture warns at
300, longer-config fixtures emit none); go test -run
TestDeriveArrayJobStats -count=1000 all green (was 118-130 failures);
full ./tools/xce-build.sh all green end to end.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Production (hand-fixed, semantics preserved with width-limited %.*s):
- persist.c (33 sites): workdir/name joins share the destination budget;
WAL segment/snapshot paths via a WAL_SEG_WALPATH_MAX macro derived from
seg_path's own declared size; trim loops use d_name's NAME_MAX (255) cap.
- config_repl.c (16 sites) and ha.c's config-hash twin: same half-split
and fixed-literal-tail budgets, byte-identical path construction kept
between ha.c and conf_save_config.
- main.c: certdir/key/cert paths leave room for their fixed suffixes.
Test suites (-Wno-format-truncation, rationale in-tree): the 68 remaining
sites are all snprintf(dir, 4096, "%s/<literal>") fixture scratch paths
under mkdtemp /tmp/xce_* dirs that never approach PATH_MAX -- gcc cannot
prove the tail fits, but every one is a false positive. Applied as a
compile_args declare_dependency on each suite's shared deps list.
Vendored c-capnproto: -Wno-switch (~40 exhaustive-dispatch false
positives), -Wno-unused-function (empty define_encode_function stub),
-Wno-format-overflow (255-byte uniontag + 3-byte prefix vs 256-byte buf,
unreachable schema lengths). Both the wrap checkout and the
packagefiles overlay patched identically.
Also: test_worker_launch_type_dispatch removes its fixed-path fake-xrun
/tmp fixture at exit (bug-924 /tmp hygiene; the mkdtemp shim can't see it).
Verified: fresh full recompile of every source = 0 warnings (gcc 13.3,
-Wall asan profile); meson test 134/134 green; no /tmp/xce_* residue.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Deduped census insight: the earlier "730 warnings" figure counted per-TU
echoes of the same source line (white-box textual inclusion recompiles
scheduler.c/limit.c/etc. dozens of times, each copy re-reporting). The
distinct warning sites in our code were 24 unused-variable + 11
unused-function + 2 Wcomment. This commit removes all of them; a clean
full rebuild now reports zero warnings outside the format-truncation
family (125 sites, deferred to its own pass) and vendored capnpc-c.
Notable non-mechanical fixes (the rest is dead-code deletion):
- schedobj.c make_sched_obj() silently swallowed its sub-constructor's
rc (ENOMEM/-EINVAL) and returned 1 unconditionally -- a half-built
schedobj with a NULL union payload could escape to callers. It now
propagates the failure, and make_sched_job() frees and returns NULL
on it instead of continuing with a broken schedobj.
- scheduler.c make_allocs(): vget() rc was set-but-unused; the r==NULL
skip below is the intended handling, so the call is now (void)'ed with
a comment saying so.
- commit.c highest_appended_lsn(): kept but scoped #ifdef XCE_TESTING --
it backs the never-beyond-the-log assert; it had no production caller.
- limit.c get_vres_value(): deleted -- its only call site sits inside a
#if 0 block and its own body was #if 0'ed to a constant.
- group.c update_group_dirty(): deleted recursive dead subtree, no
callers outside itself.
- test_sim_reconcile_unaccounted.c: test_reconcile_respects_host_
capacity_under_contention() was never registered in RUN_TEST; removed
with an NB pointing at the registered bug-1836 coverage.
Verified: clean full rebuild = 0 unused/Wcomment/implicit warnings in
our code (89 unique sites remain, all -Wformat-truncation or vendored);
controller_unit_test 1342/1342, xcelib_test 758/758, 126 scheduler
suites and 18 worker suites all green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Warning fixes (from a 730-warning full-build census; these were the
semantically risky ones):
- test_repl_majority.c:272: remove a broken "field-existence" loop whose
condition (i < ep != NULL) parsed wrong (-Wparentheses); the direct
pending_ack_count write it guarded already exists and still fails the
build if the field disappears.
- ha.c:8593: ha_rdhup_election_hook now takes const xce_candidate_t**
so RDHUP election can hand over candidates straight from the const
peers_cfg without stripping const. All 5 test-side hook impls only
(void)peers -- signature updated in ha.h + 3 test files.
- ha.c:726/872/10804 endpoint_connect: constify the host parameter of
endpoint_connect/socket_connect (netio.h/netio.c). The functions only
pass host to gethostbyname(); peer->hostname etc. can now flow in
without discarding const. All callers are read-only and unaffected.
Debug profile (src/meson.build): drop -gdwarf-4 -g3
-fvar-tracking(-assignments) in favor of -ggdb -g1. Debug objects were
~16x larger with full DWARF (scheduler.c: 1.3MB vs 0.08MB), pushing the
build tree to 3.9G; at -g1 it is 850M and a clean build drops from
6m43s to 4m36s. -g1 keeps file/line and backtraces; gdb variable/
macro inspection needs the asan profile. -fno-omit-frame-pointer kept.
Verified: controller_unit_test 1342/1342, xcelib_test 758/758, all 126
scheduler suites + 18 worker suites green; modified sources compile
with zero warnings.
Co-Authored-By: Claude Code <noreply@anthropic.com>
A committed job-state fact (WAL_JOB_UPDATE/FAIL/START/...) whose jobid no
longer exists in the jobmgr used to return -ENOENT from
apply_committed_entry(). Both of its callers (replay_wal during boot,
persist_apply_pump_run live) halt-and-fence on rc < 0, engaging the Live
Apply Resource Fence. A finished job is purged from the live table once
finished_retain_s expires, but its committed WAL entries remain; a boot
replay walks the full log and re-hits a late post-finish UPDATE for the
purged jobid on EVERY restart. No in-memory state change can ever make a
purged job re-appear, so the fence can never self-heal: the controller
logs "promotion FENCED -- durable_floor < cluster HWM=0" every 30s
forever, rejects all component joins ("standby has no primary yet"), and
cannot serve. Observed live on the t1 rollout (2026-10-21): the old
primary had logged the same entry as "duplicate UPDATE to state 0x4 --
no-op" while the job was still in memory; the first restart after the
job's purge tripped the fence. Pre-existing since the July 2026
apply_committed_entry dispatch table (2fa955b9/387ea35c) -- exposed by
any controller restart, not a de-nix regression.
Fix: all three case groups (shared START/FINISH/STOP/RESUME/RUSAGE/CKPT/
PREASON/DISPATCH group, WAL_JOB_FAIL, WAL_JOB_UPDATE) now return 0 for a
missing job, mirroring apply_wal_entry()'s existing behavior on the same
seam (restore_job_state()'s rc discarded; the FAIL branch guards on
j != NULL) and this function's own WAL_QUEUE_CTRL benign-skip precedent.
Genuine construction/ENOMEM failures keep their fence semantics.
Also:
- test_apply_committed_entry: three new tests covering the skip for all
three groups (incl. the POSTDONE-mark variant), asserting rc == 0 and
no hook fires.
- lib/meson.build: ASAN_OPTIONS=detect_leaks=0 on the capnp generation
command. capnpc-c deliberately never frees its compile-time structures
(one-shot tool); under the de-nix asan profile the wrap-built generator
carries -fsanitize=address and LSAN's end-of-process check failed the
generation step. The legacy nix build never propagated -fsanitize to
the subproject. Same exemption convention as every other one-shot/
never-torn-down boundary in this repo.
- xce-package.sh: ALWAYS-bundle name list (libcyaml/libyaml/libtirpc/
libcgroup/libhwloc/libedit) alongside the ldd-location rule -- libcyaml
and libcgroup proved missing on the t2/t3 Ubuntu 24.04 runtime hosts
even though the build host resolves them from /usr/lib.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The build no longer self-wraps into `nix develop`: src/tools/xce-build.sh now
runs meson+ninja directly on system packages, with a dependency preflight
that fails early naming the missing tool/pkg-config package (DEPS.md).
Dependencies (see DEPS.md for the full tables):
- 11 C libraries from apt (libuv, openssl, libedit, sqlite3, hwloc, libcgroup,
libtirpc, libcyaml, libyaml, zlib, libsystemd) with version floors in
meson.build mirroring the Ubuntu 24.04 baseline.
- sd-bus: default libsystemd, -Duse_elogind=true opt-back for nix-era hosts.
- 6 libraries/tools unavailable in apt become committed meson wrap
subprojects under src/subprojects/ (source closure, ~500 files): the
upstream trees plus packagefiles/ overlays and diff_files patches
registering dependency/program overrides the nix .pc files used to
provide (incl. the '<name>-raw' variable deps the Go CGO path reads).
Fresh-clone reproducibility verified: wiped trees + `meson subprojects
download` reproduce the committed state exactly (fixed librtti.wrap's
dangling patch_directory and c-capnproto's empty packagefiles overlay).
- xceadmin: self-contained venv built from requirements-build.txt (pinned,
offline-seedable via PIP_FIND_LINKS) instead of --system-site-packages.
Build/packaging:
- Makefile facade over xce-build.sh; scripts/ci-ubuntu.sh drives a bare
Ubuntu 24.04 container through deps + build + tests.
- src/tools/xce-package.sh: release-tree tarball packager. Collects
non-base DT_NEEDED libs from the BUILD-tree binaries (staged ldd lies:
the dangling build RPATH makes ldd drop the very lines the naive filter
needs), sets $ORIGIN RPATHs, rebases the xceadmin venv at install time
(kernel does not parse quotes in shebangs), ships systemd unit samples +
install.sh. Local smoke test: all 8 binaries resolve, xceadmin works from
both bin/ wrapper and venv script.
Test fixes found on the way (all environmental, not build bugs):
- harness_test.c setUp(): pin XCE_CONTROLLER_PEERS to a bound-then-closed
loopback port so the CLI-error tests assert the no-controller failure
path even on a dev box running a live cluster on localhost:9990.
- test_test_cli.c: setpgid(0,0) in the spawned-xce_test child. libedit's
sig_handler re-raises received signals via kill(0, sig) -- process GROUP
wide (strace -k evidence) -- and the child previously shared the
harness's pgrp, so an interrupted SSL handshake killed the whole suite
(exit 130).
Also fixed while re-baselining on the de-nix toolchain (pre-existing, now
deterministic): raft_init zeroes pre-vote tally state (pre_vote_in_progress/
pre_votes_granted/pre_vote_replies were left at stack garbage);
push_config_to_standbys clears stale config acks before fan-out
(repl_reset_config_acks) -- SHA-256-derived tags are not monotonic, so a
previous push's ack numerically above the new tag fabricated a barrier
quorum; capnpc-c's generated encode_* memsets the destination first
(skipped Data fields fed garbage into capn_setp).
Validation: asan profile 161/161 meson tests green (one known p2_integration
-110 timing flake on first run, green on re-run); debug + release full
builds OK; tarball smoke-tested on the build host. Cluster rollout on
t1/t2/t3 follows.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Merge origin/scheduler-framework into merge-sf-20260903 (97 commits on
ours: bug-1490/1494/1498/1509/1513/1521/1522 fixes, PEND-reason P1,
scheduler-agent Phase 2, bug-924 owner-pid shim).
Bookkeeping excluded per policy: .kiro/, .wolf/, CHANGES.md, reports/
(92 files dropped); .claude/ + CLAUDE.md keep ours.
Key theirs-side changes absorbed:
- bug-1938/1960: QUEUE_STAT_CLOSED no longer gates dispatch (closed
queue refuses new submissions only; pending jobs still run) --
supersedes our QUEUE_CLOSED preason on the dispatch path;
sched_queue_status_reason drops that branch
- bug-1846: config push now stashes and applies between scheduling
rounds (sched_stash_config/sched_apply_pending_config), applied at
the same main-thread round boundary that preserves bug-1513
serialization
- bug-1760: sched_check_pending_rejoin exposed +
xcecli_is_connection_broken public predicate
- bug-1936: execuser/exechome/execcwd/exdata carried on submission
(superset of our bug-1500 execuser fallback; our job-typed
resolveExecUser removed as dead code)
- bug-1945~1962: apiserver auth hardening (HS256 pinning, default-
secret rejection, logout revocation, bcrypt, RBAC) -- our API-key
branch folded into controllers.ParseTokenClaims so both coexist
- bug-1736: rawRunUsage job run-usage fields in the Go model
- xce_test relocated src/controller/testsuite/test.c ->
src/harness/test.c; controller suite split into
controller_test.c + fixtures; their clitest fixtures spawn xce_test
mid-suite under `meson test -j`
Ours preserved across the merge:
- PEND-reason P1 (codes 1-9, preason.h, job_preason machinery) --
gate re-ported into their relocated allocator (sched_election.c)
after plugin_job_election, with attach on the allocator thread
only (bug-1521 race discipline)
- PEND-reason P0 main_reason semantics RECONCILED to theirs: the
merge initially re-carried our first-detail-add-promotes-main_reason
fallback in add_job_preason (4e2ac86), but their whole trace/verify
stack treats "detail-only => main_reason 0" as a contract --
sched_trace_capture counts main_reason as its own occurrence
("Expected 2 Was 3" in test_trace_capture), the outcome renderer
prints "main reason: none" for detail-only PENDs (test_repl_trace_e2e),
and the committed .replay/.sim_verify baselines encode main_reason=0
(test_repl_trace_baseline byte compare, test_trace_equivalence
off-vs-traced). The fallback is dropped from the shared lib;
the user-visible "overall reason" survives as an API-layer view:
rawJobToModel (convert_pure.go) promotes the first detail row to
main_reason_name when main_reason is 0 (main_reason VALUE marshals
unchanged). add_job_preason keeps only the NULL-reasons guard
- bug-1498: sched_job_is_ready rejects non-PEND/suspended jobs
- bug-1490/1781: xce_jobctrl_t.action as capnp union discriminant
(their jobCtrlUnionDiscriminant, same semantics)
- xcecli typed PendingReason field (53c00834), bug-1522 reconnect
resend, xcecli_force_reconnect + their is_connection_broken both
exported in client.h/libclient
- t1/t2/t3 test-node hostconfig in src/configs/hosts.yaml (bug-1494),
/xce/v1/events + /xce/v1/apikeys whitelist routes (agent Phase 2)
- lib ASAN_OPTIONS=detect_leaks=0 test env + their oom-test block
- drop-audit: our bug-1504 fix covered by their bug-1680 (no action)
bug-924 follow-up (replaces the 67f3f6ad scratch-dir wipe): the
take-theirs audit's one "drop" was re-evaluated and REVERTED instead of
re-applied -- a global `find /tmp -name 'xce_*' -exec rm -rf` at suite
start/exit races the LIVE /tmp/xce_* fixtures of sibling suites under
`meson test -j` (harness_unit_test's clitest fixtures spawn xce_test as
a child mid-suite; the wipe destroyed the parent's fixtures ->
test_test_cli failures). Coverage instead:
- mkdtemp_clean shim gains mkdtemp_clean_mark/sweep_since
(testsuite_support/mkdtemp_clean.[ch])
- new case_scratch_sweep.[ch]: aggregate runners (controller_test.c,
harness_test.c -- hundreds of cases per process) call
case_scratch_sweep_begin/_end_ok from setUp/tearDown; a PASSED case's
dirs are swept immediately in tearDown (after engine reaping), FAILED
cases keep their dirs for post-mortem; XCE_MKDTEMP_KEEP_ALL=1
disables. One full controller_unit_test run was observed to leak 12G+
to ENOSPC without this, cascading into unrelated "Expected Non-NULL"
failures
- harness_unit_test_deps links the shim library as well
Test reconciliations on adopt:
- core-ready-gates: sched_queue_is_ready now gates only
INACT_ADMIN|INACT_WINDOW (bug-1938/1960); CLOSED queues elect --
covered end-to-end by test_queue_admin_control.c
- test_worker_resync: the 4 restart-replay RUN assertions predate the
bug-1509 B3 marking (8858afed) landing on the same lineage; replayed
RUN jobs now assert RUN|UNKWN|TRANSIENT (TWR_ASSERT_REPLAYED_RUN);
terminal DONE/EXIT assertions unchanged
- convert_pure.go: our job-typed resolveExecUser deleted (superseded
by their bug-1780/1936 two-arg version)
- test_preason: the 3 P0-promotion assertions reverted to the
detail-keeps-main-0 contract; new tests cover limit-code detail
rows and explicit-main-wins-over-later-details; apiserver
convert_test.go gains the detail-only name-promotion cases
Co-Authored-By: Claude Code <noreply@anthropic.com>
Cause: handle_jobsubmit() never looked at the target queue, so a
submission that bypassed the apiserver's check (libxce, harness
xce_test, or a REST submit without a queue name) was accepted into a
closed queue, an unknown queue, or the reserved lost_and_found queue.
Fix: jobsubmit_check_queue() reads the queue's in-memory state under the
qmgr/queue read locks. handle_jobsubmit() calls it after the field
checks and before the in-flight jobid registry and any WAL append, for
single jobs and job arrays. It replies -E_XCE_QUEUE_CLOSED (new code,
"queue is closed") for a closed queue and -E_XCE_BAD_QUEUE for an
unknown or reserved one. Apply/replay (restore_job) is unchanged, so
committed submit facts for a now-closed queue still apply.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: job_ctrl (job.c) sent kill/stop/resume only to allocs[0]->host.
The tasks on the other hosts are started through runtasks (by the head
worker for a parallel job, or by xrun as the MPI launcher's rsh agent)
and run under their own worker entry on each host; the head worker
signals only its local process group and forwards nothing, so they kept
running.
Fix: job_ctrl sends the request to every allocated host through
job_ctrl_to_host. The head decides the result as before; an unconnected
remote host is logged and skipped (its tasks belong to host failure
handling). Sends are non-blocking enqueues. Test:
testsuite/test_job_ctrl_hosts.c.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: job_ctrl (job.c) sent kill/stop/resume of a running job only when
the host's status was exactly 0, so any status bit (CLOSED,
CLOSED_BY_ADMIN, BUSY, LOCKED...) refused the request, and the refusal
returned -errno, i.e. whatever errno happened to hold (often 0 or
-ENOENT).
Fix: new job_ctrl_to_host sends the request whenever the worker is
connected (endpoint present, no XCE_WORKER_UNAVAIL); an unknown or
unconnected worker fails with -E_XCE_BAD_WORKER and an ERROR naming the
host. Nothing else changes for jobs on dead hosts. Test:
testsuite/test_job_ctrl_hosts.c.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The controller computed run usage (CPU time, memory, CPU peak and
efficiency, LSF bjobs -l definitions) but the capnp Job struct had no
field for it, so the job pushed to the apiserver never carried it, and
snapshots (the same struct) dropped it: a job restored from a checkpoint
lost its accumulated usage.
- controller.capnp: Job.runUsage @13 (JobRunusage); old entries decode
with no run usage.
- apiserver: map run_usage into JobInfo; max_mem, avg_mem, used_mem,
cpu_time, peak_cpu, peak_cpu_time, peak_cpu_duration, mem_efficiency,
avg_cpu_efficiency, peak_cpu_efficiency become queryable columns.
- xcecli: -o fields MAX_MEM, AVG_MEM, MEM_EFFICIENCY, CPU_PEAK,
CPU_PEAK_DURATION, CPU_EFFICIENCY, CPU_PEAK_EFFICIENCY (CPU_TIME now
renders too), and a MEMORY USAGE / CPU USAGE block in jobs list -l.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The configAck byte-delivery barrier waited 500 ms, below scheduling
delay on an oversubscribed host, so a push to a healthy majority was
rejected with -110 while its already-appended marker still committed.
CONF_PUSH_TIMEOUT_MS is now 2000 ms, the wait holds a commit-wait
admission slot (new commit_blocking_admit/release; a full budget fails
fast with not-ready), and the timeout reply no longer claims the change
was not committed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A restarted standby accepts the primary's connection during its startup
delay, before persist_recover runs. Catch-up entries written into the
un-recovered persist_t opened seg-...0001 with O_TRUNC and rewrote it,
and after recovery the stale fd kept filing new LSNs under that segment
name. The primary's filename-ordered streamer then skipped them, so a
standby needing that range never caught up and commits stalled.
persist_wal_write_replicated now returns -EBUSY until recovery completes
(the bug-1416 recovery_complete gate). ha_apply_wal_catchup_entry and
ha_handle_snapshot_chunk drop pre-recovery traffic without ACK or NACK.
Catch-up streams from min(NACK-derived join position, standby report),
and the streamer warns when a non-empty range yields no entries.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The fence re-check required !persist_recovery_truncated(), a boot-time
verdict nothing cleared, so a node that booted RECOVER_TRUNCATED stayed
fenced (fenced: true, non-promotable) until restart however far it
caught up. And the "progress is possible" branch did not re-arm, so the
fence was never re-evaluated.
persist_t gains a runtime recovery_truncation_healed latch, which
persist_recovery_truncated() honours (the raw boot verdict stays
available). The re-check marks the truncation healed once the durable
floor has advanced past the fence-time baseline (entries replicated from
the live primary) and reached the HWM, and the progress branch re-arms
with the same baseline and HWM.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A late PreVote grant after WON re-ran the WON path and started a second
real election that replaced elect_ctx, so the term-2 leader promoted no
vote endpoints. Rounds now carry an id and endpoints a round/kind; stale
or wrong-kind replies, timeouts and errors are dropped; raft ignores
replies with no round in progress. raft_cb_become_leader promotes only
with its own stamped election record (own term, majority), else steps
down and re-arms. raft_on_leader_lost keeps voted_for (one vote per
term) and the delayed election starts only as a follower in the lost
term. The failover stagger hands the PreVote start to a vtask so the
delaymgr stays trigger-only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
After an election where the vote connections are not promoted, both
standbys register later via on_standby_connected; the own-term no-op was
appended before any standby existed, and catch-up waited for heartbeat-
driven state reports (config push one 3 s cycle, WAL stream the next), so
the no-op committed ~6 s late and every client write in that window
failed -110.
ha_send_catchup_probe() sends the ordinary heartbeat request (same fields
as primary_heartbeat_fn; encode + enqueue only; primary only) at
registration and once after the standby's first accepted config-push ACK,
so the decision loop's next action follows at ~RTT. The decision loop,
its one-action-per-report pacing, commit/majority rules and the dispatch
gate are unchanged; WAL streaming still runs only on the vtask. The
one-shot arm is cleared on disconnect and on every become_leader /
become_follower.
Tests: test_late_joiner_catchup_kick.c (6). Controller suite 1285/0.
Live: catch-up starts 0-6 ms after a late registration (was up to ~6 s);
/xce-test-ha sequential 20 rounds, chaotic 6 rounds and sequential 8
rounds under 40-spinner load all PASS with zero job loss.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
update_job_cpu_stats() (misc.c), run after each usage report:
- each host's cpu_rate = CPU-seconds gained / wall seconds since its
previous report; the job's cores = the sum over hosts
- peak_cpu = highest job cores seen (peak_cpu_time = it rounded, the
wire field is an integer); peak_cpu_duration = the interval that set it
- avg_cpu_efficiency = cpu_time / elapsed / slots, peak_cpu_efficiency =
peak_cpu / slots, mem_efficiency = highest used_mem / rusage[mem]
request (job-wide total, per task by default), all in percent
aggregate_job_runusage() no longer treats cumulative cpu_time as a peak.
New fields host sample_time/cpu_rate and job peak_cpu are appended to
the capnp structs (old WAL entries still decode) and copied by
merge_job_runusage(), so a promoted standby keeps the rate baselines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
handle_jobsubmit now registers the submission first and checks the job
table second; a hit is answered -EEXIST and released through the atomic
reply-and-release step. For array submissions, array_identity_exists()
does one read-locked scan of the 32768 tagged element keys of the
array's block (element 0 included for pre-upgrade arrays), covering
both the element range and the array identity; unmarked submissions
keep jobid_range_exists. An existing X|3 now blocks a new X|5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
In-flight registry entries carry is_array and array_id; two array
entries with the same array identity overlap. inflight_reply_and_remove()
sends the reply and unlinks the entry under the registry lock (xce_reply
is non-blocking) and is used on every post-registration path, inside the
async one-shot reply guard; jobsubmit_async_ctx_release no longer touches
the registry, so a retry registered after the reply cannot be deleted by
the original's release. Test-only seams under XCE_TESTING make the
atomicity observable deterministically.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
handle_jobsubmit classifies (first_jobid, array_size) before the field
checks, the before-event hook and any registry access: a tagged array
starting at index 0 is answered with the new E_XCE_ARRAY_START0 and an
ERROR log asking for the client to be upgraded (7.1, wins per 7.9);
size 0, tags 2/3, untagged multi-element submissions and arrays whose
last element exceeds 32767 (checked without addition) get -EINVAL.
Nothing is stored for a rejected submission. This also closes a crash:
a size-0 submission used to reach payload building and overflow calloc.
Existing array tests move to tagged block|1 first IDs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The script required a pre-generated <repo>/ha/data (cluster.bin + TLS
certs) that ee8473bd removed, and called deploy-ha.sh with that dir as an
extra argument; deploy-ha.sh now generates both into the deploy dir and
takes <binary> <top-dir> <peers>. It failed at its first precondition in
every checkout.
Behind that, both committed-LSN boundaries still assumed bug-845's doubled
submits (1 + 2*N_JOBS): the commit wait never reached it and walcompare
compared an LSN past every member's WAL. Since that fix each job is one
WAL entry, so the boundary is 1 + N_JOBS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A promoted vote endpoint carries a peer index relative to the votes
array (which leaves out the primary); a late-joiner's heartbeat endpoint
carries the absolute index. repl_ack_decode_and_dispatch recorded state
reports under the raw index, so with ctrl-1 primary, ctrl-3's vote
endpoint (relative 1) and a rejoined ctrl-2's heartbeat endpoint
(absolute 1) shared standby_reported_*[1]. Each handler then acted on
the other's position over its own endpoint: the snapshot meant for the
empty, rejoining ctrl-2 went to the caught-up ctrl-3, and ctrl-2 never
caught up.
Resolve the absolute index with ha_repl_ack_abs_peer_idx (as the
WAL-ack branch already did) for the state report and for
ha_walack_on_ack, and size catchup_inflight by MAX_HA_CANDIDATES so
every absolute index fits.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its
term-1 election before its peers ran; connect failures count as "no"
votes (bug-1423), so it lost at once and became a follower with no
leader. ctrl-2/ctrl-3 hold no config and are voters only (by design),
so nothing retried until ctrl-1's 10 s standby liveness timeout: every
cold start had no primary for ~10 s.
- ha_note_leaderless() marks a config-holding node that becomes a
follower with no leader known; the liveness poll then retries the
election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to
heartbeat_timeout_ms. Primary contact or becoming leader clears it.
- ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity
then only holds the watchdog's arm time, and the failover gates took
it for a live primary.
Live cold start: primary 1.7 s after the first node started (was ~10 s).
bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a
standby-only counter a primary never advances, so every primary showed
"appliedLsn: 0". A primary now reports persist->last_applied_lsn. The
controller-status design doc's "already correct in every role" note is
corrected.
Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c
(RED: expected 4900, was 0); controller suite 1256/0. HA soaks:
sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero
job loss).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- bug-1872 (worker): "Failed to find any disk statistics in
/proc/diskstats" ERROR every 15 s on an NVMe-only host. The whole-disk
filter guessed from the name and skipped anything ending in a digit or
containing 'p' -- every nvmeXnY, mmcblkN, even sdp -- so io was never
reported. diskstats.h now counts a device only when
/sys/block/<name>/device exists (real disks; not partitions, loop, dm,
md, zram), and a host with no disk warns once.
- bug-1873 (worker): "Failed to get worker pid" WARN on job start. MainPID
reads 0 until systemd has forked the new transient unit; xcecd.c already
retries, so the miss is DEBUG and the final failure names the unit.
- bug-1874 (scheduler): deprecation WARN for the legacy xce_sched_fcfs on
every start, though builtin_fcfs is the picker. Registration still wraps
a job_elect_fn plugin but no longer warns; selecting one warns once.
- bug-1875 (apiserver): "Not running as root -- PAM authentication will not
work" with the default provider config_file; now only for provider pam.
- bug-1876 (controller): standbys logged "cluster_init_phase2 ... failed
(rc=-22)" WARN every 3 s while waiting for the primary's config push,
a state the function itself calls normal. It returns -EAGAIN for that
wait; callers log it at INFO, other failures still WARN.
- bug-1877 (controller): "WAL/snapshot stored unencrypted" warned without
looking. persist_path_encrypted() checks fscrypt (statx) and dm-crypt
(sysfs dm/uuid CRYPT-, through LVM slaves); the warning stays when the
storage is not shown to be encrypted (e.g. a plain /tmp).
Tests: worker diskstats_test (2 of 4 RED on the old filter with pc13v's
/proc/diskstats), controller test_phase2_cold_boot and
test_persist_encryption, scheduler picker tests, apiserver
TestRootWarning. Full ASAN build: all 113 suites pass (controller 319/0,
harness 141/0).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A worker reported job start, finish and failure with a fire-and-forget
OPCode_notify the controller never answers, so a report lost while the
worker stayed connected (the live incident: 6 of 7 finish reports in a
burst) left the job RUN forever; the rejoin resync (165fc158) only
recovers reports lost around a reconnect.
- controller.capnp: OPCode_notifyAcked -- the same NotifyData body,
answered. A separate opcode because replies are matched through the
client's single wbuf, not by sequence number: answering plain notifies
would feed stale statuses to the next synchronous request.
- controller (cluster.c): do_notify_acked() applies the fact like
do_notify, then waits on the commit barrier and replies 0 only once it
is majority-committed. A barrier failure (the fact is still appended and
retained) replies -E_XCE_NOT_READY, so the worker retries in place
instead of rotating away from a healthy primary (cf. bug-1850). Admitted
by the no-quorum fact-bearing gate like OPCode_notify.
- libxce: xce_notify_acked_req/xce_notify_acked (send, wait for the
status) and xcecli_notify_acked (XCECLI_FAILOVER_CALL retry/failover).
xcecli_notify_jobstart/jobfinish/jobfail use it; on -E_XCE_BAD_OPCODE
(a controller predating the opcode) they fall back to the plain
fire-and-forget notify, so workers and controllers can be upgraded in
either order.
- Resends are safe: START/FINISH are duplicate-guarded and a repeated FAIL
yields the same state.
- worker: logs a WARN when a report is still unacknowledged after the
retry budget (the rejoin resync remains the backstop). These calls run
on the worker daemon's IPC callback, so while the controller lacks
quorum other jobs' IPC messages wait (buffered, not lost) up to the
retry budget.
Tests: test_notify_acked_reply_only_after_commit and
test_notify_acked_withheld_replies_not_ready (no reply before the covering
ACK; -E_XCE_NOT_READY with the fact retained when ACKs are withheld).
Written with the opcode, so no separate RED run. Full /xce-build all:
111 suites pass; controller suite 1245/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Regression from 63fd3d2a (bug-1870), found by the --config-upgrade HA soak
(force-elect: "submit on {ctrl-2,ctrl-1} failed", ~640 standby NACKs;
bisected: 725301dd passes, 88c4310e fails).
load_snapshot() seeded current_lsn = checkpoint_lsn, but current_lsn is
the NEXT lsn to write and the snapshot covers 1..checkpoint_lsn. The
off-by-one was always there, hidden because every checkpoint also wrote a
WAL_CHECKPOINT marker ABOVE the checkpoint, so the WAL scan moved
current_lsn past it. With the marker gone, a standby whose WAL held
nothing above its config-marker checkpoint (LSN 11) recovered with
current_lsn=11: its raft layer counted entry 11 as held (ae_already_have,
not written), and persist_wal_write_replicated() then refused entry 12 as
a gap (ae_append_persist_fail) on every resend -- stuck at 11 forever.
Both of load_snapshot's commit points now set current_lsn =
checkpoint_lsn + 1, matching the apply-side catch-up path (lsn + 1).
New test_recover_from_snapshot_without_wal_resumes_after_it (RED before:
current_lsn 3, not 4). Two snapshot-entity tests asserted the old
current_lsn == checkpoint_lsn and now expect +1. Controller suite 1245/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
build_queues() only ran at boot, so a queue a live config upgrade added
had no runtime object in c->qmgr (and the scheduler/apiserver never heard
of it) until the controller restarted. The checkpoint-ordering half of
bug-1469 was fixed in 959b0cda; this is the remaining half.
- reconcile_config_queues(c, notify): creates every queue in the active
config that has no runtime object, leaves existing ones untouched,
returns the number created.
- config_activate_committed() calls it (with notify) after every
successful activation -- verified-marker and no-marker paths alike.
- persist_notify_queue_updated_no_wal(): fires HOOK_DETAIL_QUEUE_UPDATE
under the t_apply_seam_notifying guard, so the scheduler/apiserver are
told while no WAL_QUEUE_CTRL is appended -- config queues are
config-owned, built on every member from the committed config.
- build_queues() now delegates to it without notify, so boot and
activation are idempotent in either order.
New test_config_queue_reconcile.c (RED before: the added queue was never
created): creates the added queue, leaves the existing object untouched,
notifies once, and a second reconcile/build_queues adds nothing.
Controller suite passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Root cause of the "spurious duplicate reconnect" (angle a): zarray_del()
NULLs the slot and decrements count without compacting, so a live entry
can sit at any index < caps. Every standby_endpoints loop bounded by
`count` missed it. Live evidence (--config-upgrade-wal-trim soak,
2026-09-26): ctrl-2 won, promoted vote eps [ctrl-1 (dead), ctrl-3]; the
dead one's removal left [NULL, ctrl-3] with count 1, so the primary
- stopped heartbeating ctrl-3 (primary_heartbeat_fn's send loop),
- re-dialed it ("peer[2] not connected -- initiating outbound connect"
9s later, for a peer that never dropped) and registered a second,
"late-joiner" channel that burned a MAX_STANDBYS repl slot,
- could not evict the stale channel (on_standby_connected's dedup scan),
- and skipped it in config pushes and the ctrlhello broadcast, where a
hole below count also handed endpoint_ref() a NULL.
The re-dial landed in the freed slot 0, which is what made ctrl-3 visible
again -- the "spurious" reconnect was masking the lost heartbeats.
All index scans of standby_endpoints (ha.c heartbeat, sweep, dedup, two
removals; cluster.c removal and ctrlhello; config_repl.c push collection)
now run to `caps` and skip NULL slots; the push collects exactly the
non-NULL entries. `count` stays where it means "how many connected".
New test_reconnect_dedups_channel_behind_deleted_slot (RED before: the
stale channel behind the hole was never evicted). Controller suite 1241/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A winning RequestVoteReply is dispatched through peer_ctrl_classify, which
returned VTASK_LIVENESS for all peer-control traffic -- the never-blocks,
never-admission-gated class. The reply runs raft_cb_become_leader, whose
tail persist_apply_pump_run_sync() does bounded blocking WAL pread()s (it
cannot be made async without reopening the bug-1043-family job loss on a
freshly promoted primary). The fdatasync half of bug-1433 was removed in
40f510fb; this pread was the tracked residual.
peer_ctrl_classify now returns VTASK_LIVE_BLOCK while the node is
CTRL_ELECTING -- its peer-control traffic is then vote/PreVote replies,
election work like inbound RequestVote, which already uses LIVE_BLOCK.
As PRIMARY/STANDBY the traffic is WAL/config ACKs and stays VTASK_LIVENESS,
so a quorum of acks is never admission-gated (R4.5); a candidate has no
commit to advance.
New test_peer_ctrl_classify_vote_replies_while_electing (RED before).
Controller suite 1240/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
"ERROR incorrect worker idx -1" on every node after a worker-join commit
timeout (during a PreVote/election) was the visible end of a real
dangling pointer. init_worker/rejoin_worker bind the endpoint to the
worker (apply_join_state: w->ep = ep, UNAVAIL cleared) before the commit
barrier; on barrier failure handle_worker_join returned without setting
priv->u.idx, so when that connection closed handle_endpoint_error saw
idx -1, logged the ERROR and never ran handle_worker_leave -- the worker
kept w->ep pointing at the freed endpoint, available for dispatch, until
it rejoined.
- handle_worker_join: on barrier failure, clear w->ep (if still this
endpoint) and mark the worker unavailable, as if the join never
happened; the worker retries its join.
- handle_endpoint_error: an idx -1 worker endpoint is now the legitimate
"closed before its join completed" case -- WARN, nothing to clean up.
test_worker_join_withheld_acks_replies_error also asserts the worker is
unbound and unavailable after the -ETIMEDOUT reply (RED before).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
f88e4ef4 moved ha_demotion_convergence_fn's WAL truncate + self-restart
and ha_catchup_continuation_fn's WAL streaming off the delaymgr thread
(which drives heartbeats and election timeouts cluster-wide) onto the
slow lane, but left both flags OFF: flipping them ON hung pre-existing
unit fixtures that demote a primary or trigger catch-up without ever
starting c->slowlane -- the work was queued in a lane with no workers and
never ran.
- slowlane_is_started(): true while the lane has live workers.
- Both dispatchers run inline when the flag is OFF *or* the lane was never
started. Production always starts the lane, so it always offloads; the
lightweight fixtures keep their old inline behaviour, and neither
one-shot action can be silently dropped.
- g_drive_async_demotion_offload / g_drive_async_catchup_offload now
default to 1. The offload tests save and restore the previous value
instead of forcing 0.
New test_catchup_offload_without_started_lane_runs_inline (RED before: the
catch-up body never ran, the stream gate was never reached). Controller
suite 1239/0 with both offloads ON by default.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A job the controller requeued to PEND while the scheduler was disconnected
stayed RUN (with its old allocation) in the scheduler forever: the
bug-1812 guard in core_events.c's JOBS_PUSH merge refused any non-RUN,
non-terminal resync claim over a job holding a real allocation, because
it could not tell a genuine requeue from a push built before the
scheduler's own fresh dispatch.
Design (per review): the scheduler persists nothing, so on rejoin the
controller's state wins, and the race the guard protected against is
closed where the truth lives:
- scheduler JOBS_PUSH: the bug-1812 refusal is removed; the controller's
state replaces the scheduler's and its allocation is released. The
terminal-is-permanent and phantom-RUN (RUN without allocation data)
guards stay.
- controller dispatch_a_job_(): refuses a job that is not PEND (RUN or
finished) before any worker is touched -- previously nothing checked,
so a stale second dispatch would start a running job again (or, through
a retryable failure, requeue it). The job is left as is, no WAL fact,
and its true state is re-sent (HOOK_DETAIL_JOB_UPDATE). A PEND job with
a leftover allocation stays dispatchable.
- scheduler JOB_UPDATE: a RUN report carrying the controller's committed
allocation is adopted, so a corrected job is never RUN with nothing
charged.
Tests (all RED before): test_jobs_push_duplicate_jobid_controller_pend_
wins_over_local_run (replaces the bug-1812 test that pinned the
refusal), test_job_update_run_adopts_controller_allocation,
test_dispatch_of_non_pending_job_is_refused. Two tests that seeded a RUN
job to reach retry branches (a precondition their own headers called
unreachable) now use the real paths: the counter-reset test requeues via
record_job_reconcile_pending_fact() first; the disconnected-worker RUN
test asserts the refusal contract.
Controller suite 1238/0, scheduler 87/87. HA soaks: chaotic 6 rounds PASS
(0 divergence/loss, 6 worker-resync jobs), sequential 6 rounds PASS
(SIMVERIFY 7 files, 0 mismatches). The race did not occur in either run
(0 refusals logged), so the new paths are proven by the unit tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
do_update_resvals() fired the AFTER_EVENT HOOK_DETAIL_RESOURCE_UPDATE with
the raw client xce_resval_list_t. update_resval_list() rejects unknown
resource names (-ENOENT) but only logged it, so both listeners -- the
scheduler push and the apiserver store (apiserver_resource_hook_fun ->
handler.go) -- received names the controller had rejected, and the
apiserver stored them.
- update_resval_list() takes an optional `applied` view (caller-sized,
borrowed pointers) and fills it with just the accepted items.
- do_update_resvals() hands that view to the AFTER_EVENT hook, and skips
it when nothing was applied. The request still owns every item.
New test_update_resval_list_reports_only_applied_items (RED before: both
items reported). Controller suite 1237/0. Live /xce-test-resources with
mem=5 totally-fake-resource-xyz=1: mem MATCH at controller, scheduler and
apiserver; the fake name is rejected at injection and appears 0 times in
scheduler.log / apiserver.log (verdict FAIL by design, for the rejected
name).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-448: job_ctrl() took a reference on the job but only one error path
released it; every successful call leaked one (refcount 1 -> 3 after two
calls). It is now released on every exit path, including the key_find
rc<0 one.
bug-1626: job_ctrl()'s not-started branch changes the job (mark_job_update
/ mark_job_fail -> apply_job_*_transition, which require RTTI_LOCK_WRITE)
while holding only RTTI_LOCK_READ, so it could run concurrently with
readers. It now takes the write lock. (dispatch_a_job_ already did.)
bug-1631: dispatch_a_job_()'s four retryable-failure branches never reset
state/allocs, correct for a first dispatch (still PEND) but not for a job
that had already been dispatched once: it stayed RUN holding its old
allocation. Each branch now calls record_job_reconcile_pending_fact() --
a no-op without a WAL fact for a PEND job; for a RUN job it returns it to
PEND, releases allocs and appends a durable WAL_JOB_UPDATE -- and notifies
after unlocking when it applied, the same pattern as the resync reconcile
path (cfe66dde). Not reachable until something re-dispatches RUN jobs
(e.g. preemption), fixed now so it is correct when that lands.
- job_get_refcount() (jobj.c) read accessor for the leak test.
- test_apply_pump.c: test_job_ctrl_releases_its_job_reference and
test_job_ctrl_mutates_not_started_job_under_write_lock (both RED before).
- test_dispatch_fail_retry.c now asserts PEND + allocs released after a
post-success retryable failure (RED before).
Controller suite 1236/0; all 111 meson suites pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
persist_checkpoint() and persist_checkpoint_marker_triggered() wrote a
local-only WAL_CHECKPOINT marker that consumed an LSN of the replicated log
(current_lsn++) but was never replicated. A checkpoint taken by one member
only -- the primary's periodic max-wal-records checkpoint, or the one before
a snapshot send -- left a hole no standby had: the leader's next entry
carried prev_log_index = the marker's LSN, logOk failed on every standby
forever, commits timed out and writes stalled ("could not submit 110 jobs"
in the --force-snapshot soak; reproduced on the pre-bug-1867 baseline). Even
the config-marker checkpoint, taken on every member, stamped the marker with
each member's own current_lsn at apply time, which need not agree.
Nothing reads the marker except to skip it, so checkpoints simply stop
writing it. The recovery, streamer and bug-1475 log_len skip paths stay, for
WALs written by older binaries; persist_test_write_legacy_checkpoint_marker()
(XCE_TESTING only) reproduces such a WAL for the tests that cover them.
- new test_bug1870_leader_only_checkpoint_leaves_no_log_hole (RED before):
a leader-only checkpoint takes no LSN and a standby that never
checkpointed catches up past it via the real streamer/receive path.
- test_checkpoint_writes_marker_to_wal inverted to ..._writes_no_marker_...
- bug-1475 x2 and the apply-correctness trailing-marker test now build
their legacy marker explicitly.
Controller suite 1234/0. HA soaks: --force-snapshot PASS (113 jobs, real
snapshot install -- also live-verifies bug-1867), --config-upgrade-wal-trim
PASS, --sequential 6 rounds PASS (SIMVERIFY 7 files, 0 mismatches).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The capnp Job struct -- the checkpoint/entity-snapshot form of xce_job_t --
had no exit-info field, so capnp_encode_job() never wrote job.exit_info
and a job restored from a snapshot (instead of WAL replay) came back with
exit_info NULL, for DONE/EXIT and failed jobs alike. bug-1689/bug-1754
made the WAL facts carry the code; anything older than the last
checkpoint's WAL trim still lost it.
- controller.capnp: struct JobExitInfo $C.mapname("job_exitinfo_t")
{ exitcode }, and Job.exitInfo @12 $C.mapname("exit_info"). The
generator now emits encode/decode/free for it inside encode_Job /
decode_Job / free_Job; restore_job()'s create_job() already takes
exit_info via MOVE_PTR. Null stays null (no fabricated exit 0). A new
ordinal, so older snapshots/recordings decode with exit_info NULL.
New test_snapent_checkpoint_exit_info_roundtrip (RED before): exit codes
0 and 137 and a NULL survive both the raw checkpoint frames and a real
persist_load_snapshot restore. Full /xce-build all (asan): 111 meson
suites, 0 failures; all Go packages pass.
Not live-verified: the --force-snapshot HA soak cannot reach its snapshot
step because of a pre-existing checkpoint-marker replication stall,
reproduced on the baseline without this change (bug-1870).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The FAIL half of bug-1747: apply_job_fail_transition() stamps
finish_time = time(NULL) at APPLY time, and WAL_JOB_FAIL carried no time,
so a failed job's finish_time changed on every restart replay / standby
catch-up, and a worker-reported failure found on rejoin could never keep
the worker's own time.
- WAL_JOB_FAIL's trailer grows to int32_t code + uint64_t finish_time
(seconds). Decoded by length: 16 B (pre-bug-1754), 20 B (code only,
bug-1754) and 28 B (code + time) all apply as before for what they carry.
- record_job_fail_fact_at(c, job, failinfo, finish_time_ms), the fail-side
sibling of record_job_finish_fact_at(): 0 = receipt time. on_job_fail()
uses receipt time; resync's finishedFail branch passes the worker-
reported finish_time_ms (Req 4.2 now holds for failures too).
- Both apply seams adopt the wire time after the transition.
4 new tests in test_wal_job_fail_fact.c (3 RED before the fix) and the
resync round-trip now asserts the worker-reported failure time (RED
before). Controller suite 1232/0. Factstream HA soak (2 rounds, 60 jobs)
PASS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
