Cause: handle_jobsubmit() never looked at the target queue, so a
submission that bypassed the apiserver's check (libxce, harness
xce_test, or a REST submit without a queue name) was accepted into a
closed queue, an unknown queue, or the reserved lost_and_found queue.
Fix: jobsubmit_check_queue() reads the queue's in-memory state under the
qmgr/queue read locks. handle_jobsubmit() calls it after the field
checks and before the in-flight jobid registry and any WAL append, for
single jobs and job arrays. It replies -E_XCE_QUEUE_CLOSED (new code,
"queue is closed") for a closed queue and -E_XCE_BAD_QUEUE for an
unknown or reserved one. Apply/replay (restore_job) is unchanged, so
committed submit facts for a now-closed queue still apply.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A plain kill sent one SIGTERM and never escalated, and a plain stop sent
one uncatchable SIGSTOP. Following LSF's TERMINATE and SUSPEND actions, a
plain kill now sends SIGINT, then SIGTERM, then SIGKILL, and a plain stop
sends SIGTSTP (the job can catch it, prepare and stop itself), then SIGSTOP
if it has not stopped; each next signal follows after the grace period and
only while the job is alive (or not yet stopped). A resume cancels a
pending stop; the job's exit ends any sequence. An explicit signal is
delivered once; a force kill is SIGKILL at once.
- JobSignal gains grace (capnp @2, xce_signal_t.grace): > 0 asks for the
sequence, in seconds between signals.
- apiserver: job.terminate_interval (default 10 s, LSF's
JOB_TERMINATE_INTERVAL); jobCtrlParams sends kill = SIGKILL and stop =
SIGSTOP with that grace when no signal is given; the handler no longer
pre-fills a default signal.
- worker: grace travels through the job-control task, the kept-signal table
(pending_signal_*_ex) and the IPC message; the job process runs the
sequence (job_signal_seq_*, uv timer) and checks stop state in /proc.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The session-level unsatisfiable-resreq cache stored only the exhausted
attempt's main reason (an int). The allocators record shortfalls as
per-host/per-resource entries and leave main_reason at 0, so every cache
hit in the live scheduler pended its job with an empty reason. The
simulator registered no cache at all (its table lived on xce_scheduler_t),
so it searched every identical pending job again and its traces differed
from production.
- The cache table moves to xce_sched_core_t; cache_lookup/insert/clear and
the hook adapters take the core, with no cast to a driver struct.
sched_init() builds it, cache_clear() rebuilds it per session,
xce_sched_core_fini() destroys it.
- A cache entry is a deep copy of the attempt's full xce_job_preason_t
plus the job that proved the request unsatisfiable (core->alloc_jobid,
set by sched_allocator()). A hit moves a copy into ctx->reason; a miss
leaves ctx->reason untouched.
- main.c passes the same three adapters to the simulator through new
sched_sim_args_t fields; sched_sim_driver_start() sets them on the sim
core (plain field writes: white-box tests include sim.c alone).
- Decision trace: a hit renders "no plan generated: skipped, same request
as <job> proved unsatisfiable earlier this session" (new
XCE_TRACE_NOPLAN_KNOWN_UNSAT via sched_trace_known_unsat()); the outcome
shows the restored reasons.
- sim_verify keeps its main-only PEND compare (old recordings carry
main-only hits); comments and the scheduler-sim-live-allocation design
note updated.
Tests: full-reason assertions in the cache dispatch tests (RED before:
"a cache hit must restore the original attempt's FULL pend reason"),
test_alloc_cache.c ported to the new API (deep copy, clear frees, -EEXIST
frees), a sim-registered clear test, a renderer test, and a sim e2e
scenario with two identical unsatisfiable jobs (204, 205) where 205 is
skipped with 204's full reasons (RED before: no skip line).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scheduler appended one NO_RESOURCE/MATCH_FAIL entry per host per
placement plan, so a pending parallel job carried plans x hosts copies
(~93k entries in the 1000-host bounds test). libxce gains
add_job_preason_once() with a per-context hash index keyed on
(host, resource, code) that keeps first-occurrence order; the
scheduler's three builders use it. add_job_preason() is unchanged for
other callers. The parjob equivalence test now expects each short host
once; the bounds test runs in ~2 s instead of ~19 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A reconnect made from inside a task of the endpoint it re-establishes (a
request handler that failed over) waited out the full 2 s quiesce bound:
its own reference cannot drain, and the old connection's queued tasks are
serialised behind it -- and those then ran after the owner was re-armed,
reaching the new session.
Each VTASK_IO task now records the endpoint's reconnect_gen when queued and
vtask_entry drops a task whose generation is stale; the vtask manager
tracks the endpoint whose task the current thread runs
(vtask_current_io_ep), and xce_ctx_quiesce_ep skips the wait in that case.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
INVOKE_MSG_HANDLER stored NULL into ep->fsm_in.pkt after every msg
handler. Handlers that take the packet clear the slot themselves
(vtask_handle_msg, peer_ctrl_msg_handler); one that refuses it
(-EINVAL / -ENOMEM) leaves it there, and the NULL store dropped the
only reference. The slot is now left as the handler leaves it: a
refused packet is reused for the next message or freed by endpoint_fini.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The fence re-check required !persist_recovery_truncated(), a boot-time
verdict nothing cleared, so a node that booted RECOVER_TRUNCATED stayed
fenced (fenced: true, non-promotable) until restart however far it
caught up. And the "progress is possible" branch did not re-arm, so the
fence was never re-evaluated.
persist_t gains a runtime recovery_truncation_healed latch, which
persist_recovery_truncated() honours (the raw boot verdict stays
available). The re-check marks the truncation healed once the durable
floor has advanced past the fence-time baseline (entries replicated from
the live primary) and reached the HWM, and the progress branch re-arms
with the same baseline and HWM.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
vtask_handle_msg published the VTASK_IO task while ep->fsm_in.pkt still
pointed at the packet; INVOKE_MSG_HANDLER clears the slot only after the
handler returns. A vtask could decode and free the packet (xce_decode)
and wake the client, whose failover reconnect then ran endpoint_fini and
freed the dangling slot again: ASAN heap-use-after-free in free_packet,
seen in the worker and scheduler during failovers under heavy CPU load.
vtask_handle_msg now clears ep->fsm_in.pkt before vtask_enq and marks
the task the packet's sole owner (owns_io_pkt), so never-run tasks free
it at vtaskmgr_fini and an enqueue failure frees it instead of leaking;
xcectx vtask_entry frees it on its early -EINVAL exits. Same take-and-
clear discipline the controller's peer-control path already used.
Tests: test_vtask_pkt_handoff.c (3), RED reproduces the soak's ASAN stack.
libxce 733/0; xce-build all green; loaded /xce-test-ha soak (8 rounds,
40 spinners) PASS with no ASAN reports (it crashed 2/2 before).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- bug-1912: xce_parse_peers() ignores whitespace around entries and skips
blank ones, matching xceadmin's resolve_controllers(), so "a=h:1, b=h:2"
names the second candidate "b" (5 libxce tests + regression case).
- bug-1911: xcecd frees the -l log dir before the -h / --version returns
(regression case: no LeakSanitizer report, exit 0).
- bug-1910: both version scripts build the apiserver with -mod=vendor
(as the meson build does) instead of -mod=readonly, so they no longer
need a Go module cache.
- bug-1909: test-version-startup-consistency.sh accepts exactly the
dirty-content-fingerprint pairing ('<sha>-<8 hex>' in --version with
'<sha>-dirty' in the startup log), with self-test cases (e)/(f).
Both version scripts now exit 0 on a dirty tree with XCE_CONTROLLER_PEERS
set; regression script 35/35; xce-build all: C 127/0, scheduler 16/0,
xceadmin 364 passed, no ASAN reports.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
REGEX_JOBID, REGEX_JOBARRAY_ID and N_MAX_FIELDS_JOBID served the old
regex-based parse_jobid(), replaced by parse_jobsel() in
job-array-one-based-index task 1.2; nothing references them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
update_job_cpu_stats() (misc.c), run after each usage report:
- each host's cpu_rate = CPU-seconds gained / wall seconds since its
previous report; the job's cores = the sum over hosts
- peak_cpu = highest job cores seen (peak_cpu_time = it rounded, the
wire field is an integer); peak_cpu_duration = the interval that set it
- avg_cpu_efficiency = cpu_time / elapsed / slots, peak_cpu_efficiency =
peak_cpu / slots, mem_efficiency = highest used_mem / rusage[mem]
request (job-wide total, per task by default), all in percent
aggregate_job_runusage() no longer treats cumulative cpu_time as a peak.
New fields host sample_time/cpu_rate and job peak_cpu are appended to
the capnp structs (old WAL entries still decode) and copied by
merge_job_runusage(), so a promoted standby keeps the rate baselines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
idgen() now accepts only a size of 1 (anything else: -EINVAL, context
unchanged) and refuses an exhausted 20-bit sequence with -EOVERFLOW
instead of spilling into the timestamp (also after idgen_array has
taken the last block). Arrays come only from idgen_array; every caller
of idgen/GenerateJobID passes 1. The old multi-element test loop moves
to idgen_array with 1-based assertions.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
handle_jobsubmit classifies (first_jobid, array_size) before the field
checks, the before-event hook and any registry access: a tagged array
starting at index 0 is answered with the new E_XCE_ARRAY_START0 and an
ERROR log asking for the client to be upgraded (7.1, wins per 7.9);
size 0, tags 2/3, untagged multi-element submissions and arrays whose
last element exceeds 32767 (checked without addition) get -EINVAL.
Nothing is stored for a rejected submission. This also closes a crash:
a size-0 submission used to reach payload building and overflow calloc.
Existing array tests move to tagged block|1 first IDs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
idgen_array() allocates the next whole 32768 block and returns its
first element (array tag, index 1), refusing n outside 1..32767 and a
block past the 20-bit sequence (-EOVERFLOW, context unchanged). The
reset-on-new-second step is shared with idgen() via idgen_tick_locked();
idgen() itself is unchanged until task 9.1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
parse_jobsel() parses c-a-ts-seq with an optional [k] or [a-b] under a
fixed grammar and check order (syntax, field widths, array-identity
alignment, element bound, reversed range), saturating digits so an
oversized number cannot wrap; plus is_array_tag(), jobsel_element_id(),
jobsel_strerror() and JOBID_FIELD_MAX_* constants. The selection-cases
conformance table runs in the libxce suite.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The primary_resreq action ignored parse_resreq_primary()'s result, so a
primary whose select (or rusage) clause failed to parse was accepted
without that clause: select[type==] or select[mem>] matched every host.
A malformed clause now fails the whole resreq, and parse_resreq() returns
the clause's own message.
A select naming an unregistered resource is still dropped, not an
error: parse_select() reports it as -ENOENT (new select_context_t
.unknown_resource) and the action lets it pass, because jobs rely on it
-- the scheduler's own "type == any" default among them when no "type"
resource is registered (making it fatal failed 36 scheduler suites).
Test: test_resreq_primary_select_error_fails_parse RED before (all
accepted), GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- bug-1872 (worker): "Failed to find any disk statistics in
/proc/diskstats" ERROR every 15 s on an NVMe-only host. The whole-disk
filter guessed from the name and skipped anything ending in a digit or
containing 'p' -- every nvmeXnY, mmcblkN, even sdp -- so io was never
reported. diskstats.h now counts a device only when
/sys/block/<name>/device exists (real disks; not partitions, loop, dm,
md, zram), and a host with no disk warns once.
- bug-1873 (worker): "Failed to get worker pid" WARN on job start. MainPID
reads 0 until systemd has forked the new transient unit; xcecd.c already
retries, so the miss is DEBUG and the final failure names the unit.
- bug-1874 (scheduler): deprecation WARN for the legacy xce_sched_fcfs on
every start, though builtin_fcfs is the picker. Registration still wraps
a job_elect_fn plugin but no longer warns; selecting one warns once.
- bug-1875 (apiserver): "Not running as root -- PAM authentication will not
work" with the default provider config_file; now only for provider pam.
- bug-1876 (controller): standbys logged "cluster_init_phase2 ... failed
(rc=-22)" WARN every 3 s while waiting for the primary's config push,
a state the function itself calls normal. It returns -EAGAIN for that
wait; callers log it at INFO, other failures still WARN.
- bug-1877 (controller): "WAL/snapshot stored unencrypted" warned without
looking. persist_path_encrypted() checks fscrypt (statx) and dm-crypt
(sysfs dm/uuid CRYPT-, through LVM slaves); the warning stays when the
storage is not shown to be encrypted (e.g. a plain /tmp).
Tests: worker diskstats_test (2 of 4 RED on the old filter with pc13v's
/proc/diskstats), controller test_phase2_cold_boot and
test_persist_encryption, scheduler picker tests, apiserver
TestRootWarning. Full ASAN build: all 113 suites pass (controller 319/0,
harness 141/0).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A worker reported job start, finish and failure with a fire-and-forget
OPCode_notify the controller never answers, so a report lost while the
worker stayed connected (the live incident: 6 of 7 finish reports in a
burst) left the job RUN forever; the rejoin resync (165fc158) only
recovers reports lost around a reconnect.
- controller.capnp: OPCode_notifyAcked -- the same NotifyData body,
answered. A separate opcode because replies are matched through the
client's single wbuf, not by sequence number: answering plain notifies
would feed stale statuses to the next synchronous request.
- controller (cluster.c): do_notify_acked() applies the fact like
do_notify, then waits on the commit barrier and replies 0 only once it
is majority-committed. A barrier failure (the fact is still appended and
retained) replies -E_XCE_NOT_READY, so the worker retries in place
instead of rotating away from a healthy primary (cf. bug-1850). Admitted
by the no-quorum fact-bearing gate like OPCode_notify.
- libxce: xce_notify_acked_req/xce_notify_acked (send, wait for the
status) and xcecli_notify_acked (XCECLI_FAILOVER_CALL retry/failover).
xcecli_notify_jobstart/jobfinish/jobfail use it; on -E_XCE_BAD_OPCODE
(a controller predating the opcode) they fall back to the plain
fire-and-forget notify, so workers and controllers can be upgraded in
either order.
- Resends are safe: START/FINISH are duplicate-guarded and a repeated FAIL
yields the same state.
- worker: logs a WARN when a report is still unacknowledged after the
retry budget (the rejoin resync remains the backstop). These calls run
on the worker daemon's IPC callback, so while the controller lacks
quorum other jobs' IPC messages wait (buffered, not lost) up to the
retry budget.
Tests: test_notify_acked_reply_only_after_commit and
test_notify_acked_withheld_replies_not_ready (no reply before the covering
ACK; -E_XCE_NOT_READY with the fact retained when ACKs are withheld).
Written with the opcode, so no separate RED run. Full /xce-build all:
111 suites pass; controller suite 1245/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
build_queues() only ran at boot, so a queue a live config upgrade added
had no runtime object in c->qmgr (and the scheduler/apiserver never heard
of it) until the controller restarted. The checkpoint-ordering half of
bug-1469 was fixed in 959b0cda; this is the remaining half.
- reconcile_config_queues(c, notify): creates every queue in the active
config that has no runtime object, leaves existing ones untouched,
returns the number created.
- config_activate_committed() calls it (with notify) after every
successful activation -- verified-marker and no-marker paths alike.
- persist_notify_queue_updated_no_wal(): fires HOOK_DETAIL_QUEUE_UPDATE
under the t_apply_seam_notifying guard, so the scheduler/apiserver are
told while no WAL_QUEUE_CTRL is appended -- config queues are
config-owned, built on every member from the committed config.
- build_queues() now delegates to it without notify, so boot and
activation are idempotent in either order.
New test_config_queue_reconcile.c (RED before: the added queue was never
created): creates the added queue, leaves the existing object untouched,
notifies once, and a second reconcile/build_queues adds nothing.
Controller suite passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gitea #47: `xcecli jobs submit -R "rusage[mem=100000000a0]"` was accepted
with a normal "Job <...> is submitted" reply. The apiserver only
length-checked resreq; the real PEG grammar ran later, asynchronously, in
the scheduler's core_build_job(), which logged a server-side WARN and
dropped the job -- it then sat PEND forever with no sign of why.
- libxce: xce_check_rusage_syntax(resreq, &errmsg) runs parse_rusage() on
every rusage[...] clause (extracted the way section.peg does: up to the
first ']'), so primary, compound and alternative requests are all
covered. No resource manager needed, so select[]/order[] resource names
are still only checked by the scheduler (the documented scope).
- apiserver/backend: ValidateResreqSyntax() cgo wrapper (resreq_check.go);
the !cgo stub returns nil (no job can be submitted without CGO anyway).
- controllers/jobs.go preCheckJobRequest(): calls it after the length
check via a package var, so a malformed request is refused up front.
Tests: lib test_check_rusage_syntax (RED before: the reported input was
accepted) and backend TestValidateResreqSyntax (RED before) through the
real grammar; controllers TestPreCheckJobRequest_RejectsMalformedResreq
checks the call site with an injected validator (written with the call
site, so no separate RED). libxce, apiserver backend (cgo) and controllers
tests pass. apiserver/middleware TestLoadJobs fails here and on the
baseline alike: it needs a live apiserver on :8080.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-448: job_ctrl() took a reference on the job but only one error path
released it; every successful call leaked one (refcount 1 -> 3 after two
calls). It is now released on every exit path, including the key_find
rc<0 one.
bug-1626: job_ctrl()'s not-started branch changes the job (mark_job_update
/ mark_job_fail -> apply_job_*_transition, which require RTTI_LOCK_WRITE)
while holding only RTTI_LOCK_READ, so it could run concurrently with
readers. It now takes the write lock. (dispatch_a_job_ already did.)
bug-1631: dispatch_a_job_()'s four retryable-failure branches never reset
state/allocs, correct for a first dispatch (still PEND) but not for a job
that had already been dispatched once: it stayed RUN holding its old
allocation. Each branch now calls record_job_reconcile_pending_fact() --
a no-op without a WAL fact for a PEND job; for a RUN job it returns it to
PEND, releases allocs and appends a durable WAL_JOB_UPDATE -- and notifies
after unlocking when it applied, the same pattern as the resync reconcile
path (cfe66dde). Not reachable until something re-dispatches RUN jobs
(e.g. preemption), fixed now so it is correct when that lands.
- job_get_refcount() (jobj.c) read accessor for the leak test.
- test_apply_pump.c: test_job_ctrl_releases_its_job_reference and
test_job_ctrl_mutates_not_started_job_under_write_lock (both RED before).
- test_dispatch_fail_retry.c now asserts PEND + allocs released after a
post-success retryable failure (RED before).
Controller suite 1236/0; all 111 meson suites pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
persist_checkpoint() and persist_checkpoint_marker_triggered() wrote a
local-only WAL_CHECKPOINT marker that consumed an LSN of the replicated log
(current_lsn++) but was never replicated. A checkpoint taken by one member
only -- the primary's periodic max-wal-records checkpoint, or the one before
a snapshot send -- left a hole no standby had: the leader's next entry
carried prev_log_index = the marker's LSN, logOk failed on every standby
forever, commits timed out and writes stalled ("could not submit 110 jobs"
in the --force-snapshot soak; reproduced on the pre-bug-1867 baseline). Even
the config-marker checkpoint, taken on every member, stamped the marker with
each member's own current_lsn at apply time, which need not agree.
Nothing reads the marker except to skip it, so checkpoints simply stop
writing it. The recovery, streamer and bug-1475 log_len skip paths stay, for
WALs written by older binaries; persist_test_write_legacy_checkpoint_marker()
(XCE_TESTING only) reproduces such a WAL for the tests that cover them.
- new test_bug1870_leader_only_checkpoint_leaves_no_log_hole (RED before):
a leader-only checkpoint takes no LSN and a standby that never
checkpointed catches up past it via the real streamer/receive path.
- test_checkpoint_writes_marker_to_wal inverted to ..._writes_no_marker_...
- bug-1475 x2 and the apply-correctness trailing-marker test now build
their legacy marker explicitly.
Controller suite 1234/0. HA soaks: --force-snapshot PASS (113 jobs, real
snapshot install -- also live-verifies bug-1867), --config-upgrade-wal-trim
PASS, --sequential 6 rounds PASS (SIMVERIFY 7 files, 0 mismatches).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The FAIL half of bug-1747: apply_job_fail_transition() stamps
finish_time = time(NULL) at APPLY time, and WAL_JOB_FAIL carried no time,
so a failed job's finish_time changed on every restart replay / standby
catch-up, and a worker-reported failure found on rejoin could never keep
the worker's own time.
- WAL_JOB_FAIL's trailer grows to int32_t code + uint64_t finish_time
(seconds). Decoded by length: 16 B (pre-bug-1754), 20 B (code only,
bug-1754) and 28 B (code + time) all apply as before for what they carry.
- record_job_fail_fact_at(c, job, failinfo, finish_time_ms), the fail-side
sibling of record_job_finish_fact_at(): 0 = receipt time. on_job_fail()
uses receipt time; resync's finishedFail branch passes the worker-
reported finish_time_ms (Req 4.2 now holds for failures too).
- Both apply seams adopt the wire time after the transition.
4 new tests in test_wal_job_fail_fact.c (3 RED before the fix) and the
resync round-trip now asserts the worker-reported failure time (RED
before). Controller suite 1232/0. Factstream HA soak (2 rounds, 60 jobs)
PASS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A worker reports every nonzero job exit as a FAIL (xcecd_job_finished ->
xcecd_notify_job_fail(jobid, stage, exit_code)), not a FINISH. The deferred
WAL_JOB_FAIL fact carried only {jobid, STAGE_FLAG|stage} and no apply seam
touched job.exit_info, so exit_info stayed NULL for every failed job, live
and after replay, and HOOK_DETAIL_JOB_FAIL subscribers saw code 0.
- persist_submit_job_fail_fact(): header + int32_t code trailer, used by
record_job_fail_fact(). Old entries (no trailer) decode as before.
- apply_job_fail_exit_code(): records the code in exit_info for pre-exit
stages only; a POST_EXEC code is the script's and leaves exit_info alone.
Kept out of apply_job_fail_transition() so mark_job_fail()'s internal
-errno failures (not replayed with a code) stay live/replay consistent.
- Both apply seams adopt the code. apply_wal_entry()'s WAL_JOB_FAIL case
also stops assigning the wire state 0 for STAGE_FLAG entries and runs the
real transition, as apply_committed_entry() already does.
New test_wal_job_fail_fact.c (5 tests, 3 RED before the fix); the two
test_worker_resync.c assertions that pinned exit_info == NULL now assert the
worker-reported code. Controller suite: 1228 tests, 0 failures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every scheduler (re)join re-pushes the whole config, and both apply paths
-- sched_init_resources() (scheduler.c) and core_apply_config_resources()
(core_events.c) -- re-added each resource with makeResource(), which
refuses an existing name: one ERROR per resource per rejoin (33 in
data/xce_scheduler.log), and a genuinely changed definition was dropped
just as silently.
New libxce resourceApplyDef(): under one write lock, register a new
resource (ADDED), or compare an already-registered one on its
scheduling-relevant fields -- valtype, ordtype, flags, interval -- and
return SAME or CHANGED. Both call sites log SAME at debug, CHANGED as a
WARN that the definition is not applied to the running scheduler
(updating a registered resource in place would disturb its index and
accounting), and only real failures as ERROR.
Test test_resource_reapply_same_is_noop_changed_is_reported (EEXIST on
an identical re-push before). Scheduler suite 87/87, lib 1/1. Re-verify
of data/scheduler-replay-20260925-133708.replay: 0 add-resource ERRORs
(33 before), 0 changed-definition WARNs, 0 mismatches, same 60 DISPATCH /
908 PEND decisions.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_failover_classify() mapped E_XCE_NOT_READY with NOMASTER to
RETRY_BACKOFF, so xcecli_retry_next() rotated to the next master,
reconnected and fired the rejoin hook. But a NOT_READY reply comes from a
controller that answered on a working connection -- here a healthy primary
that had briefly lost its write quorum (both standbys NACKing, bug-1864).
The scheduler abandoned it for the standbys, rejoined, and got a full
re-push built from lagging committed state (21:39:24-27 in data/logs).
New action XCE_FO_RETRY_SAME (appended; existing values unchanged):
NOT_READY backs off (same jittered schedule, same -E_XCE_NOT_READY on
budget exhaustion) and re-sends on the current connection -- no rotate,
reconnect or rejoin. NOMASTER keeps rotating after backoff. If the
controller is really a standby it answers REDIRECT/STANDBY/NOMASTER next
time, and if it dies the transport error rotates, as before.
Tests (test_libxce_retry.c): taxonomy row NOT_READY -> RETRY_SAME;
decide backs off and exhausts to -E_XCE_NOT_READY; the real
xcecli_retry_next keeps cur_master on NOT_READY (rotated before) while
STANDBY still rotates; backoff-growth test moved to NOMASTER. Failover
tests 56/56; lib, controller 5/5, worker 10/10, scheduler 87/87,
harness 5/5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The dispatcher checked each queued task's endpoint busy flag one item at a
time, while the vcpu that finished the endpoint's previous message cleared
the flag without the taskq lock (a lock there deadlocked, bug-1282). Within
one scan, task N was skipped (endpoint still busy), the endpoint freed up,
and task N+1 of the SAME endpoint was dequeued first. Standby controllers
then handled WAL catch-up entry 258 before 257 and NACKed (log mismatch):
26 reorders -> ~576 NACKs in a 45-min production-like run, both standbys
at once at 21:39:24, primary lost its write quorum and answered
jobdispatch/join with E_XCE_NOT_READY, the scheduler client failed over and
rejoined, and the rejoin JOBS_PUSH carried lagging committed state
("refusing resync state ... existing job is RUN" WARNs).
dataq gains dataq_deq_ctx() (a per-scan matcher context). The scan is now
vtask_deq_eligible(taskq, is_blocked): it remembers every endpoint found
busy and keeps all later tasks of that endpoint ineligible for the rest of
the scan (past 64 distinct busy endpoints, every later IO task waits for
the next scan) -- per-endpoint FIFO without any new lock.
Test lib/testsuite/test_vtask_order.c injects is_blocked to free the
endpoint mid-scan: task N+1 was dequeued before the fix, none after; other
endpoints and FIFO/generic tasks unaffected. Suites: lib 1/1, controller
5/5, worker 10/10, scheduler 87/87, harness 5/5. Chaotic HA soak 2/2 PASS
with 0 WAL reorders (2 and 5 per run before), no ASAN.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each primary's allocator (seqjob/parjob my_alloc) sets ctx->alloc to a
fresh job allocation covering only its own hosts, so a multi-primary
COMPOUND ("1*{A} + 2*{B}") dispatched only the last primary's hosts --
leaking the earlier allocations while every primary's hosts had been
debited in ctx.uncomitted.
sched_make_compound_alloc_ now folds each primary's allocation into one
job allocation via the new libxce merge_job_alloc(): a host placed by
several primaries becomes one entry with per-resource values added
(matched on resname, instid and reserve, so distinct instances are never
summed); new hosts and shared allocations are appended/merged the same
way. Partial-failure rollback remains bug-1766.
Tests: "1*{..} + 2*{..}" on 2x2 slots now allocates 3 slots across two
distinct hosts; "1*{..} + 1*{..}" yields one host entry with 2 slots
(both kept only the last primary before). Scheduler suite 87/87, lib 1/1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A failing TEST_ASSERT longjmps out and skips the fixture teardown, leaving
the commit engine's sweep thread and persist's fsync_worker asleep on a
static struct. The next test re-inits it underneath them, corrupting the
condvar, and a later teardown blocks in pthread_cond_destroy/join until
the 300s meson timeout. The failure line itself was lost because stdout
was fully buffered.
- controller_test: line-buffer stdout; tearDown() reaps what a failed test
leaked via new XCE_TESTING registries (persist_test_reap_unstopped, then
commit_test_reap_unfinalized_engines -- persist_stop before engine fini).
Clean run reaps 0; an injected failure now fails in 2s instead of hanging.
- persist: flusher_flush_once reports only its own durable advance, so
fsync_worker no longer re-fires the durable callback for an advance a
concurrent persist_flush made (flaky "must not re-fire" test). Regression
test test_flusher_pass_does_not_claim_concurrent_flush_advance.
- tests: jap observer is atomic and waits (bounded) for a continuation that
may resolve on the flusher thread; pump catch-up tests wait for trailing
replies instead of a single scan.
- config_repl: upgrade decode temp file via mkstemp instead of a
pthread_self()-derived /tmp name (same value across processes, opened
without O_EXCL -- collisions and a symlink target).
Verified: controller_unit_test 1222/1222; 24 parallel runs (6x4), no
hangs or ASAN errors.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Worker-rejoin-resync sets XCE_WORKER_RECONCILING and pushes it to the
scheduler, which correctly freezes that worker's resInUse mid-reconciliation
(bug-1799/1801). But resync_resolve_round() only cleared the bit on the
controller's own worker_t, never notifying the scheduler -- so once a
rejoin round resolved, the scheduler's copy stayed RECONCILING forever,
permanently freezing that worker's resource accounting even after every
job on it finished. Live-observed via a replay capture: a 22-slot worker
stuck at used=22.00 for the rest of the run with zero RUN jobs left.
Added persist_notify_worker_status_updated() (mirrors bug-1679's job-side
persist_notify_job_updated_already_persisted()) and call it from
resync_resolve_round() so HOOK_DETAIL_WORKER_UPDATE fires without a
spurious WAL append. TDD: new spy-hook regression test, RED confirmed
against the unfixed code, GREEN after the fix. Full controller_unit_test
suite: 1220 Tests, 0 Failures, 0 Ignored.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CtrlHelloRequest.key was declared `Data` in controller.capnp. This
project's capnp-c backend silently no-ops encoding of Data-typed fields
without zero-initializing the caller's local struct, leaving an
uninitialized capn_ptr that write_CtrlHelloRequest unconditionally wrote
to the wire -- SIGSEGV in capn_setp/copy_ptr. Hit live: ctrl-1 won its
Raft election and crashed calling ha_send_ctrlhello_to_all_standbys().
Same bug class as bug-1504/bug-1505, both already fixed by switching the
field from Data to List(UInt8); bug-1505's own writeup had flagged
CtrlHelloRequest.key as a known-but-unfixed instance. Applied the
identical fix here: Data -> List(UInt8) + keyLen companion in the schema,
uint8_t*/key_len in typemap.h. The field is unused in production (always
NULL), so no consumer code changed. Added a round-trip regression test
covering both the NULL-key production shape and real key content.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
update_limit_res_value() computed a percent-based limit's capacity from
the resource's own live, fluctuating value (e.g. "mem"), letting the
enforced cap drift with unrelated memory pressure instead of staying
pinned to a stable fraction of total capacity. instantiate_limit() now
resolves a static capacity-source resource once at instantiation time
(mem->maxmem, swap->maxswap, table-driven), stored on a new
xce_vres_t.percent_res field; a missing counterpart falls back to
today's behavior rather than failing instantiation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds fmt_limit_units(buf, len, limit, units_mode) to src/lib/xcelimit.c,
a units-mode-aware sibling of fmt_limit() following the
fmt_jobid()/fmt_jobid_ex() extended-sibling pattern. Reuses fmt_limit()'s
exact consumer-dump loop untouched; only the final value line branches:
percent limits always render bare "%f%%" regardless of mode, byte-valued
(or "swp"-special-cased) absolute values render via
xce_format_units_value() (task 7), everything else keeps the
unconditional "%f".
fmt_limit() itself is now a one-line XCE_UNITS_RAW wrapper around
fmt_limit_units() -- provably byte-identical to its prior output for
every input shape, so its 5 existing direct unit tests needed zero
changes (the regression proof this refactor relies on).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds Requirement 6 (Human-Readable Units for Byte-Valued Limit Amounts)
to the spec: requirements, design, research, and tasks 7-10.
Task 7 (Foundation): promotes the REPL's byte-unit scaling algorithm
and byte-valued-resource classifier into src/lib/utils.c/utils.h
(xce_format_bytes_human, xce_is_bytes_valued_resource), alongside a
new display-mode-aware composer (xce_format_units_value,
xce_units_mode_t) -- following the fmt_jobid()/fmt_jobid_ex()
precedent. repl_fmt.c's scheduler-only functions become one-line
delegating wrappers. repl_info.c's repl_format_numeric_resource_value()
(shared by info worker/info resource, not just the upcoming info
limit) is retyped to delegate to the new shared composer via a small
translate_units_mode() bridge, so every consumer benefits from one
implementation going forward.
No existing symbol's signature changed; only new symbols were added.
Confirmed behavior-preserving: full scheduler suite 85/85 unchanged,
including info worker/info resource's own regression tests. One
narrow, untested, out-of-requirement-scope corner case found and
documented (swp's RAW/BOTH mode rendering under the new single-value
composer interface) -- logged in tasks.md's Implementation Notes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds CtrlPeer.status (UInt8, next free ordinal @3 after port) plus the
matching xce_ctrlpeer_t.status typemap field, verified field-for-field
against the regenerated codec with a genuine encode/decode round-trip
test. No sourcing/rendering logic yet (task 8.2/8.3) -- field defaults
to 0 (unknown) until wired.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
The primary echoes its own last_known_elected_at_ms on every outgoing
heartbeat (threaded through xce_ha_heartbeat_req's existing scalar-arg
plumbing, alongside term/committed_lsn/committed_config_tag); a standby
processing a received heartbeat (ha_consume_heartbeat_committed, the sole
production call site) stores it unconditionally into its own
last_known_elected_at_ms, reconcile-free -- no other HA field moves.
ha_observe_heartbeat_committed itself is untouched; the new store sits
next to its call site, not inside it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds CtrlStatus.electedAtMs@14 and HeartbeatMsg.electedAtMs@4 to the
capnp schema, their hand-authored typemap.h counterparts, and a new
ha_state_t.last_known_elected_at_ms field alongside last_known_committed_lsn.
Schema/state plumbing only -- nothing stamps or consumes these fields yet
(tasks 7.2-7.5). Codec round-trip tests confirm both new fields survive
encode/decode unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xce_core_push_events() WARNed on every JOBS_PUSH resync redelivery
(sched_add_job rc=-EEXIST), which is the expected outcome under
on_scheduler_join()'s full resync or the array-push dedup race
(bug-1734/bug-1714) and flooded operational logs. Only that case now
logs via a new DBG() macro (utils.h); any other add_rc (e.g. -ENOMEM)
still WARNs. DBG is named to avoid colliding with the build's own
-DDEBUG command-line define (debug/asan buildtypes).
Also collapsed hash_string()'s duplicated djb2 loop to delegate to
hash_block(), which already implements the same RUN_HASH loop.
Logged as bug-1809 in .wolf/buglog.json.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Move the one genuinely shared header (header-only, static inline, zero
link dependency) from src/controller/testsuite/ to src/include/xce/,
alongside the existing cross-daemon shared headers (cluster.h, types.h,
errnox.h). Repoint its three current consumers (harness's test.c and
joblist_display.c, controller's test_ctrlstatus.c) to the new path.
test_joblist_display.c is intentionally left on the old path -- it
relocates in task 2.2.
Also removes src/harness/meson.build's temporary '../controller/testsuite'
incdir shim, which task 1.1 added and explicitly marked for removal once
this task landed.
controller_unit_test: 1228 Tests 0 Failures 0 Ignored (unchanged from
task 1.1's baseline, confirming a header-path-only change).
Task 1.2 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
bug-1790's fix worked but put limit-plugin-specific logic in the wrong
place: seqjob_prepare_allocation()/parjob_prepare_allocation() (the
allocator plugins) unconditionally pre-computed a job-wide total for
every per-host resource in the job's rusages, "just in case" some
global limit needed one. That's the allocator plugins' concern leaking
into limit evaluation -- duplicated across both allocators (a third
would need it too), and wasteful (computed for resources no limit
actually references).
Moved ownership to sched_fill_limit_requests() (resource.c), called
once centrally from plugin_prepare_allocation() regardless of which
allocator ran:
- Removed calc_wanted_host_resval_job_totals() and its calls from both
allocator plugins entirely.
- Added calc_wanted_job_total_resval() (resreq.c): an on-demand,
single-resource lookup by name (no resmgr/key_find needed -- the
caller already has the resolved xce_resource_t), computing the same
job-wide-total formula only for the one (scope, underlying-resource)
combination that needs it -- a GLOBAL limit on a per-host resource.
The other three combinations are unchanged, still reading straight
out of alloc_req.
- sched_fill_limit_requests() gains rusages/nhosts/nslots parameters
back (a genuine need this time, not a re-added unused one).
plugin.c's two call sites pass obj->u.primary->resreq->rusages
(NULL for COMPOUND/ALTER, matching this codebase's existing scope
convention for those tags) and ctx->nhosts/ctx->nslots.
All 23 testsuite link-satisfying stubs mechanically updated. Removed
the two allocator-level regression tests from the first pass (the
behavior they proved no longer exists by design); rewrote
test_fill_limit_requests_global_limit_on_host_resource_reads_host to
exercise the new on-demand path directly, nslots=3 to prove the job's
WHOLE total is used.
Full xce-build.sh all asan --incremental: 94/94 Ok, zero regressions.
Live re-verified against the same ug1_global_mem deploy/scenario:
byte-identical scheduler trace to the pre-refactor result, confirming
behavior-preserving.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
Closes the gap bug-1789 explicitly carved out: a GLOBAL limit built on a
per-host resource other than "slots" (e.g. a future absolute global mem
cap) needs ctx->alloc_req.host to carry that resource's job-wide total
want during the global phase, but only "slots" ever got that treatment
(via seqjob/parjob_prepare_allocation()'s own unconditional store).
Adds calc_wanted_host_resval_job_totals() (resreq.c), sharing its loop
body with calc_wanted_global_resvals() via a new static helper
parameterized only by which XCE_RESF_GLOBAL direction to match -- it
computes the SAME job-wide-total formulas (PER_TASK: value*nslots, PER_JOB:
value, PER_HOST: value*nhosts) for non-global resources instead of global
ones. seqjob/parjob_prepare_allocation()'s global-phase branch now calls
it into ctx->alloc_req.host alongside the existing calc_wanted_global_
resvals() call into .shared.
Safety was traced, not assumed: plugin_prepare_allocation() only resets
.host at the top of a per-host call, never during the global phase, and
sched_fill_limit_requests() for glimits reads .host later in the SAME
global-phase invocation -- transferring the value into the limit's
permanent .shared record before any later per-host call ever resets
.host. An earlier candidate fix (reusing calc_wanted_host_resvals()
directly) was rejected during design: its PER_HOST formula and, for
parjob, its per-bucket nslots don't give a job-wide total.
New regression tests in test_seqjob_allocator.c/test_parjob_
combinatorial.c prove a non-"slots" per-host resource ("mem") now gets
correctly populated; parjob's test uses maxTasks=3 specifically to prove
the formula uses the job's whole nslots, not a bucket's share.
Full suite (xce-build.sh all asan --incremental): 92/94, both failures
pre-existing and unrelated (bug-1762, bug-1733's already-documented
xcealloc.c leak). Not live-verified -- no limits.yaml entry exercises
this path yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
While live-verifying gitea#46, found and fixed a cascade of 5 previously
undetected bugs that together made every usergroup-scoped limit in
limits.yaml silently non-functional:
- Named usergroups/hostgroups (users.yaml/hosts.yaml) were never
registered into the scheduler's group managers (groupmgr_add() had zero
production callers), so any limit referencing one by name could never
resolve real membership.
- An unconfigured limit scope (e.g. no "projects:") arrived as an
empty-but-present capnp group, not NULL, so build_consumer() built a
real empty group (matching no one) instead of falling back to "all".
- Three separate call sites conflated key_find()'s negative "not found"
return with a real error, short-circuiting before ever reaching the
"instantiate on first use" logic right below them -- the same defect
class already fixed once before in this codebase (bug-1495).
- Two call sites passed &(sched->core.resmgr) where a plain handle was
expected, one level of indirection too many, causing RTTI's key_find()
to reject it as an invalid handle.
All five are live-verified: fresh cluster deploys now correctly match and
instantiate usage tracking for all 7 limits in limits.yaml on every job
submission, reproduced across multiple independent from-scratch redeploys.
A 6th, architecturally distinct issue remains open (bug-1789, logged, not
fixed): sched_fill_limit_requests() computes a job's consumption of a
slots-scoped global limit by reading the wrong request vector, so the
numeric cap itself still never binds. This needs a design decision on
intended cross-scope semantics, not a mechanical fix, so it's left for a
follow-up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
Adds apply_job_task_count() to overwrite a standalone primary resreq's
minTasks/maxTasks with a job's real requested task count, no-op for
COMPOUND/ALTER nested primaries, with a defensive min>max clamp.
Closes the gap that lets scheduler-allocation's multi-host allocator
ever be reached (bug-1770). Wiring at live ingest call sites follows
in subsequent tasks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds rusage_hash to xce_primary_resreq_t (computed once by
parse_resreq_primary() via a new hash_rusages() helper, hashing
rusage content rather than raw struct bytes since resname is a
non-interned strdup'd pointer), the xce_resreq_fingerprint_t value
type, and resreq_fp on xce_primary_schedobj_t, populated once at real
job-ingest time in core_build_primary_schedobj().
Task 1.2 (scheduler-allocation spec). Requirement 7.1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
