The worker-rejoin-resync query sent when a worker with live jobs re-joins
had no handler in xce_test's worker context, so the controller resolved
every such round by its 15 s timeout, withholding dispatch meanwhile.
New fake_resync.{c,h} reassembles the paginated query and builds the
report like xcecd (queried jobs in order, running or unrecognized, then
unqueried running jobs); the handler sends it with a non-blocking request.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A reconnect made from inside a task of the endpoint it re-establishes (a
request handler that failed over) waited out the full 2 s quiesce bound:
its own reference cannot drain, and the old connection's queued tasks are
serialised behind it -- and those then ran after the owner was re-armed,
reaching the new session.
Each VTASK_IO task now records the endpoint's reconnect_gen when queued and
vtask_entry drops a task whose generation is stale; the vtask manager
tracks the endpoint whose task the current thread runs
(vtask_current_io_ep), and xce_ctx_quiesce_ep skips the wait in that case.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Worker-context handlers (startjob, runtasks, jobcontrol) run as tasks of
the worker's own connection, so their synchronous notify RPCs waited for
a reply queued behind themselves, timed out (-110) and failed the client
over; the session never joined again, so the controller rejected every
later worker request ("is not a worker") and job usage stayed zero.
Handlers now queue the notification (new worker_notify.{c,h} FIFO) and a
one-shot delay-manager task sends it; housekeeping retries anything left.
A recorded worker join is repeated with the saved attributes once the
connection's reconnect_gen changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The controller computed run usage (CPU time, memory, CPU peak and
efficiency, LSF bjobs -l definitions) but the capnp Job struct had no
field for it, so the job pushed to the apiserver never carried it, and
snapshots (the same struct) dropped it: a job restored from a checkpoint
lost its accumulated usage.
- controller.capnp: Job.runUsage @13 (JobRunusage); old entries decode
with no run usage.
- apiserver: map run_usage into JobInfo; max_mem, avg_mem, used_mem,
cpu_time, peak_cpu, peak_cpu_time, peak_cpu_duration, mem_efficiency,
avg_cpu_efficiency, peak_cpu_efficiency become queryable columns.
- xcecli: -o fields MAX_MEM, AVG_MEM, MEM_EFFICIENCY, CPU_PEAK,
CPU_PEAK_DURATION, CPU_EFFICIENCY, CPU_PEAK_EFFICIENCY (CPU_TIME now
renders too), and a MEMORY USAGE / CPU USAGE block in jobs list -l.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The configAck byte-delivery barrier waited 500 ms, below scheduling
delay on an oversubscribed host, so a push to a healthy majority was
rejected with -110 while its already-appended marker still committed.
CONF_PUSH_TIMEOUT_MS is now 2000 ms, the wait holds a commit-wait
admission slot (new commit_blocking_admit/release; a full budget fails
fast with not-ready), and the timeout reply no longer claims the change
was not committed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Work dispatched for the old connection (the io_worker's epoll batch,
queued VTASK_IO reply tasks) could still run while the reconnect tore
down and re-armed ctx->ep; a queued reply of the old connection then
reached the new session. The reconnect now unmonitors the old fd,
clears ep->owner (stale tasks are dropped) and waits for the endpoint's
references to drain, bounded at 2 s so a caller that is itself one of
those tasks cannot deadlock. vtask_entry reads ep->owner under ep->lock.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
INVOKE_MSG_HANDLER stored NULL into ep->fsm_in.pkt after every msg
handler. Handlers that take the packet clear the slot themselves
(vtask_handle_msg, peer_ctrl_msg_handler); one that refuses it
(-EINVAL / -ENOMEM) leaves it there, and the NULL store dropped the
only reference. The slot is now left as the handler leaves it: a
refused packet is reused for the next message or freed by endpoint_fini.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A restarted standby accepts the primary's connection during its startup
delay, before persist_recover runs. Catch-up entries written into the
un-recovered persist_t opened seg-...0001 with O_TRUNC and rewrote it,
and after recovery the stale fd kept filing new LSNs under that segment
name. The primary's filename-ordered streamer then skipped them, so a
standby needing that range never caught up and commits stalled.
persist_wal_write_replicated now returns -EBUSY until recovery completes
(the bug-1416 recovery_complete gate). ha_apply_wal_catchup_entry and
ha_handle_snapshot_chunk drop pre-recovery traffic without ACK or NACK.
Catch-up streams from min(NACK-derived join position, standby report),
and the streamer warns when a non-empty range yields no entries.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The fence re-check required !persist_recovery_truncated(), a boot-time
verdict nothing cleared, so a node that booted RECOVER_TRUNCATED stayed
fenced (fenced: true, non-promotable) until restart however far it
caught up. And the "progress is possible" branch did not re-arm, so the
fence was never re-evaluated.
persist_t gains a runtime recovery_truncation_healed latch, which
persist_recovery_truncated() honours (the raw boot verdict stays
available). The re-check marks the truncation healed once the durable
floor has advanced past the fence-time baseline (entries replicated from
the live primary) and reached the HWM, and the progress branch re-arms
with the same baseline and HWM.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A late PreVote grant after WON re-ran the WON path and started a second
real election that replaced elect_ctx, so the term-2 leader promoted no
vote endpoints. Rounds now carry an id and endpoints a round/kind; stale
or wrong-kind replies, timeouts and errors are dropped; raft ignores
replies with no round in progress. raft_cb_become_leader promotes only
with its own stamped election record (own term, majority), else steps
down and re-arms. raft_on_leader_lost keeps voted_for (one vote per
term) and the delayed election starts only as a follower in the lost
term. The failover stagger hands the PreVote start to a vtask so the
delaymgr stays trigger-only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
vtask_handle_msg published the VTASK_IO task while ep->fsm_in.pkt still
pointed at the packet; INVOKE_MSG_HANDLER clears the slot only after the
handler returns. A vtask could decode and free the packet (xce_decode)
and wake the client, whose failover reconnect then ran endpoint_fini and
freed the dangling slot again: ASAN heap-use-after-free in free_packet,
seen in the worker and scheduler during failovers under heavy CPU load.
vtask_handle_msg now clears ep->fsm_in.pkt before vtask_enq and marks
the task the packet's sole owner (owns_io_pkt), so never-run tasks free
it at vtaskmgr_fini and an enqueue failure frees it instead of leaking;
xcectx vtask_entry frees it on its early -EINVAL exits. Same take-and-
clear discipline the controller's peer-control path already used.
Tests: test_vtask_pkt_handoff.c (3), RED reproduces the soak's ASAN stack.
libxce 733/0; xce-build all green; loaded /xce-test-ha soak (8 rounds,
40 spinners) PASS with no ASAN reports (it crashed 2/2 before).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
After an election where the vote connections are not promoted, both
standbys register later via on_standby_connected; the own-term no-op was
appended before any standby existed, and catch-up waited for heartbeat-
driven state reports (config push one 3 s cycle, WAL stream the next), so
the no-op committed ~6 s late and every client write in that window
failed -110.
ha_send_catchup_probe() sends the ordinary heartbeat request (same fields
as primary_heartbeat_fn; encode + enqueue only; primary only) at
registration and once after the standby's first accepted config-push ACK,
so the decision loop's next action follows at ~RTT. The decision loop,
its one-action-per-report pacing, commit/majority rules and the dispatch
gate are unchanged; WAL streaming still runs only on the vtask. The
one-shot arm is cleared on disconnect and on every become_leader /
become_follower.
Tests: test_late_joiner_catchup_kick.c (6). Controller suite 1285/0.
Live: catch-up starts 0-6 ms after a late registration (was up to ~6 s);
/xce-test-ha sequential 20 rounds, chaotic 6 rounds and sequential 8
rounds under 40-spinner load all PASS with zero job loss.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- bug-1683 (test_wire known-good capture): the client closed its TLS
socket with the server's TLS 1.3 NewSessionTickets unread, so the kernel
sent RST and the server lost the unread payload record; the server
swallowed the OSError. The client now waits for the capture to finish,
handler errors are recorded and asserted, and a 50 ms "tickets-pending"
variant makes the RST race deterministic.
- bug-1485 (live HA push, versionTag=0): the CLI's 0 was an error reply
(-110): after a late-joiner election the primary's own-term no-op only
commits ~6 s later via heartbeat catch-up, so a push in that window
misses the majority barrier. The fixture now waits until the latest
own-term no-op has committed (wait_for_primary_write_ready); versionTag
asserts print the CLI's stdout/stderr. The controller latency itself is
bug-1913.
- bug-1725 (activation line missing): "Config upgraded" and the
activation line are logged after the reply; both tests now poll for all
three commit lines (shared assert_primary_committed).
xceadmin suite 373 passed; live push tests looped (95 runs incl. heavy
CPU load, 0 failures from these mechanisms).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- bug-1912: xce_parse_peers() ignores whitespace around entries and skips
blank ones, matching xceadmin's resolve_controllers(), so "a=h:1, b=h:2"
names the second candidate "b" (5 libxce tests + regression case).
- bug-1911: xcecd frees the -l log dir before the -h / --version returns
(regression case: no LeakSanitizer report, exit 0).
- bug-1910: both version scripts build the apiserver with -mod=vendor
(as the meson build does) instead of -mod=readonly, so they no longer
need a Go module cache.
- bug-1909: test-version-startup-consistency.sh accepts exactly the
dirty-content-fingerprint pairing ('<sha>-<8 hex>' in --version with
'<sha>-dirty' in the startup log), with self-test cases (e)/(f).
Both version scripts now exit 0 on a dirty tree with XCE_CONTROLLER_PEERS
set; regression script 35/35; xce-build all: C 127/0, scheduler 16/0,
xceadmin 364 passed, no ASAN reports.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- apiserver peek (parseJobIDParam): <array-id>[k] now resolves to the
element; it used to drop [k] and look up the array identity itself
(and panicked in parseArrayName on some inputs). A job number still
names that job exactly (an element by its own XCE_JOBID); ranges and
invalid selections are rejected.
- apiserver restart: the decimal-only pre-check (isValidJobId, now
unused and removed) rejected the written form c-a-ts-seq[k];
ParseIdentitySelection, already used for the lookup, validates it.
- xcecli jobs restart converts its argument with ToAPIJobID like the
control commands, so written IDs work and invalid ones fail locally.
- docs: the peek/restart caveat in the Job arrays section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It logged in to a hard-coded http://localhost:8080 with fixed
credentials, so 'go test ./...' failed with connection refused. It now
skips unless XCE_LOAD_TEST_URL is set, with the credentials in
XCE_LOAD_TEST_USER/XCE_LOAD_TEST_PASSWORD; 'go test ./...' is green.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
REGEX_JOBID, REGEX_JOBARRAY_ID and N_MAX_FIELDS_JOBID served the old
regex-based parse_jobid(), replaced by parse_jobsel() in
job-array-one-based-index task 1.2; nothing references them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Formatting only (gofmt -w on the 29 non-vendored, non-generated files
gofmt -l reported); swag-generated apiserver/docs/docs.go left as is.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
clitest_wait_for_exit/clitest_wait_for_text added CLITEST_POLL_SLICE_MS
per loop instead of measuring time. Once xce_test closes its pty after
'quit', poll() returns at once on every call, so the 5 s budget was used
up within milliseconds while the (ASAN, loaded) child was not yet
reapable -- measured: real exit within 6 ms while the counter reached
58800 ms. The waits now use a CLOCK_MONOTONIC deadline, and
clitest_pump_once naps 10 ms when poll wakes with nothing to read.
Under 16 parallel runs: 12/48 runs failed before, 0/96 after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
update_job_cpu_stats() (misc.c), run after each usage report:
- each host's cpu_rate = CPU-seconds gained / wall seconds since its
previous report; the job's cores = the sum over hosts
- peak_cpu = highest job cores seen (peak_cpu_time = it rounded, the
wire field is an integer); peak_cpu_duration = the interval that set it
- avg_cpu_efficiency = cpu_time / elapsed / slots, peak_cpu_efficiency =
peak_cpu / slots, mem_efficiency = highest used_mem / rusage[mem]
request (job-wide total, per task by default), all in percent
aggregate_job_runusage() no longer treats cumulative cpu_time as a peak.
New fields host sample_time/cpu_rate and job peak_cpu are appended to
the capnp structs (old WAL entries still decode) and copied by
merge_job_runusage(), so a promoted standby keeps the rate baselines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
RestartJobByID looked the job up with a job_id CompCondition, which the
store never evaluates, and restarted jqueryReply[0] -- the first stored
job -- for any ID, including unknown ones. It now parses the ID with
ParseIdentitySelection, fetches the rows by key and matches them with
foundJob, as ControlJobByID does: an ordinary job or <array-id>[k] is
restarted, an unknown job or element is not found, and a bare array
identity (several elements) is refused. Restart goes through the
generateJobID/submitJob seams so the handler fixture can test it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Follows page_token with limit=3 over the handler fixture. Before
efff6b65 the token's float-rounded array-element ID matched no row and
every page restarted at row 0 (the original repeat-the-first-page
report); it now pages through every row exactly once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The controller forwards a kill as soon as it has dispatched a job, but
the worker queues the start and only registers the job (and, for the
real worker, connects its IPC client) once it starts -- seconds later
for the first jobs after a worker starts. A signal arriving in between
was dropped ("Failed to find worker"), although the API reported it
accepted, and the job ran to completion.
New pending_signal table: the start handler records the job, a signal
for a queued or still-starting job is kept there, and it is delivered
when the job reaches RUNNING (virtual: killed_signal; real: IPC). A
failed start or a job that ends drops the entry.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The virtual worker's job-end decision lived inside the branches of the
systemd exit-code check, and only the exit-code-0 branch looked at
killed_signal. Without a system bus that call returns -1, so a kill was
recorded but ignored until the job's sleep time elapsed. A virtual job
has no systemd unit: decide its end from the kill and the timer alone,
before asking systemd.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The controller acknowledges job dispatch and pend-reason requests with
status 0 before looking the job up, and the scheduler relies on that
early acknowledgement, so the harness now looks the job up first
(job_exists_or_report) and reports "not found" like the other
single-job commands (job-array-one-based-index Req 4.7, 5.13).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
findPageTokenIndex decoded the token's unique_job_id into a float64;
array element IDs set bit 62, where float64 spacing is 1024, so paging
skipped rows or restarted at row 0 -- an endless loop for callers that
page until the token is empty (control/query of arrays over 1000
elements). Decode with UseNumber and keep integers as int64.
Found by the job-array-one-based-index feature validation.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1904: both scheduler JOB_UPDATE paths parsed "" as-is:
parse_resreq() returns success with a PRIMARY container whose body is
NULL, and apply_job_task_count() wrote through it. Use ingest's default
("type == any") instead.
bug-1906: xcecli jobs list sent local selection-parse errors through
ExtractErrorMessage, which strips brackets and the quoted selection;
report them as written.
Also logs bug-1905 (xcecd_virtual ignores kill of a running job). All
three found by job-array-one-based-index task 10.2's live REST flow.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
src/docs/xce.md gains a 'Job arrays' section: name[1-N] and name[1],
numbering from 1, the API-server limit (default 1000, at most 32767),
<array-id>[k] and <array-id>[a-b] selection (with the peek/restart
caveat), XCE_JOBINDEX, the harness jobsub -a N / $jobid[k], and every
LSF array feature this release does not support. The superseded
statements in job-array-single-wal-record and xce-test-e2e-primitives
are annotated (history kept), the controller-job-cleanup and
job-dependency briefs describe 1-based numbering, and the roadmap gains
an entry.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The comment still described arraySize > 1 as the array path; since task
9.1 idgen() rejects any size other than 1 and arrays use
GenerateArrayJobID.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
idgen() now accepts only a size of 1 (anything else: -EINVAL, context
unchanged) and refuses an exhausted 20-bit sequence with -EOVERFLOW
instead of spilling into the timestamp (also after idgen_array has
taken the last block). Arrays come only from idgen_array; every caller
of idgen/GenerateJobID passes 1. The old multi-element test loop moves
to idgen_array with 1-based assertions.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Submitting an array prints the server's message with the array ID in
written form (no [k]) and the element count, e.g.
'Job <0-0-...> is submitted to queue <normal> (3 elements).'; ordinary
submits print exactly as before. The client model mirrors array_size.
Submit help documents name[1-N] and name[1], numbering from 1 and the
API-server limit (default 1000, at most 32767).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ID lists split only at commas outside brackets and every selection is
validated before any request (control allows identity ranges, listing
refuses them). A raw -J value with brackets is checked before the
wildcard mapping, and the search rule and * stripping apply to the
name part only (-J 'foo*[1]' lists element 1 of arrays whose name
starts with foo); values without brackets map exactly as before.
Control results print each item's own job ID when the server sends it,
so a pre-upgrade element 0 shows as <array-id>[0]. Control and list
help show <array-id>[k] and <array-id>[a-b]; list help now says -J.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xcecli's format package gains ParseSelection (written or decimal key
with optional [k]/[a-b], the server's grammar and check order; 5.8 and
5.7 also on decimal keys with [...], so an upgraded client protects
itself against an older server; query ranges refused after the other
checks), the bracket-aware SplitSelections, the raw -J
ParseNameSelection and Selection.APIKey. ParseJobID is bounded;
ToAPIJobID(s, allowRange) no longer passes unrecognised text through;
peek keeps today's behaviour via ToAPIJobIDLegacy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Filter-based ControlJob validates a bracketed -J value before fetching
(400 with the parser's message; unbracketed names, including commands
with spaces, unchanged). FetchJobList evaluates bracketed job_name
entries through the selection predicates instead of unevaluated
compound conditions, so -J 'foo[1]' acts on element 1 of every array
named foo and never on ordinary jobs of that name. The most-recent-job
rule applies only without an element number and without all-jobs, and
treats the element of a one-element array like an ordinary job
(isSingleElementArray, a documented approximation). The old
index-section parser is deleted. Completes bug-1898 on the owned paths.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
