The controller computed run usage (CPU time, memory, CPU peak and
efficiency, LSF bjobs -l definitions) but the capnp Job struct had no
field for it, so the job pushed to the apiserver never carried it, and
snapshots (the same struct) dropped it: a job restored from a checkpoint
lost its accumulated usage.
- controller.capnp: Job.runUsage @13 (JobRunusage); old entries decode
with no run usage.
- apiserver: map run_usage into JobInfo; max_mem, avg_mem, used_mem,
cpu_time, peak_cpu, peak_cpu_time, peak_cpu_duration, mem_efficiency,
avg_cpu_efficiency, peak_cpu_efficiency become queryable columns.
- xcecli: -o fields MAX_MEM, AVG_MEM, MEM_EFFICIENCY, CPU_PEAK,
CPU_PEAK_DURATION, CPU_EFFICIENCY, CPU_PEAK_EFFICIENCY (CPU_TIME now
renders too), and a MEMORY USAGE / CPU USAGE block in jobs list -l.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The configAck byte-delivery barrier waited 500 ms, below scheduling
delay on an oversubscribed host, so a push to a healthy majority was
rejected with -110 while its already-appended marker still committed.
CONF_PUSH_TIMEOUT_MS is now 2000 ms, the wait holds a commit-wait
admission slot (new commit_blocking_admit/release; a full budget fails
fast with not-ready), and the timeout reply no longer claims the change
was not committed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Work dispatched for the old connection (the io_worker's epoll batch,
queued VTASK_IO reply tasks) could still run while the reconnect tore
down and re-armed ctx->ep; a queued reply of the old connection then
reached the new session. The reconnect now unmonitors the old fd,
clears ep->owner (stale tasks are dropped) and waits for the endpoint's
references to drain, bounded at 2 s so a caller that is itself one of
those tasks cannot deadlock. vtask_entry reads ep->owner under ep->lock.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
INVOKE_MSG_HANDLER stored NULL into ep->fsm_in.pkt after every msg
handler. Handlers that take the packet clear the slot themselves
(vtask_handle_msg, peer_ctrl_msg_handler); one that refuses it
(-EINVAL / -ENOMEM) leaves it there, and the NULL store dropped the
only reference. The slot is now left as the handler leaves it: a
refused packet is reused for the next message or freed by endpoint_fini.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A restarted standby accepts the primary's connection during its startup
delay, before persist_recover runs. Catch-up entries written into the
un-recovered persist_t opened seg-...0001 with O_TRUNC and rewrote it,
and after recovery the stale fd kept filing new LSNs under that segment
name. The primary's filename-ordered streamer then skipped them, so a
standby needing that range never caught up and commits stalled.
persist_wal_write_replicated now returns -EBUSY until recovery completes
(the bug-1416 recovery_complete gate). ha_apply_wal_catchup_entry and
ha_handle_snapshot_chunk drop pre-recovery traffic without ACK or NACK.
Catch-up streams from min(NACK-derived join position, standby report),
and the streamer warns when a non-empty range yields no entries.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The fence re-check required !persist_recovery_truncated(), a boot-time
verdict nothing cleared, so a node that booted RECOVER_TRUNCATED stayed
fenced (fenced: true, non-promotable) until restart however far it
caught up. And the "progress is possible" branch did not re-arm, so the
fence was never re-evaluated.
persist_t gains a runtime recovery_truncation_healed latch, which
persist_recovery_truncated() honours (the raw boot verdict stays
available). The re-check marks the truncation healed once the durable
floor has advanced past the fence-time baseline (entries replicated from
the live primary) and reached the HWM, and the progress branch re-arms
with the same baseline and HWM.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A late PreVote grant after WON re-ran the WON path and started a second
real election that replaced elect_ctx, so the term-2 leader promoted no
vote endpoints. Rounds now carry an id and endpoints a round/kind; stale
or wrong-kind replies, timeouts and errors are dropped; raft ignores
replies with no round in progress. raft_cb_become_leader promotes only
with its own stamped election record (own term, majority), else steps
down and re-arms. raft_on_leader_lost keeps voted_for (one vote per
term) and the delayed election starts only as a follower in the lost
term. The failover stagger hands the PreVote start to a vtask so the
delaymgr stays trigger-only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
vtask_handle_msg published the VTASK_IO task while ep->fsm_in.pkt still
pointed at the packet; INVOKE_MSG_HANDLER clears the slot only after the
handler returns. A vtask could decode and free the packet (xce_decode)
and wake the client, whose failover reconnect then ran endpoint_fini and
freed the dangling slot again: ASAN heap-use-after-free in free_packet,
seen in the worker and scheduler during failovers under heavy CPU load.
vtask_handle_msg now clears ep->fsm_in.pkt before vtask_enq and marks
the task the packet's sole owner (owns_io_pkt), so never-run tasks free
it at vtaskmgr_fini and an enqueue failure frees it instead of leaking;
xcectx vtask_entry frees it on its early -EINVAL exits. Same take-and-
clear discipline the controller's peer-control path already used.
Tests: test_vtask_pkt_handoff.c (3), RED reproduces the soak's ASAN stack.
libxce 733/0; xce-build all green; loaded /xce-test-ha soak (8 rounds,
40 spinners) PASS with no ASAN reports (it crashed 2/2 before).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
After an election where the vote connections are not promoted, both
standbys register later via on_standby_connected; the own-term no-op was
appended before any standby existed, and catch-up waited for heartbeat-
driven state reports (config push one 3 s cycle, WAL stream the next), so
the no-op committed ~6 s late and every client write in that window
failed -110.
ha_send_catchup_probe() sends the ordinary heartbeat request (same fields
as primary_heartbeat_fn; encode + enqueue only; primary only) at
registration and once after the standby's first accepted config-push ACK,
so the decision loop's next action follows at ~RTT. The decision loop,
its one-action-per-report pacing, commit/majority rules and the dispatch
gate are unchanged; WAL streaming still runs only on the vtask. The
one-shot arm is cleared on disconnect and on every become_leader /
become_follower.
Tests: test_late_joiner_catchup_kick.c (6). Controller suite 1285/0.
Live: catch-up starts 0-6 ms after a late registration (was up to ~6 s);
/xce-test-ha sequential 20 rounds, chaotic 6 rounds and sequential 8
rounds under 40-spinner load all PASS with zero job loss.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- bug-1683 (test_wire known-good capture): the client closed its TLS
socket with the server's TLS 1.3 NewSessionTickets unread, so the kernel
sent RST and the server lost the unread payload record; the server
swallowed the OSError. The client now waits for the capture to finish,
handler errors are recorded and asserted, and a 50 ms "tickets-pending"
variant makes the RST race deterministic.
- bug-1485 (live HA push, versionTag=0): the CLI's 0 was an error reply
(-110): after a late-joiner election the primary's own-term no-op only
commits ~6 s later via heartbeat catch-up, so a push in that window
misses the majority barrier. The fixture now waits until the latest
own-term no-op has committed (wait_for_primary_write_ready); versionTag
asserts print the CLI's stdout/stderr. The controller latency itself is
bug-1913.
- bug-1725 (activation line missing): "Config upgraded" and the
activation line are logged after the reply; both tests now poll for all
three commit lines (shared assert_primary_committed).
xceadmin suite 373 passed; live push tests looped (95 runs incl. heavy
CPU load, 0 failures from these mechanisms).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- bug-1912: xce_parse_peers() ignores whitespace around entries and skips
blank ones, matching xceadmin's resolve_controllers(), so "a=h:1, b=h:2"
names the second candidate "b" (5 libxce tests + regression case).
- bug-1911: xcecd frees the -l log dir before the -h / --version returns
(regression case: no LeakSanitizer report, exit 0).
- bug-1910: both version scripts build the apiserver with -mod=vendor
(as the meson build does) instead of -mod=readonly, so they no longer
need a Go module cache.
- bug-1909: test-version-startup-consistency.sh accepts exactly the
dirty-content-fingerprint pairing ('<sha>-<8 hex>' in --version with
'<sha>-dirty' in the startup log), with self-test cases (e)/(f).
Both version scripts now exit 0 on a dirty tree with XCE_CONTROLLER_PEERS
set; regression script 35/35; xce-build all: C 127/0, scheduler 16/0,
xceadmin 364 passed, no ASAN reports.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
18 sub-tasks in 6 groups (libxce primitives + regression script
skeleton; controller; scheduler/worker/harness; xceadmin; integration;
docs). Each C task adds its failing regression-script case before the
change. Requirements, design and tasks approved.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Iterated design review until clean (GO):
- Controller main() calls mylog_fini() before every early return, so
log lines written by tlog's background thread survive the exit.
- cluster_init_phase1() keeps its candidates_count > 0 guard around the
new xce_peers_find_identity() call (standalone stays my_priority 0).
- xceadmin adds the "(from $VAR)" source note only to network errors,
not to local cluster-bin file errors (OSError with filename set).
- Regression script: scratch --workdir/--certdir/--logfile per daemon,
refuses to run if a local cluster is up (port 9990, worker socket),
--sim case asserts the sim branch ran, 2.3 patches follow_redirects,
2.8 malformed-entry cases use a readable cluster-bin file, wrapper
test expects PEERS_STR form.
- Two existing scripts that start xce_scheduler without --peers will
unset XCE_CONTROLLER_PEERS; --interactive wording corrected.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Design review (NO-GO -> GO) and iterated requirements review until clean:
- Controller refuses to start (non-zero exit, stderr, before any network
I/O) when an env-supplied XCE_CONTROLLER_PEERS lacks its own name (1.6);
new libxce xce_peers_find_identity() replaces both identity loops in
controller/cluster.c so the check and my_priority/listen port agree.
- deploy-ha.sh's generated scripts/xceadmin wrapper exports
XCE_CONTROLLER_PEERS so an ambient value cannot redirect it (Req 4).
- Opt-out is unsetting/emptying the variable (1.7); "No --peers" log
lines say it was unset or empty (1.8); docs cover sudo/systemd not
inheriting it (3.1).
- Requirement 1 defines explicit candidate list / resolution point /
own name so harness (set masters) is covered precisely; error messages
name the value's source incl. validation failures (1.4).
- xceadmin: name= stripping and a clean malformed-entry error apply to
every --controller source; errors name the env var source (2.2, 2.8).
- Test plan fixed against real code (stderr vs stdout, log texts,
xcecd_virtual, --norc, free ports, failover order test).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- apiserver peek (parseJobIDParam): <array-id>[k] now resolves to the
element; it used to drop [k] and look up the array identity itself
(and panicked in parseArrayName on some inputs). A job number still
names that job exactly (an element by its own XCE_JOBID); ranges and
invalid selections are rejected.
- apiserver restart: the decimal-only pre-check (isValidJobId, now
unused and removed) rejected the written form c-a-ts-seq[k];
ParseIdentitySelection, already used for the lookup, validates it.
- xcecli jobs restart converts its argument with ToAPIJobID like the
control commands, so written IDs work and invalid ones fail locally.
- docs: the peek/restart caveat in the Job arrays section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It logged in to a hard-coded http://localhost:8080 with fixed
credentials, so 'go test ./...' failed with connection refused. It now
skips unless XCE_LOAD_TEST_URL is set, with the credentials in
XCE_LOAD_TEST_USER/XCE_LOAD_TEST_PASSWORD; 'go test ./...' is green.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
REGEX_JOBID, REGEX_JOBARRAY_ID and N_MAX_FIELDS_JOBID served the old
regex-based parse_jobid(), replaced by parse_jobsel() in
job-array-one-based-index task 1.2; nothing references them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
