The apiserver's job store learned of a new job only from the
controller's asynchronous OnJobCreated / array push, so a 'jobs kill'
(or query) straight after 'jobs submit' could answer 'job not found'
for the job just accepted (2 of 4 immediate kills).
services.SubmitJob now records the accepted job right after the
controller's reply (which comes only after the job is committed and
applied): one record per element, base+i for arrays, stored with the new
DataStore.PutIfAbsent and marked JobInfo.Provisional (json:"-", not a
column). An OnJobCreated that already arrived is never overwritten; the
push that arrives later replaces the provisional record -- including in
dispatchJobArray, whose duplicate check no longer treats a provisional
element as already known.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xcecli prints a submitted job's ID in the dashed
{cluster}-{apiserver}-{timestamp}-{sequence} form (format.FormatJobID,
since d6008ead), but run.sh still matched 'Job <[0-9]+>' -- so 'jobs
submit' always failed and 'jobs kill' was skipped (28/1/1). It also
extracted the ID with [0-9]{10,}, which would have picked out only the
timestamp. Match the dashed form and take the whole ID from between the
angle brackets; 'jobs kill' accepts it back.
'jobs submit' reads a job script from stdin whenever stdin is a pipe,
so the suite hung forever when run from a script or CI with an open
pipe; the submit now reads /dev/null.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The script required a pre-generated <repo>/ha/data (cluster.bin + TLS
certs) that ee8473bd removed, and called deploy-ha.sh with that dir as an
extra argument; deploy-ha.sh now generates both into the deploy dir and
takes <binary> <top-dir> <peers>. It failed at its first precondition in
every checkout.
Behind that, both committed-LSN boundaries still assumed bug-845's doubled
submits (1 + 2*N_JOBS): the commit wait never reached it and walcompare
compared an LSN past every member's WAL. Since that fix each job is one
WAL entry, so the boundary is 1 + N_JOBS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two xce-build.sh runs against the same build dir raced: one run's
setup_build rm -rf hit files the other was writing ('Directory not
empty'), or deleted a binary the other had just built and tested.
xce-build.sh now takes an exclusive per-build-dir lock right after cd-ing
into src/ and holds it until it exits; a second run against the same dir
waits, runs against different dirs are unaffected. The lock
(src/tools/build-lock.sh) is an mkdir lock holding the owner PID, outside
the build dir, keyed by a hash of its absolute path; it needs only
coreutils (the nix dev shell has no flock) and takes over a lock whose
owner died. src/tools/test-build-lock.sh tests it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
zarray_get_ex() checked only idx >= caps, so a negative index read
array->data[idx] out of bounds -- reachable through a resource whose
registration failed and kept idx == -1 (xcelimit.c add_vres()). It now
returns NULL for idx < 0.
repl_info_resource_one() bounded a shared resource's slot by
sharedInst.count / sharedInUse.count, the number of populated slots, not
the capacity: a populated slot above that count showed as value=(none).
It now relies on zarray_get()'s own bounds check.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_sim_driver_run() started the picker and allocator threads on
&sim->core and returned with them still running; only
sched_sim_driver_fini() stopped them, and 24 test files (35 call sites)
run the driver on a stack-local sim without calling it. The leftover
threads kept a pointer into the dead frame, and ASAN occasionally caught
one reading it (stack-use-after-return in cbuf_get) in
sched_sim_verify_test and sched_sim_record_output_test.
sched_sim_driver_run() now wraps the loop and stops both threads on
every return (new sched_sim_driver_stop_threads(), shared with fini). If
the allocator fails to start, the running picker is stopped too. The
stop no longer pushes a jobid 0 to wake the allocator: cbuf_get/put wait
in pthread_cond_wait, a cancellation point, and the fake end-of-session
was visible to the allocator.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A promoted vote endpoint carries a peer index relative to the votes
array (which leaves out the primary); a late-joiner's heartbeat endpoint
carries the absolute index. repl_ack_decode_and_dispatch recorded state
reports under the raw index, so with ctrl-1 primary, ctrl-3's vote
endpoint (relative 1) and a rejoined ctrl-2's heartbeat endpoint
(absolute 1) shared standby_reported_*[1]. Each handler then acted on
the other's position over its own endpoint: the snapshot meant for the
empty, rejoining ctrl-2 went to the caught-up ctrl-3, and ctrl-2 never
caught up.
Resolve the absolute index with ha_repl_ack_abs_peer_idx (as the
WAL-ack branch already did) for the state report and for
ha_walack_on_ack, and size catchup_inflight by MAX_HA_CANDIDATES so
every absolute index fits.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With Nagle on, the small TLS record that ends a burst -- a packet header,
the last short chunk of an HA snapshot, a reply -- waited for the peer's
delayed ACK, ~40 ms on Linux: seen on 5 of 40 snapshot transfers. Set
TCP_NODELAY on every netio TCP socket, outbound (socket_create) and
accepted (socket_accept).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SSL_get_error() reports SSL_ERROR_SSL whenever the calling thread's
OpenSSL error queue is non-empty. io_worker serves every endpoint from
one thread, so an error left by another connection -- a ctrlstatus
client's EOF queues 'unexpected eof while reading' in OpenSSL 3 -- turned
the next plain would-block SSL_read on a healthy connection into a fatal
error. handle_error() then tore that connection down; mid snapshot
transfer this left a rejoining standby with a truncated snap.part until
its liveness timeout forced a reconnect and a fresh transfer.
Clear the queue before each SSL_read/SSL_write/SSL_do_handshake so the
classification is about that call only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test_fmt_limit_units_percent_limit_ignores_units_mode failed on every
run: fmt_limit_units() widened the float limit value before scaling it,
so 0.8 printed as "80.000001%". The product is now taken in float and
widened after, which gives the configured 80.
The failure never counted: test_libxce_retry(), one group of test.c's
single runner, ran its own UNITY_BEGIN()/UNITY_END(), which reset Unity's
counters -- every failure in a group before it was dropped and
xcelib_test exited 0. With the pair removed the runner reports 695 tests;
this was the only hidden failure.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The primary_resreq action ignored parse_resreq_primary()'s result, so a
primary whose select (or rusage) clause failed to parse was accepted
without that clause: select[type==] or select[mem>] matched every host.
A malformed clause now fails the whole resreq, and parse_resreq() returns
the clause's own message.
A select naming an unregistered resource is still dropped, not an
error: parse_select() reports it as -ENOENT (new select_context_t
.unknown_resource) and the action lets it pass, because jobs rely on it
-- the scheduler's own "type == any" default among them when no "type"
resource is registered (making it fatal failed 36 scheduler suites).
Test: test_resreq_primary_select_error_fails_parse RED before (all
accepted), GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
build_wire_job() filled the saved job's submit->resreq with
format_resreq()'s human-readable dump ("Primary:\n Tasks: ..."). The
loader's parser accepted that as a primary with neither select nor
rusage, so every job restored from a snapshot (a replay's start snapshot,
REPL snapshots) lost both.
A scheduler job now keeps the text its resreq was parsed from
(xce_sched_job_t.resreq_str, set at ingest, JOB_UPDATE and snapshot load,
freed at teardown) and the snapshot saves that; a job without it restores
as the loader's "type == any" default.
Test: test_snapshot_round_trip_keeps_job_resreq_text RED before (saved
text was the dump), GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_init_queues() built a new xce_sched_queue_t for every configured
queue on each config apply; for an existing name the add failed, and the
unregistered queue was handed to the picker (which then tracked it
instead of the registered one) and leaked, while the registered queue
kept its old settings. It now updates the registered queue in place, as
QUEUE_UPDATE's core_upsert_queue() does.
sched_init_config() and the core's CONFIG_PUSH replaced core->config
without freeing the previous one. The core owns its config, and nothing
points into a replaced one (queues, resources, groups and limits built
from it are copies), so it is freed once the new one is applied.
Test: test_config_reapply_updates_queue_in_place_and_frees_old RED before
(priority stayed 1), GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_sched_driver_run_normal_mode_round() drained every decision, handed it
to the recorder and the decision callback, and never freed the struct
make_*_decision() allocates: 64 bytes per decision for the life of the
scheduler. The struct is freed after the callback returns; its allocation
or pend reason belongs to the job, and the callback and recorder copy
what they keep.
Test: test_round_frees_decision_after_callback (ASAN shadow check) RED
before, GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- kill_sched_resval() (every resource-value array's item killer) frees
the value, not only a string's buffer: each replaced or torn-down value
leaked its struct, including every WORKERS_PUSH value update.
- put_resvals() shares the value it creates with later entries for the
same index in one call; it created a second one that replaced (and now
freed) the first while still writing through it.
- JOB_UPDATE and the duplicate-JOBS_PUSH path kill_resreq() the old parsed
resreq, and free the rejected job's schedobj (xce_core_free_schedobj(),
exported from core.c), instead of freeing only the containers.
- The previous attempt's (or ALTER alternative's) pend reason is freed when
a new one replaces it.
- kill_resreq_ctx() frees the parser context's error message, and the
primary-resreq PEG action frees an earlier primary's message first.
- write_snapshot_entry() frees the snapshot it builds (config and resource
objects detached, they are the core's); the save side strdup()s its
strings so the generated free_SnapshotData() applies.
Test: test_cli_batch_replay_exits_clean_under_leak_detection
(test_sim_discrete_clock.c) runs the real binary on a two-session fixture
with detect_leaks=1 and requires exit 0: RED before (exit 1), GREEN now.
All nine HA-soak recordings replay with exit 0 and no LeakSanitizer report.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A failed key_add() runs the table's datakiller on the rejected value, and
callers keep using it (JOBS_PUSH merges a duplicate job into the existing
one; sched_init_queues() re-adds every queue on each config apply), so the
job, queue, worker and resource datakillers must stay shallow. Part 4 made
kill_sched_job() free the whole job; that is reverted. Instead
xce_sched_core_fini() collects each table's entries, destroys the table,
then frees each entry with a full destructor (core.c). The resource table
is released the same way (a reference per resobj).
Also:
- dup_val() tested which == XCE_VAL_STRING (0, the resource value type,
equal to Value_u_ivalue32): real string values were shallow-copied and a
nonzero int32 went to makeString() as a pointer. It now tests
Value_u_svalue and strdup()s, as free_Value() expects.
- The shared/host satisfy checks re-initialize the unmeet vector they are
given; the allocators passed the ctx's live one, dropping its buffers on
every candidate host. The callers finalize it first.
- add_vres() registers a resobj via makeResource(), like every other
resource, instead of a bare calloc(); test_limit.c drops it with
dereference().
- plugin_register_core() rejects a duplicate name before init_fn/key_add,
since a failed add now finalizes (frees) the plugin and callers free it
on failure.
- sharedInUse gets kill_sched_resval, as put_resvals() fills it.
Exit-time leak on the reproduction replay 7,469 -> 928 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- kill_sched_job() (the job table's datakiller) frees the whole job:
schedobj, parsed resreq, allocation, accounts and rusages; new
kill_sched_obj() frees the schedobj's names, limit lists, per-primary
wrappers and their pend reasons, and its last allocation/reason.
- The simulator frees each decision struct after draining it (live and
reconciled); the alloc/pend reason it points at stays the job's.
- The builtin_fcfs heap plugin gets a fini_fn that also frees the plugin;
builtin_fcfs_fini() still frees only the picker state (tests run it on
stack plugins).
- xce_sched_core_fini() frees the two pend_updates queues both drivers
calloc().
Exit-time leak on the reproduction replay 11,426 -> 7,469 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_sched_core_load_snapshot() keeps the snapshot's config as core->config
and copies everything else, so the simulator now moves the config into the
core and frees the rest of the decoded SnapshotEntry. xce_sched_core_fini()
frees core->config last, after the plugins and group managers that may
point into it; nothing freed it in either mode. Exit-time leak on the
reproduction replay 25,807 -> 11,426 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sim_process_replay_event_message() now hands the decoded replay event to
the caller, which keeps it beside the translated core event and frees it
after xce_core_push_events() with driver.c's release_translated_payload()
rules: a pushed config is kept by the core, and a rusages report is kept
by the job it found. An untranslatable event is freed at once, and events
after the last session are freed at the end. The payloads were an
"accepted leak" per event; exit-time leak on the reproduction replay
75,495 -> 25,807 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A batch --sim replay under ASAN exited 1 on 17.2 MB of LeakSanitizer
reports. This removes all but 75 KB:
- sched_sim_driver_fini() (called by main after a clean run) cancels and
joins the picker/allocator threads like sched_fini(), closes the replay,
alternate-events and output files, and finalizes the core. The threads'
ids now live in core.jobpicker/allocator, as in normal mode.
- xce_sched_core_fini() destroys the plugin, job, worker, queue, user and
project registries and the shared-resource arrays it creates.
- The seqjob/parjob sched_make_allocs() stand-ins set up only lock, nextid
and the aliased resmgr. Each ran a full xce_sched_core_init(), which also
replaced the process-wide string and selector tables mid-startup.
- plugin_unregister(), the plugin registry's datakiller, only runs fini_fn;
it key_del()'d from the registry being destroyed and skipped plugins
registered with plugin_register_core().
- The limit plugin's usage-instance table used kill_limit as datakiller on
xce_vres_t records (bad free on destroy); kill_vres now frees the record.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A COMPOUND/ALTER job's primaries share one ctx. The job-level prepare pass
reset only alloc_req.shared, and the allocators append their slots want
into alloc_req.host, so a later primary's host pre-filter (and a global
limit on slots) read the previous primary's last per-host want: a 1-task
primary after a 4-task one was filtered on slots:4 and pended although a
host with its one free slot existed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Add test_sim_replay_reproduces_recorded_alloc_from: a recorded DISPATCH at
an epoch-scale beginRound with a sub-second part replays at the default
simulation settings with no --sim-verify mismatch, and the live allocation's
from equals beginRound in whole seconds. Share the dispatch fixture setup
of the existing tests through two helpers.
Live: 2-round HA soak recordings (11 DISPATCH decisions) replayed with
--sim-verify and no --sim-speed report 0 mismatches.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
make_alloc() and make_host_alloc() assigned strdup() results unchecked,
so under memory exhaustion they returned a record whose resname/reserve/
host was NULL; merge_job_alloc()'s same_alloc_slot() (bug-1863) then
strcmp()'d it (ASAN SEGV at xcealloc.c:337). make_alloc() also passed a
NULL resname straight to strdup().
- xcealloc.c: make_alloc()/make_host_alloc() reject a NULL name (EINVAL)
and check every strdup(), freeing what they built and returning NULL
(ENOMEM); on failure make_host_alloc()'s caller keeps `allocs`.
dup_host_alloc() frees its partial copy on failure instead of leaking.
- scheduler.c sched_make_allocs(): a record that cannot be built fails
the whole array (NULL, ENOMEM) instead of leaving a NULL entry.
- seqjob/parjob my_alloc(): check every constructor; on failure free
what was built (parjob also rolls back the remaining claims) and
return -ENOMEM with ctx->alloc NULL -- the caller moves ctx.alloc onto
the job whatever the rc.
Tests: new xcealloc_oom_test (8, --wrap=strdup) and
sched_make_allocs_oom_test (real scheduler_srcs + --wrap=strdup); a new
-ENOMEM test in the seqjob and parjob allocator suites. All RED before
(SEGV / "Expected NULL" / "Expected -12 Was 1"), GREEN after.
xce-build.sh all asan: 115/115 suites OK, Go tests OK.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its
term-1 election before its peers ran; connect failures count as "no"
votes (bug-1423), so it lost at once and became a follower with no
leader. ctrl-2/ctrl-3 hold no config and are voters only (by design),
so nothing retried until ctrl-1's 10 s standby liveness timeout: every
cold start had no primary for ~10 s.
- ha_note_leaderless() marks a config-holding node that becomes a
follower with no leader known; the liveness poll then retries the
election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to
heartbeat_timeout_ms. Primary contact or becoming leader clears it.
- ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity
then only holds the watchdog's arm time, and the failover gates took
it for a live primary.
Live cold start: primary 1.7 s after the first node started (was ~10 s).
bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a
standby-only counter a primary never advances, so every primary showed
"appliedLsn: 0". A primary now reports persist->last_applied_lsn. The
controller-status design doc's "already correct in every role" note is
corrected.
Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c
(RED: expected 4900, was 0); controller suite 1256/0. HA soaks:
sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero
job loss).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- bug-1872 (worker): "Failed to find any disk statistics in
/proc/diskstats" ERROR every 15 s on an NVMe-only host. The whole-disk
filter guessed from the name and skipped anything ending in a digit or
containing 'p' -- every nvmeXnY, mmcblkN, even sdp -- so io was never
reported. diskstats.h now counts a device only when
/sys/block/<name>/device exists (real disks; not partitions, loop, dm,
md, zram), and a host with no disk warns once.
- bug-1873 (worker): "Failed to get worker pid" WARN on job start. MainPID
reads 0 until systemd has forked the new transient unit; xcecd.c already
retries, so the miss is DEBUG and the final failure names the unit.
- bug-1874 (scheduler): deprecation WARN for the legacy xce_sched_fcfs on
every start, though builtin_fcfs is the picker. Registration still wraps
a job_elect_fn plugin but no longer warns; selecting one warns once.
- bug-1875 (apiserver): "Not running as root -- PAM authentication will not
work" with the default provider config_file; now only for provider pam.
- bug-1876 (controller): standbys logged "cluster_init_phase2 ... failed
(rc=-22)" WARN every 3 s while waiting for the primary's config push,
a state the function itself calls normal. It returns -EAGAIN for that
wait; callers log it at INFO, other failures still WARN.
- bug-1877 (controller): "WAL/snapshot stored unencrypted" warned without
looking. persist_path_encrypted() checks fscrypt (statx) and dm-crypt
(sysfs dm/uuid CRYPT-, through LVM slaves); the warning stays when the
storage is not shown to be encrypted (e.g. a plain /tmp).
Tests: worker diskstats_test (2 of 4 RED on the old filter with pc13v's
/proc/diskstats), controller test_phase2_cold_boot and
test_persist_encryption, scheduler picker tests, apiserver
TestRootWarning. Full ASAN build: all 113 suites pass (controller 319/0,
harness 141/0).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stack overflows on user input:
- bug-1702 jobrun -m: host segment strncpy'd into a 256-byte buffer
unbounded (ASAN stack-buffer-overflow); now cmdutil_split_host_tasks()
refuses an over-long host.
- bug-1703 jobsub -n: min strncpy'd into a 64-byte buffer; now
cmdutil_parse_ntasks() parses in place, 1 <= min <= max.
- resupdate's 16-entry vals[] had no bound either; a 17th -n overflowed.
Wrong or silent results:
- bug-1574 resupdate type/model: set_resval() had no string case, left the
entry's name uninitialized, and crashed the encoder. Moved to
resval_typeresolve.c as resval_fill(), which fills every type.
- bug-1705 jobsub -x was overwritten (and leaked) by the default exdata;
cmdutil_default_exdata() only fills an unset one. jobsub's
execuser/exechome/execcwd copies leaked on every call.
- bug-1707 a trailing -n without -v was dropped silently; resupdate also
returned 0 after its RPC failed and "succeeded" with nothing to send.
- bug-1708 jobkill -s SIGKILL became signal 0 and was sent.
- bug-1709 jobinfo exited 0 with no -j, a malformed -j, or no hits.
- bug-1710 hello -c 0 sent nothing and reported success.
- bug-1712 a duplicate -v was reported as "no res name found".
- bug-1777 workerlist showed join-time maxmem/maxswap/maxtmp, not the
effective values (hosts.yaml overrides) the reply already carries in
resvals_to_sched and the scheduler uses.
- bug-1793 join's maxswap/maxtmp defaults were 1 KB / 2 KB in bytes; now
1 GiB / 2 GiB, and -m/-s/-t no longer overflow atoi() above 2 GB.
Error reporting and leaks:
- bug-1796 RPC failures printed perror() of a stale errno ("Connection
refused" / "Invalid argument" / "Inappropriate ioctl for device" for
the same failure). cmdutil_rpc_error() prints the RPC's own code
(E_XCE_* included) at all 19 sites.
- bug-1797 qctrl leaked the queue name on the RPC-error path; repeated
-q/-h leaked the earlier copy.
New src/harness/cmdutil.c (+ cmdutil_unit_test, 14 cases); 9 new
test_test_cli.c cases, 8 RED on the old xce_test; new resval_fill,
worker_effective_attr and join-default tests (join ones RED on the old
join_worker_args.c). Harness suite 6/6 (asan), harness_unit_test 141/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
add_worker strndup's launch_type and the four queue/job pre/post-exec
commands out of the startinfo, but remove_worker only freed exechome,
execcwd, execuser, job_cmd, pids and exdata, so every finished job leaked
up to five strings (LeakSanitizer at xcecd_virtual exit reported the
launch_type one).
New test in test_finished_job_retention.c compares ASAN's live-heap byte
count across an add/remove round trip (the worker suite runs with
detect_leaks=0): RED leaked exactly the 58 bytes of those five strings,
GREEN after. Worker suite 12/12.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
schedAdmin, jobStateQuery, jobStateReport and the new notifyAcked
(bug-1741) all logged as "opcode unknown" in the controller's request
trace -- e.g. the bug-1741 factstream soak showed 373 "unknown" requests
that were really acknowledged job reports.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A worker reported job start, finish and failure with a fire-and-forget
OPCode_notify the controller never answers, so a report lost while the
worker stayed connected (the live incident: 6 of 7 finish reports in a
burst) left the job RUN forever; the rejoin resync (165fc158) only
recovers reports lost around a reconnect.
- controller.capnp: OPCode_notifyAcked -- the same NotifyData body,
answered. A separate opcode because replies are matched through the
client's single wbuf, not by sequence number: answering plain notifies
would feed stale statuses to the next synchronous request.
- controller (cluster.c): do_notify_acked() applies the fact like
do_notify, then waits on the commit barrier and replies 0 only once it
is majority-committed. A barrier failure (the fact is still appended and
retained) replies -E_XCE_NOT_READY, so the worker retries in place
instead of rotating away from a healthy primary (cf. bug-1850). Admitted
by the no-quorum fact-bearing gate like OPCode_notify.
- libxce: xce_notify_acked_req/xce_notify_acked (send, wait for the
status) and xcecli_notify_acked (XCECLI_FAILOVER_CALL retry/failover).
xcecli_notify_jobstart/jobfinish/jobfail use it; on -E_XCE_BAD_OPCODE
(a controller predating the opcode) they fall back to the plain
fire-and-forget notify, so workers and controllers can be upgraded in
either order.
- Resends are safe: START/FINISH are duplicate-guarded and a repeated FAIL
yields the same state.
- worker: logs a WARN when a report is still unacknowledged after the
retry budget (the rejoin resync remains the backstop). These calls run
on the worker daemon's IPC callback, so while the controller lacks
quorum other jobs' IPC messages wait (buffered, not lost) up to the
retry budget.
Tests: test_notify_acked_reply_only_after_commit and
test_notify_acked_withheld_replies_not_ready (no reply before the covering
ACK; -E_XCE_NOT_READY with the fact retained when ACKs are withheld).
Written with the opcode, so no separate RED run. Full /xce-build all:
111 suites pass; controller suite 1245/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Regression from 63fd3d2a (bug-1870), found by the --config-upgrade HA soak
(force-elect: "submit on {ctrl-2,ctrl-1} failed", ~640 standby NACKs;
bisected: 725301dd passes, 88c4310e fails).
load_snapshot() seeded current_lsn = checkpoint_lsn, but current_lsn is
the NEXT lsn to write and the snapshot covers 1..checkpoint_lsn. The
off-by-one was always there, hidden because every checkpoint also wrote a
WAL_CHECKPOINT marker ABOVE the checkpoint, so the WAL scan moved
current_lsn past it. With the marker gone, a standby whose WAL held
nothing above its config-marker checkpoint (LSN 11) recovered with
current_lsn=11: its raft layer counted entry 11 as held (ae_already_have,
not written), and persist_wal_write_replicated() then refused entry 12 as
a gap (ae_append_persist_fail) on every resend -- stuck at 11 forever.
Both of load_snapshot's commit points now set current_lsn =
checkpoint_lsn + 1, matching the apply-side catch-up path (lsn + 1).
New test_recover_from_snapshot_without_wal_resumes_after_it (RED before:
current_lsn 3, not 4). Two snapshot-entity tests asserted the old
current_lsn == checkpoint_lsn and now expect +1. Controller suite 1245/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
build_queues() only ran at boot, so a queue a live config upgrade added
had no runtime object in c->qmgr (and the scheduler/apiserver never heard
of it) until the controller restarted. The checkpoint-ordering half of
bug-1469 was fixed in 959b0cda; this is the remaining half.
- reconcile_config_queues(c, notify): creates every queue in the active
config that has no runtime object, leaves existing ones untouched,
returns the number created.
- config_activate_committed() calls it (with notify) after every
successful activation -- verified-marker and no-marker paths alike.
- persist_notify_queue_updated_no_wal(): fires HOOK_DETAIL_QUEUE_UPDATE
under the t_apply_seam_notifying guard, so the scheduler/apiserver are
told while no WAL_QUEUE_CTRL is appended -- config queues are
config-owned, built on every member from the committed config.
- build_queues() now delegates to it without notify, so boot and
activation are idempotent in either order.
New test_config_queue_reconcile.c (RED before: the added queue was never
created): creates the added queue, leaves the existing object untouched,
notifies once, and a second reconcile/build_queues adds nothing.
Controller suite passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Root cause of the "spurious duplicate reconnect" (angle a): zarray_del()
NULLs the slot and decrements count without compacting, so a live entry
can sit at any index < caps. Every standby_endpoints loop bounded by
`count` missed it. Live evidence (--config-upgrade-wal-trim soak,
2026-09-26): ctrl-2 won, promoted vote eps [ctrl-1 (dead), ctrl-3]; the
dead one's removal left [NULL, ctrl-3] with count 1, so the primary
- stopped heartbeating ctrl-3 (primary_heartbeat_fn's send loop),
- re-dialed it ("peer[2] not connected -- initiating outbound connect"
9s later, for a peer that never dropped) and registered a second,
"late-joiner" channel that burned a MAX_STANDBYS repl slot,
- could not evict the stale channel (on_standby_connected's dedup scan),
- and skipped it in config pushes and the ctrlhello broadcast, where a
hole below count also handed endpoint_ref() a NULL.
The re-dial landed in the freed slot 0, which is what made ctrl-3 visible
again -- the "spurious" reconnect was masking the lost heartbeats.
All index scans of standby_endpoints (ha.c heartbeat, sweep, dedup, two
removals; cluster.c removal and ctrlhello; config_repl.c push collection)
now run to `caps` and skip NULL slots; the push collects exactly the
non-NULL entries. `count` stays where it means "how many connected".
New test_reconnect_dedups_channel_behind_deleted_slot (RED before: the
stale channel behind the hole was never evicted). Controller suite 1241/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A winning RequestVoteReply is dispatched through peer_ctrl_classify, which
returned VTASK_LIVENESS for all peer-control traffic -- the never-blocks,
never-admission-gated class. The reply runs raft_cb_become_leader, whose
tail persist_apply_pump_run_sync() does bounded blocking WAL pread()s (it
cannot be made async without reopening the bug-1043-family job loss on a
freshly promoted primary). The fdatasync half of bug-1433 was removed in
40f510fb; this pread was the tracked residual.
peer_ctrl_classify now returns VTASK_LIVE_BLOCK while the node is
CTRL_ELECTING -- its peer-control traffic is then vote/PreVote replies,
election work like inbound RequestVote, which already uses LIVE_BLOCK.
As PRIMARY/STANDBY the traffic is WAL/config ACKs and stays VTASK_LIVENESS,
so a quorum of acks is never admission-gated (R4.5); a candidate has no
commit to advance.
New test_peer_ctrl_classify_vote_replies_while_electing (RED before).
Controller suite 1240/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
