test_fmt_limit_units_percent_limit_ignores_units_mode failed on every
run: fmt_limit_units() widened the float limit value before scaling it,
so 0.8 printed as "80.000001%". The product is now taken in float and
widened after, which gives the configured 80.
The failure never counted: test_libxce_retry(), one group of test.c's
single runner, ran its own UNITY_BEGIN()/UNITY_END(), which reset Unity's
counters -- every failure in a group before it was dropped and
xcelib_test exited 0. With the pair removed the runner reports 695 tests;
this was the only hidden failure.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The primary_resreq action ignored parse_resreq_primary()'s result, so a
primary whose select (or rusage) clause failed to parse was accepted
without that clause: select[type==] or select[mem>] matched every host.
A malformed clause now fails the whole resreq, and parse_resreq() returns
the clause's own message.
A select naming an unregistered resource is still dropped, not an
error: parse_select() reports it as -ENOENT (new select_context_t
.unknown_resource) and the action lets it pass, because jobs rely on it
-- the scheduler's own "type == any" default among them when no "type"
resource is registered (making it fatal failed 36 scheduler suites).
Test: test_resreq_primary_select_error_fails_parse RED before (all
accepted), GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
build_wire_job() filled the saved job's submit->resreq with
format_resreq()'s human-readable dump ("Primary:\n Tasks: ..."). The
loader's parser accepted that as a primary with neither select nor
rusage, so every job restored from a snapshot (a replay's start snapshot,
REPL snapshots) lost both.
A scheduler job now keeps the text its resreq was parsed from
(xce_sched_job_t.resreq_str, set at ingest, JOB_UPDATE and snapshot load,
freed at teardown) and the snapshot saves that; a job without it restores
as the loader's "type == any" default.
Test: test_snapshot_round_trip_keeps_job_resreq_text RED before (saved
text was the dump), GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_init_queues() built a new xce_sched_queue_t for every configured
queue on each config apply; for an existing name the add failed, and the
unregistered queue was handed to the picker (which then tracked it
instead of the registered one) and leaked, while the registered queue
kept its old settings. It now updates the registered queue in place, as
QUEUE_UPDATE's core_upsert_queue() does.
sched_init_config() and the core's CONFIG_PUSH replaced core->config
without freeing the previous one. The core owns its config, and nothing
points into a replaced one (queues, resources, groups and limits built
from it are copies), so it is freed once the new one is applied.
Test: test_config_reapply_updates_queue_in_place_and_frees_old RED before
(priority stayed 1), GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_sched_driver_run_normal_mode_round() drained every decision, handed it
to the recorder and the decision callback, and never freed the struct
make_*_decision() allocates: 64 bytes per decision for the life of the
scheduler. The struct is freed after the callback returns; its allocation
or pend reason belongs to the job, and the callback and recorder copy
what they keep.
Test: test_round_frees_decision_after_callback (ASAN shadow check) RED
before, GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- kill_sched_resval() (every resource-value array's item killer) frees
the value, not only a string's buffer: each replaced or torn-down value
leaked its struct, including every WORKERS_PUSH value update.
- put_resvals() shares the value it creates with later entries for the
same index in one call; it created a second one that replaced (and now
freed) the first while still writing through it.
- JOB_UPDATE and the duplicate-JOBS_PUSH path kill_resreq() the old parsed
resreq, and free the rejected job's schedobj (xce_core_free_schedobj(),
exported from core.c), instead of freeing only the containers.
- The previous attempt's (or ALTER alternative's) pend reason is freed when
a new one replaces it.
- kill_resreq_ctx() frees the parser context's error message, and the
primary-resreq PEG action frees an earlier primary's message first.
- write_snapshot_entry() frees the snapshot it builds (config and resource
objects detached, they are the core's); the save side strdup()s its
strings so the generated free_SnapshotData() applies.
Test: test_cli_batch_replay_exits_clean_under_leak_detection
(test_sim_discrete_clock.c) runs the real binary on a two-session fixture
with detect_leaks=1 and requires exit 0: RED before (exit 1), GREEN now.
All nine HA-soak recordings replay with exit 0 and no LeakSanitizer report.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A failed key_add() runs the table's datakiller on the rejected value, and
callers keep using it (JOBS_PUSH merges a duplicate job into the existing
one; sched_init_queues() re-adds every queue on each config apply), so the
job, queue, worker and resource datakillers must stay shallow. Part 4 made
kill_sched_job() free the whole job; that is reverted. Instead
xce_sched_core_fini() collects each table's entries, destroys the table,
then frees each entry with a full destructor (core.c). The resource table
is released the same way (a reference per resobj).
Also:
- dup_val() tested which == XCE_VAL_STRING (0, the resource value type,
equal to Value_u_ivalue32): real string values were shallow-copied and a
nonzero int32 went to makeString() as a pointer. It now tests
Value_u_svalue and strdup()s, as free_Value() expects.
- The shared/host satisfy checks re-initialize the unmeet vector they are
given; the allocators passed the ctx's live one, dropping its buffers on
every candidate host. The callers finalize it first.
- add_vres() registers a resobj via makeResource(), like every other
resource, instead of a bare calloc(); test_limit.c drops it with
dereference().
- plugin_register_core() rejects a duplicate name before init_fn/key_add,
since a failed add now finalizes (frees) the plugin and callers free it
on failure.
- sharedInUse gets kill_sched_resval, as put_resvals() fills it.
Exit-time leak on the reproduction replay 7,469 -> 928 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- kill_sched_job() (the job table's datakiller) frees the whole job:
schedobj, parsed resreq, allocation, accounts and rusages; new
kill_sched_obj() frees the schedobj's names, limit lists, per-primary
wrappers and their pend reasons, and its last allocation/reason.
- The simulator frees each decision struct after draining it (live and
reconciled); the alloc/pend reason it points at stays the job's.
- The builtin_fcfs heap plugin gets a fini_fn that also frees the plugin;
builtin_fcfs_fini() still frees only the picker state (tests run it on
stack plugins).
- xce_sched_core_fini() frees the two pend_updates queues both drivers
calloc().
Exit-time leak on the reproduction replay 11,426 -> 7,469 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_sched_core_load_snapshot() keeps the snapshot's config as core->config
and copies everything else, so the simulator now moves the config into the
core and frees the rest of the decoded SnapshotEntry. xce_sched_core_fini()
frees core->config last, after the plugins and group managers that may
point into it; nothing freed it in either mode. Exit-time leak on the
reproduction replay 25,807 -> 11,426 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sim_process_replay_event_message() now hands the decoded replay event to
the caller, which keeps it beside the translated core event and frees it
after xce_core_push_events() with driver.c's release_translated_payload()
rules: a pushed config is kept by the core, and a rusages report is kept
by the job it found. An untranslatable event is freed at once, and events
after the last session are freed at the end. The payloads were an
"accepted leak" per event; exit-time leak on the reproduction replay
75,495 -> 25,807 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A batch --sim replay under ASAN exited 1 on 17.2 MB of LeakSanitizer
reports. This removes all but 75 KB:
- sched_sim_driver_fini() (called by main after a clean run) cancels and
joins the picker/allocator threads like sched_fini(), closes the replay,
alternate-events and output files, and finalizes the core. The threads'
ids now live in core.jobpicker/allocator, as in normal mode.
- xce_sched_core_fini() destroys the plugin, job, worker, queue, user and
project registries and the shared-resource arrays it creates.
- The seqjob/parjob sched_make_allocs() stand-ins set up only lock, nextid
and the aliased resmgr. Each ran a full xce_sched_core_init(), which also
replaced the process-wide string and selector tables mid-startup.
- plugin_unregister(), the plugin registry's datakiller, only runs fini_fn;
it key_del()'d from the registry being destroyed and skipped plugins
registered with plugin_register_core().
- The limit plugin's usage-instance table used kill_limit as datakiller on
xce_vres_t records (bad free on destroy); kill_vres now frees the record.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A COMPOUND/ALTER job's primaries share one ctx. The job-level prepare pass
reset only alloc_req.shared, and the allocators append their slots want
into alloc_req.host, so a later primary's host pre-filter (and a global
limit on slots) read the previous primary's last per-host want: a 1-task
primary after a 4-task one was filtered on slots:4 and pended although a
host with its one free slot existed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Add test_sim_replay_reproduces_recorded_alloc_from: a recorded DISPATCH at
an epoch-scale beginRound with a sub-second part replays at the default
simulation settings with no --sim-verify mismatch, and the live allocation's
from equals beginRound in whole seconds. Share the dispatch fixture setup
of the existing tests through two helpers.
Live: 2-round HA soak recordings (11 DISPATCH decisions) replayed with
--sim-verify and no --sim-speed report 0 mismatches.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
make_alloc() and make_host_alloc() assigned strdup() results unchecked,
so under memory exhaustion they returned a record whose resname/reserve/
host was NULL; merge_job_alloc()'s same_alloc_slot() (bug-1863) then
strcmp()'d it (ASAN SEGV at xcealloc.c:337). make_alloc() also passed a
NULL resname straight to strdup().
- xcealloc.c: make_alloc()/make_host_alloc() reject a NULL name (EINVAL)
and check every strdup(), freeing what they built and returning NULL
(ENOMEM); on failure make_host_alloc()'s caller keeps `allocs`.
dup_host_alloc() frees its partial copy on failure instead of leaking.
- scheduler.c sched_make_allocs(): a record that cannot be built fails
the whole array (NULL, ENOMEM) instead of leaving a NULL entry.
- seqjob/parjob my_alloc(): check every constructor; on failure free
what was built (parjob also rolls back the remaining claims) and
return -ENOMEM with ctx->alloc NULL -- the caller moves ctx.alloc onto
the job whatever the rc.
Tests: new xcealloc_oom_test (8, --wrap=strdup) and
sched_make_allocs_oom_test (real scheduler_srcs + --wrap=strdup); a new
-ENOMEM test in the seqjob and parjob allocator suites. All RED before
(SEGV / "Expected NULL" / "Expected -12 Was 1"), GREEN after.
xce-build.sh all asan: 115/115 suites OK, Go tests OK.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its
term-1 election before its peers ran; connect failures count as "no"
votes (bug-1423), so it lost at once and became a follower with no
leader. ctrl-2/ctrl-3 hold no config and are voters only (by design),
so nothing retried until ctrl-1's 10 s standby liveness timeout: every
cold start had no primary for ~10 s.
- ha_note_leaderless() marks a config-holding node that becomes a
follower with no leader known; the liveness poll then retries the
election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to
heartbeat_timeout_ms. Primary contact or becoming leader clears it.
- ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity
then only holds the watchdog's arm time, and the failover gates took
it for a live primary.
Live cold start: primary 1.7 s after the first node started (was ~10 s).
bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a
standby-only counter a primary never advances, so every primary showed
"appliedLsn: 0". A primary now reports persist->last_applied_lsn. The
controller-status design doc's "already correct in every role" note is
corrected.
Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c
(RED: expected 4900, was 0); controller suite 1256/0. HA soaks:
sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero
job loss).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- bug-1872 (worker): "Failed to find any disk statistics in
/proc/diskstats" ERROR every 15 s on an NVMe-only host. The whole-disk
filter guessed from the name and skipped anything ending in a digit or
containing 'p' -- every nvmeXnY, mmcblkN, even sdp -- so io was never
reported. diskstats.h now counts a device only when
/sys/block/<name>/device exists (real disks; not partitions, loop, dm,
md, zram), and a host with no disk warns once.
- bug-1873 (worker): "Failed to get worker pid" WARN on job start. MainPID
reads 0 until systemd has forked the new transient unit; xcecd.c already
retries, so the miss is DEBUG and the final failure names the unit.
- bug-1874 (scheduler): deprecation WARN for the legacy xce_sched_fcfs on
every start, though builtin_fcfs is the picker. Registration still wraps
a job_elect_fn plugin but no longer warns; selecting one warns once.
- bug-1875 (apiserver): "Not running as root -- PAM authentication will not
work" with the default provider config_file; now only for provider pam.
- bug-1876 (controller): standbys logged "cluster_init_phase2 ... failed
(rc=-22)" WARN every 3 s while waiting for the primary's config push,
a state the function itself calls normal. It returns -EAGAIN for that
wait; callers log it at INFO, other failures still WARN.
- bug-1877 (controller): "WAL/snapshot stored unencrypted" warned without
looking. persist_path_encrypted() checks fscrypt (statx) and dm-crypt
(sysfs dm/uuid CRYPT-, through LVM slaves); the warning stays when the
storage is not shown to be encrypted (e.g. a plain /tmp).
Tests: worker diskstats_test (2 of 4 RED on the old filter with pc13v's
/proc/diskstats), controller test_phase2_cold_boot and
test_persist_encryption, scheduler picker tests, apiserver
TestRootWarning. Full ASAN build: all 113 suites pass (controller 319/0,
harness 141/0).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stack overflows on user input:
- bug-1702 jobrun -m: host segment strncpy'd into a 256-byte buffer
unbounded (ASAN stack-buffer-overflow); now cmdutil_split_host_tasks()
refuses an over-long host.
- bug-1703 jobsub -n: min strncpy'd into a 64-byte buffer; now
cmdutil_parse_ntasks() parses in place, 1 <= min <= max.
- resupdate's 16-entry vals[] had no bound either; a 17th -n overflowed.
Wrong or silent results:
- bug-1574 resupdate type/model: set_resval() had no string case, left the
entry's name uninitialized, and crashed the encoder. Moved to
resval_typeresolve.c as resval_fill(), which fills every type.
- bug-1705 jobsub -x was overwritten (and leaked) by the default exdata;
cmdutil_default_exdata() only fills an unset one. jobsub's
execuser/exechome/execcwd copies leaked on every call.
- bug-1707 a trailing -n without -v was dropped silently; resupdate also
returned 0 after its RPC failed and "succeeded" with nothing to send.
- bug-1708 jobkill -s SIGKILL became signal 0 and was sent.
- bug-1709 jobinfo exited 0 with no -j, a malformed -j, or no hits.
- bug-1710 hello -c 0 sent nothing and reported success.
- bug-1712 a duplicate -v was reported as "no res name found".
- bug-1777 workerlist showed join-time maxmem/maxswap/maxtmp, not the
effective values (hosts.yaml overrides) the reply already carries in
resvals_to_sched and the scheduler uses.
- bug-1793 join's maxswap/maxtmp defaults were 1 KB / 2 KB in bytes; now
1 GiB / 2 GiB, and -m/-s/-t no longer overflow atoi() above 2 GB.
Error reporting and leaks:
- bug-1796 RPC failures printed perror() of a stale errno ("Connection
refused" / "Invalid argument" / "Inappropriate ioctl for device" for
the same failure). cmdutil_rpc_error() prints the RPC's own code
(E_XCE_* included) at all 19 sites.
- bug-1797 qctrl leaked the queue name on the RPC-error path; repeated
-q/-h leaked the earlier copy.
New src/harness/cmdutil.c (+ cmdutil_unit_test, 14 cases); 9 new
test_test_cli.c cases, 8 RED on the old xce_test; new resval_fill,
worker_effective_attr and join-default tests (join ones RED on the old
join_worker_args.c). Harness suite 6/6 (asan), harness_unit_test 141/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
add_worker strndup's launch_type and the four queue/job pre/post-exec
commands out of the startinfo, but remove_worker only freed exechome,
execcwd, execuser, job_cmd, pids and exdata, so every finished job leaked
up to five strings (LeakSanitizer at xcecd_virtual exit reported the
launch_type one).
New test in test_finished_job_retention.c compares ASAN's live-heap byte
count across an add/remove round trip (the worker suite runs with
detect_leaks=0): RED leaked exactly the 58 bytes of those five strings,
GREEN after. Worker suite 12/12.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
schedAdmin, jobStateQuery, jobStateReport and the new notifyAcked
(bug-1741) all logged as "opcode unknown" in the controller's request
trace -- e.g. the bug-1741 factstream soak showed 373 "unknown" requests
that were really acknowledged job reports.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A worker reported job start, finish and failure with a fire-and-forget
OPCode_notify the controller never answers, so a report lost while the
worker stayed connected (the live incident: 6 of 7 finish reports in a
burst) left the job RUN forever; the rejoin resync (165fc158) only
recovers reports lost around a reconnect.
- controller.capnp: OPCode_notifyAcked -- the same NotifyData body,
answered. A separate opcode because replies are matched through the
client's single wbuf, not by sequence number: answering plain notifies
would feed stale statuses to the next synchronous request.
- controller (cluster.c): do_notify_acked() applies the fact like
do_notify, then waits on the commit barrier and replies 0 only once it
is majority-committed. A barrier failure (the fact is still appended and
retained) replies -E_XCE_NOT_READY, so the worker retries in place
instead of rotating away from a healthy primary (cf. bug-1850). Admitted
by the no-quorum fact-bearing gate like OPCode_notify.
- libxce: xce_notify_acked_req/xce_notify_acked (send, wait for the
status) and xcecli_notify_acked (XCECLI_FAILOVER_CALL retry/failover).
xcecli_notify_jobstart/jobfinish/jobfail use it; on -E_XCE_BAD_OPCODE
(a controller predating the opcode) they fall back to the plain
fire-and-forget notify, so workers and controllers can be upgraded in
either order.
- Resends are safe: START/FINISH are duplicate-guarded and a repeated FAIL
yields the same state.
- worker: logs a WARN when a report is still unacknowledged after the
retry budget (the rejoin resync remains the backstop). These calls run
on the worker daemon's IPC callback, so while the controller lacks
quorum other jobs' IPC messages wait (buffered, not lost) up to the
retry budget.
Tests: test_notify_acked_reply_only_after_commit and
test_notify_acked_withheld_replies_not_ready (no reply before the covering
ACK; -E_XCE_NOT_READY with the fact retained when ACKs are withheld).
Written with the opcode, so no separate RED run. Full /xce-build all:
111 suites pass; controller suite 1245/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Regression from 63fd3d2a (bug-1870), found by the --config-upgrade HA soak
(force-elect: "submit on {ctrl-2,ctrl-1} failed", ~640 standby NACKs;
bisected: 725301dd passes, 88c4310e fails).
load_snapshot() seeded current_lsn = checkpoint_lsn, but current_lsn is
the NEXT lsn to write and the snapshot covers 1..checkpoint_lsn. The
off-by-one was always there, hidden because every checkpoint also wrote a
WAL_CHECKPOINT marker ABOVE the checkpoint, so the WAL scan moved
current_lsn past it. With the marker gone, a standby whose WAL held
nothing above its config-marker checkpoint (LSN 11) recovered with
current_lsn=11: its raft layer counted entry 11 as held (ae_already_have,
not written), and persist_wal_write_replicated() then refused entry 12 as
a gap (ae_append_persist_fail) on every resend -- stuck at 11 forever.
Both of load_snapshot's commit points now set current_lsn =
checkpoint_lsn + 1, matching the apply-side catch-up path (lsn + 1).
New test_recover_from_snapshot_without_wal_resumes_after_it (RED before:
current_lsn 3, not 4). Two snapshot-entity tests asserted the old
current_lsn == checkpoint_lsn and now expect +1. Controller suite 1245/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
build_queues() only ran at boot, so a queue a live config upgrade added
had no runtime object in c->qmgr (and the scheduler/apiserver never heard
of it) until the controller restarted. The checkpoint-ordering half of
bug-1469 was fixed in 959b0cda; this is the remaining half.
- reconcile_config_queues(c, notify): creates every queue in the active
config that has no runtime object, leaves existing ones untouched,
returns the number created.
- config_activate_committed() calls it (with notify) after every
successful activation -- verified-marker and no-marker paths alike.
- persist_notify_queue_updated_no_wal(): fires HOOK_DETAIL_QUEUE_UPDATE
under the t_apply_seam_notifying guard, so the scheduler/apiserver are
told while no WAL_QUEUE_CTRL is appended -- config queues are
config-owned, built on every member from the committed config.
- build_queues() now delegates to it without notify, so boot and
activation are idempotent in either order.
New test_config_queue_reconcile.c (RED before: the added queue was never
created): creates the added queue, leaves the existing object untouched,
notifies once, and a second reconcile/build_queues adds nothing.
Controller suite passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Root cause of the "spurious duplicate reconnect" (angle a): zarray_del()
NULLs the slot and decrements count without compacting, so a live entry
can sit at any index < caps. Every standby_endpoints loop bounded by
`count` missed it. Live evidence (--config-upgrade-wal-trim soak,
2026-09-26): ctrl-2 won, promoted vote eps [ctrl-1 (dead), ctrl-3]; the
dead one's removal left [NULL, ctrl-3] with count 1, so the primary
- stopped heartbeating ctrl-3 (primary_heartbeat_fn's send loop),
- re-dialed it ("peer[2] not connected -- initiating outbound connect"
9s later, for a peer that never dropped) and registered a second,
"late-joiner" channel that burned a MAX_STANDBYS repl slot,
- could not evict the stale channel (on_standby_connected's dedup scan),
- and skipped it in config pushes and the ctrlhello broadcast, where a
hole below count also handed endpoint_ref() a NULL.
The re-dial landed in the freed slot 0, which is what made ctrl-3 visible
again -- the "spurious" reconnect was masking the lost heartbeats.
All index scans of standby_endpoints (ha.c heartbeat, sweep, dedup, two
removals; cluster.c removal and ctrlhello; config_repl.c push collection)
now run to `caps` and skip NULL slots; the push collects exactly the
non-NULL entries. `count` stays where it means "how many connected".
New test_reconnect_dedups_channel_behind_deleted_slot (RED before: the
stale channel behind the hole was never evicted). Controller suite 1241/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A winning RequestVoteReply is dispatched through peer_ctrl_classify, which
returned VTASK_LIVENESS for all peer-control traffic -- the never-blocks,
never-admission-gated class. The reply runs raft_cb_become_leader, whose
tail persist_apply_pump_run_sync() does bounded blocking WAL pread()s (it
cannot be made async without reopening the bug-1043-family job loss on a
freshly promoted primary). The fdatasync half of bug-1433 was removed in
40f510fb; this pread was the tracked residual.
peer_ctrl_classify now returns VTASK_LIVE_BLOCK while the node is
CTRL_ELECTING -- its peer-control traffic is then vote/PreVote replies,
election work like inbound RequestVote, which already uses LIVE_BLOCK.
As PRIMARY/STANDBY the traffic is WAL/config ACKs and stays VTASK_LIVENESS,
so a quorum of acks is never admission-gated (R4.5); a candidate has no
commit to advance.
New test_peer_ctrl_classify_vote_replies_while_electing (RED before).
Controller suite 1240/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
"ERROR incorrect worker idx -1" on every node after a worker-join commit
timeout (during a PreVote/election) was the visible end of a real
dangling pointer. init_worker/rejoin_worker bind the endpoint to the
worker (apply_join_state: w->ep = ep, UNAVAIL cleared) before the commit
barrier; on barrier failure handle_worker_join returned without setting
priv->u.idx, so when that connection closed handle_endpoint_error saw
idx -1, logged the ERROR and never ran handle_worker_leave -- the worker
kept w->ep pointing at the freed endpoint, available for dispatch, until
it rejoined.
- handle_worker_join: on barrier failure, clear w->ep (if still this
endpoint) and mark the worker unavailable, as if the join never
happened; the worker retries its join.
- handle_endpoint_error: an idx -1 worker endpoint is now the legitimate
"closed before its join completed" case -- WARN, nothing to clean up.
test_worker_join_withheld_acks_replies_error also asserts the worker is
unbound and unavailable after the -ETIMEDOUT reply (RED before).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
f88e4ef4 moved ha_demotion_convergence_fn's WAL truncate + self-restart
and ha_catchup_continuation_fn's WAL streaming off the delaymgr thread
(which drives heartbeats and election timeouts cluster-wide) onto the
slow lane, but left both flags OFF: flipping them ON hung pre-existing
unit fixtures that demote a primary or trigger catch-up without ever
starting c->slowlane -- the work was queued in a lane with no workers and
never ran.
- slowlane_is_started(): true while the lane has live workers.
- Both dispatchers run inline when the flag is OFF *or* the lane was never
started. Production always starts the lane, so it always offloads; the
lightweight fixtures keep their old inline behaviour, and neither
one-shot action can be silently dropped.
- g_drive_async_demotion_offload / g_drive_async_catchup_offload now
default to 1. The offload tests save and restore the previous value
instead of forcing 0.
New test_catchup_offload_without_started_lane_runs_inline (RED before: the
catch-up body never ran, the stream gate was never reached). Controller
suite 1239/0 with both offloads ON by default.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A job the controller requeued to PEND while the scheduler was disconnected
stayed RUN (with its old allocation) in the scheduler forever: the
bug-1812 guard in core_events.c's JOBS_PUSH merge refused any non-RUN,
non-terminal resync claim over a job holding a real allocation, because
it could not tell a genuine requeue from a push built before the
scheduler's own fresh dispatch.
Design (per review): the scheduler persists nothing, so on rejoin the
controller's state wins, and the race the guard protected against is
closed where the truth lives:
- scheduler JOBS_PUSH: the bug-1812 refusal is removed; the controller's
state replaces the scheduler's and its allocation is released. The
terminal-is-permanent and phantom-RUN (RUN without allocation data)
guards stay.
- controller dispatch_a_job_(): refuses a job that is not PEND (RUN or
finished) before any worker is touched -- previously nothing checked,
so a stale second dispatch would start a running job again (or, through
a retryable failure, requeue it). The job is left as is, no WAL fact,
and its true state is re-sent (HOOK_DETAIL_JOB_UPDATE). A PEND job with
a leftover allocation stays dispatchable.
- scheduler JOB_UPDATE: a RUN report carrying the controller's committed
allocation is adopted, so a corrected job is never RUN with nothing
charged.
Tests (all RED before): test_jobs_push_duplicate_jobid_controller_pend_
wins_over_local_run (replaces the bug-1812 test that pinned the
refusal), test_job_update_run_adopts_controller_allocation,
test_dispatch_of_non_pending_job_is_refused. Two tests that seeded a RUN
job to reach retry branches (a precondition their own headers called
unreachable) now use the real paths: the counter-reset test requeues via
record_job_reconcile_pending_fact() first; the disconnected-worker RUN
test asserts the refusal contract.
Controller suite 1238/0, scheduler 87/87. HA soaks: chaotic 6 rounds PASS
(0 divergence/loss, 6 worker-resync jobs), sequential 6 rounds PASS
(SIMVERIFY 7 files, 0 mismatches). The race did not occur in either run
(0 refusals logged), so the new paths are proven by the unit tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gitea #47: `xcecli jobs submit -R "rusage[mem=100000000a0]"` was accepted
with a normal "Job <...> is submitted" reply. The apiserver only
length-checked resreq; the real PEG grammar ran later, asynchronously, in
the scheduler's core_build_job(), which logged a server-side WARN and
dropped the job -- it then sat PEND forever with no sign of why.
- libxce: xce_check_rusage_syntax(resreq, &errmsg) runs parse_rusage() on
every rusage[...] clause (extracted the way section.peg does: up to the
first ']'), so primary, compound and alternative requests are all
covered. No resource manager needed, so select[]/order[] resource names
are still only checked by the scheduler (the documented scope).
- apiserver/backend: ValidateResreqSyntax() cgo wrapper (resreq_check.go);
the !cgo stub returns nil (no job can be submitted without CGO anyway).
- controllers/jobs.go preCheckJobRequest(): calls it after the length
check via a package var, so a malformed request is refused up front.
Tests: lib test_check_rusage_syntax (RED before: the reported input was
accepted) and backend TestValidateResreqSyntax (RED before) through the
real grammar; controllers TestPreCheckJobRequest_RejectsMalformedResreq
checks the call site with an injected validator (written with the call
site, so no separate RED). libxce, apiserver backend (cgo) and controllers
tests pass. apiserver/middleware TestLoadJobs fails here and on the
baseline alike: it needs a live apiserver on :8080.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
do_update_resvals() fired the AFTER_EVENT HOOK_DETAIL_RESOURCE_UPDATE with
the raw client xce_resval_list_t. update_resval_list() rejects unknown
resource names (-ENOENT) but only logged it, so both listeners -- the
scheduler push and the apiserver store (apiserver_resource_hook_fun ->
handler.go) -- received names the controller had rejected, and the
apiserver stored them.
- update_resval_list() takes an optional `applied` view (caller-sized,
borrowed pointers) and fills it with just the accepted items.
- do_update_resvals() hands that view to the AFTER_EVENT hook, and skips
it when nothing was applied. The request still owns every item.
New test_update_resval_list_reports_only_applied_items (RED before: both
items reported). Controller suite 1237/0. Live /xce-test-resources with
mem=5 totally-fake-resource-xyz=1: mem MATCH at controller, scheduler and
apiserver; the fake name is rejected at injection and appears 0 times in
scheduler.log / apiserver.log (verdict FAIL by design, for the rejected
name).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
snapshot_load.c carried its own field-for-field copy of core_events.c's
schedobj builders (primary/compound/alter + hlimits killer), kept separate
by a "never share builders" convention. The copy drifted from the original
three times, each a field-completeness bug in restored jobs (bug-1772's
all-zero resreq_fp among them). The convention's reasons apply to the
job/queue/worker builders, not to schedobj, which is never wire-captured.
- core_build_schedobj() becomes the exported xce_core_build_schedobj(obj,
user, queue, project, resreq) (core.h), taking identity strings instead
of a wire job; core_build_job() and snapshot_load.c both call it.
- snapshot_load.c's five copied functions are removed.
- test_sim_startup.c #includes snapshot_load.c without core_events.c; it
now also #includes core_events.c, plus the same
sched_reconcile_job_resource_usage link stub test_driver_translate.c
uses.
- test_fresh_ingest_and_snapshot_restore_derive_identical_... now compares
the whole schedobj (tag, user/queue/project, configver, glimits, owner,
resreq binding, resreq_fp), guarding against future drift. No RED phase:
there is no live gap left to catch; this removes the duplication that
kept producing them.
Scheduler suite 87/87; full build links.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
JOB_UPDATE (core_events.c) and the legacy sched_update_existing_job()
(schedobj.c) replaced job->resreq, but the allocator reads job->schedobj,
whose primary->resreq still pointed into the old parsed request -- so an
update never changed scheduling, for any field. The old primary was also
leaked (FREEUP only freed the container).
- core_bind_primary_resreq(): the resreq pointer + fingerprint computation
factored out of core_build_primary_schedobj(), reused on update.
- core_rebind_schedobj_resreq(): primary->primary re-points in place and
recomputes the fingerprint, keeping the schedobj's runtime state; a kind
change rebuilds the kind-specific part (the old part is left alone, as
nothing frees these today -- never a dangling pointer).
- JOB_UPDATE now applies the job's ingest-time task count to the new
resreq, as core_build_job() does, so resreq-maxtasks-wiring Req 1.4 (an
update never changes the dispatched task count) still holds -- it used to
hold only because the update never reached the schedobj at all.
- schedobj.c's legacy path gets the same re-binding.
test_resource_request_update_never_changes_dispatched_task_count asserted
the old behaviour (schedobj keeps the ingest-time object); it now asserts
the schedobj uses the updated resreq and fingerprint while the task count
stays 4/7 (RED before). Scheduler suite 87/87.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-448: job_ctrl() took a reference on the job but only one error path
released it; every successful call leaked one (refcount 1 -> 3 after two
calls). It is now released on every exit path, including the key_find
rc<0 one.
bug-1626: job_ctrl()'s not-started branch changes the job (mark_job_update
/ mark_job_fail -> apply_job_*_transition, which require RTTI_LOCK_WRITE)
while holding only RTTI_LOCK_READ, so it could run concurrently with
readers. It now takes the write lock. (dispatch_a_job_ already did.)
bug-1631: dispatch_a_job_()'s four retryable-failure branches never reset
state/allocs, correct for a first dispatch (still PEND) but not for a job
that had already been dispatched once: it stayed RUN holding its old
allocation. Each branch now calls record_job_reconcile_pending_fact() --
a no-op without a WAL fact for a PEND job; for a RUN job it returns it to
PEND, releases allocs and appends a durable WAL_JOB_UPDATE -- and notifies
after unlocking when it applied, the same pattern as the resync reconcile
path (cfe66dde). Not reachable until something re-dispatches RUN jobs
(e.g. preemption), fixed now so it is correct when that lands.
- job_get_refcount() (jobj.c) read accessor for the leak test.
- test_apply_pump.c: test_job_ctrl_releases_its_job_reference and
test_job_ctrl_mutates_not_started_job_under_write_lock (both RED before).
- test_dispatch_fail_retry.c now asserts PEND + allocs released after a
post-success retryable failure (RED before).
Controller suite 1236/0; all 111 meson suites pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
persist_checkpoint() and persist_checkpoint_marker_triggered() wrote a
local-only WAL_CHECKPOINT marker that consumed an LSN of the replicated log
(current_lsn++) but was never replicated. A checkpoint taken by one member
only -- the primary's periodic max-wal-records checkpoint, or the one before
a snapshot send -- left a hole no standby had: the leader's next entry
carried prev_log_index = the marker's LSN, logOk failed on every standby
forever, commits timed out and writes stalled ("could not submit 110 jobs"
in the --force-snapshot soak; reproduced on the pre-bug-1867 baseline). Even
the config-marker checkpoint, taken on every member, stamped the marker with
each member's own current_lsn at apply time, which need not agree.
Nothing reads the marker except to skip it, so checkpoints simply stop
writing it. The recovery, streamer and bug-1475 log_len skip paths stay, for
WALs written by older binaries; persist_test_write_legacy_checkpoint_marker()
(XCE_TESTING only) reproduces such a WAL for the tests that cover them.
- new test_bug1870_leader_only_checkpoint_leaves_no_log_hole (RED before):
a leader-only checkpoint takes no LSN and a standby that never
checkpointed catches up past it via the real streamer/receive path.
- test_checkpoint_writes_marker_to_wal inverted to ..._writes_no_marker_...
- bug-1475 x2 and the apply-correctness trailing-marker test now build
their legacy marker explicitly.
Controller suite 1234/0. HA soaks: --force-snapshot PASS (113 jobs, real
snapshot install -- also live-verifies bug-1867), --config-upgrade-wal-trim
PASS, --sequential 6 rounds PASS (SIMVERIFY 7 files, 0 mismatches).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
