Commit graph

1410 commits

Author SHA1 Message Date
b4470b2b94
test(scheduler-allocator-runner): characterize the parjob allocator (task 2.2)
15 tests drive the real parjob plugin through job_alloc_fn: success,
short shared, pending on hosts, mid-plan rollback, task counts 0/1,
missing inputs, six OOM steps with their debits, COMPOUND. Values the
runner changes on purpose sit in *_today tests.

Spec: the task showed that parjob records host reasons only for a bucket
search's first rejected candidate (the rejected host's request stays in
alloc_req.host and the host iterator pre-filters later hosts against
it). Decision: the runner clears alloc_req.host before every candidate,
for both allocators, so every short host gets its reasons (Req 3.6/3.7);
publishing reasons only when they change is a follow-up.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 08:40:35 +00:00
91a8580948
test(scheduler-allocator-runner): characterize the seqjob allocator (task 2.1)
12 tests drive the real seqjob plugin through job_alloc_fn and record
today's results, reasons and debits, with switch-stable OOM injection
(sched_make_allocs stub, --wrap of make_host_alloc/make_job_alloc).
Assertions the runner changes on purpose sit in *_today tests.

The characterization showed bug-1893's real effect: vget() returns 0 on
success, so the shared-reason loop (rc <= 0 skip) never records a shared
NO_RESOURCE reason. Bug record and spec wording corrected; the runner
skips only on rc < 0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 08:11:26 +00:00
31ad79d325
feat(scheduler-allocator-runner): declare the allocator plugin contract (task 1.2)
xce_sched_allocator_iface_t (optional accepts guard + plan_init/next/fini)
and the sched_allocator_run() entry point in sched/alloc_iter.h; the
plugin's const allocator_iface reference, forward-declared in plugin.h
with the picker_iface ownership rule. No plugin uses them yet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 07:49:25 +00:00
86ee3491dd
feat(scheduler-allocator-runner): plan generators initialise from the scheduling core (task 1.1)
xce_alloc_plan_iter_t.init takes xce_sched_core_t* instead of
xce_scheduler_t*; both built-in generators, both allocators' call sites
and the contract-test stub follow. The parjob purity test now shows an
identical plan sequence for a NULL and a poisoned core.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 07:44:27 +00:00
4e48462968
fix(tools): a failing apiserver Go test fails xce-build.sh (bug-1895)
The apiserver and all components ran the frontend Go tests as one list,
'cd frontend/apiserver && go test ... && cd ../xcecli && go test ...'.
set -e ignores a failure of any command in an && list except the last,
so a failing apiserver go test only short-circuited the list: the build
reported success and the xcecli tests never ran. Run each suite as its
own command in a subshell (which also keeps the cd from leaking).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 04:31:25 +00:00
141ac22c54
fix(apiserver): an accepted job is in the job store at once (bug-1894)
The apiserver's job store learned of a new job only from the
controller's asynchronous OnJobCreated / array push, so a 'jobs kill'
(or query) straight after 'jobs submit' could answer 'job not found'
for the job just accepted (2 of 4 immediate kills).

services.SubmitJob now records the accepted job right after the
controller's reply (which comes only after the job is committed and
applied): one record per element, base+i for arrays, stored with the new
DataStore.PutIfAbsent and marked JobInfo.Provisional (json:"-", not a
column). An OnJobCreated that already arrived is never overwritten; the
push that arrives later replaces the provisional record -- including in
dispatchJobArray, whose duplicate check no longer treats a provisional
element as already known.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 04:23:10 +00:00
0f4d23e3cd
fix(frontend-test): regression suite accepts dashed job IDs and no longer hangs on stdin (bug-1765)
xcecli prints a submitted job's ID in the dashed
{cluster}-{apiserver}-{timestamp}-{sequence} form (format.FormatJobID,
since d6008ead), but run.sh still matched 'Job <[0-9]+>' -- so 'jobs
submit' always failed and 'jobs kill' was skipped (28/1/1). It also
extracted the ID with [0-9]{10,}, which would have picked out only the
timestamp. Match the dashed form and take the whole ID from between the
angle brackets; 'jobs kill' accepts it back.

'jobs submit' reads a job script from stdin whenever stdin is a pipe,
so the suite hung forever when run from a script or CI with an open
pipe; the submit now reads /dev/null.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 04:13:26 +00:00
f5c887dc50
fix(controller-test): clean_failover_preservation.sh runs again (bug-1721)
The script required a pre-generated <repo>/ha/data (cluster.bin + TLS
certs) that ee8473bd removed, and called deploy-ha.sh with that dir as an
extra argument; deploy-ha.sh now generates both into the deploy dir and
takes <binary> <top-dir> <peers>. It failed at its first precondition in
every checkout.

Behind that, both committed-LSN boundaries still assumed bug-845's doubled
submits (1 + 2*N_JOBS): the commit wait never reached it and walcompare
compared an LSN past every member's WAL. Since that fix each job is one
WAL entry, so the boundary is 1 + N_JOBS.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 03:51:20 +00:00
980a0054d1
fix(tools): one xce-build.sh run per build dir at a time (bug-1604)
Two xce-build.sh runs against the same build dir raced: one run's
setup_build rm -rf hit files the other was writing ('Directory not
empty'), or deleted a binary the other had just built and tested.

xce-build.sh now takes an exclusive per-build-dir lock right after cd-ing
into src/ and holds it until it exits; a second run against the same dir
waits, runs against different dirs are unaffected. The lock
(src/tools/build-lock.sh) is an mkdir lock holding the owner PID, outside
the build dir, keyed by a hash of its absolute path; it needs only
coreutils (the nix dev shell has no flock) and takes over a lock whose
owner died. src/tools/test-build-lock.sh tests it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 03:43:48 +00:00
16b7aa4b60
fix(lib,scheduler): zarray_get rejects negative indexes; info resource bounds by capacity (bug-1824)
zarray_get_ex() checked only idx >= caps, so a negative index read
array->data[idx] out of bounds -- reachable through a resource whose
registration failed and kept idx == -1 (xcelimit.c add_vres()). It now
returns NULL for idx < 0.

repl_info_resource_one() bounded a shared resource's slot by
sharedInst.count / sharedInUse.count, the number of populated slots, not
the capacity: a populated slot above that count showed as value=(none).
It now relies on zarray_get()'s own bounds check.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 03:34:57 +00:00
cfa0a94166
fix(scheduler): the simulator's threads stop when its run returns (bug-1632, bug-1763)
sched_sim_driver_run() started the picker and allocator threads on
&sim->core and returned with them still running; only
sched_sim_driver_fini() stopped them, and 24 test files (35 call sites)
run the driver on a stack-local sim without calling it. The leftover
threads kept a pointer into the dead frame, and ASAN occasionally caught
one reading it (stack-use-after-return in cbuf_get) in
sched_sim_verify_test and sched_sim_record_output_test.

sched_sim_driver_run() now wraps the loop and stops both threads on
every return (new sched_sim_driver_stop_threads(), shared with fini). If
the allocator fails to start, the running picker is stopped too. The
stop no longer pushes a jobid 0 to wake the allocator: cbuf_get/put wait
in pthread_cond_wait, a cancellation point, and the fake end-of-session
was visible to the allocator.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 02:47:24 +00:00
4922f6c844
fix(controller): record standby state under its absolute peer index (bug-1891)
A promoted vote endpoint carries a peer index relative to the votes
array (which leaves out the primary); a late-joiner's heartbeat endpoint
carries the absolute index. repl_ack_decode_and_dispatch recorded state
reports under the raw index, so with ctrl-1 primary, ctrl-3's vote
endpoint (relative 1) and a rejoined ctrl-2's heartbeat endpoint
(absolute 1) shared standby_reported_*[1]. Each handler then acted on
the other's position over its own endpoint: the snapshot meant for the
empty, rejoining ctrl-2 went to the caught-up ctrl-3, and ctrl-2 never
caught up.

Resolve the absolute index with ha_repl_ack_abs_peer_idx (as the
WAL-ack branch already did) for the state report and for
ha_walack_on_ack, and size catchup_inflight by MAX_HA_CANDIDATES so
every absolute index fits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 15:12:53 +00:00
8b561547dd
fix(lib): netio TCP connections run with TCP_NODELAY (bug-1890)
With Nagle on, the small TLS record that ends a burst -- a packet header,
the last short chunk of an HA snapshot, a reply -- waited for the peer's
delayed ACK, ~40 ms on Linux: seen on 5 of 40 snapshot transfers. Set
TCP_NODELAY on every netio TCP socket, outbound (socket_create) and
accepted (socket_accept).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 14:38:57 +00:00
754d5bcf3e
fix(lib): a stale OpenSSL error no longer kills a healthy connection (bug-1871)
SSL_get_error() reports SSL_ERROR_SSL whenever the calling thread's
OpenSSL error queue is non-empty. io_worker serves every endpoint from
one thread, so an error left by another connection -- a ctrlstatus
client's EOF queues 'unexpected eof while reading' in OpenSSL 3 -- turned
the next plain would-block SSL_read on a healthy connection into a fatal
error. handle_error() then tore that connection down; mid snapshot
transfer this left a rejoining standby with a truncated snap.part until
its liveness timeout forced a reconnect and a fresh transfer.

Clear the queue before each SSL_read/SSL_write/SSL_do_handshake so the
classification is about that call only.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 13:59:44 +00:00
9c51de88c6
fix(lib): percent limits render as configured; xcelib_test counts every failure again
test_fmt_limit_units_percent_limit_ignores_units_mode failed on every
run: fmt_limit_units() widened the float limit value before scaling it,
so 0.8 printed as "80.000001%". The product is now taken in float and
widened after, which gives the configured 80.

The failure never counted: test_libxce_retry(), one group of test.c's
single runner, ran its own UNITY_BEGIN()/UNITY_END(), which reset Unity's
counters -- every failure in a group before it was dropped and
xcelib_test exited 0. With the pair removed the runner reports 695 tests;
this was the only hidden failure.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 10:31:54 +00:00
b5712c053d
fix(lib): a malformed select in a resreq fails the parse instead of being dropped (bug-1887)
The primary_resreq action ignored parse_resreq_primary()'s result, so a
primary whose select (or rusage) clause failed to parse was accepted
without that clause: select[type==] or select[mem>] matched every host.
A malformed clause now fails the whole resreq, and parse_resreq() returns
the clause's own message.

A select naming an unregistered resource is still dropped, not an
error: parse_select() reports it as -ENOENT (new select_context_t
.unknown_resource) and the action lets it pass, because jobs rely on it
-- the scheduler's own "type == any" default among them when no "type"
resource is registered (making it fatal failed 36 scheduler suites).

Test: test_resreq_primary_select_error_fails_parse RED before (all
accepted), GREEN after.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 10:26:35 +00:00
077b1213d3
fix(scheduler): snapshots save a job's resreq text, not its debug dump (bug-1888)
build_wire_job() filled the saved job's submit->resreq with
format_resreq()'s human-readable dump ("Primary:\n  Tasks: ..."). The
loader's parser accepted that as a primary with neither select nor
rusage, so every job restored from a snapshot (a replay's start snapshot,
REPL snapshots) lost both.

A scheduler job now keeps the text its resreq was parsed from
(xce_sched_job_t.resreq_str, set at ingest, JOB_UPDATE and snapshot load,
freed at teardown) and the snapshot saves that; a job without it restores
as the loader's "type == any" default.

Test: test_snapshot_round_trip_keeps_job_resreq_text RED before (saved
text was the dump), GREEN after.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 10:26:21 +00:00
3e873f0a66
fix(scheduler): a re-applied config updates queues in place and frees the config it replaces (bug-1886)
sched_init_queues() built a new xce_sched_queue_t for every configured
queue on each config apply; for an existing name the add failed, and the
unregistered queue was handed to the picker (which then tracked it
instead of the registered one) and leaked, while the registered queue
kept its old settings. It now updates the registered queue in place, as
QUEUE_UPDATE's core_upsert_queue() does.

sched_init_config() and the core's CONFIG_PUSH replaced core->config
without freeing the previous one. The core owns its config, and nothing
points into a replaced one (queues, resources, groups and limits built
from it are copies), so it is freed once the new one is applied.

Test: test_config_reapply_updates_queue_in_place_and_frees_old RED before
(priority stayed 1), GREEN after.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 10:03:04 +00:00
d21b3d8f98
fix(scheduler): the normal-mode round frees each decision after publishing it (bug-1885)
xce_sched_driver_run_normal_mode_round() drained every decision, handed it
to the recorder and the decision callback, and never freed the struct
make_*_decision() allocates: 64 bytes per decision for the life of the
scheduler. The struct is freed after the callback returns; its allocation
or pend reason belongs to the job, and the callback and recorder copy
what they keep.

Test: test_round_frees_decision_after_callback (ASAN shadow check) RED
before, GREEN after.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 09:57:26 +00:00
90b99d91e9
fix(scheduler): a batch replay exits 0 under LeakSanitizer (bug-1884, part 6)
- kill_sched_resval() (every resource-value array's item killer) frees
  the value, not only a string's buffer: each replaced or torn-down value
  leaked its struct, including every WORKERS_PUSH value update.
- put_resvals() shares the value it creates with later entries for the
  same index in one call; it created a second one that replaced (and now
  freed) the first while still writing through it.
- JOB_UPDATE and the duplicate-JOBS_PUSH path kill_resreq() the old parsed
  resreq, and free the rejected job's schedobj (xce_core_free_schedobj(),
  exported from core.c), instead of freeing only the containers.
- The previous attempt's (or ALTER alternative's) pend reason is freed when
  a new one replaces it.
- kill_resreq_ctx() frees the parser context's error message, and the
  primary-resreq PEG action frees an earlier primary's message first.
- write_snapshot_entry() frees the snapshot it builds (config and resource
  objects detached, they are the core's); the save side strdup()s its
  strings so the generated free_SnapshotData() applies.

Test: test_cli_batch_replay_exits_clean_under_leak_detection
(test_sim_discrete_clock.c) runs the real binary on a two-session fixture
with detect_leaks=1 and requires exit 0: RED before (exit 1), GREEN now.
All nine HA-soak recordings replay with exit 0 and no LeakSanitizer report.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 09:44:43 +00:00
c1cd8f917d
fix(scheduler): free registry entries after their tables at core teardown; fix dup_val and the unmeet re-init leak (bug-1884, part 5)
A failed key_add() runs the table's datakiller on the rejected value, and
callers keep using it (JOBS_PUSH merges a duplicate job into the existing
one; sched_init_queues() re-adds every queue on each config apply), so the
job, queue, worker and resource datakillers must stay shallow. Part 4 made
kill_sched_job() free the whole job; that is reverted. Instead
xce_sched_core_fini() collects each table's entries, destroys the table,
then frees each entry with a full destructor (core.c). The resource table
is released the same way (a reference per resobj).

Also:
- dup_val() tested which == XCE_VAL_STRING (0, the resource value type,
  equal to Value_u_ivalue32): real string values were shallow-copied and a
  nonzero int32 went to makeString() as a pointer. It now tests
  Value_u_svalue and strdup()s, as free_Value() expects.
- The shared/host satisfy checks re-initialize the unmeet vector they are
  given; the allocators passed the ctx's live one, dropping its buffers on
  every candidate host. The callers finalize it first.
- add_vres() registers a resobj via makeResource(), like every other
  resource, instead of a bare calloc(); test_limit.c drops it with
  dereference().
- plugin_register_core() rejects a duplicate name before init_fn/key_add,
  since a failed add now finalizes (frees) the plugin and callers free it
  on failure.
- sharedInUse gets kill_sched_resval, as put_resvals() fills it.

Exit-time leak on the reproduction replay 7,469 -> 928 bytes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 09:28:08 +00:00
ebfe4794f0
fix(scheduler): free jobs, schedobjs, sim decisions and the fcfs plugin at teardown (bug-1884, part 4)
- kill_sched_job() (the job table's datakiller) frees the whole job:
  schedobj, parsed resreq, allocation, accounts and rusages; new
  kill_sched_obj() frees the schedobj's names, limit lists, per-primary
  wrappers and their pend reasons, and its last allocation/reason.
- The simulator frees each decision struct after draining it (live and
  reconciled); the alloc/pend reason it points at stays the job's.
- The builtin_fcfs heap plugin gets a fini_fn that also frees the plugin;
  builtin_fcfs_fini() still frees only the picker state (tests run it on
  stack plugins).
- xce_sched_core_fini() frees the two pend_updates queues both drivers
  calloc().

Exit-time leak on the reproduction replay 11,426 -> 7,469 bytes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 09:09:12 +00:00
8134b75b10
fix(scheduler): the core owns and frees its config; the simulator frees its start snapshot (bug-1884, part 3)
xce_sched_core_load_snapshot() keeps the snapshot's config as core->config
and copies everything else, so the simulator now moves the config into the
core and frees the rest of the decoded SnapshotEntry. xce_sched_core_fini()
frees core->config last, after the plugins and group managers that may
point into it; nothing freed it in either mode. Exit-time leak on the
reproduction replay 25,807 -> 11,426 bytes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 09:00:13 +00:00
c89e4e4520
fix(scheduler): release replay-event payloads after the simulator pushes them (bug-1884, part 2)
sim_process_replay_event_message() now hands the decoded replay event to
the caller, which keeps it beside the translated core event and frees it
after xce_core_push_events() with driver.c's release_translated_payload()
rules: a pushed config is kept by the core, and a rusages report is kept
by the job it found. An untranslatable event is freed at once, and events
after the last session are freed at the end. The payloads were an
"accepted leak" per event; exit-time leak on the reproduction replay
75,495 -> 25,807 bytes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 08:55:34 +00:00
9b61fd99e4
fix(scheduler): tear down the simulator and the core's registries at exit (bug-1884, part 1)
A batch --sim replay under ASAN exited 1 on 17.2 MB of LeakSanitizer
reports. This removes all but 75 KB:

- sched_sim_driver_fini() (called by main after a clean run) cancels and
  joins the picker/allocator threads like sched_fini(), closes the replay,
  alternate-events and output files, and finalizes the core. The threads'
  ids now live in core.jobpicker/allocator, as in normal mode.
- xce_sched_core_fini() destroys the plugin, job, worker, queue, user and
  project registries and the shared-resource arrays it creates.
- The seqjob/parjob sched_make_allocs() stand-ins set up only lock, nextid
  and the aliased resmgr. Each ran a full xce_sched_core_init(), which also
  replaced the process-wide string and selector tables mid-startup.
- plugin_unregister(), the plugin registry's datakiller, only runs fini_fn;
  it key_del()'d from the registry being destroyed and skipped plugins
  registered with plugin_register_core().
- The limit plugin's usage-instance table used kill_limit as datakiller on
  xce_vres_t records (bad free on destroy); kill_vres now frees the record.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 08:50:59 +00:00
1fb1bcc971
docs(scheduler): parjob bucket-reset comment describes the job-level .host want after bug-1883
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 08:11:52 +00:00
c10f32ccd8
fix(scheduler): reset the per-host want at each primary's job-level prepare pass (bug-1883)
A COMPOUND/ALTER job's primaries share one ctx. The job-level prepare pass
reset only alloc_req.shared, and the allocators append their slots want
into alloc_req.host, so a later primary's host pre-filter (and a global
limit on slots) read the previous primary's last per-host want: a 1-task
primary after a 4-task one was filtered on slots:4 and pended although a
host with its one free slot existed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 08:09:27 +00:00
4e57bde693
feat(libxce-alloc-preason-equality): replay reproduces the reservation start time (task 5.3)
Add test_sim_replay_reproduces_recorded_alloc_from: a recorded DISPATCH at
an epoch-scale beginRound with a sub-second part replays at the default
simulation settings with no --sim-verify mismatch, and the live allocation's
from equals beginRound in whole seconds. Share the dispatch fixture setup
of the existing tests through two helpers.

Live: 2-round HA soak recordings (11 DISPATCH decisions) replayed with
--sim-verify and no --sim-speed report 0 mismatches.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 06:48:38 +00:00
d5527c665b
docs(scheduler-sim-discrete-clock): scheduler-core spec drops speed modes, --sim-speed deprecated, REPL speed removed (task 4.2)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 05:26:03 +00:00
66435d100d
docs(scheduler-sim-discrete-clock): deployment guide describes the event-driven sim clock and deprecated --sim-speed (task 4.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 05:21:11 +00:00
aa2bb6d090
feat(scheduler-sim-discrete-clock): remove the simulator speed argument and shadow copy (task 3.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 05:19:34 +00:00
eb5f339541
feat(scheduler-sim-discrete-clock): retire --sim-speed as a warn-once no-op, end-to-end CLI tests (task 2.3)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 05:06:06 +00:00
05320e777b
feat(scheduler-sim-discrete-clock): remove the REPL speed parameter (task 2.2)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 04:53:20 +00:00
3a6b503f07
feat(scheduler-sim-discrete-clock): pin the simulation clock so it moves only at recorded timestamps (task 2.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 04:41:14 +00:00
4f653d8fbe
feat(scheduler-sim-discrete-clock): --sim-speed deprecation-warning helper and discrete-clock test binary (task 1.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 04:23:46 +00:00
a9c984775f
feat(libxce-alloc-preason-equality): drop sim verify tests that duplicate libxce alloc_eq coverage (task 5.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 11:04:19 +00:00
24b057189f
feat(libxce-alloc-preason-equality): sim verify uses public job_alloc_eq and logs the first difference (task 4.3)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 10:58:41 +00:00
69112c2d22
feat(libxce-alloc-preason-equality): recorded dispatch fixtures carry the session-start from (task 4.2)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 10:44:00 +00:00
6110d12e5a
feat(libxce-alloc-preason-equality): white-box equivalence tests use public job_alloc_eq/job_preason_eq (task 4.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 10:27:46 +00:00
c1960457f8
feat(libxce-alloc-preason-equality): stamp every allocation's from with the deciding session start (task 3.2)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 10:14:35 +00:00
2d7ba66019
feat(libxce-alloc-preason-equality): record session begin from the session-start clock reading (task 3.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 09:59:56 +00:00
62d0b8b686
feat(libxce-alloc-preason-equality): full-match job_preason_eq with first-difference description (task 2.2)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 09:49:57 +00:00
a6cbb323b0
feat(libxce-alloc-preason-equality): public alloc_eq/host_alloc_eq/job_alloc_eq with first-difference description (task 2.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 09:35:08 +00:00
8f25c1c3f3
feat(libxce-alloc-preason-equality): NULL-safe str_eq helper in libxce utils (task 1.1)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 09:21:04 +00:00
f9146864a3
fix(lib,scheduler): allocation constructors and allocators report OOM instead of building NULL-name records (bug-1882)
make_alloc() and make_host_alloc() assigned strdup() results unchecked,
so under memory exhaustion they returned a record whose resname/reserve/
host was NULL; merge_job_alloc()'s same_alloc_slot() (bug-1863) then
strcmp()'d it (ASAN SEGV at xcealloc.c:337). make_alloc() also passed a
NULL resname straight to strdup().
- xcealloc.c: make_alloc()/make_host_alloc() reject a NULL name (EINVAL)
  and check every strdup(), freeing what they built and returning NULL
  (ENOMEM); on failure make_host_alloc()'s caller keeps `allocs`.
  dup_host_alloc() frees its partial copy on failure instead of leaking.
- scheduler.c sched_make_allocs(): a record that cannot be built fails
  the whole array (NULL, ENOMEM) instead of leaving a NULL entry.
- seqjob/parjob my_alloc(): check every constructor; on failure free
  what was built (parjob also rolls back the remaining claims) and
  return -ENOMEM with ctx->alloc NULL -- the caller moves ctx.alloc onto
  the job whatever the rc.

Tests: new xcealloc_oom_test (8, --wrap=strdup) and
sched_make_allocs_oom_test (real scheduler_srcs + --wrap=strdup); a new
-ENOMEM test in the seqjob and parjob allocator suites. All RED before
(SEGV / "Expected NULL" / "Expected -12 Was 1"), GREEN after.
xce-build.sh all asan: 115/115 suites OK, Go tests OK.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 08:49:51 +00:00
aa7fc65a36
fix(controller): cold start elects in ~1.7 s, not ~10 s; ctrlstatus appliedLsn on the primary (bug-1879, bug-1878)
bug-1879: on a fresh HA cluster the config-holding ctrl-1 started its
term-1 election before its peers ran; connect failures count as "no"
votes (bug-1423), so it lost at once and became a follower with no
leader. ctrl-2/ctrl-3 hold no config and are voters only (by design),
so nothing retried until ctrl-1's 10 s standby liveness timeout: every
cold start had no primary for ~10 s.
- ha_note_leaderless() marks a config-holding node that becomes a
  follower with no leader known; the liveness poll then retries the
  election (PreVote) after HA_LEADERLESS_RETRY_MS = 1 s, doubling up to
  heartbeat_timeout_ms. Primary contact or becoming leader clears it.
- ha_heartbeat_is_fresh() answers "no" while leaderless: last_activity
  then only holds the watchdog's arm time, and the failover gates took
  it for a live primary.
Live cold start: primary 1.7 s after the first node started (was ~10 s).

bug-1878: ctrlstatus_fill() reported ha->applied_lsn for every role, a
standby-only counter a primary never advances, so every primary showed
"appliedLsn: 0". A primary now reports persist->last_applied_lsn. The
controller-status design doc's "already correct in every role" note is
corrected.

Tests: 5 new in test_ha_failover_prevote.c, 1 new in test_ctrlstatus.c
(RED: expected 4900, was 0); controller suite 1256/0. HA soaks:
sequential 6 rounds PASS, chaotic 6 rounds PASS (zero divergence, zero
job loss).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 07:35:45 +00:00
44173f74ae
fix: spurious errors/warnings from a pc13v HA deploy (bug-1872..1877)
- bug-1872 (worker): "Failed to find any disk statistics in
  /proc/diskstats" ERROR every 15 s on an NVMe-only host. The whole-disk
  filter guessed from the name and skipped anything ending in a digit or
  containing 'p' -- every nvmeXnY, mmcblkN, even sdp -- so io was never
  reported. diskstats.h now counts a device only when
  /sys/block/<name>/device exists (real disks; not partitions, loop, dm,
  md, zram), and a host with no disk warns once.
- bug-1873 (worker): "Failed to get worker pid" WARN on job start. MainPID
  reads 0 until systemd has forked the new transient unit; xcecd.c already
  retries, so the miss is DEBUG and the final failure names the unit.
- bug-1874 (scheduler): deprecation WARN for the legacy xce_sched_fcfs on
  every start, though builtin_fcfs is the picker. Registration still wraps
  a job_elect_fn plugin but no longer warns; selecting one warns once.
- bug-1875 (apiserver): "Not running as root -- PAM authentication will not
  work" with the default provider config_file; now only for provider pam.
- bug-1876 (controller): standbys logged "cluster_init_phase2 ... failed
  (rc=-22)" WARN every 3 s while waiting for the primary's config push,
  a state the function itself calls normal. It returns -EAGAIN for that
  wait; callers log it at INFO, other failures still WARN.
- bug-1877 (controller): "WAL/snapshot stored unencrypted" warned without
  looking. persist_path_encrypted() checks fscrypt (statx) and dm-crypt
  (sysfs dm/uuid CRYPT-, through LVM slaves); the warning stays when the
  storage is not shown to be encrypted (e.g. a plain /tmp).

Tests: worker diskstats_test (2 of 4 RED on the old filter with pc13v's
/proc/diskstats), controller test_phase2_cold_boot and
test_persist_encryption, scheduler picker tests, apiserver
TestRootWarning. Full ASAN build: all 113 suites pass (controller 319/0,
harness 141/0).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 04:35:19 +00:00
da1a663f05
fix(harness): xce_test argument validation, RPC error text and leaks (13 bugs)
Stack overflows on user input:
- bug-1702 jobrun -m: host segment strncpy'd into a 256-byte buffer
  unbounded (ASAN stack-buffer-overflow); now cmdutil_split_host_tasks()
  refuses an over-long host.
- bug-1703 jobsub -n: min strncpy'd into a 64-byte buffer; now
  cmdutil_parse_ntasks() parses in place, 1 <= min <= max.
- resupdate's 16-entry vals[] had no bound either; a 17th -n overflowed.

Wrong or silent results:
- bug-1574 resupdate type/model: set_resval() had no string case, left the
  entry's name uninitialized, and crashed the encoder. Moved to
  resval_typeresolve.c as resval_fill(), which fills every type.
- bug-1705 jobsub -x was overwritten (and leaked) by the default exdata;
  cmdutil_default_exdata() only fills an unset one. jobsub's
  execuser/exechome/execcwd copies leaked on every call.
- bug-1707 a trailing -n without -v was dropped silently; resupdate also
  returned 0 after its RPC failed and "succeeded" with nothing to send.
- bug-1708 jobkill -s SIGKILL became signal 0 and was sent.
- bug-1709 jobinfo exited 0 with no -j, a malformed -j, or no hits.
- bug-1710 hello -c 0 sent nothing and reported success.
- bug-1712 a duplicate -v was reported as "no res name found".
- bug-1777 workerlist showed join-time maxmem/maxswap/maxtmp, not the
  effective values (hosts.yaml overrides) the reply already carries in
  resvals_to_sched and the scheduler uses.
- bug-1793 join's maxswap/maxtmp defaults were 1 KB / 2 KB in bytes; now
  1 GiB / 2 GiB, and -m/-s/-t no longer overflow atoi() above 2 GB.

Error reporting and leaks:
- bug-1796 RPC failures printed perror() of a stale errno ("Connection
  refused" / "Invalid argument" / "Inappropriate ioctl for device" for
  the same failure). cmdutil_rpc_error() prints the RPC's own code
  (E_XCE_* included) at all 19 sites.
- bug-1797 qctrl leaked the queue name on the RPC-error path; repeated
  -q/-h leaked the earlier copy.

New src/harness/cmdutil.c (+ cmdutil_unit_test, 14 cases); 9 new
test_test_cli.c cases, 8 RED on the old xce_test; new resval_fill,
worker_effective_attr and join-default tests (join ones RED on the old
join_worker_args.c). Harness suite 6/6 (asan), harness_unit_test 141/0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 04:17:04 +00:00
e84a8d4332
fix(worker): remove_worker frees every string add_worker copied (bug-1866)
add_worker strndup's launch_type and the four queue/job pre/post-exec
commands out of the startinfo, but remove_worker only freed exechome,
execcwd, execuser, job_cmd, pids and exdata, so every finished job leaked
up to five strings (LeakSanitizer at xcecd_virtual exit reported the
launch_type one).

New test in test_finished_job_retention.c compares ASAN's live-heap byte
count across an add/remove round trip (the worker suite runs with
detect_leaks=0): RED leaked exactly the 58 bytes of those five strings,
GREEN after. Worker suite 12/12.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 13:14:14 +00:00
e77968bd7e
fix(lib): name the newer opcodes in xce_request_op_name
schedAdmin, jobStateQuery, jobStateReport and the new notifyAcked
(bug-1741) all logged as "opcode unknown" in the controller's request
trace -- e.g. the bug-1741 factstream soak showed 373 "unknown" requests
that were really acknowledged job reports.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 07:01:38 +00:00