Two xce-build.sh runs against the same build dir raced: one run's
setup_build rm -rf hit files the other was writing ('Directory not
empty'), or deleted a binary the other had just built and tested.
xce-build.sh now takes an exclusive per-build-dir lock right after cd-ing
into src/ and holds it until it exits; a second run against the same dir
waits, runs against different dirs are unaffected. The lock
(src/tools/build-lock.sh) is an mkdir lock holding the owner PID, outside
the build dir, keyed by a hash of its absolute path; it needs only
coreutils (the nix dev shell has no flock) and takes over a lock whose
owner died. src/tools/test-build-lock.sh tests it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
zarray_get_ex() checked only idx >= caps, so a negative index read
array->data[idx] out of bounds -- reachable through a resource whose
registration failed and kept idx == -1 (xcelimit.c add_vres()). It now
returns NULL for idx < 0.
repl_info_resource_one() bounded a shared resource's slot by
sharedInst.count / sharedInUse.count, the number of populated slots, not
the capacity: a populated slot above that count showed as value=(none).
It now relies on zarray_get()'s own bounds check.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1580's evidence was a ':FAIL:' grep that cannot match Unity's
colored FAIL lines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_sim_driver_run() started the picker and allocator threads on
&sim->core and returned with them still running; only
sched_sim_driver_fini() stopped them, and 24 test files (35 call sites)
run the driver on a stack-local sim without calling it. The leftover
threads kept a pointer into the dead frame, and ASAN occasionally caught
one reading it (stack-use-after-return in cbuf_get) in
sched_sim_verify_test and sched_sim_record_output_test.
sched_sim_driver_run() now wraps the loop and stops both threads on
every return (new sched_sim_driver_stop_threads(), shared with fini). If
the allocator fails to start, the running picker is stopped too. The
stop no longer pushes a jobid 0 to wake the allocator: cbuf_get/put wait
in pthread_cond_wait, a cancellation point, and the fake end-of-session
was visible to the allocator.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A promoted vote endpoint carries a peer index relative to the votes
array (which leaves out the primary); a late-joiner's heartbeat endpoint
carries the absolute index. repl_ack_decode_and_dispatch recorded state
reports under the raw index, so with ctrl-1 primary, ctrl-3's vote
endpoint (relative 1) and a rejoined ctrl-2's heartbeat endpoint
(absolute 1) shared standby_reported_*[1]. Each handler then acted on
the other's position over its own endpoint: the snapshot meant for the
empty, rejoining ctrl-2 went to the caught-up ctrl-3, and ctrl-2 never
caught up.
Resolve the absolute index with ha_repl_ack_abs_peer_idx (as the
WAL-ack branch already did) for the state report and for
ha_walack_on_ack, and size catchup_inflight by MAX_HA_CANDIDATES so
every absolute index fits.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With Nagle on, the small TLS record that ends a burst -- a packet header,
the last short chunk of an HA snapshot, a reply -- waited for the peer's
delayed ACK, ~40 ms on Linux: seen on 5 of 40 snapshot transfers. Set
TCP_NODELAY on every netio TCP socket, outbound (socket_create) and
accepted (socket_accept).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SSL_get_error() reports SSL_ERROR_SSL whenever the calling thread's
OpenSSL error queue is non-empty. io_worker serves every endpoint from
one thread, so an error left by another connection -- a ctrlstatus
client's EOF queues 'unexpected eof while reading' in OpenSSL 3 -- turned
the next plain would-block SSL_read on a healthy connection into a fatal
error. handle_error() then tore that connection down; mid snapshot
transfer this left a rejoining standby with a truncated snap.part until
its liveness timeout forced a reconnect and a fresh transfer.
Clear the queue before each SSL_read/SSL_write/SSL_do_handshake so the
classification is about that call only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New specs from /kiro-discovery (2026-09-29), based on the LSF 10.1 Admin
Guide section "Job dependency and job priority". Dependency only; job priority
is deferred to a later discovery. Both are added to roadmap.md.
- job-dependency: -w expressions by job ID (done, ended, exit(id[,op code]),
started, post_done, post_err, bare id; && || ! and parentheses). One libxce
grammar, used from Go through cgo. The apiserver validates early (syntax,
parent exists, job:view). The controller re-validates, stores the
expression, and keeps an in-memory reverse index rebuilt on replay. The
scheduler evaluates in sched_job_is_ready(); unsatisfiable dependencies stay
PEND. Also closes the scheduler exit-code field-completeness gap. Adds
xcecli -w / modify -w -wn and harness jobsub -w / jobmod.
- job-dependency-insight: bjdepinfo-style query, dependency info in job
details, and optional orphan termination (-ti or a cluster grace period).
- controller-job-cleanup brief: job-dependency will replace the
job_has_dependents() stub.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
All required tasks are done and feature-level GO is recorded (d19e9f27);
only optional task 8.5 (observing transient catching-up live) remains.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Its false-ERROR-on-reconnect symptom was already fixed by bug-1854 and
bug-1886; today's HA soak shows zero scheduler ERRORs on reconnect.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scheduler-sim-diagnostics, scheduler-sim-live-allocation, xce-test-cli-ux
and eliminate-frontend-makefile had every sub-task checked but their
parent boxes and spec.json phase still read tasks-generated.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test_fmt_limit_units_percent_limit_ignores_units_mode failed on every
run: fmt_limit_units() widened the float limit value before scaling it,
so 0.8 printed as "80.000001%". The product is now taken in float and
widened after, which gives the configured 80.
The failure never counted: test_libxce_retry(), one group of test.c's
single runner, ran its own UNITY_BEGIN()/UNITY_END(), which reset Unity's
counters -- every failure in a group before it was dropped and
xcelib_test exited 0. With the pair removed the runner reports 695 tests;
this was the only hidden failure.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The primary_resreq action ignored parse_resreq_primary()'s result, so a
primary whose select (or rusage) clause failed to parse was accepted
without that clause: select[type==] or select[mem>] matched every host.
A malformed clause now fails the whole resreq, and parse_resreq() returns
the clause's own message.
A select naming an unregistered resource is still dropped, not an
error: parse_select() reports it as -ENOENT (new select_context_t
.unknown_resource) and the action lets it pass, because jobs rely on it
-- the scheduler's own "type == any" default among them when no "type"
resource is registered (making it fatal failed 36 scheduler suites).
Test: test_resreq_primary_select_error_fails_parse RED before (all
accepted), GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
build_wire_job() filled the saved job's submit->resreq with
format_resreq()'s human-readable dump ("Primary:\n Tasks: ..."). The
loader's parser accepted that as a primary with neither select nor
rusage, so every job restored from a snapshot (a replay's start snapshot,
REPL snapshots) lost both.
A scheduler job now keeps the text its resreq was parsed from
(xce_sched_job_t.resreq_str, set at ingest, JOB_UPDATE and snapshot load,
freed at teardown) and the snapshot saves that; a job without it restores
as the loader's "type == any" default.
Test: test_snapshot_round_trip_keeps_job_resreq_text RED before (saved
text was the dump), GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_init_queues() built a new xce_sched_queue_t for every configured
queue on each config apply; for an existing name the add failed, and the
unregistered queue was handed to the picker (which then tracked it
instead of the registered one) and leaked, while the registered queue
kept its old settings. It now updates the registered queue in place, as
QUEUE_UPDATE's core_upsert_queue() does.
sched_init_config() and the core's CONFIG_PUSH replaced core->config
without freeing the previous one. The core owns its config, and nothing
points into a replaced one (queues, resources, groups and limits built
from it are copies), so it is freed once the new one is applied.
Test: test_config_reapply_updates_queue_in_place_and_frees_old RED before
(priority stayed 1), GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_sched_driver_run_normal_mode_round() drained every decision, handed it
to the recorder and the decision callback, and never freed the struct
make_*_decision() allocates: 64 bytes per decision for the life of the
scheduler. The struct is freed after the callback returns; its allocation
or pend reason belongs to the job, and the callback and recorder copy
what they keep.
Test: test_round_frees_decision_after_callback (ASAN shadow check) RED
before, GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- kill_sched_resval() (every resource-value array's item killer) frees
the value, not only a string's buffer: each replaced or torn-down value
leaked its struct, including every WORKERS_PUSH value update.
- put_resvals() shares the value it creates with later entries for the
same index in one call; it created a second one that replaced (and now
freed) the first while still writing through it.
- JOB_UPDATE and the duplicate-JOBS_PUSH path kill_resreq() the old parsed
resreq, and free the rejected job's schedobj (xce_core_free_schedobj(),
exported from core.c), instead of freeing only the containers.
- The previous attempt's (or ALTER alternative's) pend reason is freed when
a new one replaces it.
- kill_resreq_ctx() frees the parser context's error message, and the
primary-resreq PEG action frees an earlier primary's message first.
- write_snapshot_entry() frees the snapshot it builds (config and resource
objects detached, they are the core's); the save side strdup()s its
strings so the generated free_SnapshotData() applies.
Test: test_cli_batch_replay_exits_clean_under_leak_detection
(test_sim_discrete_clock.c) runs the real binary on a two-session fixture
with detect_leaks=1 and requires exit 0: RED before (exit 1), GREEN now.
All nine HA-soak recordings replay with exit 0 and no LeakSanitizer report.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New spec from /kiro-discovery (2026-09-29), added to roadmap.md:
- Begin time: the scheduler won't start a job before it (gate in
sched_job_is_ready() using the core clock). New pend reason
BEGIN_TIME_NOT_REACHED.
- End time: the worker enforces it for running jobs (SIGTERM, grace, SIGKILL)
and records a persisted END_TIME_REACHED exit reason. The primary is not
involved. A pending job whose end time can't be met stays PEND with
END_TIME_UNMEETABLE until the user kills or modifies it.
- Runtime estimate (-We): a scheduling hint only. It sets the rule
now + (estimate, else run limit, else 0) > end; an estimate larger than the
run limit is rejected.
- Times and estimate can be changed only while the job is pending, through a
new `xce jobs modify` in xcecli and a new `jobmod` in the harness. Harness
jobsub gains all four options.
- Fixes the silent drop of xcecli -b/-t/-W and REST run_time_estimation,
which the apiserver's modelJobToSubmit never passes to the controller.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A failed key_add() runs the table's datakiller on the rejected value, and
callers keep using it (JOBS_PUSH merges a duplicate job into the existing
one; sched_init_queues() re-adds every queue on each config apply), so the
job, queue, worker and resource datakillers must stay shallow. Part 4 made
kill_sched_job() free the whole job; that is reverted. Instead
xce_sched_core_fini() collects each table's entries, destroys the table,
then frees each entry with a full destructor (core.c). The resource table
is released the same way (a reference per resobj).
Also:
- dup_val() tested which == XCE_VAL_STRING (0, the resource value type,
equal to Value_u_ivalue32): real string values were shallow-copied and a
nonzero int32 went to makeString() as a pointer. It now tests
Value_u_svalue and strdup()s, as free_Value() expects.
- The shared/host satisfy checks re-initialize the unmeet vector they are
given; the allocators passed the ctx's live one, dropping its buffers on
every candidate host. The callers finalize it first.
- add_vres() registers a resobj via makeResource(), like every other
resource, instead of a bare calloc(); test_limit.c drops it with
dereference().
- plugin_register_core() rejects a duplicate name before init_fn/key_add,
since a failed add now finalizes (frees) the plugin and callers free it
on failure.
- sharedInUse gets kill_sched_resval, as put_resvals() fills it.
Exit-time leak on the reproduction replay 7,469 -> 928 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- kill_sched_job() (the job table's datakiller) frees the whole job:
schedobj, parsed resreq, allocation, accounts and rusages; new
kill_sched_obj() frees the schedobj's names, limit lists, per-primary
wrappers and their pend reasons, and its last allocation/reason.
- The simulator frees each decision struct after draining it (live and
reconciled); the alloc/pend reason it points at stays the job's.
- The builtin_fcfs heap plugin gets a fini_fn that also frees the plugin;
builtin_fcfs_fini() still frees only the picker state (tests run it on
stack plugins).
- xce_sched_core_fini() frees the two pend_updates queues both drivers
calloc().
Exit-time leak on the reproduction replay 11,426 -> 7,469 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_sched_core_load_snapshot() keeps the snapshot's config as core->config
and copies everything else, so the simulator now moves the config into the
core and frees the rest of the decoded SnapshotEntry. xce_sched_core_fini()
frees core->config last, after the plugins and group managers that may
point into it; nothing freed it in either mode. Exit-time leak on the
reproduction replay 25,807 -> 11,426 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New specs from /kiro-discovery (2026-09-28/29), added to roadmap.md:
- job-status-detail-display: show pending reasons and reserved-vs-used
resources in REST and xcecli. Requirements generated (7 requirements).
- scheduler-alter-pend-reasons: keep a pending reason for every alternative
resreq (per-alternative JobPendReason) and fix the obj->reason leak.
- controller-job-cleanup: primary-only sweep that removes finished jobs past
a configurable retention, via a replicated WAL removal fact.
- application-profile: admin-defined application profiles (execution mode,
launch type, env, resource defaults) chosen per job, with app:use/app:view
RBAC. Queues and profiles stay orthogonal.
- job-sandbox-execution / job-sandbox-multinode: run jobs in a systemd
DynamicUser sandbox behind a pluggable execution backend.
- rbac-persistence: make RBAC changes made through the API survive an
apiserver restart (closes the security.md gap).
roadmap.md also records the scheduler-sim-diagnostics info job update and a
security item: remote-task units appear to run as root (found by code
reading).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scheduler-allocator-runner: brief and requirements for moving the duplicated
seqjob/parjob my_alloc() body into one framework-owned allocator runner.
scheduler-sim-decision-trace: brief and requirements for a per-job decision
trace in the simulator REPL (converged after 9 review rounds). The
decision-reasoning seed capability in scheduler-sim-diagnostics now points to
this spec.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sim_process_replay_event_message() now hands the decoded replay event to
the caller, which keeps it beside the translated core event and frees it
after xce_core_push_events() with driver.c's release_translated_payload()
rules: a pushed config is kept by the core, and a rusages report is kept
by the job it found. An untranslatable event is freed at once, and events
after the last session are freed at the end. The payloads were an
"accepted leak" per event; exit-time leak on the reproduction replay
75,495 -> 25,807 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A batch --sim replay under ASAN exited 1 on 17.2 MB of LeakSanitizer
reports. This removes all but 75 KB:
- sched_sim_driver_fini() (called by main after a clean run) cancels and
joins the picker/allocator threads like sched_fini(), closes the replay,
alternate-events and output files, and finalizes the core. The threads'
ids now live in core.jobpicker/allocator, as in normal mode.
- xce_sched_core_fini() destroys the plugin, job, worker, queue, user and
project registries and the shared-resource arrays it creates.
- The seqjob/parjob sched_make_allocs() stand-ins set up only lock, nextid
and the aliased resmgr. Each ran a full xce_sched_core_init(), which also
replaced the process-wide string and selector tables mid-startup.
- plugin_unregister(), the plugin registry's datakiller, only runs fini_fn;
it key_del()'d from the registry being destroyed and skipped plugins
registered with plugin_register_core().
- The limit plugin's usage-instance table used kill_limit as datakiller on
xce_vres_t records (bad free on destroy); kill_vres now frees the record.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A COMPOUND/ALTER job's primaries share one ctx. The job-level prepare pass
reset only alloc_req.shared, and the allocators append their slots want
into alloc_req.host, so a later primary's host pre-filter (and a global
limit on slots) read the previous primary's last per-host want: a 1-task
primary after a 4-task one was filtered on slots:4 and pended although a
host with its one free slot existed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gitea ignores the singular milestone= query parameter and returns the
newest issues repo-wide, so list_issues() never filtered by milestone:
phase-state updates only saw issues on the first page. Use milestones=.
sync_spec() adopted an existing milestone but always created its three
phase issues, so a run killed between creating an issue and writing
spec.json created duplicates on the next run. Reuse an issue already in
the milestone with the same title.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Add test_sim_replay_reproduces_recorded_alloc_from: a recorded DISPATCH at
an epoch-scale beginRound with a sub-second part replays at the default
simulation settings with no --sim-verify mismatch, and the live allocation's
from equals beginRound in whole seconds. Share the dispatch fixture setup
of the existing tests through two helpers.
Live: 2-round HA soak recordings (11 DISPATCH decisions) replayed with
--sim-verify and no --sim-speed report 0 mismatches.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
