A winning RequestVoteReply is dispatched through peer_ctrl_classify, which
returned VTASK_LIVENESS for all peer-control traffic -- the never-blocks,
never-admission-gated class. The reply runs raft_cb_become_leader, whose
tail persist_apply_pump_run_sync() does bounded blocking WAL pread()s (it
cannot be made async without reopening the bug-1043-family job loss on a
freshly promoted primary). The fdatasync half of bug-1433 was removed in
40f510fb; this pread was the tracked residual.
peer_ctrl_classify now returns VTASK_LIVE_BLOCK while the node is
CTRL_ELECTING -- its peer-control traffic is then vote/PreVote replies,
election work like inbound RequestVote, which already uses LIVE_BLOCK.
As PRIMARY/STANDBY the traffic is WAL/config ACKs and stays VTASK_LIVENESS,
so a quorum of acks is never admission-gated (R4.5); a candidate has no
commit to advance.
New test_peer_ctrl_classify_vote_replies_while_electing (RED before).
Controller suite 1240/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
"ERROR incorrect worker idx -1" on every node after a worker-join commit
timeout (during a PreVote/election) was the visible end of a real
dangling pointer. init_worker/rejoin_worker bind the endpoint to the
worker (apply_join_state: w->ep = ep, UNAVAIL cleared) before the commit
barrier; on barrier failure handle_worker_join returned without setting
priv->u.idx, so when that connection closed handle_endpoint_error saw
idx -1, logged the ERROR and never ran handle_worker_leave -- the worker
kept w->ep pointing at the freed endpoint, available for dispatch, until
it rejoined.
- handle_worker_join: on barrier failure, clear w->ep (if still this
endpoint) and mark the worker unavailable, as if the join never
happened; the worker retries its join.
- handle_endpoint_error: an idx -1 worker endpoint is now the legitimate
"closed before its join completed" case -- WARN, nothing to clean up.
test_worker_join_withheld_acks_replies_error also asserts the worker is
unbound and unavailable after the -ETIMEDOUT reply (RED before).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
f88e4ef4 moved ha_demotion_convergence_fn's WAL truncate + self-restart
and ha_catchup_continuation_fn's WAL streaming off the delaymgr thread
(which drives heartbeats and election timeouts cluster-wide) onto the
slow lane, but left both flags OFF: flipping them ON hung pre-existing
unit fixtures that demote a primary or trigger catch-up without ever
starting c->slowlane -- the work was queued in a lane with no workers and
never ran.
- slowlane_is_started(): true while the lane has live workers.
- Both dispatchers run inline when the flag is OFF *or* the lane was never
started. Production always starts the lane, so it always offloads; the
lightweight fixtures keep their old inline behaviour, and neither
one-shot action can be silently dropped.
- g_drive_async_demotion_offload / g_drive_async_catchup_offload now
default to 1. The offload tests save and restore the previous value
instead of forcing 0.
New test_catchup_offload_without_started_lane_runs_inline (RED before: the
catch-up body never ran, the stream gate was never reached). Controller
suite 1239/0 with both offloads ON by default.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A job the controller requeued to PEND while the scheduler was disconnected
stayed RUN (with its old allocation) in the scheduler forever: the
bug-1812 guard in core_events.c's JOBS_PUSH merge refused any non-RUN,
non-terminal resync claim over a job holding a real allocation, because
it could not tell a genuine requeue from a push built before the
scheduler's own fresh dispatch.
Design (per review): the scheduler persists nothing, so on rejoin the
controller's state wins, and the race the guard protected against is
closed where the truth lives:
- scheduler JOBS_PUSH: the bug-1812 refusal is removed; the controller's
state replaces the scheduler's and its allocation is released. The
terminal-is-permanent and phantom-RUN (RUN without allocation data)
guards stay.
- controller dispatch_a_job_(): refuses a job that is not PEND (RUN or
finished) before any worker is touched -- previously nothing checked,
so a stale second dispatch would start a running job again (or, through
a retryable failure, requeue it). The job is left as is, no WAL fact,
and its true state is re-sent (HOOK_DETAIL_JOB_UPDATE). A PEND job with
a leftover allocation stays dispatchable.
- scheduler JOB_UPDATE: a RUN report carrying the controller's committed
allocation is adopted, so a corrected job is never RUN with nothing
charged.
Tests (all RED before): test_jobs_push_duplicate_jobid_controller_pend_
wins_over_local_run (replaces the bug-1812 test that pinned the
refusal), test_job_update_run_adopts_controller_allocation,
test_dispatch_of_non_pending_job_is_refused. Two tests that seeded a RUN
job to reach retry branches (a precondition their own headers called
unreachable) now use the real paths: the counter-reset test requeues via
record_job_reconcile_pending_fact() first; the disconnected-worker RUN
test asserts the refusal contract.
Controller suite 1238/0, scheduler 87/87. HA soaks: chaotic 6 rounds PASS
(0 divergence/loss, 6 worker-resync jobs), sequential 6 rounds PASS
(SIMVERIFY 7 files, 0 mismatches). The race did not occur in either run
(0 refusals logged), so the new paths are proven by the unit tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gitea #47: `xcecli jobs submit -R "rusage[mem=100000000a0]"` was accepted
with a normal "Job <...> is submitted" reply. The apiserver only
length-checked resreq; the real PEG grammar ran later, asynchronously, in
the scheduler's core_build_job(), which logged a server-side WARN and
dropped the job -- it then sat PEND forever with no sign of why.
- libxce: xce_check_rusage_syntax(resreq, &errmsg) runs parse_rusage() on
every rusage[...] clause (extracted the way section.peg does: up to the
first ']'), so primary, compound and alternative requests are all
covered. No resource manager needed, so select[]/order[] resource names
are still only checked by the scheduler (the documented scope).
- apiserver/backend: ValidateResreqSyntax() cgo wrapper (resreq_check.go);
the !cgo stub returns nil (no job can be submitted without CGO anyway).
- controllers/jobs.go preCheckJobRequest(): calls it after the length
check via a package var, so a malformed request is refused up front.
Tests: lib test_check_rusage_syntax (RED before: the reported input was
accepted) and backend TestValidateResreqSyntax (RED before) through the
real grammar; controllers TestPreCheckJobRequest_RejectsMalformedResreq
checks the call site with an injected validator (written with the call
site, so no separate RED). libxce, apiserver backend (cgo) and controllers
tests pass. apiserver/middleware TestLoadJobs fails here and on the
baseline alike: it needs a live apiserver on :8080.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
do_update_resvals() fired the AFTER_EVENT HOOK_DETAIL_RESOURCE_UPDATE with
the raw client xce_resval_list_t. update_resval_list() rejects unknown
resource names (-ENOENT) but only logged it, so both listeners -- the
scheduler push and the apiserver store (apiserver_resource_hook_fun ->
handler.go) -- received names the controller had rejected, and the
apiserver stored them.
- update_resval_list() takes an optional `applied` view (caller-sized,
borrowed pointers) and fills it with just the accepted items.
- do_update_resvals() hands that view to the AFTER_EVENT hook, and skips
it when nothing was applied. The request still owns every item.
New test_update_resval_list_reports_only_applied_items (RED before: both
items reported). Controller suite 1237/0. Live /xce-test-resources with
mem=5 totally-fake-resource-xyz=1: mem MATCH at controller, scheduler and
apiserver; the fake name is rejected at injection and appears 0 times in
scheduler.log / apiserver.log (verdict FAIL by design, for the rejected
name).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
snapshot_load.c carried its own field-for-field copy of core_events.c's
schedobj builders (primary/compound/alter + hlimits killer), kept separate
by a "never share builders" convention. The copy drifted from the original
three times, each a field-completeness bug in restored jobs (bug-1772's
all-zero resreq_fp among them). The convention's reasons apply to the
job/queue/worker builders, not to schedobj, which is never wire-captured.
- core_build_schedobj() becomes the exported xce_core_build_schedobj(obj,
user, queue, project, resreq) (core.h), taking identity strings instead
of a wire job; core_build_job() and snapshot_load.c both call it.
- snapshot_load.c's five copied functions are removed.
- test_sim_startup.c #includes snapshot_load.c without core_events.c; it
now also #includes core_events.c, plus the same
sched_reconcile_job_resource_usage link stub test_driver_translate.c
uses.
- test_fresh_ingest_and_snapshot_restore_derive_identical_... now compares
the whole schedobj (tag, user/queue/project, configver, glimits, owner,
resreq binding, resreq_fp), guarding against future drift. No RED phase:
there is no live gap left to catch; this removes the duplication that
kept producing them.
Scheduler suite 87/87; full build links.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
JOB_UPDATE (core_events.c) and the legacy sched_update_existing_job()
(schedobj.c) replaced job->resreq, but the allocator reads job->schedobj,
whose primary->resreq still pointed into the old parsed request -- so an
update never changed scheduling, for any field. The old primary was also
leaked (FREEUP only freed the container).
- core_bind_primary_resreq(): the resreq pointer + fingerprint computation
factored out of core_build_primary_schedobj(), reused on update.
- core_rebind_schedobj_resreq(): primary->primary re-points in place and
recomputes the fingerprint, keeping the schedobj's runtime state; a kind
change rebuilds the kind-specific part (the old part is left alone, as
nothing frees these today -- never a dangling pointer).
- JOB_UPDATE now applies the job's ingest-time task count to the new
resreq, as core_build_job() does, so resreq-maxtasks-wiring Req 1.4 (an
update never changes the dispatched task count) still holds -- it used to
hold only because the update never reached the schedobj at all.
- schedobj.c's legacy path gets the same re-binding.
test_resource_request_update_never_changes_dispatched_task_count asserted
the old behaviour (schedobj keeps the ingest-time object); it now asserts
the schedobj uses the updated resreq and fingerprint while the task count
stays 4/7 (RED before). Scheduler suite 87/87.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-448: job_ctrl() took a reference on the job but only one error path
released it; every successful call leaked one (refcount 1 -> 3 after two
calls). It is now released on every exit path, including the key_find
rc<0 one.
bug-1626: job_ctrl()'s not-started branch changes the job (mark_job_update
/ mark_job_fail -> apply_job_*_transition, which require RTTI_LOCK_WRITE)
while holding only RTTI_LOCK_READ, so it could run concurrently with
readers. It now takes the write lock. (dispatch_a_job_ already did.)
bug-1631: dispatch_a_job_()'s four retryable-failure branches never reset
state/allocs, correct for a first dispatch (still PEND) but not for a job
that had already been dispatched once: it stayed RUN holding its old
allocation. Each branch now calls record_job_reconcile_pending_fact() --
a no-op without a WAL fact for a PEND job; for a RUN job it returns it to
PEND, releases allocs and appends a durable WAL_JOB_UPDATE -- and notifies
after unlocking when it applied, the same pattern as the resync reconcile
path (cfe66dde). Not reachable until something re-dispatches RUN jobs
(e.g. preemption), fixed now so it is correct when that lands.
- job_get_refcount() (jobj.c) read accessor for the leak test.
- test_apply_pump.c: test_job_ctrl_releases_its_job_reference and
test_job_ctrl_mutates_not_started_job_under_write_lock (both RED before).
- test_dispatch_fail_retry.c now asserts PEND + allocs released after a
post-success retryable failure (RED before).
Controller suite 1236/0; all 111 meson suites pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
persist_checkpoint() and persist_checkpoint_marker_triggered() wrote a
local-only WAL_CHECKPOINT marker that consumed an LSN of the replicated log
(current_lsn++) but was never replicated. A checkpoint taken by one member
only -- the primary's periodic max-wal-records checkpoint, or the one before
a snapshot send -- left a hole no standby had: the leader's next entry
carried prev_log_index = the marker's LSN, logOk failed on every standby
forever, commits timed out and writes stalled ("could not submit 110 jobs"
in the --force-snapshot soak; reproduced on the pre-bug-1867 baseline). Even
the config-marker checkpoint, taken on every member, stamped the marker with
each member's own current_lsn at apply time, which need not agree.
Nothing reads the marker except to skip it, so checkpoints simply stop
writing it. The recovery, streamer and bug-1475 log_len skip paths stay, for
WALs written by older binaries; persist_test_write_legacy_checkpoint_marker()
(XCE_TESTING only) reproduces such a WAL for the tests that cover them.
- new test_bug1870_leader_only_checkpoint_leaves_no_log_hole (RED before):
a leader-only checkpoint takes no LSN and a standby that never
checkpointed catches up past it via the real streamer/receive path.
- test_checkpoint_writes_marker_to_wal inverted to ..._writes_no_marker_...
- bug-1475 x2 and the apply-correctness trailing-marker test now build
their legacy marker explicitly.
Controller suite 1234/0. HA soaks: --force-snapshot PASS (113 jobs, real
snapshot install -- also live-verifies bug-1867), --config-upgrade-wal-trim
PASS, --sequential 6 rounds PASS (SIMVERIFY 7 files, 0 mismatches).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The capnp Job struct -- the checkpoint/entity-snapshot form of xce_job_t --
had no exit-info field, so capnp_encode_job() never wrote job.exit_info
and a job restored from a snapshot (instead of WAL replay) came back with
exit_info NULL, for DONE/EXIT and failed jobs alike. bug-1689/bug-1754
made the WAL facts carry the code; anything older than the last
checkpoint's WAL trim still lost it.
- controller.capnp: struct JobExitInfo $C.mapname("job_exitinfo_t")
{ exitcode }, and Job.exitInfo @12 $C.mapname("exit_info"). The
generator now emits encode/decode/free for it inside encode_Job /
decode_Job / free_Job; restore_job()'s create_job() already takes
exit_info via MOVE_PTR. Null stays null (no fabricated exit 0). A new
ordinal, so older snapshots/recordings decode with exit_info NULL.
New test_snapent_checkpoint_exit_info_roundtrip (RED before): exit codes
0 and 137 and a NULL survive both the raw checkpoint frames and a real
persist_load_snapshot restore. Full /xce-build all (asan): 111 meson
suites, 0 failures; all Go packages pass.
Not live-verified: the --force-snapshot HA soak cannot reach its snapshot
step because of a pre-existing checkpoint-marker replication stall,
reproduced on the baseline without this change (bug-1870).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The FAIL half of bug-1747: apply_job_fail_transition() stamps
finish_time = time(NULL) at APPLY time, and WAL_JOB_FAIL carried no time,
so a failed job's finish_time changed on every restart replay / standby
catch-up, and a worker-reported failure found on rejoin could never keep
the worker's own time.
- WAL_JOB_FAIL's trailer grows to int32_t code + uint64_t finish_time
(seconds). Decoded by length: 16 B (pre-bug-1754), 20 B (code only,
bug-1754) and 28 B (code + time) all apply as before for what they carry.
- record_job_fail_fact_at(c, job, failinfo, finish_time_ms), the fail-side
sibling of record_job_finish_fact_at(): 0 = receipt time. on_job_fail()
uses receipt time; resync's finishedFail branch passes the worker-
reported finish_time_ms (Req 4.2 now holds for failures too).
- Both apply seams adopt the wire time after the transition.
4 new tests in test_wal_job_fail_fact.c (3 RED before the fix) and the
resync round-trip now asserts the worker-reported failure time (RED
before). Controller suite 1232/0. Factstream HA soak (2 rounds, 60 jobs)
PASS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xcecd runs as root and kept files at fixed names in world-writable /tmp,
opened by path:
- main.c created <logdir or /tmp>/xcecd_worker.log and then chmod(path,
0666)'d it: a symlink planted there made root chmod 0666 any file
(e.g. /etc/shadow).
- process_worker_manager.c appended xcecd_worker's stdout/stderr to
/tmp/xcecd_worker.log through any planted symlink/hardlink.
- xcecd_prepare_restart() created /tmp/.xcecd.restarting through a
dangling symlink (an empty root-owned file anywhere), and
xcecd_check_restart_type() trusted access() on it, so any user could make
an unplanned restart look planned.
xcecd_tmpfile.h:
- xcecd_open_owned_file(): reuses an existing file only if it is our own
regular single-link file (so a legitimate log keeps being appended to);
otherwise unlinks whatever is there and creates a fresh file with
O_EXCL|O_NOFOLLOW. O_NONBLOCK so a planted FIFO cannot hang startup.
- xcecd_owned_file_exists(): lstat-based; only our own regular file counts
as the restart flag. Anything else at the name is removed and ignored.
- The worker log's 0666 mode is now set with fchmod() on the fd.
New test_xcecd_tmpfile.c (7 tests; with the old behaviour the symlink,
dangling-symlink and hardlink cases FAIL and the FIFO case hangs);
worker suite 12/12. Sequential HA soak (5 rounds, xcecd_virtual) PASS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
uv_proc_exec() created /tmp/job_<jobid>_task<N>.out as root with
O_WRONLY|O_CREAT|O_TRUNC. The name is predictable and /tmp is world-
writable, so a local user could plant a symlink (or hardlink) there before
the job started and make the root worker truncate and write job output
into the target. Only the host's fs.protected_* sysctls stood in the way.
xcecd_create_job_output() (job_output_path.h) now unlinks whatever is at
the name and creates a fresh file with O_EXCL|O_NOFOLLOW|O_CLOEXEC, mode
0600 as before; O_EXCL never follows a symlink and fails if anything
reappears, so a race can only fail the create, never redirect it (retried
3 times). A re-run job still starts with an empty file, as O_TRUNC gave.
New test_job_output_path.c (4 tests, 2 RED before the fix); worker suite
11/11.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
handle_fetchfile() runs as root in xcecd and open()ed whatever filepath
the request carried. Any libxce client could read any file on a worker
(/etc/shadow, keys) with a fetchfile RPC, and so could any REST user: the
apiserver resolves the path from the job's user-supplied out_file/err_file,
so submitting a job with out_file=/etc/shadow and then reading its output
passed the apiserver's ownership check and read the file as root.
- job_output_path.h: the per-task output name uv_proc.c writes,
<dir>/job_<jobid>_task<N>.out, now shared by the writer (xcecd_worker)
and the reader (xcecd), plus xcecd_is_job_output_path(), an exact
string match for that job's names only.
- fetchfile_open_checked(): rejects any other path with -EACCES, opens
with O_NOFOLLOW|O_NONBLOCK|O_CLOEXEC and requires a regular file with a
single link, so a symlink, hardlink, FIFO or directory planted at the
output name is refused.
Three new tests in test_controller_connector.c (2 RED before the fix);
worker suite 10/10. Sequential HA soak (5 rounds, xcecd_virtual) PASS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
gitea-sync.py: --ids syncs named bugs regardless of dates (OR'd with
--since). Bulk triage bumped last_seen/status_updated across most of the
buglog, so --since 2026-09-01 matched ~1150 bugs while recently fixed older
bugs (bug-1798..1804, 1714, 1754, 1850, 1854) could never be reached.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A worker reports every nonzero job exit as a FAIL (xcecd_job_finished ->
xcecd_notify_job_fail(jobid, stage, exit_code)), not a FINISH. The deferred
WAL_JOB_FAIL fact carried only {jobid, STAGE_FLAG|stage} and no apply seam
touched job.exit_info, so exit_info stayed NULL for every failed job, live
and after replay, and HOOK_DETAIL_JOB_FAIL subscribers saw code 0.
- persist_submit_job_fail_fact(): header + int32_t code trailer, used by
record_job_fail_fact(). Old entries (no trailer) decode as before.
- apply_job_fail_exit_code(): records the code in exit_info for pre-exit
stages only; a POST_EXEC code is the script's and leaves exit_info alone.
Kept out of apply_job_fail_transition() so mark_job_fail()'s internal
-errno failures (not replayed with a code) stay live/replay consistent.
- Both apply seams adopt the code. apply_wal_entry()'s WAL_JOB_FAIL case
also stops assigning the wire state 0 for STAGE_FLAG entries and runs the
real transition, as apply_committed_entry() already does.
New test_wal_job_fail_fact.c (5 tests, 3 RED before the fix); the two
test_worker_resync.c assertions that pinned exit_info == NULL now assert the
worker-reported code. Controller suite: 1228 tests, 0 failures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every scheduler (re)join re-pushes the whole config, and both apply paths
-- sched_init_resources() (scheduler.c) and core_apply_config_resources()
(core_events.c) -- re-added each resource with makeResource(), which
refuses an existing name: one ERROR per resource per rejoin (33 in
data/xce_scheduler.log), and a genuinely changed definition was dropped
just as silently.
New libxce resourceApplyDef(): under one write lock, register a new
resource (ADDED), or compare an already-registered one on its
scheduling-relevant fields -- valtype, ordtype, flags, interval -- and
return SAME or CHANGED. Both call sites log SAME at debug, CHANGED as a
WARN that the definition is not applied to the running scheduler
(updating a registered resource in place would disturb its index and
accounting), and only real failures as ERROR.
Test test_resource_reapply_same_is_noop_changed_is_reported (EEXIST on
an identical re-push before). Scheduler suite 87/87, lib 1/1. Re-verify
of data/scheduler-replay-20260925-133708.replay: 0 add-resource ERRORs
(33 before), 0 changed-definition WARNs, 0 mismatches, same 60 DISPATCH /
908 PEND decisions.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_failover_classify() mapped E_XCE_NOT_READY with NOMASTER to
RETRY_BACKOFF, so xcecli_retry_next() rotated to the next master,
reconnected and fired the rejoin hook. But a NOT_READY reply comes from a
controller that answered on a working connection -- here a healthy primary
that had briefly lost its write quorum (both standbys NACKing, bug-1864).
The scheduler abandoned it for the standbys, rejoined, and got a full
re-push built from lagging committed state (21:39:24-27 in data/logs).
New action XCE_FO_RETRY_SAME (appended; existing values unchanged):
NOT_READY backs off (same jittered schedule, same -E_XCE_NOT_READY on
budget exhaustion) and re-sends on the current connection -- no rotate,
reconnect or rejoin. NOMASTER keeps rotating after backoff. If the
controller is really a standby it answers REDIRECT/STANDBY/NOMASTER next
time, and if it dies the transport error rotates, as before.
Tests (test_libxce_retry.c): taxonomy row NOT_READY -> RETRY_SAME;
decide backs off and exhausts to -E_XCE_NOT_READY; the real
xcecli_retry_next keeps cur_master on NOT_READY (rotated before) while
STANDBY still rotates; backoff-growth test moved to NOMASTER. Failover
tests 56/56; lib, controller 5/5, worker 10/10, scheduler 87/87,
harness 5/5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
handle_ctrlhello stored the netio endpoint's fd (primary_ctrl_fd = ep->fd)
and ha_on_primary_ctrl_fd_rdhup close()d it in three branches. netio owns
that socket and closes it on endpoint teardown, so by the time the
liveness poll fired the RDHUP the number could be reused -- in a chaotic
soak by the standby's WAL segment: HA's close() shut the WAL, fdatasync
failed 2305 times (EBADF, then EINVAL once a socket took the number),
durable_lsn stuck, and one round's jobsub failed EIO.
handle_ctrlhello now also holds a reference on the endpoint
(primary_ctrl_ep), which keeps its socket open and its fd number ours.
ha_drop_primary_ctrl_conn() replaces the raw closes: shutdown() the
still-owned socket so netio sees the hang-up and tears it down itself,
then release the reference. A bare fd with no endpoint is only forgotten.
Test test_rdhup_never_closes_an_fd_netio_already_released: real
ctrlhello -> netio teardown -> file opened on the freed number -> RDHUP;
the file was closed before, survives now. test_rdhup_closes_and_nulls_fd
updated to the new contract (peer reads EOF, fields cleared). Controller
suite 5/5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The dispatcher checked each queued task's endpoint busy flag one item at a
time, while the vcpu that finished the endpoint's previous message cleared
the flag without the taskq lock (a lock there deadlocked, bug-1282). Within
one scan, task N was skipped (endpoint still busy), the endpoint freed up,
and task N+1 of the SAME endpoint was dequeued first. Standby controllers
then handled WAL catch-up entry 258 before 257 and NACKed (log mismatch):
26 reorders -> ~576 NACKs in a 45-min production-like run, both standbys
at once at 21:39:24, primary lost its write quorum and answered
jobdispatch/join with E_XCE_NOT_READY, the scheduler client failed over and
rejoined, and the rejoin JOBS_PUSH carried lagging committed state
("refusing resync state ... existing job is RUN" WARNs).
dataq gains dataq_deq_ctx() (a per-scan matcher context). The scan is now
vtask_deq_eligible(taskq, is_blocked): it remembers every endpoint found
busy and keeps all later tasks of that endpoint ineligible for the rest of
the scan (past 64 distinct busy endpoints, every later IO task waits for
the next scan) -- per-endpoint FIFO without any new lock.
Test lib/testsuite/test_vtask_order.c injects is_blocked to free the
endpoint mid-scan: task N+1 was dequeued before the fix, none after; other
endpoints and FIFO/generic tasks unaffected. Suites: lib 1/1, controller
5/5, worker 10/10, scheduler 87/87, harness 5/5. Chaotic HA soak 2/2 PASS
with 0 WAL reorders (2 and 5 per run before), no ASAN.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When a later primary of a COMPOUND failed, sched_make_compound_alloc_
still moved the earlier primaries' merged allocation onto the schedobj,
so a job decided PEND carried a partial allocation that the next
successful round overwrote and leaked. The merged allocation is now
published only when every primary succeeds and freed otherwise. No
resource-state rollback is needed: the primaries' uncommitted debits live
in ctx.uncomitted, per-attempt scratch freed by sched_ctx_fini(), and
schedobj->alloc's only reader is the dispatch path. The ALTER loop's
per-alternative cleanup (added with bug-1769) is now redundant and removed.
Test: "1*{..} + 5*{..}" on 2x2 slots PENDs with schedobj->alloc NULL
(held the first primary's allocation before). Scheduler suite 87/87.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The 16 commits since 534895c5 were re-signed (rebase --exec amend -S; tree
unchanged), which changed their hashes. Update the buglog and the
scheduler-allocation completion note to the new ones.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each primary's allocator (seqjob/parjob my_alloc) sets ctx->alloc to a
fresh job allocation covering only its own hosts, so a multi-primary
COMPOUND ("1*{A} + 2*{B}") dispatched only the last primary's hosts --
leaking the earlier allocations while every primary's hosts had been
debited in ctx.uncomitted.
sched_make_compound_alloc_ now folds each primary's allocation into one
job allocation via the new libxce merge_job_alloc(): a host placed by
several primaries becomes one entry with per-resource values added
(matched on resname, instid and reserve, so distinct instances are never
summed); new hosts and shared allocations are appended/merged the same
way. Partial-failure rollback remains bug-1766.
Tests: "1*{..} + 2*{..}" on 2x2 slots now allocates 3 slots across two
distinct hosts; "1*{..} + 1*{..}" yields one host entry with 2 slots
(both kept only the last primary before). Scheduler suite 87/87, lib 1/1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seqjob_prepare_allocation() and parjob_prepare_allocation() returned 0
whenever ctx->schedobj was not PRIMARY, so a primary nested in a COMPOUND
(or ALTER) job never had its slot count or rusage checked: a 5-task
compound dispatched onto 4 slots, and an ALTER job was "placed" by its
first alternative with an empty (0-slot) allocation. Both hooks now
compute the want from ctx->primary -- the primary being allocated, set by
sched_ctx_set_primary() for nested primaries and identical to
obj->u.primary->resreq for PRIMARY jobs.
sched_ctx_set_primary() also clears ctx->worker, so the next primary of a
compound computes its job-level want rather than a per-host want left
over from the previous primary's host (PRIMARY jobs set it once, with
worker already NULL).
Tests (real ingest + live allocator): a 5-task COMPOUND on 2x2 slots now
PENDs; "{5*{..}} || {1*{..}}" falls back to the second alternative (1
slot). Both failed before. Scheduler suite 87/87.
Follow-up logged: bug-1863 (multi-primary COMPOUND keeps only the last
primary's allocation).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1769: every builder stores xce_compound_schedobj_t* (24 bytes) in
obj->u.alters, but sched_make_alter_allocation() read them as
xce_sched_obj_t* (176 bytes) and ran the compound allocation on them -- a
heap-buffer-overflow read of ->u.compound and write of ->alloc for any real
`{..} || {..}` resreq. The compound allocation now takes its resreq
explicitly (sched_make_compound_alloc_), and each ALTER alternative is
tried on behalf of the ALTER obj itself; a failed alternative's partial
allocation is dropped before the next (rollback itself is bug-1766).
sched_ctx_init() rejected ALTER objs ("bad sched obj type"), so ALTER jobs
could never allocate at all -- it now accepts them like COMPOUND (all
downstream checks only distinguish PRIMARY).
bug-1771: plugin_commit_decision() fans every DISPATCH out to the legacy
"xce_sched_fcfs" plugin even when it never tracked the job; fcfs_del_job()
(and fcfs's job-add hook) dereferenced job->queue->name with no NULL
check, SEGV'ing the allocator for a job whose queue was never registered.
Tests (test_parjob_e2e_multislot.c, real ingest + live allocator thread):
test_alter_resreq_job_dispatches_via_real_allocator (ASAN
heap-buffer-overflow before, DISPATCH after) and
test_dispatch_of_job_with_unregistered_queue_does_not_crash (SEGV at
fcfs.c:318 before). Scheduler suite 87/87.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
All 16 subtasks were already done; check the 6 parent tasks and set
phase implementation-complete. Verified 2026-09-24/25: scheduler suite
87/87 (ASAN), live multi-host e2e PASS, HA soak sequential + chaotic PASS.
Out-of-scope follow-ups stay open: bug-1766, bug-1768, bug-1769,
bug-1771, bug-1860.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
