ParseSubmitIndexExpr accepts only name[1-N] and name[1] (leading zeros
allowed) and otherwise reports the lowest-numbered broken rule with its
specific message: interactive arrays (2.1), invalid name or malformed
expression (2.2), lists/steps (2.3), %N (2.4), start 0 (2.5), reversed
(2.6), start above 1 (2.7), above the limit (2.8). Oversized numbers
saturate for the bound checks while reversal compares the digit
strings, so the rule order holds for any input. Not yet wired into
CreateJob (task 6.2).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CreateJob's ID generation and submission and both control dispatch
sites now go through package variables (generateJobID,
generateArrayJobID, submitJob, controlJob) that default to the real
backend/services functions, so production behaviour is unchanged. A new
non-cgo handler test file provides a reusable fixture: it saves and
restores every global it touches, sets up auth/RBAC, seeds jobs and
their queues, and records each dispatched job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
QueryOptions gains Predicates; store.ExecuteQuery keeps a row only if
every predicate returns true, after the plain conditions and before
sort, pagination and field selection, so page boundaries and tokens
count only matching rows. Nil/empty predicates leave every existing
query unchanged. Foundation for evaluating element selections
(bug-1898).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
job.max_array_size is resolved by a pure resolveMaxArraySize: unset or 0
gives 1000, 1..32767 is used as is, values above 32767 are clamped to
32767 with a startup warning naming the configured and effective value,
and a negative value refuses startup. The old >2147483646 rejection is
gone; the shipped config documents the bounds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GenerateArrayJobID wraps libxce idgen_array (first element, index 1;
one-element arrays included) with a non-cgo stub; pure helpers
ParseHumanJobIDStrict (written identity with exact field-width checks,
uint64 overflow reported as a width error), IsArrayCapableKey and
MakeElementID (bit-identical to libxce jobsel_element_id).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every harness command that acts on one job (jobinfo per -j, jobrun,
jobkill, jobckpt, taskrun, setpend, jstart, jfinish, jfail) and
expect-job parse their job argument with cmdutil_job_arg on top of the
libxce selection parser: element ranges (reversed too) are refused with
the single-element message, invalid identities are rejected with the
parser's reason, and no command sends job 0 for a rejected argument.
Not-found results (controller -ENOENT, or jobinfo's empty reply) name
the job and, for bare identities, hint at <array-id>[k]. expect-job
matches job numbers instead of formatted text.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Auto-variable substitution moves out of test.c into its own unit-tested
module (autovars.c) with an explicit context for $jobid, $? and
$jobcount, keeping today's token rule, messages and buffer handling.
New: a token $jobid[...] expands $jobid and keeps the rest of the token,
so $jobid[3] names element 3 of the array just submitted; it fails like
an unset $jobid when none is set.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
jobsub -a parses its value strictly (digits only, leading zeros ok):
1 submits an ordinary job; 2..32767 allocate with idgen_array, whose
result is checked; missing, non-digit and 0 values are refused with the
1-32767 range and values above 32767 with the limit, submitting nothing.
The success line and $jobid name the array identity; the random
sequence seed is bounded so the first array cannot overflow; help text
updated. Fixes bug-1896 for the harness.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The worker sets XCE_JOBINDEX to the element number for jobs carrying the
array marking (a one-element array's element gets 1) and to 0 otherwise.
A submission-supplied XCE_JOBINDEX is not applied and a WARN names the
job and the ignored variable; XCE_JOBID handling is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scheduler expands an array push only when the first ID carries the
array marking and the size fits in its block (size <= 32768 - idx, no
addition); otherwise it logs an ERROR naming the array identity and
expands nothing. Pre-upgrade arrays starting at index 0 still expand.
The INFO log names the array identity (array_jobid=...). This also
closes a crash: a malformed push with a huge size overflowed calloc.
Expansion tests move to tagged block|1 first IDs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
handle_jobsubmit now registers the submission first and checks the job
table second; a hit is answered -EEXIST and released through the atomic
reply-and-release step. For array submissions, array_identity_exists()
does one read-locked scan of the 32768 tagged element keys of the
array's block (element 0 included for pre-upgrade arrays), covering
both the element range and the array identity; unmarked submissions
keep jobid_range_exists. An existing X|3 now blocks a new X|5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
In-flight registry entries carry is_array and array_id; two array
entries with the same array identity overlap. inflight_reply_and_remove()
sends the reply and unlinks the entry under the registry lock (xce_reply
is non-blocking) and is used on every post-registration path, inside the
async one-shot reply guard; jobsubmit_async_ctx_release no longer touches
the registry, so a retry registered after the reply cannot be deleted by
the original's release. Test-only seams under XCE_TESTING make the
atomicity observable deterministically.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
handle_jobsubmit classifies (first_jobid, array_size) before the field
checks, the before-event hook and any registry access: a tagged array
starting at index 0 is answered with the new E_XCE_ARRAY_START0 and an
ERROR log asking for the client to be upgraded (7.1, wins per 7.9);
size 0, tags 2/3, untagged multi-element submissions and arrays whose
last element exceeds 32767 (checked without addition) get -EINVAL.
Nothing is stored for a rejected submission. This also closes a crash:
a size-0 submission used to reach payload building and overflow calloc.
Existing array tests move to tagged block|1 first IDs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
idgen_array() allocates the next whole 32768 block and returns its
first element (array tag, index 1), refusing n outside 1..32767 and a
block past the 20-bit sequence (-EOVERFLOW, context unchanged). The
reset-on-new-second step is shared with idgen() via idgen_tick_locked();
idgen() itself is unchanged until task 9.1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
parse_jobid() keeps its signature but is now bounded and NULL-safe: a
bare identity returns the exact ID, [k] the element ID, and a range, any
rejected selection (5.7, 5.8, 5.14, 5.15) or a NULL argument returns
-EINVAL with *jobid = 0. Fixes the strlen-before-NULL-check crash and
the atoi overflow; regex parsing is gone.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
parse_jobsel() parses c-a-ts-seq with an optional [k] or [a-b] under a
fixed grammar and check order (syntax, field widths, array-identity
alignment, element bound, reversed range), saturating digits so an
oversized number cannot wrap; plus is_array_tag(), jobsel_element_id(),
jobsel_strerror() and JOBID_FIELD_MAX_* constants. The selection-cases
conformance table runs in the libxce suite.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Spec for numbering job-array elements from 1 (bug-1896): approved
requirements, a design reviewed over 20 rounds until clean (log in
research.md), and a 30-task plan. The design review found two live
apiserver defects, logged here: bracketed job_id/job_name filters are
never evaluated (bug-1898) and ControlJobByID re-dispatches earlier
selections (bug-1899).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
unsatisfy_() compared positional total/inuse/wanted arrays (entry i is
resource wanted->idxs[i]) with uncomitted[i], but the allocation
context's uncommitted vectors are dense and keyed by resource index.
From the second primary of a COMPOUND/ALTER job on, the check ignored
earlier primaries' claim on the same resource (over-admit) or charged
another resource's (wrong pend), with the matching wrong shortfall in
the pend reason.
unsatisfy_() now takes the vector and reads entry wanted->idxs[i]; an
index past its end (sim_loop.c's short all-zero vectors) or a NULL
vector counts as nothing uncommitted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A job whose snapshot resreq fails to parse (any replay captured before
the bug-1888 fix) is kept by xce_sched_core_load_snapshot() with no
schedobj, but was still made a picker candidate -- by the loader and, in
simulation mode, by sim_bootstrap_picker()'s picker rebuild. The
allocator thread then dereferenced the NULL schedobj in
limit_job_election() and xce_scheduler --sim crashed.
Both push sites now skip such a job (it stays in the core so a restored
allocation is still charged), and sched_allocator() skips any picked job
without a schedobj as a backstop.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Feature validation GO (ASAN scheduler suite 96/96, smoke replay clean on
the allocation path). Roadmap entry marked implemented with follow-ups.
bug-1897 (found by the validation smoke run, not caused by this feature):
xce_scheduler --sim segfaults in limit_job_election on a job whose
snapshot resreq failed to parse (pre-bug-1888-fix replays).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Clean ASAN scheduler suite 96/96 with no expectation changes in existing
tests beyond wiring; scheduler invariant audit of the allocation path
clean (driver state-equivalence, core I/O isolation, picker/allocator
rendezvous, snapshot parity). issues/new_requirements.md item 4 marked
resolved.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comments in scheduler sources and tests that presented the deleted
my_alloc()/parjob claim helpers as current code now name
sched_allocator_run() and its runner helpers; test helpers and tests
named after my_alloc are renamed. No logic or expectation changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seqjob registers sched_allocator_run with seqjob_alloc_iface (no accepts
guard, plus its one-bucket plan iterator); its private my_alloc() is
gone. New fixtures pin the intended changes: a reason for every short
host, the bug-1893 shared reason, the host-stage OOM debit release, and
-EINVAL for missing inputs; seqjob's plan is asserted to be one bucket of
one slot. No other test expectation changed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
parjob registers sched_allocator_run with parjob_alloc_iface (accepts
maxTasks > 1, bug-1767, plus its plan iterator); its private my_alloc()
and claim helpers are gone. Characterization now pins the intended
changes: a reason for every short host (2/6/5) and the bug-1893 shared
reason. No other test expectation changed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_finalize_job_alloc_() turns a satisfied plan's claims into the job
allocation exactly as parjob's commit tail does (host records from the
claim-time snapshots in claim order, shared records, jobid 0, no
duration) and publishes ctx->alloc only on success. Every out-of-memory
step frees what it built and releases the debits of claims not yet
turned into host allocations; each claim snapshot is freed exactly once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The runner tries the plugin's plans in order and claims one distinct host
per bucket, with parjob's rollback and claim snapshots, nslots/nhosts per
candidate, a growable claims buffer and malformed-plan skipping. It
clears alloc_req.host before every candidate so each short host gets its
pend reasons (Req 3.6/3.7). Result assembly follows in task 3.3; until
then a satisfied plan's claims are released and the attempt pends.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New alloc_runner.c: sched_allocator_run() validates its inputs (-EINVAL
without touching the pend reason), applies the plugin's optional accepts
guard, creates the pend-reason record, then checks shared resources: one
host-less NO_RESOURCE reason per unmet entry (skip only on vget < 0,
fixing bug-1893) and the met amounts counted once as uncommitted. Plan
search follows in task 3.2. Runner tests use a fake allocator contract;
the pend-reason factory audit now scans the runner.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
15 tests drive the real parjob plugin through job_alloc_fn: success,
short shared, pending on hosts, mid-plan rollback, task counts 0/1,
missing inputs, six OOM steps with their debits, COMPOUND. Values the
runner changes on purpose sit in *_today tests.
Spec: the task showed that parjob records host reasons only for a bucket
search's first rejected candidate (the rejected host's request stays in
alloc_req.host and the host iterator pre-filters later hosts against
it). Decision: the runner clears alloc_req.host before every candidate,
for both allocators, so every short host gets its reasons (Req 3.6/3.7);
publishing reasons only when they change is a follow-up.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
12 tests drive the real seqjob plugin through job_alloc_fn and record
today's results, reasons and debits, with switch-stable OOM injection
(sched_make_allocs stub, --wrap of make_host_alloc/make_job_alloc).
Assertions the runner changes on purpose sit in *_today tests.
The characterization showed bug-1893's real effect: vget() returns 0 on
success, so the shared-reason loop (rc <= 0 skip) never records a shared
NO_RESOURCE reason. Bug record and spec wording corrected; the runner
skips only on rc < 0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_sched_allocator_iface_t (optional accepts guard + plan_init/next/fini)
and the sched_allocator_run() entry point in sched/alloc_iter.h; the
plugin's const allocator_iface reference, forward-declared in plugin.h
with the picker_iface ownership rule. No plugin uses them yet.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_alloc_plan_iter_t.init takes xce_sched_core_t* instead of
xce_scheduler_t*; both built-in generators, both allocators' call sites
and the contract-test stub follow. The parjob purity test now shows an
identical plan sequence for a NULL and a poisoned core.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Requirements reviewed and revised (seqjob per-bucket reset, nslots/nhosts,
bug-1893 fix in the runner), design and 11-task plan generated and reviewed.
bug-1892: unsatisfy_() reads index-keyed uncommitted amounts by position.
bug-1893: allocators' shared pend-reason loop reads past the unmet list.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The apiserver and all components ran the frontend Go tests as one list,
'cd frontend/apiserver && go test ... && cd ../xcecli && go test ...'.
set -e ignores a failure of any command in an && list except the last,
so a failing apiserver go test only short-circuited the list: the build
reported success and the xcecli tests never ran. Run each suite as its
own command in a subshell (which also keeps the cd from leaking).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The apiserver's job store learned of a new job only from the
controller's asynchronous OnJobCreated / array push, so a 'jobs kill'
(or query) straight after 'jobs submit' could answer 'job not found'
for the job just accepted (2 of 4 immediate kills).
services.SubmitJob now records the accepted job right after the
controller's reply (which comes only after the job is committed and
applied): one record per element, base+i for arrays, stored with the new
DataStore.PutIfAbsent and marked JobInfo.Provisional (json:"-", not a
column). An OnJobCreated that already arrived is never overwritten; the
push that arrives later replaces the provisional record -- including in
dispatchJobArray, whose duplicate check no longer treats a provisional
element as already known.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xcecli prints a submitted job's ID in the dashed
{cluster}-{apiserver}-{timestamp}-{sequence} form (format.FormatJobID,
since d6008ead), but run.sh still matched 'Job <[0-9]+>' -- so 'jobs
submit' always failed and 'jobs kill' was skipped (28/1/1). It also
extracted the ID with [0-9]{10,}, which would have picked out only the
timestamp. Match the dashed form and take the whole ID from between the
angle brackets; 'jobs kill' accepts it back.
'jobs submit' reads a job script from stdin whenever stdin is a pipe,
so the suite hung forever when run from a script or CI with an open
pipe; the submit now reads /dev/null.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The script required a pre-generated <repo>/ha/data (cluster.bin + TLS
certs) that ee8473bd removed, and called deploy-ha.sh with that dir as an
extra argument; deploy-ha.sh now generates both into the deploy dir and
takes <binary> <top-dir> <peers>. It failed at its first precondition in
every checkout.
Behind that, both committed-LSN boundaries still assumed bug-845's doubled
submits (1 + 2*N_JOBS): the commit wait never reached it and walcompare
compared an LSN past every member's WAL. Since that fix each job is one
WAL entry, so the boundary is 1 + N_JOBS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
