18 sub-tasks in 6 groups (libxce primitives + regression script
skeleton; controller; scheduler/worker/harness; xceadmin; integration;
docs). Each C task adds its failing regression-script case before the
change. Requirements, design and tasks approved.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Iterated design review until clean (GO):
- Controller main() calls mylog_fini() before every early return, so
log lines written by tlog's background thread survive the exit.
- cluster_init_phase1() keeps its candidates_count > 0 guard around the
new xce_peers_find_identity() call (standalone stays my_priority 0).
- xceadmin adds the "(from $VAR)" source note only to network errors,
not to local cluster-bin file errors (OSError with filename set).
- Regression script: scratch --workdir/--certdir/--logfile per daemon,
refuses to run if a local cluster is up (port 9990, worker socket),
--sim case asserts the sim branch ran, 2.3 patches follow_redirects,
2.8 malformed-entry cases use a readable cluster-bin file, wrapper
test expects PEERS_STR form.
- Two existing scripts that start xce_scheduler without --peers will
unset XCE_CONTROLLER_PEERS; --interactive wording corrected.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Design review (NO-GO -> GO) and iterated requirements review until clean:
- Controller refuses to start (non-zero exit, stderr, before any network
I/O) when an env-supplied XCE_CONTROLLER_PEERS lacks its own name (1.6);
new libxce xce_peers_find_identity() replaces both identity loops in
controller/cluster.c so the check and my_priority/listen port agree.
- deploy-ha.sh's generated scripts/xceadmin wrapper exports
XCE_CONTROLLER_PEERS so an ambient value cannot redirect it (Req 4).
- Opt-out is unsetting/emptying the variable (1.7); "No --peers" log
lines say it was unset or empty (1.8); docs cover sudo/systemd not
inheriting it (3.1).
- Requirement 1 defines explicit candidate list / resolution point /
own name so harness (set masters) is covered precisely; error messages
name the value's source incl. validation failures (1.4).
- xceadmin: name= stripping and a clean malformed-entry error apply to
every --controller source; errors name the env var source (2.2, 2.8).
- Test plan fixed against real code (stderr vs stdout, log texts,
xcecd_virtual, --norc, free ports, failover order test).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
src/docs/xce.md gains a 'Job arrays' section: name[1-N] and name[1],
numbering from 1, the API-server limit (default 1000, at most 32767),
<array-id>[k] and <array-id>[a-b] selection (with the peek/restart
caveat), XCE_JOBINDEX, the harness jobsub -a N / $jobid[k], and every
LSF array feature this release does not support. The superseded
statements in job-array-single-wal-record and xce-test-e2e-primitives
are annotated (history kept), the controller-job-cleanup and
job-dependency briefs describe 1-based numbering, and the roadmap gains
an entry.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The HA array soak checks small-array elements 1..N, large-array first,
middle and last as 1, N/2 and N, status/cancel/completion on 2, N/2+1
and N-1, and the chaos-retry round over seq 1 N; the multihost
verification drives element [1]. Comments now say that after jobsub -a N
the success line and $jobid name the array identity. Historical
live-run quotes that show [0] are kept and annotated as predating the
change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
idgen() now accepts only a size of 1 (anything else: -EINVAL, context
unchanged) and refuses an exhausted 20-bit sequence with -EOVERFLOW
instead of spilling into the timestamp (also after idgen_array has
taken the last block). Arrays come only from idgen_array; every caller
of idgen/GenerateJobID passes 1. The old multi-element test loop moves
to idgen_array with 1-based assertions.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Submitting an array prints the server's message with the array ID in
written form (no [k]) and the element count, e.g.
'Job <0-0-...> is submitted to queue <normal> (3 elements).'; ordinary
submits print exactly as before. The client model mirrors array_size.
Submit help documents name[1-N] and name[1], numbering from 1 and the
API-server limit (default 1000, at most 32767).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ID lists split only at commas outside brackets and every selection is
validated before any request (control allows identity ranges, listing
refuses them). A raw -J value with brackets is checked before the
wildcard mapping, and the search rule and * stripping apply to the
name part only (-J 'foo*[1]' lists element 1 of arrays whose name
starts with foo); values without brackets map exactly as before.
Control results print each item's own job ID when the server sends it,
so a pre-upgrade element 0 shows as <array-id>[0]. Control and list
help show <array-id>[k] and <array-id>[a-b]; list help now says -J.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xcecli's format package gains ParseSelection (written or decimal key
with optional [k]/[a-b], the server's grammar and check order; 5.8 and
5.7 also on decimal keys with [...], so an upgraded client protects
itself against an older server; query ranges refused after the other
checks), the bracket-aware SplitSelections, the raw -J
ParseNameSelection and Selection.APIKey. ParseJobID is bounded;
ToAPIJobID(s, allowRange) no longer passes unrecognised text through;
peek keeps today's behaviour via ToAPIJobIDLegacy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Filter-based ControlJob validates a bracketed -J value before fetching
(400 with the parser's message; unbracketed names, including commands
with spaces, unchanged). FetchJobList evaluates bracketed job_name
entries through the selection predicates instead of unevaluated
compound conditions, so -J 'foo[1]' acts on element 1 of every array
named foo and never on ordinary jobs of that name. The most-recent-job
rule applies only without an element number and without all-jobs, and
treats the element of a one-element array like an ordinary job
(isSingleElementArray, a documented approximation). The old
index-section parser is deleted. Completes bug-1898 on the owned paths.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ControlJobByID parses every selection with ParseIdentitySelection before
any store access (first rejection: 400, nothing dispatched), collects
the jobs to act on de-duplicated by unique_job_id plus one not-found
result per missing number per selection (carrying the would-be element
ID), validates each job once after the loop and dispatches each
accepted job once. foundJob gains an explicit-index flag so
<array-id>[0] matches only element 0 and never falls back to the whole
array or an ordinary job. Fixes bug-1899 (earlier selections were
re-dispatched for every later one).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GetJobsInfo validates every filter entry in request order before any
store access (including the exec_hosts lookup) and GetJobByIDs every key
before querying; the first rejection returns 400 with the parser's
message. Bracketed job_id/job_name entries are evaluated as row
predicates before sorting and pagination, so job_id:eq:X[2] lists only
element 2, GET /jobs/X[1-3] lists elements 1-3 and <ordinary-id>[0]
lists nothing. The unevaluated compound conditions and the old index
parsing are gone from both handlers. Fixes bug-1898 on the query paths.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
BuildSelectionPredicates converts each job_id/job_name filter entry
containing a bracketed value into one row predicate (values OR'ed,
entries AND'ed with each other and the plain conditions), removing only
that entry. Operators are honoured instead of dropped (bug-1898):
eq/in/between select, notin excludes, name pattern operators reuse
today's condition semantics on the name part, and ne / comparison /
null operators keep today's rejection. Element selections match only
array elements in range; bare identities match the whole array and any
ordinary job; an empty name under exact operators matches nothing.
Bare written keys are normalised to decimal on every path. Not yet
wired into the handlers.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ParseIdentitySelection (strict written form or base-10 decimal key,
optional [k] or [a-b]) and ParseNameSelection (name part may be empty,
no whitespace or brackets) apply the shared selection grammar and check
order (5.16 syntax, 5.14 width, 5.17 marked decimal without [k], 5.8
alignment only with [...], 5.7 element bound, 5.5 reversed), reporting
only the first failure with a rule tag and a message quoting the
selection. Runs the shared selection-cases table plus REST-only rows.
Not yet wired into the handlers.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
preCheckJobRequest returns (n, isArray, err): a bracketed job name is
parsed by ParseSubmitIndexExpr first, so name[1-N] and name[1] create
arrays (one-element arrays included) allocated with
generateArrayJobID, stored under the bare name; expression errors are
answered 400 with their specific message and create nothing; names
without brackets take the unchanged createArrayName path. The response
names the array (jobid = array ID, new array_size, "(n elements)"),
submit-time %I/%J use the first element, and the submit and dispatch
logs identify arrays by array ID. Fixes bug-1896 for REST and xcecli.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ParseSubmitIndexExpr accepts only name[1-N] and name[1] (leading zeros
allowed) and otherwise reports the lowest-numbered broken rule with its
specific message: interactive arrays (2.1), invalid name or malformed
expression (2.2), lists/steps (2.3), %N (2.4), start 0 (2.5), reversed
(2.6), start above 1 (2.7), above the limit (2.8). Oversized numbers
saturate for the bound checks while reversal compares the digit
strings, so the rule order holds for any input. Not yet wired into
CreateJob (task 6.2).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CreateJob's ID generation and submission and both control dispatch
sites now go through package variables (generateJobID,
generateArrayJobID, submitJob, controlJob) that default to the real
backend/services functions, so production behaviour is unchanged. A new
non-cgo handler test file provides a reusable fixture: it saves and
restores every global it touches, sets up auth/RBAC, seeds jobs and
their queues, and records each dispatched job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
QueryOptions gains Predicates; store.ExecuteQuery keeps a row only if
every predicate returns true, after the plain conditions and before
sort, pagination and field selection, so page boundaries and tokens
count only matching rows. Nil/empty predicates leave every existing
query unchanged. Foundation for evaluating element selections
(bug-1898).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
job.max_array_size is resolved by a pure resolveMaxArraySize: unset or 0
gives 1000, 1..32767 is used as is, values above 32767 are clamped to
32767 with a startup warning naming the configured and effective value,
and a negative value refuses startup. The old >2147483646 rejection is
gone; the shipped config documents the bounds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GenerateArrayJobID wraps libxce idgen_array (first element, index 1;
one-element arrays included) with a non-cgo stub; pure helpers
ParseHumanJobIDStrict (written identity with exact field-width checks,
uint64 overflow reported as a width error), IsArrayCapableKey and
MakeElementID (bit-identical to libxce jobsel_element_id).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every harness command that acts on one job (jobinfo per -j, jobrun,
jobkill, jobckpt, taskrun, setpend, jstart, jfinish, jfail) and
expect-job parse their job argument with cmdutil_job_arg on top of the
libxce selection parser: element ranges (reversed too) are refused with
the single-element message, invalid identities are rejected with the
parser's reason, and no command sends job 0 for a rejected argument.
Not-found results (controller -ENOENT, or jobinfo's empty reply) name
the job and, for bare identities, hint at <array-id>[k]. expect-job
matches job numbers instead of formatted text.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Auto-variable substitution moves out of test.c into its own unit-tested
module (autovars.c) with an explicit context for $jobid, $? and
$jobcount, keeping today's token rule, messages and buffer handling.
New: a token $jobid[...] expands $jobid and keeps the rest of the token,
so $jobid[3] names element 3 of the array just submitted; it fails like
an unset $jobid when none is set.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
jobsub -a parses its value strictly (digits only, leading zeros ok):
1 submits an ordinary job; 2..32767 allocate with idgen_array, whose
result is checked; missing, non-digit and 0 values are refused with the
1-32767 range and values above 32767 with the limit, submitting nothing.
The success line and $jobid name the array identity; the random
sequence seed is bounded so the first array cannot overflow; help text
updated. Fixes bug-1896 for the harness.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The worker sets XCE_JOBINDEX to the element number for jobs carrying the
array marking (a one-element array's element gets 1) and to 0 otherwise.
A submission-supplied XCE_JOBINDEX is not applied and a WARN names the
job and the ignored variable; XCE_JOBID handling is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scheduler expands an array push only when the first ID carries the
array marking and the size fits in its block (size <= 32768 - idx, no
addition); otherwise it logs an ERROR naming the array identity and
expands nothing. Pre-upgrade arrays starting at index 0 still expand.
The INFO log names the array identity (array_jobid=...). This also
closes a crash: a malformed push with a huge size overflowed calloc.
Expansion tests move to tagged block|1 first IDs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
handle_jobsubmit now registers the submission first and checks the job
table second; a hit is answered -EEXIST and released through the atomic
reply-and-release step. For array submissions, array_identity_exists()
does one read-locked scan of the 32768 tagged element keys of the
array's block (element 0 included for pre-upgrade arrays), covering
both the element range and the array identity; unmarked submissions
keep jobid_range_exists. An existing X|3 now blocks a new X|5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
In-flight registry entries carry is_array and array_id; two array
entries with the same array identity overlap. inflight_reply_and_remove()
sends the reply and unlinks the entry under the registry lock (xce_reply
is non-blocking) and is used on every post-registration path, inside the
async one-shot reply guard; jobsubmit_async_ctx_release no longer touches
the registry, so a retry registered after the reply cannot be deleted by
the original's release. Test-only seams under XCE_TESTING make the
atomicity observable deterministically.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
handle_jobsubmit classifies (first_jobid, array_size) before the field
checks, the before-event hook and any registry access: a tagged array
starting at index 0 is answered with the new E_XCE_ARRAY_START0 and an
ERROR log asking for the client to be upgraded (7.1, wins per 7.9);
size 0, tags 2/3, untagged multi-element submissions and arrays whose
last element exceeds 32767 (checked without addition) get -EINVAL.
Nothing is stored for a rejected submission. This also closes a crash:
a size-0 submission used to reach payload building and overflow calloc.
Existing array tests move to tagged block|1 first IDs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
idgen_array() allocates the next whole 32768 block and returns its
first element (array tag, index 1), refusing n outside 1..32767 and a
block past the 20-bit sequence (-EOVERFLOW, context unchanged). The
reset-on-new-second step is shared with idgen() via idgen_tick_locked();
idgen() itself is unchanged until task 9.1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
parse_jobid() keeps its signature but is now bounded and NULL-safe: a
bare identity returns the exact ID, [k] the element ID, and a range, any
rejected selection (5.7, 5.8, 5.14, 5.15) or a NULL argument returns
-EINVAL with *jobid = 0. Fixes the strlen-before-NULL-check crash and
the atoi overflow; regex parsing is gone.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
parse_jobsel() parses c-a-ts-seq with an optional [k] or [a-b] under a
fixed grammar and check order (syntax, field widths, array-identity
alignment, element bound, reversed range), saturating digits so an
oversized number cannot wrap; plus is_array_tag(), jobsel_element_id(),
jobsel_strerror() and JOBID_FIELD_MAX_* constants. The selection-cases
conformance table runs in the libxce suite.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Spec for numbering job-array elements from 1 (bug-1896): approved
requirements, a design reviewed over 20 rounds until clean (log in
research.md), and a 30-task plan. The design review found two live
apiserver defects, logged here: bracketed job_id/job_name filters are
never evaluated (bug-1898) and ControlJobByID re-dispatches earlier
selections (bug-1899).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Feature validation GO (ASAN scheduler suite 96/96, smoke replay clean on
the allocation path). Roadmap entry marked implemented with follow-ups.
bug-1897 (found by the validation smoke run, not caused by this feature):
xce_scheduler --sim segfaults in limit_job_election on a job whose
snapshot resreq failed to parse (pre-bug-1888-fix replays).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Clean ASAN scheduler suite 96/96 with no expectation changes in existing
tests beyond wiring; scheduler invariant audit of the allocation path
clean (driver state-equivalence, core I/O isolation, picker/allocator
rendezvous, snapshot parity). issues/new_requirements.md item 4 marked
resolved.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comments in scheduler sources and tests that presented the deleted
my_alloc()/parjob claim helpers as current code now name
sched_allocator_run() and its runner helpers; test helpers and tests
named after my_alloc are renamed. No logic or expectation changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seqjob registers sched_allocator_run with seqjob_alloc_iface (no accepts
guard, plus its one-bucket plan iterator); its private my_alloc() is
gone. New fixtures pin the intended changes: a reason for every short
host, the bug-1893 shared reason, the host-stage OOM debit release, and
-EINVAL for missing inputs; seqjob's plan is asserted to be one bucket of
one slot. No other test expectation changed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
parjob registers sched_allocator_run with parjob_alloc_iface (accepts
maxTasks > 1, bug-1767, plus its plan iterator); its private my_alloc()
and claim helpers are gone. Characterization now pins the intended
changes: a reason for every short host (2/6/5) and the bug-1893 shared
reason. No other test expectation changed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_finalize_job_alloc_() turns a satisfied plan's claims into the job
allocation exactly as parjob's commit tail does (host records from the
claim-time snapshots in claim order, shared records, jobid 0, no
duration) and publishes ctx->alloc only on success. Every out-of-memory
step frees what it built and releases the debits of claims not yet
turned into host allocations; each claim snapshot is freed exactly once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The runner tries the plugin's plans in order and claims one distinct host
per bucket, with parjob's rollback and claim snapshots, nslots/nhosts per
candidate, a growable claims buffer and malformed-plan skipping. It
clears alloc_req.host before every candidate so each short host gets its
pend reasons (Req 3.6/3.7). Result assembly follows in task 3.3; until
then a satisfied plan's claims are released and the attempt pends.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New alloc_runner.c: sched_allocator_run() validates its inputs (-EINVAL
without touching the pend reason), applies the plugin's optional accepts
guard, creates the pend-reason record, then checks shared resources: one
host-less NO_RESOURCE reason per unmet entry (skip only on vget < 0,
fixing bug-1893) and the met amounts counted once as uncommitted. Plan
search follows in task 3.2. Runner tests use a fake allocator contract;
the pend-reason factory audit now scans the runner.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
