myxce/.kiro/specs/job-array-one-based-index/requirements.md

Requirements Document

Project Description (Input)

Who has the problem. Users who submit job arrays through xcecli/REST (-J 'name[...]') or the test harness (xce_test jobsub -a N), and anyone porting LSF job-array scripts: those scripts read the element index (LSB_JOBINDEX, %I) and expect indices that start at 1.

Current situation. XCE silently ignores the indices the user writes. The apiserver parses arr[1-3] or arr[1,5,9] correctly, but sends only the element count to the controller, and the controller numbers elements 0..N-1 -- so both become [0], [1], [2] (bug-1896). The harness's jobsub -a N also produces 0..N-1. The worker sets XCE_JOBINDEX to a hardcoded 0 for every job, so a running element cannot learn its index. LSF defines array indices as positive integers (index 0 is invalid) and sets LSB_JOBINDEX to 0 only for non-array jobs (LSF 10.1 Admin Guide, "Chunk Jobs and Job Arrays", p.519).

What should change. Array elements are numbered 1..N, as in LSF, everywhere they are created, shown, parsed and exposed to the running job; anything this spec does not support is rejected with a clear message instead of being silently renumbered.

Decisions already made (with the user, 2026-09-30)

1. Option B -- the index lives in the job ID. A job ID stays one uint64_t; an element's index is the existing 15-bit array-index field. The first element's ID is submitted as base | 1, and the existing contiguous expansion (first_jobid + i) in the controller and scheduler yields elements 1..N. No new start field and no wire, WAL or snapshot format change. fmt_jobid, parse_jobid, array_jobid and array_idx keep their signatures.
2. Start is fixed at 1 in this spec. Only name[1-N] is accepted. Any other start ([1001-2000], a single [n] with n > 1), index 0, comma lists and steps are rejected with a clear message. Start > 1 is deferred to a future spec; this spec is a strict subset of LSF behavior so that extension will not change anything accepted here.
3. The submit reply identifies the whole array (array_jobid(), as LSF's Job <123> is submitted), not the first element; the harness's $jobid after jobsub -a N likewise names the array. (Round 3: the controller keeps replying with the first element's ID -- the apiserver's clib.go:653-657 and RecordAcceptedJob rely on it -- and the clients derive the array identity with array_jobid() for what they show.)
4. A one-element array name[1-1] is accepted (as in LSF) and behaves like any other array.
5. Index 0 is rejected only at submission. Arrays submitted before this change keep their element [0], which stays selectable and controllable.
6. XCE_JOBINDEX is set by the system and cannot be overridden by a user-supplied environment variable of the same name.

Resolved during requirements review (round 1, 2026-09-30), following the decisions above: 7. The cluster rejects a new array whose first element number is 0, and accepts any first element number from 1 to 32767, so a future start > 1 needs no cluster-side change; "start = 1" is enforced where users submit (command-line client, REST, harness). Consequence for a rolling upgrade: an un-upgraded client's array submissions are rejected with an error until it is upgraded; ordinary jobs are unaffected. An un-upgraded command-line client is not protected by 5.7, 5.8 and 5.14: selections it resolves before sending them to the REST API reach the API server as ordinary job numbers, until the client is upgraded. 8. Whole-array control by the array ID stays a command-line-client/REST feature (the apiserver fans it out; the controller and harness act on exact IDs only). The harness's job commands refuse a bare array ID with a message pointing to <array-id>[k]. 9. Pre-upgrade element [0] is covered for single-element references (<array-id>[0]), including listing and control results; ranges or lists that include 0 stay with the separately tracked duplicate-signal bug. 10. Syntax details: bounds are plain decimal digits (no sign); whitespace inside the brackets is rejected; leading zeros are accepted as the same number (LSF reads indices as numbers); interactive jobs cannot be arrays (as in LSF). Harness jobsub -a 1 stays an ordinary job; -a 0, negative or non-numeric values are refused.

Boundary (from a four-area investigation, 2026-09-30)

• libxce: lib/idgen.c (a size-1 array currently gets a plain untagged ID; bound size so that indices 1..N fit in 15 bits, i.e. N <= 32767); lib/utils.c parse_jobid and include/config.h REGEX_JOBARRAY_ID (bound the index to 32767, reject a misaligned array block, replace the unchecked atoi -- and must reject any ID field wider than its bit width (cluster <= 63, API server <= 15, timestamp < 2^32, sequence < 2^20), since MAKE_JOBID ORs fields unmasked (include/utils.h:20-24) and e.g. 0-0-T-1048576[1] would otherwise resolve to 0-0-(T+1)-0[1]).
• Controller: controller/job.c handle_jobsubmit (:2461) needs pre-commit validation -- array tag set, size >= 1, array_idx(jobid) + size - 1 <= 32767 -- because the controller trusts the client today; it also needs a check that no existing array already has the submission's array identity (7.7, 7.10), since today's duplicate check compares element identities only (jobid_range_exists job.c:214-229); the in-flight submission registry (inflight_range_overlaps_locked/inflight_range_try_add, job.c:117-158, bug-1419) compares element ranges only and must also treat a pending array submission with the same array identity as an overlap, or two concurrent submissions such as X|1 (2 elements) and X|5 (1 element) are both accepted; today the job-table check (job.c:2463) runs before registration (job.c:2482), which leaves a window in which a submission applied and answered between the two steps is seen by neither, so the job-table checks (element and array identity) must run after the submission is registered (and release it on a hit) — a submission that commits in time is in the job table before it leaves the registry (persist.c:~5930 apply, then on_applied → jobsubmit_apply_complete → release at job.c:2321); one answered with a commit failure or timeout leaves the registry at job.c:2356, and any later apply of it is the out-of-scope bug-1419 class; the array-identity check is bounded work (up to 32768 element keys of the block, element 0 included for pre-upgrade arrays); the controller's submit reply keeps carrying the first element's ID (:2284), and the clients derive the array identity from it.
• Scheduler: scheduler/src/driver.c :477-478 overflow/tag guard on expansion; :509-512 the whole-array INFO log must print array_jobid() instead of the first element's ID.
• Worker: worker/src/xcecd_exdata.c :419-420 XCE_JOBINDEX becomes array_idx(jobid) (0 for non-array jobs); user environment variables must not override it (:431-457) -- only XCE_JOBINDEX becomes non-overridable; a user-supplied XCE_JOBID keeps winning as today (:408, :431-436; out of scope); XCE_JOBID stays the raw element ID (xrun.c:70 parses it).
• Apiserver: controllers/jobs.go createArrayName (:1824-1979: accept only [1-N], explicit rejection messages, cap at 32767); CreateJob sizing and ID (:1419-1434, :1454-1457: always generate an array ID, submit base + 1); the submit response (:1517-1520: report the array); utils/config.go:214-225 and config/app.yaml:71 (max_array_size capped at 32767); parseArrayIndexSection (:3290-3291: allow [k-k], fix the reversed error text).
• xcecli: help text (cmd/jobs/submit.go:626-684, controller.go:133-240, including checkpoint's help at :236-240, list.go:329 vs :360); common/format/jobid.go ParseJobID/ToAPIJobID (index unbounded: X[40000] resolves into the next array block; no alignment check: ParseJobID adds idJob+idIdx (:102) and ToAPIJobID clears the low 15 bits (:141, via ExtractBaseJobID :116-118), so a misaligned c-a-ts-5[3] becomes element 8 of another array before REST ever sees it -- both must reject an index above 32767 and an <array-id> whose sequence has any low 15 bit set -- and must reject any ID field wider than its bit width (cluster <= 63, API server <= 15, timestamp < 2^32, sequence < 2^20), since MAKE_JOBID ORs fields unmasked (include/utils.h:20-24) and e.g. 0-0-T-1048576[1] would otherwise resolve to 0-0-(T+1)-0[1]) and FormatArrayChildID (:124-127, prints the bare array ID for index 0); cmd/jobs/list.go:257 (array_idx:ne:0) is dead code -- its -W flag registration is commented out (list.go:360, used at :90) -- so it hides nothing today and does not affect 8.3; interactive submit waits on jobResp.Data.JobID (submit.go:505-507). The ID-list splits at cmd/jobs/controller.go:114 and filters/job_filter.go:43 cut at every comma, even inside [...]; they must split only at commas outside brackets, as the API server's extractJobNames (jobs.go:1043-1068) does, so that a rejected selection such as X[1,3] is reported as the user wrote it (Terms, 5.12, 5.16).
• Name-based element selection: xcecli jobs list -J (list.go:355) and kill/stop/resume -J (controller.go:154,190,225) reach the REST job_name filter, which maps name[a-b] to array_idx between (a,b) (jobs.go:265-300, :3362-3390) -- the same selection rules apply.
• "Most recent job" control: the filter-based ControlJob (jobs.go:2730-2830) acts on only the last matching job when array_idx == 0 and on all matching jobs otherwise (:2753-2757); "last" is by unique_job_id ascending (FetchJobList sort, :3458), not submit time. foo[1] used to create an ordinary job and now creates a one-element array element, so this rule must keep acting on that element alone (5.10).
• Explicit element 0 in ControlJobByID: the query already narrows a single <array-id>[0] to rows with index 0 (jobs.go:2380-2409), but foundJob's index-0 branch (jobs.go:3111-3127) matches every row with that job_id, so when the same request also brings in other rows of that array (e.g. bare X plus X[0], or X[0] plus X[2]) it acts on the whole array; that branch must fire only for a selection with no [...] (5.4, 8.2).
• Reversed-range message: every caller of parseArrayIndexSection replaces its error with "bad job array index: ..." (jobs.go:300, :407, :844, :2392, :3389), so the fix must pass the specific message through (5.5).
• Element-only matching: an ordinary job is stored with job_id = its full ID and array_idx 0 (backend/convert_pure.go:50-52, jobs.go:1455-1456), so <ordinary-id>[0] matches it on REST (jobs.go:2380-2409); whenever a selection has [...], GetJobByIDs, GetJobsInfo's job_id branch (jobs.go:366-420, the path xcecli jobs list <id>[...] uses via xcecli/filters/job_filter.go:55), ControlJobByID and foundJob's explicit-index path must match only array elements (e.g. IsArrayJob(unique_job_id)) (5.2, 5.4, 5.11).
• REST identity alignment: ControlJobByID accepts a numeric or ParseHumanJobID key (jobs.go:2369-2384; backend/convert_pure.go:120-133); GetJobByIDs (jobs.go:799) and GetJobsInfo's job_id branch (jobs.go:377) accept numeric keys only; a job_id filter value without both [ and ] never reaches :377 — it continues at jobs.go:354-356 into the generic job_id in (...) filter, which is the path xcecli jobs list <decimal> takes (filters/job_filter.go:49-58), so the checks below must also cover every key in that branch before the continue. All three pass the key to job_id eq without an array-alignment check, so the 5.8 check (and, for ControlJobByID's written form, the field-width check) is needed there, and all three must also reject a numeric key that carries the array marking and has no [...] (5.17).
• Job-name rule: the part before [ must satisfy the existing isValidArrayName rule (at most 4094 characters, ^[a-zA-Z0-9._-]+$, jobs.go:1981-1987); the apiserver already enforces it (jobs.go:1568-1572).
• Controller rejection text: error replies carry only a status (job.c:2451-2454) and older clients print only an rc or errno text (lib/client.c:352-355, clib.go:629-631, test.c:3825-3835), so the "client must be upgraded" explanation goes in a distinct error status plus a controller log entry.
• Apiserver parsers and logs: the element-number bound for REST selections belongs in controllers/jobs.go parseArrayName/parseArrayIndexSection (they strip and parse [...]; ParseHumanJobID never sees the index); those parsers are reached only when a value contains both [ and ] (job_name in GetJobsInfo jobs.go:250-252 and FetchJobList jobs.go:3356-3358, the latter reached by filter-based ControlJob; job_id at jobs.go:354), so a name or identity containing only one of [ or ] falls through to a plain match today and the 5.16 syntax check must run on any value containing [ or ] before that fallback; backend/handler.go:272-275 dispatchJobArray logs the raw first-element ID where it means the array.
• One-element arrays vs ordinary jobs: idgen(..., 1, ...) is how ordinary jobs get IDs (harness jobsub -a 1, RestartJobByID via GenerateJobID(...,1) at jobs.go:1185) and must keep returning an untagged ID; a one-element array needs a distinct request. C array_idx/array_jobid (any non-zero tag, lib/utils.c:694,705), C fmt_jobid and xcecli IsArrayJob/FormatJobID (tag & 1, lib/utils.c:591, xcecli/common/format/jobid.go:46,60-62) and apiserver IsArrayJob/FormatJobID (tag == 1, backend/convert_pure.go:78-80,88) disagree on what marks an array; the cluster must accept only tag value 1 as an array.
• Harness: substitute_autovars (test.c:1283-1360) expands only a whole token named exactly jobid (:1306-1308, :1315) and copies $jobid[3] verbatim (:1268-1272, :1344-1350), so it must also expand $jobid followed by a [...] suffix, keeping the suffix, and fail as for an unset $jobid when it is unset (4.10, 4.11); single-job commands look up exact IDs only and ignore parse_jobid's return code (jobkill, jobinfo, jobckpt :4276, jobrun :3517, taskrun :4921, setpend :5073, jobstart :5268, jobfinish :5377, jobfail :5481, and others at :3327, :4140), so a rejected ID silently becomes job 0; expect-job compares the text against the job list (cmd_expect_job :2787-2817 → joblist_query fmt_jobid + strcmp), so it must instead parse its argument with the bounded parse_jobid and match the job number exactly, which makes it a harness command that parses job identities (Terms); -a is read with atoi and silently turns 0/negative/non-numeric into a plain job (test.c:3670-3674). src/harness/test.c jobsub -a N yields elements 1..N (:3721 submit base | 1 and check the idgen return code; :3670-3675 cap -a at 32767; :3840 the ack comparison must match the controller's reply; :3866 $jobid = the array ID; help text :556-562, :603, :3680).
• Skill scripts and docs: .claude/skills/xce-test-ha/run-ha-soak.sh (:2601, :2731-2798, :2808-2815, :2975-2976 verify elements 0..N-1) and .claude/skills/xce-test-multihost/run-multihost-verify.sh:792 (verifies element 0), plus both SKILL.md files.
• Tests: controller/testsuite/test_commit_barrier_rollout.c:2323-2370 (32768 -> 32767 elements); controller/testsuite/test_commit_barrier_reply.c (cbrp_arrange_jobsubmit :1025-1040, tests at :1640ff and :1875ff) submits untagged index-0 arrays straight to handle_jobsubmit, which the new validation (7.1, 7.3) rejects -- its IDs must become array-tagged base|1; new tests for createArrayName/CreateJob sizing, harness -a bounds and $jobid, controller submit validation, and parse_jobid bounds. Existing base + i expansion tests stay valid under option B.
• Already index-agnostic (no change expected): WAL JOB_ARRAY_SUBMIT append/apply/replay, standby apply, HA/raft replication, snapshots, controller-to-scheduler/apiserver pushes, scheduler snapshot/replay/REPL formatter (expansion arithmetic apart from the guard in the Scheduler bullet), fmt_jobid and the Go FormatJobID, the apiserver's array_idx between filter construction itself, RecordAcceptedJob.
• Docs: the XCE user documentation is src/docs/xce.md (Requirement 9.4); this spec supersedes job-array-single-wal-record Requirement 5 and the $jobid/expect-job definitions in xce-test-e2e-primitives/requirements.md:43-44, :66-67, :83 (which make $jobid the acknowledged element ID) and design.md:77 (whole-token-only substitution, relaxed by 4.10) (no client-visible identity change; indices [0, size)); update .kiro/specs/controller-job-cleanup/brief.md:86-87 and .kiro/specs/job-dependency/brief.md:45-46; add a roadmap entry.

Out of scope (future spec or separate bugs)

• Start > 1, comma lists and steps (rejected with a clear message in this spec).
• Setting LSF's own LSB_JOBINDEX/LSB_JOBID variables (scripts read XCE_JOBINDEX).
• Whole-array control by the bare array ID through the harness or directly at the controller.
• A user-supplied XCE_JOBID overriding the system value (e.g. xcecli jobs submit --env from inside a running job copies the parent's XCE_JOBID, and xrun.c:70 then starts remote tasks for the parent job) -- a separate defect to log.
• Missing-number reporting for query (as opposed to control) commands (GetJobByIDs returns only matching rows).
• The array statistics view (xce jobs list -A), which is empty because store.ArrayJobs is never written.
• Job output / peek by identity (GET /jobs/:jobids/output, xcecli jobs peek; controllers/output.go:224-244 parseJobIDParam ignores [k] and looks up the raw key): element peek works today only with the element's decimal ID and is left as is; 5.2, 5.7, 5.8, 5.14, 5.16 and 5.17 do not apply to it.
• Job restart by identity (POST /jobs/:jobids/restart, xcecli jobs restart; jobs.go:1074-1175): left as is; 5.2, 5.6, 5.7, 5.8, 5.14, 5.16 and 5.17 do not apply to it.
• The scheduler REPL's info job identity argument (scheduler/src/repl_info.c:114-117, falls back to strtoull and reports job 0) -- a developer diagnostic, not a user surface.
• Selecting a range of elements by identity in queries (xcecli jobs list <array-id>[a-b]): rejected by the client today (xcecli/filters/job_filter.go:175-187 isValidJobId; ToAPIJobID passes ranges through and GetJobsInfo's job_id branch requires a numeric key, jobs.go:~372).
• Log entries that print a job ID as a raw number rather than in its written form (e.g. controller/resync.c:591, controller/job.c:3250, scheduler/src/core_events.c:314, snapshot_load.c:341,398, snapshot_wire.c:185, repl_info.c:139).
• Missing-number reporting for name-based (-J 'name[...]') control selections: the filter-based ControlJob returns only matching rows (jobs.go:2734-2748).
• Selecting element 0 by job name (-J 'name[0]', -J 'name[0-0]'), including its also matching ordinary jobs of that name (stored with array_idx 0, jobs.go:1456, :3380-3395).
• Selections that combine element 0 with other element numbers (e.g. <array-id>[0-3]), for new and pre-upgrade arrays alike.
• The "most recent job" rule widening to all matching jobs when the latest job is an element of a multi-element array (jobs.go:2753-2757); only the one-element case this feature creates is covered (5.10).
• The "most recent job" rule ordering by job identity rather than submit time, so any array element (tagged ID) outranks a newer ordinary job (jobs.go:3458).
• job.max_array_size values that are not whole numbers: viper.GetInt silently turns them into 0 (then 1000) or truncates them (utils/config.go:217) -- a separate config-parsing defect.
• The %N slot limit (parsed, then silently dropped today -- reject it for now).
• Per-element %I/%J expansion: blocked on the out_file/err_file decision (bug-1508) -- those fields never reach the controller or worker today. The apiserver's existing substitution is left as it is: at submit time processJobRequest (jobs.go:2065-2096, %J/%EJ from FormatJobID(primaryJobID), :1434-1435) and postJobRequest (jobs.go:2217-2234, %I from the first element's ArrayIdx) substitute once for the whole array, and resolveOutputPath (output.go:209-221) substitutes per element when serving output. For a new array the submit-time values therefore become the first element's (1 / <array-id>[1]) instead of 0 / <array-id>[0]. (LSF expands %I/%J at execution time; %J is the array ID.)
• Pre-existing problems found during the investigation, to be logged separately: preCheckJobRequest returning early and skipping validation for named jobs (jobs.go:1578), element restart and peek broken, store.ArrayJobs never written, duplicate signal on X[0],X[3], the transfer_files key mismatch, idgen sequence overflow; duplicate identities from a submission whose commit timed out or that an earlier primary admitted (the in-flight registry forgets it before it is applied, bug-1419 class), and a resent submission rejected as a duplicate being reported to the client as success (lib/client_failover.c:51-53); control commands silently ignoring extra operands after the first (xcecli/cmd/jobs/controller.go:108-116, harness cmd_jobkill getopt loop test.c:4121-4151 and the other single-job commands).

Touches the controller submit path and job-identity semantics, so it follows the full three-phase workflow with human review.

Introduction

XCE lets users submit a job array: one submission that creates many element jobs sharing a command, each distinguished by an element number. XCE follows LSF's -J 'name[...]' syntax, but today it discards the numbers the user writes and numbers every array's elements from 0, and a running element cannot learn its own number. This feature numbers array elements from 1, as in LSF, consistently for every way of submitting, referring to, displaying and running them. It supports exactly one form, name[1-N] (and its one-element spelling name[1]); every other index form is rejected with an explanation rather than silently renumbered, so that later support for other starting numbers, lists or steps can be added without changing anything this feature accepts.

Terms used below:

• An array identity names a whole array; an element identity is an array identity followed by an element number in brackets, <array-id>[k]. An array identity never equals any element's identity, including element 0 of an array submitted before this change: every element carries the array marking and the array identity does not, so an array identity is never the identity of any array element. (An ordinary job may happen to share that number; 5.2, 5.4, 5.6 and 5.11 cover that case.)
• A harness command that acts on one job is any test-harness command that takes one job identity, or one per repeated -j (jobinfo), including expect-job; every such command parses job identities, so 4.7-4.9, 5.7, 5.8 and 5.13-5.15 apply to it. For jobinfo, each -j is judged on its own, as today: a rejection or refusal under 4.7, 4.8, 5.7, 5.8, 5.14 or 5.15 applies to that -j only ("act on no job" means no job for that argument), and the other -j arguments are still looked up. For expect-job, "act on the job" means the assertion passes, and "report not found", "refuse", "reject" and "act on no job" mean the assertion fails and the harness records the run as failed.
• An array submission still being accepted, relative to a submission being judged, is one that declares itself an array, that the current primary controller found to break none of 7.1-7.5, whose collision check it began before it began the collision check of the submission being judged, and that it had not yet answered (success or failure) to its client at the moment the collision check of the submission being judged began; whether it is answered later does not change this. Of two such submissions, the one whose collision check began first therefore blocks the other, even if it later fails (for example a failed submission check or a commit failure); the blocked one is not retried. A submission already answered with a commit timeout, or admitted by an earlier primary before a failover, is not one; collisions with such submissions are the separately tracked defect listed under Out of scope.
• A submission declares itself an array when its identity carries the "array" marking (see 7.5); a submission with any other non-ordinary marking is neither an ordinary job nor an array.
• An index expression is the job-name form defined in 2.2.
• An array submitted before this change is any array accepted by a controller without this feature.
• When one command is given several identities or element selections, each is judged on its own under Requirements 4, 5 and 8, and the command acts on the union of the jobs they select; every "only", "exactly", "never", "no job", "list no job" and "not the whole array" clause in Requirements 4, 5 and 8 (e.g. 5.2, 5.3, 5.4, 5.11, 5.13, 8.2) limits only what that one selection contributes (no job for that selection), and the other selections in the same command are still acted on -- except that, in a query or control command, a selection rejected under 5.5, 5.7, 5.8, 5.14, 5.16 or 5.17 makes "act on no job" apply to the whole command (for jobinfo, the Terms bullet on harness commands that act on one job applies instead), as the Terms bullet beginning "If any identity or element selection in a query or control command is rejected" states. Whether an element selected more than once receives the action once is out of scope (duplicate-signal defect).
• A query command lists jobs or shows job details: xcecli jobs list and REST GET /jobs and GET /jobs/:jobids. Job output (xcecli jobs peek, GET /jobs/:jobids/output) and job restart are not query commands here.
• If any identity or element selection in a query or control command is rejected under 5.5, 5.7, 5.8, 5.14, 5.16 or 5.17, XCE rejects the whole command, reports the rejection of at least one rejected selection, naming that selection (with the rejection chosen for it by 5.12; reporting the others as well is allowed but not required, as today's command-line client and API server stop at the first error), and acts on no job; for such a command XCE reports no "not found" result (5.3 and 5.4 do not apply to its other selections); "not found" results (5.3, 5.4) are reported per selection, and a selection under 5.11 contributes no job without a report; in both cases the command's other selections are still acted on.
• A control command is kill, stop, resume or checkpoint (xcecli jobs kill|stop|resume|checkpoint, REST /jobs/control and /jobs/:jobids/control, and the harness commands that kill, stop, resume or checkpoint a job, such as jobkill and jobckpt); jobinfo, expect-job and job restart are not control commands here.
• An element selection refers to array elements either by identity (<array-id>[...], the elements of that one array) or by job name (name[...], e.g. -J 'name[2-5]' on query and control commands: the elements with those numbers in every array with that name).

Boundary Context

• In scope: submitting arrays as name[1-N] or name[1] through the command-line client and REST API; the test harness's jobsub -a N; rejection of every other index form, of malformed index expressions and of interactive array submissions, each with a specific message; the identity reported for a submitted array; how elements are shown, selected and controlled; the element number a running element sees in its environment, and that the submission's own environment cannot override it; the cluster refusing array submissions it cannot represent and accepting any start from 1 to 32767; continued handling of arrays submitted before this change; help text and user documentation of the supported syntax and its differences from LSF.
• Out of scope: support for starting numbers other than 1 at the command-line client, REST API and harness, comma-separated index lists, index steps and per-array running limits (%N) -- these are rejected (Requirement 2), not implemented; per-element %I/%J substitution in file names (depends on per-job input/output file support, which XCE does not provide today; the existing submit-time substitution is unchanged and, for a new array, uses the first element's values (%I = 1, %J = <array-id>[1]) for every element); setting LSF's own LSB_JOBINDEX/LSB_JOBID variables; whole-array control by the bare array identity through the test harness or directly at the cluster; maximum-array-size settings that are not whole numbers; selections that combine element 0 with other elements (except that they are still rejected under 5.5, 5.7, 5.8 and 5.14-5.16, and refused by the test harness under 4.8); reporting of missing element numbers by query commands and by name-based control selections; selecting a range of elements by identity in queries (except that such selections are still rejected under 5.7, 5.8, 5.14 and 5.16); the scheduler REPL's job-identity argument; job output (peek) and job restart by identity; selecting element 0 by job name; log entries that print raw job numbers; the "most recent job" control rule for multi-element arrays and its ordering by job identity rather than submit time; the array statistics view; a user-supplied XCE_JOBID overriding the system value; extra operands after the first to single-job control commands (e.g. unquoted kill X [1], which acts on X today because the extra [1] operand is ignored: xcecli/cmd/jobs/controller.go:108-116, harness getopt loops such as test.c:4121-4151); identity collisions with a submission that is no longer an array submission still being accepted (see Terms). Other defects found while scoping this feature are tracked on their own and are not changed here.
• Adjacent expectations: the stored format of jobs, their replication between controllers and their recovery after restart are unchanged and are relied on as they are. During a rolling upgrade, once the controller is upgraded, multi-element array submissions from an un-upgraded API server or test harness are rejected with an error status, and the controller log explains that the client must be upgraded (Requirement 7); one-element submissions written name[k] through an un-upgraded API server still create an ordinary job, and name[k-k] is still rejected, as before, until it is upgraded; ordinary jobs are unaffected. Arrays that a not-yet-upgraded controller accepts are treated as arrays submitted before this change (Requirement 8); they are numbered from 0 when an un-upgraded client submitted them and from 1 when an upgraded client did. An element that starts on a worker not yet upgraded sees XCE_JOBINDEX = 0 (or a user-supplied value) until that worker is upgraded; 6.1-6.4 apply to upgraded workers. Until the API server is upgraded, query and control commands sent through it by an un-upgraded command-line client or directly by a REST caller behave as before this feature; once the API server is upgraded, 5.2-5.8, 5.10-5.12, 5.14, 5.16, 5.17 and 8.2 apply to them, except that for an un-upgraded command-line client the checks it resolves itself (5.7, 5.8, 5.14) are not enforced, as stated below. An upgraded command-line client applies its own checks (5.7, 5.8, 5.14) and its element display whatever the API server's version; the other criteria for its commands apply once the API server is upgraded. An un-upgraded command-line client is not bound by 5.7, 5.8 and 5.14: an element selection it resolves itself (e.g. X[40000], a misaligned <array-id>[k]) can reach the API server as a valid selection of another array until the client is upgraded. This feature replaces the earlier commitment in job-array-single-wal-record (Requirement 5) that array element identities would not change for clients and would always be numbered from 0, and the xce-test-e2e-primitives definition of the harness's $jobid after an array submission.

Requirements

Requirement 1: Submitting a job array numbered from 1

Objective: As a user submitting a job array, I want name[1-N] to create elements numbered 1 through N, so that the element numbers match what I wrote and what LSF would create.

Acceptance Criteria

1. When a user submits through the command-line client or REST API a non-interactive job whose name is an index expression (2.2) with exactly one item, 1-N, and no % suffix, where 1 <= N <= the array size limit (both inclusive), XCE shall create one array with exactly N elements, numbered 1 through N.
2. When a user submits through the command-line client or REST API a non-interactive job whose name is an index expression with exactly one item, 1, and no % suffix, XCE shall create an array with one element numbered 1.
3. When XCE creates an array, XCE shall give each of its elements a distinct element identity.
4. When a user submits a job through the command-line client or REST API whose name contains no [ or ], XCE shall handle it exactly as before this feature, including the existing job-name validation, and shall never create an array for it.
5. When a number in an index expression or in an element selection (<array-id>[...] or name[...]) has leading zeros, XCE shall read it as the same number without the zeros.
6. While the API server's configured maximum array size is a whole number from 1 to 32767, XCE shall use it as the array size limit.
7. If the API server's configured maximum array size is greater than 32767, then XCE shall use 32767 as the array size limit and write a warning to the API server's startup log stating the configured and the effective value.
8. If the API server's configured maximum array size is not set or is 0, then XCE shall use 1000 as the array size limit, as before this feature.
9. If the API server's configured maximum array size is negative, then XCE shall refuse to start the API server and report the invalid setting, as before this feature.
10. When a controller restarts or fails over, XCE shall recover every array accepted under this feature with the same array identity, the same element numbers it was created with (1 through N for arrays submitted as name[1-N], name[1] or harness jobsub -a N), and the state each element had when it was last stored.

Requirement 2: Rejecting index forms this feature does not support

Objective: As a user, I want any array syntax XCE cannot honor to be rejected with a specific explanation, so that my jobs are never silently renumbered, resized or turned into ordinary jobs.

Acceptance Criteria

1. If a submission through the command-line client or REST API requests an interactive job whose name contains [ or ], then XCE shall reject the submission with a message stating that interactive jobs cannot be arrays.
2. If a job name submitted through the command-line client or REST API contains [ or ] and is not an index expression, then XCE shall reject the submission with a message that quotes the name and states that the array name is invalid when the part before the first [ (the whole name if it contains no [) breaks the array-name rule, or otherwise that the array index expression is malformed. An index expression is: a name of 1 to 4094 characters, each an ASCII letter (a-z, A-Z), digit (0-9), ., _ or - (the existing array-name rule, which may differ from the rule for ordinary job names); then [; then one or more items separated by ,; then ]; then optionally % followed by a number; and nothing after that. An item is a number, or number-number, or number-number:number. A number is one or more decimal digits with no sign. The index expression contains no whitespace anywhere (including before the name and after ] or the % number).
3. If a submitted index expression contains more than one item or an item with a step (:), then XCE shall reject the submission with a message stating that only a single range name[1-N] is supported.
4. If a submitted index expression includes a per-array running limit (%N), then XCE shall reject the submission with a message stating that running limits are not supported.
5. If a submitted index expression's first element number is 0, then XCE shall reject the submission with a message stating that element numbers start at 1.
6. If a submitted index expression's last element number is less than its first, then XCE shall reject the submission with a message stating that the range is reversed.
7. If a submitted index expression's first element number is greater than 1 (for example name[1001-2000] or name[5]), then XCE shall reject the submission with a message stating that only name[1-N] and name[1] are currently supported.
8. If a submitted index expression's last element number is greater than the array size limit (including numbers too large to represent), then XCE shall reject the submission with a message stating the effective array size limit.
9. If a submission breaks more than one of 2.1-2.8, then XCE shall report only the broken rule with the lowest number.
10. If XCE rejects a submission under any criterion of this requirement, then XCE shall create no job and no array element for it. (Raw cluster submissions, which carry no job-name index expression to check, are governed by Requirement 7 only.)

Requirement 3: Identity reported for a submitted array

Objective: As a user, I want the submission result to identify the whole array and say how many elements it created, so that I can refer to the array and its elements the way LSF users do.

Acceptance Criteria

1. When an array submission is accepted, XCE shall report the array identity, without an element number, as the identity of the submitted job, both in the command-line client's output and in the REST response.
2. When an array submission is accepted, XCE shall include the number of elements created in the command-line client's output and in the REST response.
3. When an ordinary (non-array) job submission is accepted, XCE shall report its identity exactly as before this feature.

Requirement 4: Submitting arrays from the test harness

Objective: As a developer or tester using the XCE test harness, I want jobsub -a N to create elements 1 through N, so that harness-based tests exercise the same numbering users get.

Acceptance Criteria

1. When a harness user runs jobsub -a N, with N written as one or more decimal digits (leading zeros allowed) and 2 <= N <= 32767, the XCE test harness shall submit one array whose elements are numbered 1 through N, regardless of any configured maximum array size.
2. When a harness user runs jobsub -a N where N is one or more decimal digits whose value is 1 (for example 1 or 01), the XCE test harness shall submit an ordinary non-array job, as it does today.
3. If a harness user runs jobsub -a N with N greater than 32767, then the XCE test harness shall refuse the command with a message stating the limit, and shall submit nothing.
4. If a harness user runs jobsub -a with a value that is missing, or that is not one or more decimal digits (for example -3, +5, 5, 5.0, 0x5, abc), or whose value is 0, then the XCE test harness shall refuse the command with a message stating the accepted range (1 to 32767), and shall submit nothing.
5. When an array submitted with jobsub -a N is accepted, the XCE test harness shall print a success line containing the array identity without an element number and shall set its $jobid variable to that identity (so expect-job $jobid after jobsub -a N is handled by 4.7, or by 4.9 if an ordinary job has that identity; elements are checked as $jobid[k]).
6. If the XCE test harness fails to obtain an identity for an array submission, then the XCE test harness shall print an error naming the failure and shall submit nothing.
7. If a harness command that acts on one job is given an identity without a [k] suffix that 5.14 and 5.15 do not reject, and no job has that identity, then the XCE test harness shall report that the job was not found and state that array elements must be named as <array-id>[k].
8. If a harness command that acts on one job is given an element selection of a range (<array-id>[a-b]) that 5.7, 5.8, 5.14 and 5.15 do not reject, then the XCE test harness shall refuse the command with a message stating that it takes a single element <array-id>[k], and shall act on no job.
9. When a harness command that acts on one job is given an identity without a [k] suffix that 5.14 and 5.15 do not reject, and a job has exactly that identity, the XCE test harness shall act on that job only, as before this feature (for expect-job, the assertion now compares job numbers rather than text, so any written form of the identity that 5.14 and 5.15 accept passes).
10. When a harness command line other than a ! shell-escape line or a # comment line (both handled as today: the shell-escape line is passed to the shell unchanged, the comment line is ignored) contains a token (a run of non-whitespace characters preceded by the line start or by a space or tab, the same token rule as for $jobid today) that is $jobid followed immediately by [, and $jobid is set, the XCE test harness shall replace $jobid with its current value and copy the rest of the token unchanged, so that $jobid[3] after jobsub -a N names element 3 of that array (an invalid remainder such as $jobid[3]x is then handled by 5.15).
11. If a harness command line other than a ! shell-escape line or a # comment line contains such a token (4.10) while $jobid is not set, then the XCE test harness shall fail the command with the same message as for an unset $jobid, and shall act on no job.

Requirement 5: Referring to, showing and controlling array elements

Objective: As a user or operator, I want element numbers to mean the same thing in every command, listing and log, so that the element I see is the element I act on.

Acceptance Criteria

1. XCE shall display each array element as <array-id>[k] in job listings, job detail output, control command results, and those controller, scheduler and worker log entries that print a job identity in its standard written form.
2. When a user makes, in a query or control command, an element selection of a single element ([k]) or a range of elements ([a-b], a equal to b allowed; a range by identity only in control commands) through the command-line client or REST API, and the selection's numbers are all at most 32767, its last number is not less than its first, it neither combines element 0 with other element numbers nor selects element 0 by job name, it does not select element 0 by identity (<array-id>[0] or <array-id>[0-0]; see 5.4, 5.11 and 8.2), and any <array-id> in it is valid (5.8), XCE shall act on exactly the existing array elements with those numbers (and never on an ordinary job with identity <array-id>; when no array has identity <array-id>, on no job), including when the most recent matching job is the element of a one-element array (5.10 applies only to filters without an element number).
3. If a user's control command, through the command-line client or REST API, makes an element selection by an identity that is valid (5.8) whose numbers are all at most 32767, whose last number is not less than its first, and which does not combine element 0 with other element numbers, and some selected numbers from 1 to 32767 are not elements of an array with that identity (all of them when no array has that identity), then XCE shall report each missing number as not found and shall act only on the existing elements in the selection.
4. If a user's control command, through the command-line client or the REST API, selects <array-id>[0] (or [0-0]), where <array-id> is valid (5.8), and no array with identity <array-id> has an element numbered 0 (whether or not an ordinary job has identity <array-id>), then XCE shall report element 0 as not found and shall act on no job, in particular not on the whole array or on an ordinary job with identity <array-id>.
5. If a user makes, in a query or control command, an element selection of a range through the command-line client or the REST API (by job name, or by identity in a control command) whose first and last numbers are both at most 32767 and whose last number is less than its first, then XCE shall reject the selection with a message stating that the range is reversed.
6. When a user gives, through the command-line client or REST API, a query or control command an identity written without any [k] suffix that 5.14, 5.16 and 5.17 do not reject, XCE shall act on every element of the array with that identity and on any ordinary job with that identity, as before this feature.
7. If a user makes an element selection of a single element whose number is greater than 32767, or of a range whose first or last number is greater than 32767, then XCE shall reject the selection as invalid, instead of resolving it to any other job, in query and control commands on the command-line client and the REST API and in every harness command that parses job identities.
8. If a user makes an element selection by identity (<array-id>[k] or <array-id>[a-b]) where <array-id> is not an identity XCE could have assigned to an array -- for example, it lies inside another array's range of element identities, or it is an ordinary job's identity that no array could have (most ordinary identities); an <array-id> written with any field wider than 5.14 allows is never an identity XCE could have assigned, even if its value would equal one -- then XCE shall reject the selection as an invalid job identity, instead of resolving it to elements of another array, in query and control commands on the command-line client and the REST API and in every harness command that parses job identities.
9. When the scheduler or the API server logs that it received a new array, XCE shall identify the array in that log entry by its array identity rather than by an element identity or raw number.
10. When a user applies a control command whose filter (for example -J foo, -u, -q) contains no element number ([...]) and does not use the all-jobs option, and the job that command's "most recent job" rule selects (the matching job with the highest job identity) is the element of a one-element array, XCE shall act on that element alone, as it does when the selected job is an ordinary job.
11. When a user's query selects <array-id>[0], where <array-id> is valid (5.8), and no array with identity <array-id> has an element numbered 0 (whether or not an ordinary job has identity <array-id>), XCE shall list no job, in particular not the elements of the whole array or an ordinary job with identity <array-id>.
12. If a job identity or element selection breaks more than one of 5.5, 5.7, 5.8, 5.14, 5.15, 5.16 and 5.17, then XCE shall report only the rejection of the criterion checked first in the order 5.15, 5.16, 5.14, 5.17, 5.8, 5.7, 5.5.
13. When a harness user gives a command that acts on one job an element identity <array-id>[k], where 0 <= k <= 32767 and <array-id> is valid (5.8), the XCE test harness shall act on that element if an element with that number exists in that array (8.2 for a pre-upgrade element 0), and otherwise shall report that the job was not found and act on no job, in particular not on the whole array or on an ordinary job with identity <array-id>.
14. If a user gives a job identity (with or without [...]) in which any field is wider than it can be (cluster above 63, API server above 15, timestamp of 2^32 or more, sequence of 2^20 or more), then XCE shall reject it as an invalid job identity, instead of resolving it to any other job, in query and control commands on the command-line client and the REST API and in every harness command that parses job identities.
15. If a harness command that parses job identities is given an argument that, after removing any spaces or tabs before or after it, is not a valid written job identity optionally followed by [k] or [a-b] (k, a and b each one or more decimal digits, with no sign and no whitespace) (for example 'X [1]' given as one argument, X[a], X[-1], X[+1], X[ 1], X[1, X[]), then the XCE test harness shall reject it as an invalid job identity and shall act on no job.
16. If a user gives, through the command-line client or REST API, to a query or control command a job identity or element selection whose [...] part is not one number or two numbers joined by - (a number: one or more decimal digits, with no sign and no whitespace), or that has an unmatched [ or ] or any non-whitespace character after the closing ] (whitespace before the identity or name, or after the closing ], is ignored, as before this feature; whitespace anywhere else, including between the identity or name and [, makes it invalid) (for example 'X [1]' given as one argument, -J 'foo [1]', X[a], X[], X[1, X[+1], X[1-3:2], -J 'name[ 1-3]', -J 'name[a]', -J 'name[1-3]x'), then XCE shall reject it as an invalid job identity or element selection and shall act on no job.
17. If a user gives, through the command-line client or REST API, to a query or control command a decimal job number without a [k] suffix that carries the array marking (for example the value of XCE_JOBID), whether or not an element with that number exists, then XCE shall reject it with a message stating that array elements must be named as <array-id>[k], and shall act on no job.

Requirement 6: Element number visible to a running element

Objective: As a user whose job script depends on the element number, I want a running element to know its own number, so that each element can select its own input and parameters, as LSF scripts do with LSB_JOBINDEX.

Acceptance Criteria

1. When an array element starts running, XCE shall set the environment variable XCE_JOBINDEX to that element's number in the environment XCE sets up for that element on each host it runs on.
2. When an ordinary (non-array) job starts running, XCE shall set XCE_JOBINDEX to 0 in the environment XCE sets up for that job on each host it runs on.
3. If a job's submission supplies its own environment variable named XCE_JOBINDEX, then XCE shall still set XCE_JOBINDEX to the value defined in 6.1 or 6.2.
4. If a job's submission supplies an environment variable named XCE_JOBINDEX, whatever its value, then XCE shall write a warning naming the job and the ignored variable to the log of each host on which XCE sets up that job's environment.
5. XCE shall set XCE_JOBID for every running job to the decimal number of that job's identity, as before this feature, unless the submission supplies its own XCE_JOBID (unchanged and out of scope).

Requirement 7: Cluster-side protection against unrepresentable arrays

Objective: As an operator, I want the cluster itself to refuse array submissions that it cannot represent, so that no client, including a faulty or outdated one, can create elements that collide with other jobs.

Acceptance Criteria

1. If any client makes a submission that declares itself an array and whose first element number is 0, then XCE shall reject the submission with an error status distinct from other submission errors, and shall write a controller log entry stating that array element numbers start at 1 and that the submitting client must be upgraded.
2. If any client makes a submission that declares itself an array, whose first element number is not 0 and whose last element number would be greater than 32767, then XCE shall reject the submission and return an error to that client.
3. If any client submits more than one element in a submission that does not declare itself an array, then XCE shall reject the submission and return an error to that client.
4. If any client submits a job with an element count of 0, then XCE shall reject the submission and return an error to that client.
5. If any client submits a job whose identity carries a marking other than "ordinary job" or "array", then XCE shall reject the submission and return an error to that client.
6. If XCE rejects a submission under 7.1-7.5, then XCE shall store no job for it and shall continue to accept valid submissions.
7. When any client makes a submission that declares itself an array, whose first element number is between 1 and 32767, whose element count is at least 1 and whose last element number is at most 32767, and the submission breaks none of 7.1-7.5, and no existing job has any of the element identities the submission would create, and no existing array, and no other array submission still being accepted (judged when this submission's collision check began; see Terms), has its array identity, and the submission passes the controller's other submission checks and is committed, XCE shall create it with elements numbered consecutively from that first number.
8. When XCE accepts (7.7) a one-element submission that declares itself an array, XCE shall treat it as a one-element array whose element behaves like an element of any larger array, including its listing and its XCE_JOBINDEX.
9. If a submission breaks 7.1 and also breaks any of 7.2-7.5 or collides (an existing job already has any of the element identities it would create, or it declares itself an array and an existing array or another array submission still being accepted already has its array identity), then XCE shall respond only as 7.1 requires. If a submission does not break 7.1 but breaks more than one of 7.2-7.5, or one of them and also collides (an existing job already has any of the element identities it would create, or it declares itself an array and an existing array or another array submission still being accepted already has its array identity), then XCE shall reject it with a single error that is not the duplicate-submission error of 7.10, and store no job for it.
10. If a submission breaks none of 7.1-7.5 and an existing job already has any of the element identities it would create, or it declares itself an array and an existing array or another array submission still being accepted (judged when this submission's collision check began; see Terms) already has the array identity it would have, then XCE shall reject it with the same duplicate-submission error as before this feature, or, if it also fails another of the controller's submission checks (7.7) and that check is reported first, with that check's error; XCE shall store no job for it and leave the existing jobs unchanged.
11. If XCE receives two or more submissions that declare themselves arrays, break none of 7.1-7.5 and have the same array identity, and each of them begins its collision check while every one of them whose collision check began earlier is still an array submission still being accepted (see Terms), then XCE shall create at most one of them: the one whose collision check began first, unless that one is rejected under 7.10 (because an existing job or array, or another array submission still being accepted, already has one of its identities), fails another submission check or fails to commit, in which case XCE shall create none of them; XCE shall reject each of the others with the duplicate-submission error of 7.10, or with the error of its own failed check if that check is reported first.

Requirement 8: Arrays submitted before this change

Objective: As an operator upgrading a running cluster, I want arrays that already exist to keep working unchanged, so that the upgrade does not disturb jobs already in the system.

Acceptance Criteria

1. While a cluster holds arrays submitted before this change, XCE shall keep those arrays' elements, including any element numbered 0, with their original numbers after the upgrade.
2. While an array submitted before this change contains an element numbered 0, when a user selects <array-id>[0] in a query command or controls <array-id>[0] (or, in a control command through the command-line client or the REST API, <array-id>[0-0]) as a single element through the command-line client, the REST API or the test harness, where <array-id> is that array's identity and is valid (5.8), XCE shall act on that element only and not on the whole array.
3. While an array submitted before this change contains an element numbered 0, XCE shall list that element and report control results for it as <array-id>[0].
4. When a controller restarts or fails over after the upgrade, XCE shall recover every job and array stored before the upgrade with the same identities and element numbers, and with the state each had when it was last stored.

Requirement 9: Documentation of the supported syntax

Objective: As a user, I want the help text and documentation to state exactly which array syntax is supported and how it differs from LSF, so that I know what to write and what to expect.

Acceptance Criteria

1. The XCE command-line help for job submission shall describe the name[1-N] and name[1] forms, state that element numbers start at 1, and state that the maximum array size is set by the API server's configuration (default 1000, never above 32767).
2. The XCE command-line help for job control commands shall show how to refer to one element and to a range of elements of an array.
3. The XCE test harness help for jobsub -a shall state that elements are numbered 1 through N and that N may be at most 32767.
4. The XCE user documentation shall list the LSF array features this release does not support: starting numbers other than 1, index lists, index steps, %N running limits, per-element %I/%J file-name substitution (at submission every element's file names get the first element's values, 1 and <array-id>[1]), interactive arrays, and the LSB_JOBINDEX/LSB_JOBID variables (scripts read XCE_JOBINDEX instead).