lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/job-array-one-based-index/design.md
Rongsong Shen 590860f73d
docs(job-array-one-based-index): requirements, design and tasks; log bug-1898, bug-1899 
Spec for numbering job-array elements from 1 (bug-1896): approved
requirements, a design reviewed over 20 rounds until clean (log in
research.md), and a 30-task plan. The design review found two live
apiserver defects, logged here: bracketed job_id/job_name filters are
never evaluated (bug-1898) and ControlJobByID re-dispatches earlier
selections (bug-1899).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 08:54:06 +00:00

80 KiB
Raw Blame History

Design Document: job-array-one-based-index

Overview

Purpose: This feature numbers job-array elements from 1, as LSF does. Elements are numbered 1 to N everywhere they are created, shown, selected, controlled and exposed to the running job. Every index form the feature does not support is rejected with a specific message instead of being silently renumbered (bug-1896).

Users: Users submitting arrays through xcecli or REST, operators controlling and listing elements, developers driving the xce_test harness, and job scripts reading XCE_JOBINDEX.

Impact: The element number already lives in the 15-bit index field of the 64-bit job ID. Under option B (requirements decision 1), an array's first element ID becomes array-id | 1. The existing contiguous expansion first + i in the controller, WAL apply and scheduler then yields elements 1 to N, with no change to the wire, WAL, snapshot or recording formats. The design also bounds every job-identity parser in the system, and adds cluster-side validation of array submissions.

Goals

  • name[1-N], name[1] and harness jobsub -a N create elements 1 to N. Every other index form is rejected with the message its requirement names.
  • One bounded identity and selection grammar, with one check order (5.12), per language: C (libxce), Go apiserver, Go xcecli.
  • The controller refuses array submissions it cannot represent. It also detects array-identity collisions, including against concurrent submissions.
  • A running element sees its own number in XCE_JOBINDEX, and the submission cannot override it.
  • Arrays submitted before this change keep working, including their element 0.

Non-Goals

  • Starts other than 1 at the clients, comma lists, steps, %N limits (all rejected, not implemented).
  • Per-element %I/%J expansion (blocked on bug-1508), and LSF LSB_JOBINDEX/LSB_JOBID.
  • Whole-array control through the harness or at the controller; job output (peek) and restart by identity; the array statistics view.
  • The idgen sequence overflow in general. The preCheckJobRequest early return for named jobs (unchanged; the new 2.1 interactive-array check runs inside ParseSubmitIndexExpr, before it, while today's interactive-value check at jobs.go:1734-1736 stays where it is). All other defects listed as out of scope in the requirements.

Boundary Commitments

This Spec Owns

  • Job identity grammar and bounds: the written form c-a-ts-seq, with an optional [k] or [a-b], and the decimal form. The libxce, apiserver and xcecli parsers for them, and their check order.
  • Array ID allocation: idgen_array and its Go wrapper. idgen reduced to ordinary IDs only.
  • Array-submission validation and collision detection in the controller's handle_jobsubmit, including the in-flight registry's array-identity awareness.
  • The apiserver's handling of submit index expressions, the submit response shape (jobid, array_size), REST selection parsing and matching, the unique_job_id of not-found control results, and the array-size limit setting.
  • ControlJobByID's dispatch loop (bug-1899: moved after the selection loop, de-duplicated).
  • Evaluation of selection predicates in the apiserver's in-memory query pipeline (store.ExecuteQuery). Today bracketed selections are built as CompConditions that are never evaluated (bug-1898); this spec replaces them with evaluated row predicates.
  • xcecli identity parsing, ID-list splitting, submit output and help text.
  • Harness: jobsub -a, $jobid and $jobid[...], the shared single-job argument helper, and expect-job matching.
  • The worker's XCE_JOBINDEX.
  • Scheduler: the array-expansion guard and its log line.
  • Skill scripts, src/docs/xce.md, and the cross-spec document updates listed in the File Structure Plan.

Out of Boundary

  • Storage, replication and apply: WAL JOB_ARRAY_SUBMIT append, apply and replay; snapshots; HA replication and standby apply; controller-to-scheduler and controller-to-apiserver pushes; scheduler recording and replay. They are relied on unchanged.
  • Display helpers: fmt_jobid, the Go FormatJobIDs, and array_jobid/array_idx (display semantics unchanged).
  • RecordAcceptedJob. (The apiserver's array_idx between construction and the unevaluated CompConditions are replaced on the selection paths this spec owns, see ApiMatch; other CompConditions users, if any appear, are untouched.)
  • Peek (output.go, parseJobIDParam, the sixth parseArrayName caller; xcecli peek.go, which keeps a legacy ToAPIJobIDLegacy with today's passthrough behaviour) and restart (RestartJobByID), which keep idgen with size 1.
  • The scheduler REPL info job argument.

Allowed Dependencies

  • libxce (utils.c, idgen.c, errnox.h, config.h) is the lowest layer. The controller, harness, worker and scheduler depend on it. The apiserver reaches it only through backend/clib.go cgo, plus pure-Go mirrors in backend/convert_pure.go.
  • The apiserver's selection parsers (ParseIdentitySelection, ParseNameSelection in controllers/jobsel.go) depend only on backend ID helpers and the standard library. Predicate building (ApiMatch, in controllers/jobs.go) may additionally use utils (BuildFilterConditions, OrderMap, QueryOptions) and store (MatchesCondition). Neither utils nor store depends on controllers.
  • The xcecli format package stays dependency-free; cmd/ and filters/ depend on it.
  • The harness helpers in cmdutil.c and autovars.c depend on libxce only; test.c depends on them.
  • No new third-party dependencies.

Revalidation Triggers

  • Job ID bit layout (config.h, the Go constants): any change invalidates all three parsers and idgen_array.
  • Controller submit reply semantics (still the first element ID). clib.go, RecordAcceptedJob and the harness ack check depend on it.
  • REST response shapes: JobSubmitResponse gains array_size; jobid now names the array. Not-found control result items gain unique_job_id (found items already carry it). Any REST consumer (xcecli, the regression suite) must re-check.
  • utils.QueryOptions shape: gains Predicates; any other query builder must re-check.
  • The new controller error status E_XCE_ARRAY_START0, which every client error printer must know about.
  • Specs that defined $jobid or element numbering: xce-test-e2e-primitives and job-array-single-wal-record Requirement 5 are superseded here.

Architecture

Existing Architecture Analysis

  • Submission: the client (apiserver or harness) allocates IDs with idgen. It sends (first_jobid, array_size, submit) through OPCode_jobsub. The controller checks for duplicates, commits one JOB_ARRAY_SUBMIT WAL record, and expands first + i on apply. It replies with first_jobid. The scheduler expands the pushed array in driver.c translation, before core events.
  • Identity text is parsed in about ten places with differing leniency (research.md, "Discovery corrections").
  • Constraints kept:
    • The controller vtask handler never blocks (ha-async-io.md).
    • Controllers never self-exit.
    • REST handlers use ReturnError(...); return.
    • The middleware chain is unchanged.
    • The scheduler driver state-equivalence holds, because expansion stays before core events in driver.c only.

Architecture Pattern & Boundary Map

graph TB
    subgraph Clients
        Xcecli[xcecli format jobsel]
        Harness[harness cmdutil autovars]
    end
    subgraph ApiServer
        ApiSel[controllers jobsel]
        ApiSubmit[controllers jobarray submit]
        Clib[backend clib idgen wrapper]
    end
    subgraph Libxce
        Parse[parse jobsel]
        Idgen[idgen array]
    end
    subgraph Daemons
        Ctrl[controller handle jobsubmit]
        Sched[scheduler driver expansion]
        Worker[worker exdata env]
    end
    Xcecli --> ApiSel
    Xcecli --> ApiSubmit
    ApiSubmit --> Clib
    Clib --> Idgen
    Clib --> Ctrl
    ApiSel --> Ctrl
    Harness --> Parse
    Harness --> Idgen
    Harness --> Ctrl
    Ctrl --> Sched
    Sched --> Worker

Architecture Integration:

  • Selected pattern: one bounded parser per language behind each surface, plus a single validation gate at the controller.
  • Domain boundaries: identity text belongs to the parsers. Admission of arrays belongs to the controller. Element-number exposure belongs to the worker.
  • Existing patterns preserved:
    • Option B expansion.
    • The preCheck* and ReturnError patterns.
    • Hook variables for test seams (validateResreqSyntax).
    • Unity C tests in testsuite/ and table-driven Go tests.
  • New components and why they exist:
    • parse_jobsel and the two Go jobsel modules collapse ten lenient parsers into one grammar each.
    • idgen_array exists because one-element arrays need a distinct request.
    • autovars.c exists to make $jobid[...] substitution unit-testable.
  • Steering compliance: TDD per component; errors are negative errno or errnox in C and ReturnError plus zap in Go.

Technology Stack

Layer Choice / Version Role in Feature Notes
CLI Go (xcecli module), Cobra Identity parsing, submit output, help No new dependencies
Backend services Go (apiserver, gin) + cgo to libxce Submit expression, REST selections, config GenerateArrayJobID cgo wrapper plus non-cgo stub
Daemons C gnu17 (controller, scheduler, worker), libxce Validation, expansion guard, env strtoull with errno replaces atoi
Test harness C (xce_test) + Unity jobsub -a, $jobid, element commands New autovars.c; cmdutil.c extended
Data / Storage WAL, snapshots Unchanged No format change (option B)

File Structure Plan

New Files

src/lib/testsuite/test_jobsel.c               # C conformance cases for parse_jobsel/parse_jobid (selection-cases table)
src/harness/autovars.c                        # autovar_substitute(): $jobid and $jobid[...] token substitution (moved out of test.c)
src/harness/include/autovars.h
src/harness/testsuite/test_autovars.c         # 4.10, 4.11 unit tests
src/frontend/apiserver/controllers/jobsel.go          # REST identity/name selection parser + key validator
src/frontend/apiserver/controllers/jobsel_test.go     # selection-cases table + REST-only cases (5.17, decimal keys)
src/frontend/apiserver/controllers/jobarray.go        # submit index-expression parser (2.1-2.9) for bracketed names; createArrayName keeps plain names
src/frontend/apiserver/controllers/jobarray_test.go
src/frontend/apiserver/controllers/jobs_handler_test.go # //go:build !cgo: handler-level submit/query/control tests through the test-seam variables
src/frontend/xcecli/common/format/jobsel.go           # client identity selection parser + list splitter
src/frontend/xcecli/common/format/jobsel_test.go
src/frontend/xcecli/cmd/jobs/controller_test.go      # controlJobsByIDs against an httptest server (request.NewAPIClient(srv.URL)): X[1-3] reaches the path as <decimal>[1-3]
src/controller/testsuite/test_jobsubmit_validate.c    # 7.1-7.11 handle_jobsubmit validation/collision cases
src/worker/testsuite/test_exdata_jobindex.c           # 6.1-6.4

Modified Files

  • libxce
    • src/include/xce/errnox.h: add E_XCE_ARRAY_START0 (before E_XCE_MAX) and its message text wherever errnox strings are defined.
    • src/include/utils.h, src/lib/utils.c:
      • Add jobsel_t, jobsel_err_t, parse_jobsel(), jobsel_strerror() and is_array_tag().
      • Re-implement parse_jobid() over parse_jobsel() with the same signature: NULL-safe, bounded, ranges rejected.
      • Add JOBID_FIELD_MAX_* constants.
    • src/include/idgen.h, src/lib/idgen.c:
      • Add idgen_array().
      • idgen() accepts size 1 only.
    • src/lib/testsuite/test_idgen.c: array cases move to idgen_array (first element index 1, bounds 1..32767, block overflow refused).
    • src/lib/meson.build (libxce_test_src) and the runner src/lib/testsuite/test.c: register test_jobsel.
  • Controller
    • src/controller/job.c:
      • Add jobsubmit_classify() (pure) and array_identity_exists().
      • Registry entries gain array_id, and the overlap test considers it.
      • handle_jobsubmit order: classify, then register, then table checks with release on a hit. Add the 7.1 log line.
    • src/controller/testsuite/test_commit_barrier_reply.c: array cases use tagged block|1 IDs.
    • src/controller/testsuite/test_commit_barrier_rollout.c: arrays use at most 32767 elements with tagged first IDs.
    • src/controller/meson.build and the runner src/controller/testsuite/controller_test.c: register test_jobsubmit_validate.
    • src/controller/job.c (also): inflight_reply_and_remove(); jobsubmit_async_ctx_release no longer removes registry entries; under XCE_TESTING, jobsubmit_test_reply_hook, jobsubmit_inflight_overlaps_for_test(), jobsubmit_inflight_overlaps_locked_for_test() and jobsubmit_inflight_trylock_for_test() (declared under #ifdef XCE_TESTING, as in cluster.h:110-118).
  • Scheduler
    • src/scheduler/src/driver.c:
      • Expansion guard (tag 1 and size <= 32768 - idx, else log ERROR and skip the array).
      • The INFO log prints array_jobid(first).
    • src/scheduler/testsuite/test_driver_translate.c: tagged IDs, plus guard cases.
    • src/scheduler/testsuite/test_job_array_expand.c (bases 5000, 1, 8000 at :182, :213, :283) and test_job_array_wire_e2e.c (42000, 43000 at :295, :338-339): move to tagged block|1 bases, since the guard skips untagged arrays.
  • Worker
    • src/worker/src/xcecd_exdata.c:
      • XCE_JOBINDEX is array_idx(jobid) for tag 1 and 0 otherwise.
      • A user XCE_JOBINDEX is skipped with a WARN.
    • src/worker/testsuite/meson.build: register the new test.
  • Harness
    • src/harness/cmdutil.c, src/harness/include/cmdutil.h: add cmdutil_parse_array_size(), cmdutil_job_arg() and cmdutil_job_not_found().
    • src/harness/testsuite/test_cmdutil.c: cases for the above.
    • src/harness/test.c:
      • jobsub: -a through the new parser; idgen_array with its return code checked; $jobid set to the array ID; help text.
      • substitute_autovars replaced by autovar_substitute.
      • expect-job and joblist_query match by number.
      • Every single-job command uses cmdutil_job_arg and cmdutil_job_not_found.
    • src/harness/meson.build: add autovars.c to xce_test_src and to a new test_autovars unit-test target.
    • src/harness/testsuite/test_test_cli.c: pty cases (no controller needed) for the jobsub -a refusals and for jobkill -j 1-1-1-0[1-3] (the 4.8 refusal, raised during option parsing before any RPC).
  • Apiserver
    • src/frontend/apiserver/backend/clib.go, stub.go: add GenerateArrayJobID(cluster, apiserver, n).
    • src/frontend/apiserver/backend/convert_pure.go: add ParseHumanJobIDStrict (field widths), IsArrayCapableKey, and MakeElementID(key, k).
    • src/frontend/apiserver/backend/handler.go: the dispatchJobArray log names the array ID (FormatJobID(int64(ExtractArrayBaseID(baseJobID))); the parameter baseJobID holds the first element ID).
    • src/frontend/apiserver/utils/types.go: QueryOptions gains Predicates []func(OrderMap) bool.
    • src/frontend/apiserver/store/query.go: ExecuteQuery applies Predicates after Conditions and before sort and pagination (fixes bug-1898 on the owned paths).
    • src/frontend/apiserver/controllers/jobs.go:
      • preCheckJobRequest: the interactive-name check runs first; then the jobarray.go parser.
      • CreateJob: sizing and ID through GenerateArrayJobID; the response; test-seam variables generateJobID, generateArrayJobID, submitJob, controlJob (used at both control dispatch sites).
      • GetJobsInfo, GetJobByIDs, ControlJobByID, FetchJobList and filter-based ControlJob route through jobsel.go.
      • foundJob gains an explicitIndex flag (full signature in ApiControl).
      • Not-found control result maps (ControlJobByID and ControlJob) gain unique_job_id.
      • The most-recent-job rule (5.10).
      • createArrayName is kept, called only for names without [ or ] (1.4); delete parseArrayIndexSection and the IsNumeric uses on these paths.
    • src/frontend/apiserver/models/jobs.go: JobSubmitResponse gains ArraySize int json:"array_size,omitempty". (Control results are map[string]interface{} built in jobs.go; there is no apiserver control-item struct.)
    • src/frontend/apiserver/utils/config.go, config/app.yaml: clamp the limit to 32767 with a startup warning. The rule moves into a pure resolveMaxArraySize(v int) (effective int, clamped bool, err error); InitConf calls it and emits the utils.Warn.
    • controllers/jobs_test.go, utils/config_test.go, backend/convert_test.go, store/query_test.go: new cases. TestFoundJob_MatchesRealArrayIdxType (jobs_test.go:179) moves to the new foundJob signature and gets a tagged Unique_JobID in its fixture; the four preCheckJobRequest call sites (jobs_test.go:28, :100, :118, :217) become _, _, err := preCheckJobRequest(req).
  • xcecli
    • src/frontend/xcecli/common/format/jobid.go:
      • ParseJobID becomes bounded (delegating to jobsel.go).
      • Signature change: ToAPIJobID(s string) (jobid.go:134) becomes ToAPIJobID(s string, allowRange bool), implemented as ParseSelection(s, allowRange) then APIKey(). Call sites updated: cmd/jobs/controller.go:52 (controlJobsByIDs) passes allowRange=true; filters/job_filter.go:49 and common/format/jobid_test.go:83, :96, :109 pass allowRange=false; peek.go:81 moves to ToAPIJobIDLegacy. unrecognised text is no longer passed through. Control (cmd/jobs/controller.go controlJobsByIDs) calls it with allowRange=true, so kill X[1-3] still reaches ControlJobByID (5.2, 5.3, 5.5); queries (filters/job_filter.go) call it with allowRange=false. Plain decimal keys still pass through to the server's checks, after the client's own 5.7/5.8 checks when [...] is present.
      • ToAPIJobIDLegacy(s) keeps today's passthrough behaviour for cmd/jobs/peek.go:81 (peek is out of scope).
      • FormatArrayChildID keeps its signature and is used only as a fallback.
    • src/frontend/xcecli/cmd/jobs/controller.go:
      • ParseCmdArgs uses SplitSelections.
      • controlJobsByIDs validates every selection before sending.
      • Results print FormatJobID(UniqueJobID) when it is present.
      • Help text (9.2).
    • src/frontend/xcecli/filters/job_filter.go: SplitSelections, isValidJobId replaced by ParseSelection, and the raw -J value checked with ParseNameSelection before the wildcard mapping (:81-91); for a bracketed value the search rule and * stripping apply to the name part only.
    • src/frontend/xcecli/cmd/jobs/controller.go (also): the raw control -J value checked with ParseNameSelection before controlJobsByFilter.
    • src/frontend/xcecli/cmd/jobs/submit.go:
      • Prints the array ID and element count.
      • Help text (9.1).
    • src/frontend/xcecli/cmd/jobs/list.go: help text -J correction.
    • src/frontend/xcecli/models/jobs.go: JobSubmitData gains ArraySize; JobControlResponseItem gains UniqueJobID int64 json:"unique_job_id,omitempty".
    • src/frontend/xcecli/cmd/jobs/peek.go and peek_test.go (:18,23): switch to ToAPIJobIDLegacy (no behaviour change).
    • common/format/jobid_test.go: in TestToAPIJobIDPassthrough (:107-117), "123456" stays a passthrough case; "123456[3]" is misaligned (123456 & 0x7FFF != 0) and is replaced by the aligned "32768[3]" → "32768[3]", with a new rejection case asserting "123456[3]" returns the 5.8 error; malformed text is now rejected.
  • Skills and docs
    • .claude/skills/xce-test-ha/run-ha-soak.sh, .claude/skills/xce-test-ha/SKILL.md: elements 1..N.
    • .claude/skills/xce-test-multihost/run-multihost-verify.sh, .claude/skills/xce-test-multihost/SKILL.md: element 1.
    • src/docs/xce.md: new "Job arrays" section (9.4).
    • .kiro/specs/job-array-single-wal-record/requirements.md (Requirement 5) and .kiro/specs/xce-test-e2e-primitives/requirements.md / design.md: superseded notes.
    • .kiro/specs/controller-job-cleanup/brief.md:86-87, .kiro/specs/job-dependency/brief.md:45-46: numbering updated.
    • .kiro/steering/roadmap.md: new entry.

System Flows

Controller array admission

sequenceDiagram
    participant Client
    participant Handler as handle jobsubmit
    participant Registry as inflight registry
    participant Table as jobmgr
    participant Wal as commit barrier
    Client->>Handler: jobsub first id and size
    Handler->>Handler: classify
    alt start zero
        Handler-->>Client: E XCE ARRAY START0 and log
    else invalid
        Handler-->>Client: EINVAL
    else ok
        Handler->>Handler: field checks and hook
        Handler->>Registry: try add range and array id
        alt overlap
            Registry-->>Handler: busy
            Handler-->>Client: EEXIST
        else added
            Handler->>Table: element range exists or array id exists
            alt collision
                Handler->>Registry: reply EEXIST and release atomically
                Registry-->>Client: EEXIST
            else free
                Handler->>Wal: commit array record
                Wal-->>Handler: applied
                Handler->>Registry: reply and release atomically
                Registry-->>Client: first element id
            end
        end
    end

  • classify evaluates the checks in this order:

    1. 7.1 (tag 1, index 0) → INDEX0.
    2. 7.4 (size 0), 7.5 (tag 2 or 3), 7.3 (tag 0 and size > 1), 7.2 (tag 1 and size > 32768 - idx, written without addition so a UInt64 size cannot wrap) → INVALID.

    This gives 7.9: INDEX0 alone wins, and any INVALID wins over a collision, using the single error -EINVAL.

  • Registration precedes the table checks, so any submission that commits is in the job table before it leaves the registry. Two concurrent submissions with the same array ID are ordered by who registers first (7.11).

  • An array submission is registered for its whole element range with is_array = 1 and its array_id (tag cleared, low 15 bits cleared). It is compared with every other array entry's array_id. Ordinary entries carry is_array = 0 and take part in range overlap only.

Selection evaluation (all surfaces)

graph TB
    Raw[raw argument] --> Syntax{grammar ok}
    Syntax -- no --> E516[reject invalid selection]
    Syntax -- yes --> Width{fields within width}
    Width -- no --> E514[reject invalid identity]
    Width -- yes --> Marked{decimal key with array marking and no index}
    Marked -- yes --> E517[reject name elements as array id k]
    Marked -- no --> Align{index given and key not array capable}
    Align -- yes --> E58[reject invalid identity]
    Align -- no --> Bound{number above 32767}
    Bound -- yes --> E57[reject invalid selection]
    Bound -- no --> Rev{range reversed}
    Rev -- yes --> E55[reject range reversed]
    Rev -- no --> Ok[parsed selection]
  • The check order is exactly 5.12: 5.15/5.16, then 5.14, then 5.17, then 5.8, then 5.7, then 5.5.
  • The harness parser has no decimal form, so the 5.17 node never applies there. Its syntax node is 5.15.
  • For name selections (-J, job_name), the width, marked and alignment nodes do not apply. The part before [ is not checked against the submission array-name rule (2.2): a name no array can have simply selects no job (5.2).

Requirements Traceability

Requirement Summary Components Interfaces Flows
1.1 name[1-N] creates 1..N ApiSubmitExpr, ApiCreateJob, IdgenArray, ControllerValidate ParseSubmitIndexExpr, GenerateArrayJobID Admission
1.2 name[1] makes a one-element array ApiSubmitExpr, ApiCreateJob, IdgenArray same Admission
1.3 distinct element identities IdgenArray, ControllerValidate idgen_array, first + i Admission
1.4 names without brackets unchanged ApiCreateJob (preCheckJobRequest) createArrayName path unchanged; isArray=false → GenerateJobID(..., 1) —
1.5 leading zeros ApiSubmitExpr, ApiSel, CliSel, LibJobsel number rule Selection
1.6 configured limit used ApiConfig MaxArraySize —
1.7 clamp to 32767 with warning ApiConfig startup utils.Warn —
1.8 unset or 0 → 1000 ApiConfig — —
1.9 negative refuses start ApiConfig InitConf error —
1.10 recovery keeps numbering (unchanged WAL/apply; option B) — —
2.1 interactive arrays rejected ApiSubmitExpr interactive check before name branch —
2.2 malformed expression or bad name ApiSubmitExpr ErrArrayName, ErrArrayMalformed —
2.3 lists or steps rejected ApiSubmitExpr ErrArraySingleRange —
2.4 %N rejected ApiSubmitExpr ErrArrayRunLimit —
2.5 start 0 rejected ApiSubmitExpr ErrArrayStartZero —
2.6 reversed range ApiSubmitExpr ErrArrayReversed —
2.7 start > 1 rejected ApiSubmitExpr ErrArrayStartNotOne —
2.8 above limit ApiSubmitExpr ErrArrayTooLarge —
2.9 lowest-numbered rule reported ApiSubmitExpr ordered evaluation —
2.10 nothing created on rejection ApiCreateJob early ReturnError before GenerateArrayJobID —
3.1 response names the array ApiCreateJob, CliSubmit JobSubmitResponse.jobid —
3.2 element count shown ApiCreateJob, CliSubmit array_size —
3.3 ordinary submit unchanged ApiCreateJob — —
4.1 jobsub -a N makes 1..N HarnessJobsub, IdgenArray cmdutil_parse_array_size, idgen_array Admission
4.2 -a 1 stays ordinary HarnessJobsub idgen size 1 —
4.3 N > 32767 refused HarnessJobsub -ERANGE message —
4.4 bad, missing or 0 refused HarnessJobsub -EINVAL message —
4.5 success line and $jobid are the array ID HarnessJobsub array_jobid —
4.6 ID failure reported HarnessJobsub idgen_array rc —
4.7 not-found hint HarnessJobArg cmdutil_job_not_found —
4.8 ranges refused HarnessJobArg cmdutil_job_arg Selection
4.9 exact ID acts as before HarnessJobArg, HarnessExpectJob numeric match —
4.10 $jobid[...] substitution HarnessAutovars autovar_substitute —
4.11 unset $jobid fails HarnessAutovars same —
5.1 <array-id>[k] display LibJobsel (display unchanged), CliControl, ApiControl unique_job_id, fmt_jobid —
5.2 element selection acts on elements only ApiSel, ApiMatch, CliSel Selection, element-only filter Selection
5.3 missing numbers reported ApiControl per-number not-found items —
5.4 missing [0] reported, no fallback ApiControl foundJob(explicitIndex=true) —
5.5 reversed selection range ApiSel, CliSel ErrSelReversed Selection
5.6 bare ID acts on whole array and ordinary job ApiMatch bare form —
5.7 number > 32767 rejected ApiSel, CliSel, LibJobsel *_EINDEX Selection
5.8 non-array-capable <array-id> rejected ApiSel, CliSel, LibJobsel *_EALIGN Selection
5.9 array logs use the array ID SchedGuard, ApiDispatchLog (dispatch + CreateJob log lines) log lines —
5.10 most-recent rule for one-element array ApiControl isSingleElementArray —
5.11 query of missing [0] lists nothing ApiMatch explicit-index filter —
5.12 first failed check reported ApiSel, CliSel, LibJobsel ordered evaluation Selection
5.13 harness [k] acts on that element HarnessJobArg exact element ID —
5.14 field-width rejection ApiSel, CliSel, LibJobsel *_EWIDTH Selection
5.15 harness syntax rejection LibJobsel, HarnessJobArg JOBSEL_ESYNTAX Selection
5.16 CLI/REST syntax rejection ApiSel, CliSel, CliSplit ErrSelSyntax, bracket-aware split Selection
5.17 marked decimal without [k] rejected ApiSel ErrSelMarked Selection
6.1 XCE_JOBINDEX is the element number WorkerEnv array_idx —
6.2 ordinary jobs get 0 WorkerEnv tag test —
6.3 user value cannot override WorkerEnv skip in user loop —
6.4 warning per host WorkerEnv XCECD_WORKER_LOG WARN —
6.5 XCE_JOBID unchanged WorkerEnv — —
7.1 start-0 array → distinct error and log ControllerValidate E_XCE_ARRAY_START0 Admission
7.2 last > 32767 rejected ControllerValidate INVALID Admission
7.3 untagged size > 1 rejected ControllerValidate INVALID Admission
7.4 size 0 rejected ControllerValidate INVALID Admission
7.5 other tags rejected ControllerValidate INVALID Admission
7.6 nothing stored; service continues ControllerValidate early reply, no registry Admission
7.7 valid array created from its first number ControllerValidate, ControllerCollision classify OK, then commit Admission
7.8 one-element array behaves as array ControllerValidate, WorkerEnv, ApiMatch tagged single path —
7.9 error precedence ControllerValidate classify order Admission
7.10 collisions → EEXIST ControllerCollision array_identity_exists, registry array_id Admission
7.11 concurrent same array ID: first wins ControllerCollision registry before table Admission
8.1 pre-upgrade element 0 kept (unchanged storage), LibJobsel accepts [0] — —
8.2 <array-id>[0] selects that element only ApiMatch, ApiControl, HarnessJobArg explicit-index flag Selection
8.3 element 0 shown as [0] CliControl, ApiControl unique_job_id, FormatJobID —
8.4 recovery after upgrade (unchanged WAL/apply) — —
9.1 submit help CliSubmit help text —
9.2 control help CliControl help text —
9.3 harness help HarnessJobsub usage strings —
9.4 user documentation Docs src/docs/xce.md —

Components and Interfaces

Component Domain/Layer Intent Req Coverage Key Dependencies (P0/P1) Contracts
LibJobsel libxce Bounded identity/selection parser 1.5, 5.1, 5.7, 5.8, 5.12, 5.14, 5.15, 8.1 config.h layout (P0) Service
IdgenArray libxce Allocate an array's first element ID 1.1-1.3, 4.1 idgen ctx (P0) Service
ControllerValidate controller Classify array submissions 7.1-7.9 LibJobsel is_array_tag (P0) Service
ControllerCollision controller Element and array-ID collision checks 7.7, 7.10, 7.11 jobmgr, registry (P0) Service, State
SchedGuard scheduler Guard expansion; log the array ID 5.9 driver translation (P1) Service
WorkerEnv worker XCE_JOBINDEX 6.1-6.5, 7.8 array_idx (P0) Service
HarnessJobsub harness jobsub -a and $jobid 4.1-4.6, 9.3 IdgenArray (P0) Service
HarnessJobArg harness Single-job argument parsing and not-found reporting 4.7-4.9, 5.13, 5.15, 8.2 LibJobsel (P0) Service
HarnessExpectJob harness Numeric expect-job 4.9 HarnessJobArg (P0) Service
HarnessAutovars harness $jobid and $jobid[...] 4.10, 4.11 — Service
ApiSubmitExpr apiserver Submit index expression 1.1, 1.2, 1.5, 2.1-2.9 ApiConfig (P0) Service
ApiCreateJob apiserver IDs, submit, response 1.1-1.4, 2.10, 3.1-3.3 IdgenArray via cgo (P0) API
ApiConfig apiserver Array-size limit 1.6-1.9 zap logger (P1) State
ApiSel apiserver REST selection parser and key validator 1.5, 5.2, 5.5, 5.7, 5.8, 5.12, 5.14, 5.16, 5.17 backend ID helpers (P0) Service
ApiMatch apiserver Selection → store filter 5.2, 5.6, 5.11, 7.8, 8.2 ApiSel (P0) Service
ApiControl apiserver Control results, missing numbers, most-recent rule 5.1, 5.3, 5.4, 5.10, 8.2, 8.3 ApiMatch (P0) API
ApiDispatchLog apiserver Array log line 5.9 — —
CliSel xcecli Client selection parser 1.5, 5.2, 5.5, 5.7, 5.8, 5.12, 5.14, 5.16 — Service
CliSplit xcecli Bracket-aware ID-list split 5.16 — Service
CliSubmit xcecli Submit output and help 3.1, 3.2, 9.1 ApiCreateJob (P0) —
CliControl xcecli Control by ID; result display; help 5.1, 8.3, 9.2 CliSel, ApiControl (P0) —
Docs docs Syntax and LSF differences 9.4 — —

libxce

LibJobsel

Field Detail
Intent Parse one written job identity, optionally with [k] or [a-b], under fixed bounds and check order
Requirements 1.5, 5.1, 5.7, 5.8, 5.12, 5.14, 5.15, 8.1

Responsibilities & Constraints

  • Grammar: optional leading/trailing spaces or tabs, num-num-num-num, optionally [num] or [num-num]. num is one or more decimal digits. There is no other whitespace and no sign.
  • Checks, in order: syntax (5.15), field widths (5.14: cluster <= 63, API server <= 15, timestamp < 2^32, sequence < 2^20), alignment when [...] is present (5.8: seq % 32768 == 0), index bound (5.7: each number <= 32767), reversed range (JOBSEL_EREVERSED). The harness maps JOBSEL_EREVERSED to the 4.8 single-element refusal, because 5.5 applies only to the CLI and REST.
  • Overflow: digits are accumulated with an overflow guard (strtoull with errno, or a digit-count bound), so oversized numbers fail 5.14 or 5.7 rather than wrapping.
  • Display unchanged: fmt_jobid, array_jobid and array_idx keep their behaviour.

Contracts: Service [x]

Service Interface
typedef enum {
    JOBSEL_OK = 0,
    JOBSEL_ESYNTAX,   /* 5.15 */
    JOBSEL_EWIDTH,    /* 5.14 */
    JOBSEL_EALIGN,    /* 5.8  */
    JOBSEL_EINDEX,    /* 5.7  */
    JOBSEL_EREVERSED  /* 5.5  */
} jobsel_err_t;

typedef enum { JOBSEL_BARE = 0, JOBSEL_SINGLE, JOBSEL_RANGE } jobsel_form_t;

typedef struct {
    uint64_t      key;  /* BARE: the exact (untagged) ID; else the array ID */
    jobsel_form_t form;
    uint32_t      lo;   /* SINGLE: lo == hi == k */
    uint32_t      hi;
} jobsel_t;

jobsel_err_t parse_jobsel(const char* text, jobsel_t* out);
const char*  jobsel_strerror(jobsel_err_t err);
int          is_array_tag(uint64_t jobid);            /* TAG(jobid) == TAG_JOB_ARRAY */
uint64_t     jobsel_element_id(const jobsel_t* s, uint32_t k); /* MAKE_JOBID(.., TAG_JOB_ARRAY, key|k) */

/* unchanged signature; now bounded, NULL-safe, and returns -EINVAL for a
 * range, a rejected selection, or NULL; 1 on success */
int parse_jobid(uint64_t* jobid, char* buf);
  • Preconditions: out != NULL. A NULL text returns JOBSEL_ESYNTAX.
  • Postconditions: on JOBSEL_OK, key satisfies the field widths. For SINGLE and RANGE, key has tag 0 and its low 15 bits clear.
  • Invariants: fields are masked to their widths before composition. MAKE_JOBID itself is unchanged.

Implementation Notes

  • Validation: test_jobsel.c runs the selection-cases table (Testing Strategy).
  • Callers of parse_jobid: the nine harness single-job commands (test.c:3327, 3517, 4140, 4276, 4921, 5073, 5268, 5377, 5481), which move to cmdutil_job_arg, and scheduler/src/repl_info.c:114. The REPL info job argument is out of scope, and its behaviour does not change: on any parse_jobid failure it falls back to strtoull (repl_info.c:114-117), which still yields the decimal value, 0, or the leading digits as today.

IdgenArray

Field Detail
Intent Return the first element ID of a new array of n elements, numbered from 1
Requirements 1.1, 1.2, 1.3, 4.1

Contracts: Service [x]

/* 1 <= n <= MAX_ARRAY_IDX (32767). On success *first = MAKE_JOBID(c, a, ts,
 * TAG_JOB_ARRAY, (block << 15) | 1) and returns n. -EINVAL for bad args or n;
 * -EOVERFLOW if the next block would leave the 20-bit sequence (the context
 * is left unchanged). Advances the context past the whole block. */
int idgen_array(idgen_ctx_t* ctx, size_t clusterId, size_t apiserverId,
                size_t n, uint64_t* first);

/* Ordinary IDs only: arraySize must be 1 (else -EINVAL). Returns 1. */
int idgen(idgen_ctx_t* ctx, size_t clusterId, size_t apiserverId,
          size_t arraySize, uint64_t* pidx);
  • Postconditions: first + n - 1 has the same array ID and index n.
  • Go wrapper: GenerateArrayJobID(clusterID, apiserverID int, n int) (uint64, error) in backend/clib.go, plus a matching stub.go stub that returns an error.
  • GenerateJobID: its only callers are CreateJob for ordinary jobs and RestartJobByID, so it is always called with size 1; the arraySize parameter stays for signature stability.
  • idgen callers: clib.go:341 (through GenerateJobID) and the harness test.c:3721, which switches to idgen_array for N >= 2.

Controller

ControllerValidate and ControllerCollision

Field Detail
Intent Admit only representable array submissions; detect identity collisions without a window
Requirements 7.1-7.11

Responsibilities & Constraints

  • Pure classification: jobsubmit_classify runs first -- as the first statement after init_reply_by_req in handle_jobsubmit (there is no NULL check on submit today, and the existing field checks dereference it the same way), before the existing field checks (NULLSTR user/queue, start_info, empty cmdline, job.c:2445-2454), the HOOK_BEFORE_EVENT hook and any registry or table access. It reads only submit->jobid and submit->array_size, so this is safe, and it makes 7.9 hold for every submission: an index-0 array is answered E_XCE_ARRAY_START0 with its log line even when a field is also empty, and a 7.2-7.5 break is answered -EINVAL before any field check. Consequence: HOOK_BEFORE_EVENT no longer fires for submissions rejected under 7.1-7.5 (today it fires for every submission that passes the field checks, including ones a later check rejects); such submissions create no job, so no hook consumer loses an event for a job that exists.
  • Registry and table: registry add first, then the table checks (7.10, 7.11). Answering a registered submission and removing its registry entry are one atomic step: a new inflight_reply_and_remove(endpoint_t* ep, xce_peerinfo_t* from, xce_reply_t* reply, uint64_t base, uint64_t size) (the synchronous paths build reply with init_reply_by_req; the async paths build it from actx->seq/actx->opcode and pass actx->ep, actx->c->myhost, as jobsubmit_reply_*_once do today, job.c:2234-2288) takes s_inflight_submits_lock, calls xce_reply (always non-blocking, so allowed under that lock by ha-async-io.md), unlinks the entry, and unlocks. Every path after registration uses it -- the new table-hit path, the existing payload-build and allocation-failure early-outs (today inflight_range_remove then xce_reply), commit failure or timeout and apply success (today jobsubmit_reply_*_once then jobsubmit_async_ctx_release). It is the only place an entry is removed: jobsubmit_async_ctx_release stops calling inflight_range_remove (job.c:2224) and only frees the async context, because a removal by exact (base, size) after the reply could delete a retry of the same submission that registered in between (7.7 allows that retry). On the async paths, jobsubmit_reply_*_once calls inflight_reply_and_remove inside its one-shot guard, so reply and removal happen together exactly once; a suppressed second reply removes nothing because the first already did. So another submission's collision check (which takes the same lock) sees an entry exactly while its submission is unanswered, as the Terms' "array submission still being accepted" requires: neither a just-answered submission blocking a later one (7.7) nor an unanswered one failing to block it (7.10, 7.11).
  • Array-ID check: for a tag-1 submission, one scan takes lock(c->jobmgr, RTTI_LOCK_READ) once and looks up keys MAKE_JOBID(.., TAG_JOB_ARRAY, A|k) for k = 0..32767. That one scan answers both the element-range check (keys inside [first, first + size)) and the array-identity check (any key), replacing jobid_range_exists for tag-1 submissions; tag-0 submissions keep jobid_range_exists unchanged. It is bounded CPU work (32768 hash lookups under one read-lock hold, no I/O, no blocking), so it is compatible with the vtask handler (ha-async-io.md, audit dimension 1). An array-ID index maintained at apply and snapshot load was considered and rejected: O(1), but it adds a derived structure that every apply, replay and snapshot-load seam must keep complete (dimension 7), for a cost paid only by array submissions.
  • No exits: no exit or abort path is added (dimension 2). Errors become reply statuses (dimension 5).

Contracts: Service [x] / State [x]

typedef enum { JOBSUBMIT_OK, JOBSUBMIT_INDEX0, JOBSUBMIT_INVALID } jobsubmit_class_t;

/* 7.1 first; then 7.4, 7.5, 7.3, 7.2; pure, no locks. size is the wire
 * UInt64 (controller.capnp arraySize). 7.2 is evaluated without addition,
 * as size > 32768 - idx (idx >= 1 here), so neither 2^32+2 nor UINT64_MAX
 * can wrap past it. */
jobsubmit_class_t jobsubmit_classify(uint64_t first_jobid, uint64_t size);

/* One read-locked scan of the 32768 element keys of array_id's block.
 * Returns 1 if any of them exists (array identity taken -- this covers the
 * element range [first, first+size) too), 0 otherwise. */
int array_identity_exists(cluster_t* c, uint64_t array_id);

/* registry entry: { base_jobid, array_size, is_array, array_id }. Overlap if
 * element ranges intersect, or both entries have is_array and equal array_id. */

  • Replies:

    Class Reply status Log
    INDEX0 -E_XCE_ARRAY_START0 ERROR naming the job; says array element numbers start at 1 and the submitting client must be upgraded
    INVALID -EINVAL —
    Collision -EEXIST (unchanged) —
    Success reply jobid = first element ID (unchanged) —

  • State: the registry keeps its mutex. Entries gain one field. Removal is by exact (base, size), now only inside inflight_reply_and_remove.

  • Invariant cross-check:

    Dimension Outcome
    1: async I/O Pass: bounded CPU only
    2: lifecycle Pass
    4: commit/apply Pass: unchanged commit path; registry ordering tightened
    5: errors Pass: every rejection replies
    7: field-complete apply Not affected: no WAL field change
    9: redirect Not affected: no new primary-only operation

Scheduler

SchedGuard

  • Expansion guard: for XCE_SCHED_OP_UPDATE_JOB_ARRAY, if !is_array_tag(first) or size > 32768 - array_idx(first) (no addition, so a UInt64 size cannot wrap), log ERROR (naming array_jobid(first) and the size) and expand nothing. The controller never sends such an array, so this guard is defensive only.
  • INFO log: prints array_jobid(first) as array_jobid=%s (5.9).
  • Scheduler invariants: expansion stays in driver.c translation before core events, so the recording and the simulator see expanded jobs and driver state-equivalence (scheduler dimension 1) holds. Dimensions 2-4 are not touched.

Worker

WorkerEnv

  • System value: XCE_JOBINDEX is array_idx(jobid) when is_array_tag(jobid), and 0 otherwise (6.1, 6.2, 7.8).
  • User value: in the user-variable loop, a variable whose name is exactly XCE_JOBINDEX is not applied, and XCECD_WORKER_LOG WARN names the job ID and the ignored variable (6.3, 6.4).
  • XCE_JOBID: handling is unchanged (6.5).
  • Unchanged structure: the function's other caller (the JSON-parse failure path) is unchanged.

Harness

HarnessJobsub, HarnessJobArg, HarnessExpectJob, HarnessAutovars

/* digits only (leading zeros ok); -EINVAL missing/non-digit/0, -ERANGE > 32767 */
int cmdutil_parse_array_size(const char* text, uint32_t* n);

/* Trims spaces/tabs, parse_jobsel(). RANGE -> refuse (4.8). Writes a message
 * into errbuf on rejection. On success *jobid is the exact ID (element ID for
 * [k]), *had_index says whether [k] was given. Returns 0 or -EINVAL. */
int cmdutil_job_arg(const char* arg, uint64_t* jobid, int* had_index,
                    char* errbuf, size_t errlen);

/* Prints "job <arg> not found" and, when !had_index, "array elements must be
 * named as <array-id>[k]" (4.7). */
void cmdutil_job_not_found(FILE* out, const char* arg, int had_index);

/* Values today's substitute_autovars() reads from test.c globals. */
typedef struct {
    const char* jobid;      /* NULL when unset */
    int         last_status;/* $? */
    int         jobcount;   /* $jobcount, -1 when unset */
} autovar_ctx_t;

/* $?, $jobcount, $jobid and (new) $jobid[...] token substitution (4.10,
 * 4.11); same token rule as today; '!' and '#' lines untouched. Returns 0,
 * -EINVAL for an unset $jobid / $jobcount with today's messages, or -ENOSPC
 * when out is too small (as today). The caller passes jobid = NULL when
 * g_last_jobid is empty. */
int autovar_substitute(const char* line, const autovar_ctx_t* ctx, char* out,
                       size_t outlen, char* errbuf, size_t errlen);
  • jobsub -a N:
    • N = 1 → idgen (ordinary).
    • N >= 2 → idgen_array; a failure prints the error and submits nothing (4.6).
    • The ack check compares against the first element ID.
    • Sequence seed: test.c:6896-6899 seeds idctx.sequence randomly in [0, 2^20). Any seed above 31 * 32768 would make the first idgen_array of that second return -EOVERFLOW (about 1 run in 32). The seed is bounded to [0, 30 * 32768), which leaves room for at least one whole block. (The apiserver's context starts at 0, clib.go:332.)
    • The success line and $jobid use fmt_jobid(array_jobid(first)) (4.5).
  • Single-job commands: every command listed in the requirements' Harness boundary note calls cmdutil_job_arg and stops on its error. The controller's not-found replies go through cmdutil_job_not_found. For jobinfo, each -j is judged on its own.
  • expect-job: parses its argument with cmdutil_job_arg. joblist_query matches jobs[i]->jobid == id. "Not found" and "refused" fail the assertion.

Apiserver

ApiSubmitExpr

type ArrayExprError struct {
    Rule string // "2.1" ... "2.8"
    Msg  string // user-facing, quotes the name where 2.2 requires it
}
func (e *ArrayExprError) Error() string

// Called only for names containing '[' or ']' (names without them keep the
// existing createArrayName path, 1.4).
// Checks in requirement order 2.1 .. 2.8 and returns the first failure (2.9).
func ParseSubmitIndexExpr(name string, interactive bool, limit int) (
    arrayName string, n int, isArray bool, err *ArrayExprError)
  • Grammar: exactly 2.2's. Numbers are read digit-wise and saturate at a sentinel above any limit instead of overflowing. Rules 2.5-2.8 are then evaluated in order on the saturated values, so name[99999999999999999999] reports 2.7, name[99999999999999999999-5] reports 2.6, and name[0-99999999999999999999] reports 2.5 (2.9). Leading zeros are accepted (1.5).
  • 2.3 / 2.4 / 2.5 / 2.6 / 2.7 / 2.8: 2.3 lists or steps; 2.4 %N; 2.5 first number 0; 2.6 last < first; 2.7 first > 1; 2.8 last > limit.
  • Contract with CreateJob: preCheckJobRequest becomes preCheckJobRequest(req) (n int, isArray bool, err error). CreateJob branches on isArray, not on a count: isArray == true → GenerateArrayJobID(c, a, n) (n may be 1, so name[1] is a one-element array, 1.2, 7.8); isArray == false → GenerateJobID(..., 1) as today. Every existing error return in preCheckJobRequest (jobs.go:1546, :1553, :1558, :1565, :1571, :1597, :1735, ...) becomes 0, false, err; the plain-name success path (the named-job early return, :1578) and the unnamed-job success path (the final return, :1746) return 1, false, nil. On success for a bracketed name, preCheckJobRequest sets req.JobName = arrayName (as today at jobs.go:1574, so the stored name is arr, not arr[1-3], and -J 'arr[2]' matches) and returns (n, true, nil) at the same point as today's named-job early return (jobs.go:1578), so the checks it skips are the same as today. The old sizing rule (arraySize := 1; if len(parsedIndices) > 1, jobs.go:1419-1423) is removed.
  • Names without [ or ] (1.4): keep today's path unchanged -- createArrayName stays and is called only for such names, so isValidArrayName on the whole name, the slot-limit errors for a % suffix (jobs.go:1840-1851), the "job array list format is invalid: " prefix and their 500 status are exactly as before (they are not *ArrayExprError). ParseSubmitIndexExpr is called only for names containing [ or ].
  • Caller: preCheckJobRequest (jobs.go:1537, no gin context) calls ParseSubmitIndexExpr first for a bracketed name, before the existing early return for named jobs, so the interactive check (2.1) is reached. It passes interactive = (req.Interactive != 0) and limit = utils.JobYamlConfig.MaxArraySize (the value resolveMaxArraySize produced at startup; today read at jobs.go:1901) (xcecli sends 1, 2 or 3, submit.go:245-249). When the returned *ArrayExprError is non-nil it returns 0, false, ae unwrapped (no "job array list format is invalid: " prefix, jobs.go:1565, 1571); a nil *ArrayExprError is never converted into a non-nil error (typed-nil).
  • HTTP mapping: CreateJob (jobs.go:1394-1398, today ReturnError(ctx, common.ERROR_CODE, ...) = 500 for every preCheck error) first checks var ae *ArrayExprError; if errors.As(err, &ae) { ReturnError(ctx, common.INVALID_PARAMS_CODE, ae.Msg, nil); return }, then falls through to the existing 500 path for other errors.

ApiCreateJob

Method Endpoint Request Response Errors
POST /xce/v1/jobs JobSubmitRequest (unchanged) JobSubmitResponse{jobid, queue_name, message, array_size} 400 (2.x), 500 (ID allocation/submit)
  • Test seams: controllers/jobs.go adds package variables in the existing validateResreqSyntax hook pattern -- var generateJobID = backend.GenerateJobID, var generateArrayJobID = backend.GenerateArrayJobID, var submitJob = services.SubmitJob, var controlJob = backend.ControlJob -- and CreateJob and both control dispatch sites (ControlJobByID, jobs.go:2611; ControlJob, :2847) call them, so non-CGO handler tests can replace them and observe dispatch.
  • Arrays: GenerateArrayJobID(c, a, n) returns the first element.
    • The model has Unique_JobID = first, JobID = the array ID, ArrayIdx = 1.
    • primaryJobID (used by processJobRequest and postJobRequest for the submit-time %J/%I substitution) stays the first element ID, so a new array's file names get %I = 1 and %J = <array-id>[1], as 9.4 documents.
    • It submits with SubmitJob(&job, n) (unchanged).
    • The response jobid is the array ID's decimal, array_size is n, and message is Job <array-id-decimal> is submitted to queue <q> (n elements).
  • Ordinary jobs: unchanged, with array_size omitted (3.3).

ApiConfig

  • job.max_array_size absent or 0 → 1000 (1.8).
  • Negative → startup error (1.9).
  • 1..32767 → used as is (1.6).
  • Above 32767 → 32767, plus utils.Warn("job.max_array_size above supported maximum", zap.Int("configured", v), zap.Int("effective", 32767)) (1.7). The existing > 2147483646 startup rejection (utils/config.go:221) is deleted; only a negative value is an error.

ApiSel and ApiMatch

type SelForm int
const ( SelBare SelForm = iota; SelSingle; SelRange )

type Selection struct {
    Raw    string  // as the user wrote it (trimmed)
    Name   string  // name selections only (job_name / -J)
    Key    uint64  // identity selections: exact ID (bare) or array ID
    Form   SelForm
    Lo, Hi int     // element numbers, Lo == Hi for SelSingle
}

type SelError struct {
    Rule string // "5.16", "5.14", "5.17", "5.8", "5.7", "5.5"
    Msg  string // names the selection as written
}

// Identity: written c-a-ts-seq (strict widths) or decimal uint64 (ParseUint, base 10).
// Ranges are always allowed at the server.
// An identity range in a query is evaluated by its predicate (today it was
// accepted and silently dropped, bug-1898); only xcecli's own ParseSelection
// refuses query ranges, after 5.16, 5.14, 5.8 and 5.7 pass.
func ParseIdentitySelection(raw string) (Selection, *SelError)
// Name: 5.16 syntax, 5.7 and 5.5 only; no array-name rule check. The name
// part is zero or more characters, none of them whitespace, '[' or ']'. An
// empty name part is not a 5.16 violation: with contains/startwith/endwith it
// matches every name (today's semantics of an empty pattern); with eq/in/notin
// the name predicate requires a non-empty name, so it selects no job (eq/in)
// -- jobs submitted without a name (e.g. by the harness) are stored with
// job_name "" (backend/convert.go goString) and must not be matched by
// -J '[1]'.
func ParseNameSelection(raw string) (Selection, *SelError)
  • Triggers: any value containing [ or ] goes through these parsers before the existing plain-match fallback (5.16). Error messages pass through to ReturnError unchanged, so they are no longer replaced by "bad job array index: " (5.5).
  • Callers: GetJobsInfo (job_id and job_name), GetJobByIDs, ControlJobByID, FetchJobList (and through it filter-based ControlJob).
  • Validate-all-first: each of these handlers parses and validates every selection and key in the request before any store query or control dispatch. In GetJobsInfo this is a separate pass over all filter entries -- ParseIdentitySelection / ParseNameSelection on every value that needs it, plus the operator checks above -- run before the existing filter loop, which can query the store (exec_hosts resolution via FetchJobIdList, jobs.go:465-480) and return early with count: 0. Filter-based ControlJob validates req.Filter.JobName with ParseNameSelection itself -- only when it contains [ or ], since unnamed jobs carry their command (with spaces) as their name (jobs.go:1582-1584) and kill -J 'sleep 100' must keep working -- before calling FetchJobList, so a rejection returns 400 with the specific message (today every FetchJobList error is mapped to 500 ERROR_QUERY_JOBINFO_MSG, jobs.go:2734-2738). The first rejection (in request order) is returned with ReturnError, and the command acts on no job and reports no "not found" results (Terms).
  • Matching (ApiMatch): selections become row predicates passed in QueryOptions.Predicates, evaluated by store.ExecuteQuery before sorting and pagination (bug-1898). The bracketed job_id/job_name values no longer produce CompFilterConditions.
    • Shape: Predicates []func(OrderMap) bool, one composed predicate per filter field. Within one field (one job_id or job_name filter), the values' predicates are OR'ed; across fields, and with the plain Conditions, they are AND'ed, matching today's per-field filter semantics.
    • Whole-filter conversion: when any value of a filter entry has [ or ], every value of that entry becomes a predicate (a bare value becomes a SelBare predicate) and that entry, and only that entry, is removed from Conditions. Other entries on the same field with another operator (utils.ParseFilters allows them, utils/utils.go:230-241) stay in Conditions and are AND'ed. Today's removal is keyed by field name and drops them too (jobs.go:494-512, :3420-3436). So job_id:in:(100,X[1]) matches job 100 or element X[1].
    • Bare identity keys, one rule on both paths: every value of a job_id filter entry whose operator is an identity operator (eq, in, notin, ne), and every value of a converted bracketed entry, goes through ParseIdentitySelection. Values of other operators (between bounds, lt/lte/gt/gte operands, string operators) are numbers or text, not identities, and stay on the plain path exactly as today when they carry no brackets. A written-form bare key (c-a-ts-seq) is normalised to its decimal value both on the plain job_id in (...) path and inside a converted filter, so the same key matches the same job whether or not a bracketed value shares its filter (5.6).
    • Operators: the predicate honours the filter's operator instead of dropping it (bug-1898). It adds exactly one new rejection, for operator/value combinations that have no meaning with an element selection and were silently dropped until now: bracketed job_id values with comparison or null operators and bracketed job_name values with null operators (last bullets). They reuse today's "operator doesn't support" error:
      • eq and in (both fields), and between on job_id: the row matches any of the values' selections. (job_id:between with bracketed values had no defined meaning and was dropped; it is evaluated like in. job_name:between is not in the string table and stays rejected, below.)
      • notin: the row matches none of them.
      • job_name with contains, notcontains, startwith or endwith: the name part is evaluated with today's condition semantics -- built with utils.BuildFilterConditions for that operator and evaluated with store.MatchesCondition (today contains/startwith/endwith all map to a case-insensitive substring OpLIKE, utils/types.go:143-148), so a bracketed and a plain value with the same operator match names identically -- and the row must also be an array element within the range (e.g. REST job_name:startwith:foo[1] selects element 1 of every array whose name matches foo under that operator). For notcontains the negation covers the name part only: the row is an array element within the range whose name does not contain the value. (notin, by contrast, negates whole selections, because its values are complete selections rather than one name pattern.) job_name:between stays rejected with today's "unsupported operator" error, as for plain values. xcecli's list path sends these operators for a leading and/or trailing * (generateSearchRule, filters/job_filter.go:191-205); for a bracketed -J value it checks the raw text first (CliSel ParseNameSelection, so * can only appear in the name part) and then takes the rule from, and strips * from, the name part only.
      • ne: rejected with today's "operator doesn't support" error (jobs.go:254-261 for job_name, :357-364 for job_id), unchanged.
      • Any operator not in the field type's existing mapping table (IntOperatorMappingTable for job_id, StringOperatorMappingTable for job_name) gets that table's existing "unsupported operator" error, as a plain value does today.
      • job_id with lt, lte, gt, gte, null, notnull (in the int table, utils/types.go:117-128), and job_name with null or notnull (in the string table, but they ignore the value, utils/types.go:382-388), with a bracketed value: rejected with the existing "operator doesn't support" error -- the error class ne already gets today, not a new rejection category. These combinations had no meaning and were silently dropped (bug-1898); xcecli never sends them.
      • Control by name (getFilters, jobs.go:2965-2970) always sends eq with the raw value, so kill -J 'foo*[1]' matches the literal name foo*. Wildcards are a list-only feature today, with or without brackets; that difference is pre-existing and out of scope.
    • SelBare → job_id == Key (all elements and any ordinary job; 5.6).
    • SelSingle / SelRange → job_id == Key and IsArrayJob(unique_job_id) and array_idx in [Lo, Hi] (5.2, 5.11, 8.2).
    • Name selections → job_name == Name and array_idx in [Lo, Hi] with IsArrayJob (5.2). For the exact-match operators (eq, in, notin) an empty Name matches no row, even though unnamed jobs are stored with job_name == "" (backend/convert.go:28-31); so kill -J '[1]' (control always sends eq) acts on no job.
    • No special case for element 0: <array-id>[0] / [0-0] get the identity predicate with Lo = Hi = 0 (8.2, 5.11); -J 'name[0]' and <array-id>[0-3] (out of scope in the requirements) get the general name or range predicate, which matches array elements only.
  • 5.17: a decimal key with tag 1 and no [...] is rejected with "array elements must be named as <array-id>[k]". The written form never carries the tag.
  • Pre-store key checks: GetJobByIDs (every key) and the GetJobsInfo job_id branch (values of identity operators, and every value of a converted bracketed entry including job_id:between, see Matching), including the job_id in (...) path, apply 5.14 and 5.17 to every such key, and 5.8 only to keys written with [...], before querying the store. A bare ordinary job ID is therefore never rejected for not being array-capable (5.6).

ApiControl

  • Dispatch once (bug-1899): ControlJobByID today dispatches the accumulated successJobs inside its per-selection loop (jobs.go:2464, :2575-2617), so earlier selections are re-sent for every later one. After validate-all-first and matching, the selection loop collects (a) the jobs to act on, de-duplicated by unique_job_id, and (b) the not-found maps, one per missing number per selection, in selection order (not de-duplicated; 5.3, 5.4). It does not dispatch inside the loop. After the loop, for each job in (a), in order, it calls validateJobControl(job, action, signal) once (signature unchanged, jobs.go:3015; it checks one row) and appends its accepted and failed items; it then appends every not-found map from (b). It then dispatches each accepted job once through controlJob; a dispatch error still returns 500 as today (jobs.go:2611-2616), it is not turned into a failed item.
  • foundJob: becomes foundJob(jobID int64, idx int, explicitIndex bool, rows []utils.OrderMap) ([]utils.OrderMap, error) (today: foundJob(int64, int, rows) ([]utils.OrderMap, error)). When explicitIndex is true, only rows whose unique_job_id is an array element (IsArrayJob) with array_idx == idx match (5.4, 8.2). When it is false, today's whole-job_id match applies (5.6).
  • Not-found items: a missing number in a SINGLE or RANGE selection yields one "job not found" map, whose unique_job_id is the would-be element ID (5.3, 5.4). Accepted and failed result maps already carry unique_job_id (jobs.go:2575-2588, :2813-2826).
  • Result items: every result map therefore carries unique_job_id = the element's own (or would-be) ID, so clients render <array-id>[k], including [0] (5.1, 8.3).
  • Most-recent rule (5.10): applies only when the control filter contains no element number ([...]) and the all-jobs option is not set. With an element number, the command acts on exactly the selected elements (5.2), whatever the newest row is. Without one, when the last matching row has array_idx == 0, or is the only stored element of its array, act on it alone.
  • isSingleElementArray(row): counts the stored rows with that job_id that are array elements. The API server's store records no array size, and the controller pushes a one-element array as a single job, so the stored row count is the only available signal. Deliberate approximation: a multi-element array whose other elements have all been purged from the store is treated as one-element. The multi-element rule is out of scope (requirements), so this only affects arrays already reduced to one stored element.

ApiDispatchLog

  • dispatchJobArray replaces its zap.Uint64("base_jobid", baseJobID) field (handler.go:273) with zap.String("array_jobid", FormatJobID(int64(ExtractArrayBaseID(baseJobID)))) (5.9).
  • CreateJob's two log entries (jobs.go:1434 "Generated job ID" and :1515 "Job submission successfully"), for an array, log array_jobid (formatted) and array_size instead of the raw jobID; ordinary jobs log as today (5.9).

xcecli

CliSel and CliSplit

// Same grammar and order as ApiSel for identities (5.16, 5.14, 5.8, 5.7, 5.5).
// 5.8 and 5.7 apply to decimal keys with [...] too (tag 0, low 15 bits clear),
// so an upgraded client is protected against an un-upgraded server. Only 5.17
// (a marked decimal key without [k]) is left to the server.
func ParseSelection(raw string, allowRange bool) (Selection, error)
// Splits at commas outside [...] (as extractJobNames does).
func SplitSelections(list string) []string
// 5.16 / 5.7 / 5.5 for a raw -J value that contains '[' or ']', run before the
// wildcard mapping and '*' removal: the name part is zero or more characters,
// none whitespace, '[' or ']' ('*' allowed there only); after it exactly one
// [num] or [num-num] and nothing else. So -J 'foo[1]*' and -J 'foo[1*]' are
// rejected (5.16). A name part made only of '*' is accepted: -J '*[1]' is
// sent as job_name:contains:[1] (empty name part), i.e. element 1 of every
// array, as -J '*' already means every name today. For an accepted bracketed value the list path picks the
// search rule (generateSearchRule) from the name part alone and strips '*'
// from the name part only: -J 'foo*[1]' is sent as job_name:startwith:foo[1],
// -J '*foo*[1]' as job_name:contains:foo[1]. Values without brackets keep
// today's mapping. Used by list and control -J (control sends eq as today).
func ParseNameSelection(raw string) (Selection, error)
// "<decimal>" or "<decimal>[k]" or "<decimal>[a-b]" for the REST path.
func (s Selection) APIKey() string
// allowRange=false refuses a range ("a range of elements is accepted only by
// control commands") only after 5.16, 5.14, 5.8 and 5.7 have passed, so a
// query range breaking one of those reports that rule (5.12).
  • ToAPIJobID(s, allowRange): becomes ParseSelection(s, allowRange).APIKey(). Control passes true, queries pass false. Text that is neither a written ID nor a decimal key is rejected rather than passed through.
  • Callers: controlJobsByIDs and filters/job_filter.go both validate every selection before any request. The error message quotes the trimmed selection.

CliSubmit and CliControl

  • Submit output: xcecli prints the server's message after replacing <jobid-decimal> with <FormatJobID(jobid)>, as today. The count comes from the server message's (n elements) (3.2); the client does not append it again. array_size is mirrored in the client model for scripted use.
  • Control result lines: FormatJobID(UniqueJobID) when it is non-zero, otherwise today's FormatArrayChildID (compatibility with an older server).
  • Help (9.1): name[1-N], name[1], numbering starts at 1, the limit is set by the API server (default 1000, at most 32767).
  • Help (9.2): kill, stop, resume and checkpoint show <array-id>[k] and <array-id>[a-b].

Data Models

Domain Model

  • Job ID (uint64) = tag (2) | cluster (6) | API server (4) | timestamp (32) | sequence (20).
  • Array ID = tag 0, with the low 15 sequence bits 0.
  • Element ID = tag 1 | array ID | k, with 1 <= k <= 32767 for new arrays and k = 0 only for arrays submitted before this change.
  • Invariants:
    • An array ID never equals an element ID.
    • An array occupies element IDs first .. first + n - 1 within one 32768 block.
    • The controller accepts tag 0 (size 1) and tag 1 only.

Data Contracts & Integration

  • JobSubmitResponse (REST): jobid (int64, the array ID for arrays), queue_name, message, array_size (int, omitted for ordinary jobs).
  • JobControlResponseItem (REST): adds unique_job_id (int64, omitempty).
  • Controller OPCode_jobsub: wire format unchanged. New possible reply status -E_XCE_ARRAY_START0.
  • Persistence: no field is added to, or removed from, any WAL record, snapshot or recording. The save/restore completeness review does not apply: the per-field round trip is untouched, and 1.10 and 8.4 rely on the existing JOB_ARRAY_SUBMIT apply, which stores (first, size, submit) and expands first + i.

Error Handling

Error Strategy

  • C: negative errno or -E_XCE_* reply statuses. Harness messages are written to stderr, and the command returns non-zero. Worker WARN goes through XCECD_WORKER_LOG.
  • Go apiserver: ReturnError(ctx, common.INVALID_PARAMS_CODE, msg, nil); return for 2.x, 5.x and 4xx-class rejections. ID allocation or submit failures use common.ERROR_CODE as today. Each rejection is logged at Debug with zap.String("selection", raw). Startup clamping is logged at Warn.
  • xcecli: a client-side rejection exits non-zero before any HTTP request.

Error Categories and Responses

Condition Surface Response
2.1-2.8 REST/CLI submit 400 with the specific message
5.5, 5.7, 5.8, 5.14, 5.16, 5.17 REST/CLI query or control 400; whole command rejected; no job acted on
5.3, 5.4 REST/CLI control 200 with per-number "job not found" items
7.1 controller -E_XCE_ARRAY_START0 + controller ERROR log
7.2-7.5 controller -EINVAL
7.10 controller -EEXIST
4.3, 4.4, 4.6, 4.8, 4.11, 5.7, 5.8, 5.14, 5.15 harness stderr message; non-zero; nothing submitted or acted on
identity range in a query (client-side, as today) xcecli list non-zero exit before any HTTP request
bracketed value with ne, a comparison or null operator, or an operator outside the field's mapping table REST/CLI query or control 400 with the existing "operator doesn't support" / "unsupported operator" message; whole command rejected

Monitoring

  • Controller: one ERROR line per 7.1 rejection.
  • Scheduler: an ERROR on a guarded expansion. Expected never; it would indicate a broken controller.
  • API server: a startup WARN when the limit is clamped.

Testing Strategy

Shared conformance table selection-cases

One table of (input, expected outcome) rows is copied verbatim into test_jobsel.c, controllers/jobsel_test.go and common/format/jobsel_test.go. Each implementation runs the rows its surface supports.

Group Cases Expected outcome
Valid X, X[1], X[0], X[01], X[1-3], X[3-3], X[32767] accepted
Syntax (5.15/5.16) X [1], X[a], X[], X[1, X], X[+1], X[-1], X[ 1], X[1-3:2], X[1]x, X[1-2-3] rejected
Widths (5.14) cluster 64, API server 16, timestamp 2^32, sequence 2^20 (e.g. 0-0-T-1048576[1]) rejected
Alignment (5.8) c-a-ts-5[3] rejected
Index (5.7) X[32768], X[1-40000], X[99999999999999999999] rejected
Reversed (5.5) X[3-1] rejected
Names (5.16; name surfaces only) foo[1]; [1] (empty name: eq/in match no job, including unnamed jobs stored with job_name ""; contains matches every array); xcecli raw -J 'foo*[1]' (list sends job_name:startwith:foo[1]) and -J '*[1]' (list sends job_name:contains:[1]) accepted
Names (5.16) foo [1], my job[1], foo[1]x; xcecli raw -J 'foo[1]*', -J 'foo[1*]' rejected
Precedence (5.12) 0-0-T-1048576[40000] → 5.14, not 5.7; c-a-ts-5[3-1] → 5.8 first failing check reported

Unit Tests

  • libxce:
    • test_jobsel.c runs the table.
    • parse_jobid NULL safety and range rejection.
    • test_idgen.c: idgen_array from a seed in (31 * 32768, 2^20) returns -EOVERFLOW with the context unchanged; a seed of exactly 31 * 32768 and seeds below 30 * 32768 succeed. Because idgen resets the sequence when the second changes (idgen.c:36-41), each seeded case sets ctx->timestamp = time(NULL) immediately before the call and retries if the second changed in between; idgen_array returns index 1; the last element has index n; n = 32767 is accepted and 0 or 32768 is rejected; block overflow returns -EOVERFLOW and leaves the context unchanged; idgen with size 2 returns -EINVAL.
  • Controller (test_jobsubmit_validate.c):
    • handle_jobsubmit with an index-0 tagged array and an empty queue is answered -E_XCE_ARRAY_START0 and writes the 7.1 ERROR line (classification precedes the field checks, 7.9).
    • jobsubmit_classify over all 7.1-7.5 combinations, including the 7.9 precedence, a size of 2^32+2 and idx = 2, size = UINT64_MAX (both rejected under 7.2, neither truncated nor wrapped).
    • handle_jobsubmit with an existing array ID (an existing element X|3 blocks a new X|5), giving 7.10.
    • Two registered concurrent submissions with the same array ID; the second gets -EEXIST (7.11).
    • Reply and release are atomic. Seam (project convention for test-only symbols, testing.md): under #ifdef XCE_TESTING, job.c exposes void (*jobsubmit_test_reply_hook)(void) -- called inside inflight_reply_and_remove after xce_reply and before the unlink, with the registry lock held -- plus int jobsubmit_inflight_overlaps_for_test(uint64_t base, uint64_t size, int is_array, uint64_t array_id) (takes the lock), jobsubmit_inflight_overlaps_locked_for_test(...) (same check, caller already holds the lock) and int jobsubmit_inflight_trylock_for_test(void) (returns pthread_mutex_trylock's result, unlocking on success); declared under #ifdef XCE_TESTING as cluster.h:110-118 does for other test-only symbols. (There is no xce_reply test hook in src/lib, test_commit_barrier_reply.c:27, and replies are read from ep->outq only after the handler returns, so neither order is observable without this seam.)
      • Deterministic checks inside the hook (no second thread, since a thread would just block on the lock and pass even against a split implementation): (1) jobsubmit_inflight_trylock_for_test() returns EBUSY (the registry lock is held across the reply); (2) A's reply is already queued on the endpoint's outq; (3) jobsubmit_inflight_overlaps_locked_for_test() (lock-free variant) still finds A's entry. After inflight_reply_and_remove returns, jobsubmit_inflight_overlaps_for_test() finds no entry.
      • A retry of the same (base, size) registered after A's reply is not removed by A's async-context release (jobsubmit_async_ctx_release no longer touches the registry).
      • Once A has been answered with a failure, a same-array-ID submission B is accepted (7.7); while A is registered and unanswered, B gets -EEXIST (7.11).
    • A size-0 submission is answered -EINVAL (7.4).
  • Apiserver:
    • jobs_handler_test.go (new, //go:build !cgo): every handler-level case (CreateJob, GetJobsInfo, GetJobByIDs, ControlJobByID, filter-based ControlJob). It is !cgo because the JWT/RBAC fixtures signTestJWT/setupRBACTest live in the !cgo-only controllers/integration_test.go:54, :83. It replaces the seams generateJobID, generateArrayJobID, submitJob and controlJob (the last records each dispatched unique_job_id, so "acts on" is observable), seeds store.Jobs (with a controllable job_status, since validateJobControl checks it, jobs.go:3045) and store.Queues for every seeded job's queue (the role's allowed queues come from store.GetAllQueueNames(), otherwise control returns 403 before dispatch; pattern rbac_convention_test.go:86), and sets utils.JobYamlConfig = &utils.JobConfig{MaxArraySize: 1000}, restoring every replaced global and store in t.Cleanup. Order matters: setupRBACTest calls resetIntegrationState() → store.InitStores() (integration_test.go:33-36, :83-86), which replaces store.Jobs and store.Queues with empty stores; so each test saves the original store pointers, calls setupRBACTest, then seeds, and restores the saved pointers in t.Cleanup. Every control test also asserts at least one expected dispatched unique_job_id (or, for the act-on-no-job cases, that the fixture rows are present and nothing was dispatched), so an empty store cannot make it pass.
      • Submit: arr[1-3] leaves req.JobName == "arr"; one 2.x row returns 400 with the exact message; a valid ordinary submit is not rejected (typed-nil guard); name[1] goes through generateArrayJobID with n = 1 and the response has array_size: 1; plain names my job, foo%5 and foo% get today's errors and 500 status; submit-time %I = 1 and %J = <array-id>[1] for a new array (9.4).
      • Query: GetJobsInfo?filter=job_id:eq:X[2] returns only element 2 (bug-1898); GET /jobs/X[1-3] returns elements 1-3; element-only matching for <ordinary-id>[0] (5.11); with an unnamed (job_name "") array element in the store, job_name:eq:[1] lists nothing while job_name:contains:[1] lists it; an exec_hosts filter matching no host together with job_id:eq:X[40000] returns 400 (5.7), not count: 0; job_name:notnull:foo[1] gets the existing "operator doesn't support" error; job_name:notcontains:foo[1] selects element 1 of arrays whose name does not contain foo; {job_id:in:(X[1])},{job_id:ne:100} keeps the ne entry; job_id:in:(X[1]);job_name:eq:foo[2] is an intersection; job_id:in:(100,X[1]) matches both; job_id:notin:(X[1]) excludes element X[1]; job_name:startwith:foo[1] selects element 1 of arrays named foo...; job_id:ne:X[1] keeps today's rejection; the same written-form bare key matches alone and mixed with a bracketed value.
      • Control: two selections in one request (X[1],Y[2]) dispatch each element exactly once and list each once (bug-1899); X[1-3] dispatches to the existing elements and reports missing numbers as not-found items carrying unique_job_id (5.3); X[0] with no element 0 dispatches nothing and reports it not found (5.4); a pre-upgrade element 0 is dispatched alone (8.2); a request with a valid selection followed by an invalid one dispatches nothing (Terms); filter-based -J 'foo[40000]' returns 400 with the 5.7 message; filter-based -J '[1]' with an unnamed array in the store dispatches nothing; the most-recent rule with a one-element array dispatches that element alone (5.10); -J 'foo[1]' with two one-element arrays named foo dispatches both (5.2).
    • jobs_test.go (untagged, pure functions only): foundJob with explicitIndex ([0], 5.4, 8.2); the predicate builder for each operator rule in ApiMatch, evaluated over store.Jobs-shaped rows; isSingleElementArray.
  • xcecli:
    • jobsel_test.go: the table plus SplitSelections("X[1,3],Y").
    • jobid_test.go: passthrough cases "123456" and "32768[3]"; "123456[3]" rejected under 5.8; ToAPIJobID("32768[1-3]", true) passes through while ToAPIJobID("32768[1-3]", false) is refused.
    • cmd/jobs/controller_test.go (new): controlJobsByIDs with request.NewAPIClient(srv.URL) pointed at an httptest server; X[1-3] reaches the request path as <decimal>[1-3] (guards allowRange=true at controller.go:52).
  • Harness:
    • test_cmdutil.c: cmdutil_parse_array_size (1, 01, 0, -3, +5, 5, 5.0, 0x5, abc, 32767, 32768) and cmdutil_job_arg (ranges refused including X[3-1] with the 4.8 message, [0] accepted).
    • test_autovars.c: $jobid[3], $jobid[3]x, unset $jobid, $?, $jobcount (set and unset), and ! / # lines.
  • Worker (test_exdata_jobindex.c): element index, ordinary 0, a user XCE_JOBINDEX ignored with a WARN, XCE_JOBID unchanged.
  • Scheduler (test_driver_translate.c): tagged block|1 arrays expand to indices 1..n; untagged arrays, overflowing arrays and idx = 2, size = UINT64_MAX are not expanded.

Integration Tests

  • Controller suite: existing commit-barrier tests pass with tagged first IDs (1.10 and 8.4 rely on the existing WAL replay tests, which still pass).
  • Apiserver: CGO --suite apiserver submit through GenerateArrayJobID (real libxce), asserting the response jobid equals array_jobid(first) and array_size.
  • Harness pty (test_test_cli.c): jobsub -a 0 and jobsub -a 40000 are refused; jobkill -j 1-1-1-0[1-3] is refused with the 4.8 message before any RPC.

E2E Tests

  • /xce-test-multihost: jobsub -a 2 elements [1] and [2] reach DONE on both hosts.
  • /xce-test-ha: a 6-round soak verifies elements 1..N survive failover (1.10).
  • Manual REST flow (recorded in validation):
    • xcecli jobs submit -J 'arr[1-3]' lists [1]..[3].
    • xcecli jobs kill X[2] acts on element 2 only.
    • A job script prints XCE_JOBINDEX.

Migration Strategy

  • Recommended order for clusters that use arrays:
    1. Upgrade the API servers and the harness. An un-upgraded controller accepts their array-id | 1 submissions unchanged (it does not validate tags), and those arrays are numbered from 1 (Adjacent expectations).
    2. Upgrade the controllers.
    3. Upgrade the workers, schedulers and xcecli in any order. The scheduler's expansion guard accepts arrays from un-upgraded clients (tagged, starting at index 0).
  • Controllers first is also supported: until the API servers and harness are upgraded, their multi-element array submissions are rejected with E_XCE_ARRAY_START0, and the controller log explains why (7.1). Ordinary jobs are unaffected. Operators choosing this order accept that rejection window.
  • No data migration: pre-upgrade arrays keep element 0 (8.1-8.4).