lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(job-array-one-based-index): requirements, design and tasks; log bug-1898, bug-1899

Browse source 
Spec for numbering job-array elements from 1 (bug-1896): approved
requirements, a design reviewed over 20 rounds until clean (log in
research.md), and a 30-task plan. The design review found two live
apiserver defects, logged here: bracketed job_id/job_name filters are
never evaluated (bug-1898) and ControlJobByID re-dispatches earlier
selections (bug-1899).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-01 08:54:06 +00:00
parent 8f5ba635c5
commit 590860f73d
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
8 changed files with 1725 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

857
.kiro/specs/job-array-one-based-index/design.md Normal file
View file

@ -0,0 +1,857 @@
# Design Document: job-array-one-based-index
## Overview
**Purpose**: This feature numbers job-array elements from 1, as LSF does. Elements are numbered 1 to N everywhere they are created, shown, selected, controlled and exposed to the running job. Every index form the feature does not support is rejected with a specific message instead of being silently renumbered (bug-1896).
**Users**: Users submitting arrays through `xcecli` or REST, operators controlling and listing elements, developers driving the `xce_test` harness, and job scripts reading `XCE_JOBINDEX`.
**Impact**: The element number already lives in the 15-bit index field of the 64-bit job ID. Under option B (requirements decision 1), an array's first element ID becomes `array-id | 1`. The existing contiguous expansion `first + i` in the controller, WAL apply and scheduler then yields elements 1 to N, with no change to the wire, WAL, snapshot or recording formats. The design also bounds every job-identity parser in the system, and adds cluster-side validation of array submissions.
### Goals
- `name[1-N]`, `name[1]` and harness `jobsub -a N` create elements 1 to N. Every other index form is rejected with the message its requirement names.
- One bounded identity and selection grammar, with one check order (5.12), per language: C (libxce), Go apiserver, Go xcecli.
- The controller refuses array submissions it cannot represent. It also detects array-identity collisions, including against concurrent submissions.
- A running element sees its own number in `XCE_JOBINDEX`, and the submission cannot override it.
- Arrays submitted before this change keep working, including their element 0.
### Non-Goals
- Starts other than 1 at the clients, comma lists, steps, `%N` limits (all rejected, not implemented).
- Per-element `%I`/`%J` expansion (blocked on bug-1508), and LSF `LSB_JOBINDEX`/`LSB_JOBID`.
- Whole-array control through the harness or at the controller; job output (peek) and restart by identity; the array statistics view.
- The `idgen` sequence overflow in general. The `preCheckJobRequest` early return for named jobs (unchanged; the new 2.1 interactive-array check runs inside `ParseSubmitIndexExpr`, before it, while today's interactive-value check at `jobs.go:1734-1736` stays where it is). All other defects listed as out of scope in the requirements.
## Boundary Commitments
### This Spec Owns
- **Job identity grammar and bounds:** the written form `c-a-ts-seq`, with an optional `[k]` or `[a-b]`, and the decimal form. The libxce, apiserver and xcecli parsers for them, and their check order.
- **Array ID allocation:** `idgen_array` and its Go wrapper. `idgen` reduced to ordinary IDs only.
- **Array-submission validation and collision detection** in the controller's `handle_jobsubmit`, including the in-flight registry's array-identity awareness.
- **The apiserver's handling of submit index expressions, the submit response shape (`jobid`, `array_size`), REST selection parsing and matching, the `unique_job_id` of not-found control results, and the array-size limit setting.**
- **`ControlJobByID`'s dispatch loop** (bug-1899: moved after the selection loop, de-duplicated).
- **Evaluation of selection predicates in the apiserver's in-memory query pipeline** (`store.ExecuteQuery`). Today bracketed selections are built as `CompConditions` that are never evaluated (bug-1898); this spec replaces them with evaluated row predicates.
- **xcecli** identity parsing, ID-list splitting, submit output and help text.
- **Harness:** `jobsub -a`, `$jobid` and `$jobid[...]`, the shared single-job argument helper, and `expect-job` matching.
- **The worker's `XCE_JOBINDEX`.**
- **Scheduler:** the array-expansion guard and its log line.
- **Skill scripts, `src/docs/xce.md`, and the cross-spec document updates** listed in the File Structure Plan.
### Out of Boundary
- **Storage, replication and apply:** WAL `JOB_ARRAY_SUBMIT` append, apply and replay; snapshots; HA replication and standby apply; controller-to-scheduler and controller-to-apiserver pushes; scheduler recording and replay. They are relied on unchanged.
- **Display helpers:** `fmt_jobid`, the Go `FormatJobID`s, and `array_jobid`/`array_idx` (display semantics unchanged).
- `RecordAcceptedJob`. (The apiserver's `array_idx between` construction and the unevaluated `CompConditions` are replaced on the selection paths this spec owns, see ApiMatch; other `CompConditions` users, if any appear, are untouched.)
- **Peek** (`output.go`, `parseJobIDParam`, the sixth `parseArrayName` caller; xcecli `peek.go`, which keeps a legacy `ToAPIJobIDLegacy` with today's passthrough behaviour) and **restart** (`RestartJobByID`), which keep `idgen` with size 1.
- **The scheduler REPL `info job` argument.**
### Allowed Dependencies
- **libxce** (`utils.c`, `idgen.c`, `errnox.h`, `config.h`) is the lowest layer. The controller, harness, worker and scheduler depend on it. The apiserver reaches it only through `backend/clib.go` cgo, plus pure-Go mirrors in `backend/convert_pure.go`.
- **The apiserver's selection parsers** (`ParseIdentitySelection`, `ParseNameSelection` in `controllers/jobsel.go`) depend only on `backend` ID helpers and the standard library. **Predicate building** (ApiMatch, in `controllers/jobs.go`) may additionally use `utils` (`BuildFilterConditions`, `OrderMap`, `QueryOptions`) and `store` (`MatchesCondition`). Neither `utils` nor `store` depends on controllers.
- **The xcecli `format` package** stays dependency-free; `cmd/` and `filters/` depend on it.
- **The harness helpers in `cmdutil.c` and `autovars.c`** depend on libxce only; `test.c` depends on them.
- **No new third-party dependencies.**
### Revalidation Triggers
- **Job ID bit layout** (`config.h`, the Go constants): any change invalidates all three parsers and `idgen_array`.
- **Controller submit reply semantics** (still the first element ID). `clib.go`, `RecordAcceptedJob` and the harness ack check depend on it.
- **REST response shapes:** `JobSubmitResponse` gains `array_size`; `jobid` now names the array. Not-found control result items gain `unique_job_id` (found items already carry it). Any REST consumer (xcecli, the regression suite) must re-check.
- **`utils.QueryOptions` shape:** gains `Predicates`; any other query builder must re-check.
- **The new controller error status `E_XCE_ARRAY_START0`**, which every client error printer must know about.
- **Specs that defined `$jobid` or element numbering:** `xce-test-e2e-primitives` and `job-array-single-wal-record` Requirement 5 are superseded here.
## Architecture
### Existing Architecture Analysis
- **Submission:** the client (apiserver or harness) allocates IDs with `idgen`. It sends `(first_jobid, array_size, submit)` through `OPCode_jobsub`. The controller checks for duplicates, commits one `JOB_ARRAY_SUBMIT` WAL record, and expands `first + i` on apply. It replies with `first_jobid`. The scheduler expands the pushed array in `driver.c` translation, before core events.
- **Identity text** is parsed in about ten places with differing leniency (research.md, "Discovery corrections").
- **Constraints kept:**
- The controller vtask handler never blocks (`ha-async-io.md`).
- Controllers never self-exit.
- REST handlers use `ReturnError(...); return`.
- The middleware chain is unchanged.
- The scheduler driver state-equivalence holds, because expansion stays before core events in `driver.c` only.
### Architecture Pattern & Boundary Map
```mermaid
graph TB
subgraph Clients
Xcecli[xcecli format jobsel]
Harness[harness cmdutil autovars]
end
subgraph ApiServer
ApiSel[controllers jobsel]
ApiSubmit[controllers jobarray submit]
Clib[backend clib idgen wrapper]
end
subgraph Libxce
Parse[parse jobsel]
Idgen[idgen array]
end
subgraph Daemons
Ctrl[controller handle jobsubmit]
Sched[scheduler driver expansion]
Worker[worker exdata env]
end
Xcecli --> ApiSel
Xcecli --> ApiSubmit
ApiSubmit --> Clib
Clib --> Idgen
Clib --> Ctrl
ApiSel --> Ctrl
Harness --> Parse
Harness --> Idgen
Harness --> Ctrl
Ctrl --> Sched
Sched --> Worker
```
**Architecture Integration**:
- **Selected pattern:** one bounded parser per language behind each surface, plus a single validation gate at the controller.
- **Domain boundaries:** identity text belongs to the parsers. Admission of arrays belongs to the controller. Element-number exposure belongs to the worker.
- **Existing patterns preserved:**
- Option B expansion.
- The `preCheck*` and `ReturnError` patterns.
- Hook variables for test seams (`validateResreqSyntax`).
- Unity C tests in `testsuite/` and table-driven Go tests.
- **New components and why they exist:**
- `parse_jobsel` and the two Go `jobsel` modules collapse ten lenient parsers into one grammar each.
- `idgen_array` exists because one-element arrays need a distinct request.
- `autovars.c` exists to make `$jobid[...]` substitution unit-testable.
- **Steering compliance:** TDD per component; errors are negative errno or `errnox` in C and `ReturnError` plus zap in Go.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| CLI | Go (xcecli module), Cobra | Identity parsing, submit output, help | No new dependencies |
| Backend services | Go (apiserver, gin) + cgo to libxce | Submit expression, REST selections, config | `GenerateArrayJobID` cgo wrapper plus non-cgo stub |
| Daemons | C gnu17 (controller, scheduler, worker), libxce | Validation, expansion guard, env | `strtoull` with errno replaces `atoi` |
| Test harness | C (`xce_test`) + Unity | `jobsub -a`, `$jobid`, element commands | New `autovars.c`; `cmdutil.c` extended |
| Data / Storage | WAL, snapshots | Unchanged | No format change (option B) |
## File Structure Plan
### New Files
```
src/lib/testsuite/test_jobsel.c # C conformance cases for parse_jobsel/parse_jobid (selection-cases table)
src/harness/autovars.c # autovar_substitute(): $jobid and $jobid[...] token substitution (moved out of test.c)
src/harness/include/autovars.h
src/harness/testsuite/test_autovars.c # 4.10, 4.11 unit tests
src/frontend/apiserver/controllers/jobsel.go # REST identity/name selection parser + key validator
src/frontend/apiserver/controllers/jobsel_test.go # selection-cases table + REST-only cases (5.17, decimal keys)
src/frontend/apiserver/controllers/jobarray.go # submit index-expression parser (2.1-2.9) for bracketed names; createArrayName keeps plain names
src/frontend/apiserver/controllers/jobarray_test.go
src/frontend/apiserver/controllers/jobs_handler_test.go # //go:build !cgo: handler-level submit/query/control tests through the test-seam variables
src/frontend/xcecli/common/format/jobsel.go # client identity selection parser + list splitter
src/frontend/xcecli/common/format/jobsel_test.go
src/frontend/xcecli/cmd/jobs/controller_test.go # controlJobsByIDs against an httptest server (request.NewAPIClient(srv.URL)): X[1-3] reaches the path as <decimal>[1-3]
src/controller/testsuite/test_jobsubmit_validate.c # 7.1-7.11 handle_jobsubmit validation/collision cases
src/worker/testsuite/test_exdata_jobindex.c # 6.1-6.4
```
### Modified Files
- **libxce**
- `src/include/xce/errnox.h`: add `E_XCE_ARRAY_START0` (before `E_XCE_MAX`) and its message text wherever `errnox` strings are defined.
- `src/include/utils.h`, `src/lib/utils.c`:
- Add `jobsel_t`, `jobsel_err_t`, `parse_jobsel()`, `jobsel_strerror()` and `is_array_tag()`.
- Re-implement `parse_jobid()` over `parse_jobsel()` with the same signature: NULL-safe, bounded, ranges rejected.
- Add `JOBID_FIELD_MAX_*` constants.
- `src/include/idgen.h`, `src/lib/idgen.c`:
- Add `idgen_array()`.
- `idgen()` accepts size 1 only.
- `src/lib/testsuite/test_idgen.c`: array cases move to `idgen_array` (first element index 1, bounds 1..32767, block overflow refused).
- `src/lib/meson.build` (`libxce_test_src`) and the runner `src/lib/testsuite/test.c`: register `test_jobsel`.
- **Controller**
- `src/controller/job.c`:
- Add `jobsubmit_classify()` (pure) and `array_identity_exists()`.
- Registry entries gain `array_id`, and the overlap test considers it.
- `handle_jobsubmit` order: classify, then register, then table checks with release on a hit. Add the 7.1 log line.
- `src/controller/testsuite/test_commit_barrier_reply.c`: array cases use tagged `block|1` IDs.
- `src/controller/testsuite/test_commit_barrier_rollout.c`: arrays use at most 32767 elements with tagged first IDs.
- `src/controller/meson.build` and the runner `src/controller/testsuite/controller_test.c`: register `test_jobsubmit_validate`.
- `src/controller/job.c` (also): `inflight_reply_and_remove()`; `jobsubmit_async_ctx_release` no longer removes registry entries; under `XCE_TESTING`, `jobsubmit_test_reply_hook`, `jobsubmit_inflight_overlaps_for_test()`, `jobsubmit_inflight_overlaps_locked_for_test()` and `jobsubmit_inflight_trylock_for_test()` (declared under `#ifdef XCE_TESTING`, as in `cluster.h:110-118`).
- **Scheduler**
- `src/scheduler/src/driver.c`:
- Expansion guard (tag 1 and `size <= 32768 - idx`, else log ERROR and skip the array).
- The INFO log prints `array_jobid(first)`.
- `src/scheduler/testsuite/test_driver_translate.c`: tagged IDs, plus guard cases.
- `src/scheduler/testsuite/test_job_array_expand.c` (bases 5000, 1, 8000 at :182, :213, :283) and `test_job_array_wire_e2e.c` (42000, 43000 at :295, :338-339): move to tagged `block|1` bases, since the guard skips untagged arrays.
- **Worker**
- `src/worker/src/xcecd_exdata.c`:
- `XCE_JOBINDEX` is `array_idx(jobid)` for tag 1 and 0 otherwise.
- A user `XCE_JOBINDEX` is skipped with a WARN.
- `src/worker/testsuite/meson.build`: register the new test.
- **Harness**
- `src/harness/cmdutil.c`, `src/harness/include/cmdutil.h`: add `cmdutil_parse_array_size()`, `cmdutil_job_arg()` and `cmdutil_job_not_found()`.
- `src/harness/testsuite/test_cmdutil.c`: cases for the above.
- `src/harness/test.c`:
- `jobsub`: `-a` through the new parser; `idgen_array` with its return code checked; `$jobid` set to the array ID; help text.
- `substitute_autovars` replaced by `autovar_substitute`.
- `expect-job` and `joblist_query` match by number.
- Every single-job command uses `cmdutil_job_arg` and `cmdutil_job_not_found`.
- `src/harness/meson.build`: add `autovars.c` to `xce_test_src` and to a new `test_autovars` unit-test target.
- `src/harness/testsuite/test_test_cli.c`: pty cases (no controller needed) for the `jobsub -a` refusals and for `jobkill -j 1-1-1-0[1-3]` (the 4.8 refusal, raised during option parsing before any RPC).
- **Apiserver**
- `src/frontend/apiserver/backend/clib.go`, `stub.go`: add `GenerateArrayJobID(cluster, apiserver, n)`.
- `src/frontend/apiserver/backend/convert_pure.go`: add `ParseHumanJobIDStrict` (field widths), `IsArrayCapableKey`, and `MakeElementID(key, k)`.
- `src/frontend/apiserver/backend/handler.go`: the `dispatchJobArray` log names the array ID (`FormatJobID(int64(ExtractArrayBaseID(baseJobID)))`; the parameter `baseJobID` holds the first element ID).
- `src/frontend/apiserver/utils/types.go`: `QueryOptions` gains `Predicates []func(OrderMap) bool`.
- `src/frontend/apiserver/store/query.go`: `ExecuteQuery` applies `Predicates` after `Conditions` and before sort and pagination (fixes bug-1898 on the owned paths).
- `src/frontend/apiserver/controllers/jobs.go`:
- `preCheckJobRequest`: the interactive-name check runs first; then the `jobarray.go` parser.
- `CreateJob`: sizing and ID through `GenerateArrayJobID`; the response; test-seam variables `generateJobID`, `generateArrayJobID`, `submitJob`, `controlJob` (used at both control dispatch sites).
- `GetJobsInfo`, `GetJobByIDs`, `ControlJobByID`, `FetchJobList` and filter-based `ControlJob` route through `jobsel.go`.
- `foundJob` gains an `explicitIndex` flag (full signature in ApiControl).
- Not-found control result maps (`ControlJobByID` and `ControlJob`) gain `unique_job_id`.
- The most-recent-job rule (5.10).
- `createArrayName` is kept, called only for names without `[` or `]` (1.4); delete `parseArrayIndexSection` and the `IsNumeric` uses on these paths.
- `src/frontend/apiserver/models/jobs.go`: `JobSubmitResponse` gains `ArraySize int json:"array_size,omitempty"`. (Control results are `map[string]interface{}` built in `jobs.go`; there is no apiserver control-item struct.)
- `src/frontend/apiserver/utils/config.go`, `config/app.yaml`: clamp the limit to 32767 with a startup warning. The rule moves into a pure `resolveMaxArraySize(v int) (effective int, clamped bool, err error)`; `InitConf` calls it and emits the `utils.Warn`.
- `controllers/jobs_test.go`, `utils/config_test.go`, `backend/convert_test.go`, `store/query_test.go`: new cases. `TestFoundJob_MatchesRealArrayIdxType` (`jobs_test.go:179`) moves to the new `foundJob` signature and gets a tagged `Unique_JobID` in its fixture; the four `preCheckJobRequest` call sites (`jobs_test.go:28, :100, :118, :217`) become `_, _, err := preCheckJobRequest(req)`.
- **xcecli**
- `src/frontend/xcecli/common/format/jobid.go`:
- `ParseJobID` becomes bounded (delegating to `jobsel.go`).
- Signature change: `ToAPIJobID(s string)` (`jobid.go:134`) becomes `ToAPIJobID(s string, allowRange bool)`, implemented as `ParseSelection(s, allowRange)` then `APIKey()`. Call sites updated: `cmd/jobs/controller.go:52` (`controlJobsByIDs`) passes `allowRange=true`; `filters/job_filter.go:49` and `common/format/jobid_test.go:83, :96, :109` pass `allowRange=false`; `peek.go:81` moves to `ToAPIJobIDLegacy`. unrecognised text is no longer passed through. Control (`cmd/jobs/controller.go` `controlJobsByIDs`) calls it with `allowRange=true`, so `kill X[1-3]` still reaches `ControlJobByID` (5.2, 5.3, 5.5); queries (`filters/job_filter.go`) call it with `allowRange=false`. Plain decimal keys still pass through to the server's checks, after the client's own 5.7/5.8 checks when `[...]` is present.
- `ToAPIJobIDLegacy(s)` keeps today's passthrough behaviour for `cmd/jobs/peek.go:81` (peek is out of scope).
- `FormatArrayChildID` keeps its signature and is used only as a fallback.
- `src/frontend/xcecli/cmd/jobs/controller.go`:
- `ParseCmdArgs` uses `SplitSelections`.
- `controlJobsByIDs` validates every selection before sending.
- Results print `FormatJobID(UniqueJobID)` when it is present.
- Help text (9.2).
- `src/frontend/xcecli/filters/job_filter.go`: `SplitSelections`, `isValidJobId` replaced by `ParseSelection`, and the raw `-J` value checked with `ParseNameSelection` before the wildcard mapping (`:81-91`); for a bracketed value the search rule and `*` stripping apply to the name part only.
- `src/frontend/xcecli/cmd/jobs/controller.go` (also): the raw control `-J` value checked with `ParseNameSelection` before `controlJobsByFilter`.
- `src/frontend/xcecli/cmd/jobs/submit.go`:
- Prints the array ID and element count.
- Help text (9.1).
- `src/frontend/xcecli/cmd/jobs/list.go`: help text `-J` correction.
- `src/frontend/xcecli/models/jobs.go`: `JobSubmitData` gains `ArraySize`; `JobControlResponseItem` gains `UniqueJobID int64 json:"unique_job_id,omitempty"`.
- `src/frontend/xcecli/cmd/jobs/peek.go` and `peek_test.go` (`:18,23`): switch to `ToAPIJobIDLegacy` (no behaviour change).
- `common/format/jobid_test.go`: in `TestToAPIJobIDPassthrough` (`:107-117`), `"123456"` stays a passthrough case; `"123456[3]"` is misaligned (`123456 & 0x7FFF != 0`) and is replaced by the aligned `"32768[3]"` → `"32768[3]"`, with a new rejection case asserting `"123456[3]"` returns the 5.8 error; malformed text is now rejected.
- **Skills and docs**
- `.claude/skills/xce-test-ha/run-ha-soak.sh`, `.claude/skills/xce-test-ha/SKILL.md`: elements 1..N.
- `.claude/skills/xce-test-multihost/run-multihost-verify.sh`, `.claude/skills/xce-test-multihost/SKILL.md`: element 1.
- `src/docs/xce.md`: new "Job arrays" section (9.4).
- `.kiro/specs/job-array-single-wal-record/requirements.md` (Requirement 5) and `.kiro/specs/xce-test-e2e-primitives/requirements.md` / `design.md`: superseded notes.
- `.kiro/specs/controller-job-cleanup/brief.md:86-87`, `.kiro/specs/job-dependency/brief.md:45-46`: numbering updated.
- `.kiro/steering/roadmap.md`: new entry.
## System Flows
### Controller array admission
```mermaid
sequenceDiagram
participant Client
participant Handler as handle jobsubmit
participant Registry as inflight registry
participant Table as jobmgr
participant Wal as commit barrier
Client->>Handler: jobsub first id and size
Handler->>Handler: classify
alt start zero
Handler-->>Client: E XCE ARRAY START0 and log
else invalid
Handler-->>Client: EINVAL
else ok
Handler->>Handler: field checks and hook
Handler->>Registry: try add range and array id
alt overlap
Registry-->>Handler: busy
Handler-->>Client: EEXIST
else added
Handler->>Table: element range exists or array id exists
alt collision
Handler->>Registry: reply EEXIST and release atomically
Registry-->>Client: EEXIST
else free
Handler->>Wal: commit array record
Wal-->>Handler: applied
Handler->>Registry: reply and release atomically
Registry-->>Client: first element id
end
end
end
```
- `classify` evaluates the checks in this order:
1. 7.1 (tag 1, index 0) → `INDEX0`.
2. 7.4 (size 0), 7.5 (tag 2 or 3), 7.3 (tag 0 and size > 1), 7.2 (tag 1 and `size > 32768 - idx`, written without addition so a `UInt64` size cannot wrap) → `INVALID`.
This gives 7.9: `INDEX0` alone wins, and any `INVALID` wins over a collision, using the single error `-EINVAL`.
- Registration precedes the table checks, so any submission that commits is in the job table before it leaves the registry. Two concurrent submissions with the same array ID are ordered by who registers first (7.11).
- An array submission is registered for its whole element range with `is_array = 1` and its `array_id` (tag cleared, low 15 bits cleared). It is compared with every other array entry's `array_id`. Ordinary entries carry `is_array = 0` and take part in range overlap only.
### Selection evaluation (all surfaces)
```mermaid
graph TB
Raw[raw argument] --> Syntax{grammar ok}
Syntax -- no --> E516[reject invalid selection]
Syntax -- yes --> Width{fields within width}
Width -- no --> E514[reject invalid identity]
Width -- yes --> Marked{decimal key with array marking and no index}
Marked -- yes --> E517[reject name elements as array id k]
Marked -- no --> Align{index given and key not array capable}
Align -- yes --> E58[reject invalid identity]
Align -- no --> Bound{number above 32767}
Bound -- yes --> E57[reject invalid selection]
Bound -- no --> Rev{range reversed}
Rev -- yes --> E55[reject range reversed]
Rev -- no --> Ok[parsed selection]
```
- The check order is exactly 5.12: 5.15/5.16, then 5.14, then 5.17, then 5.8, then 5.7, then 5.5.
- The harness parser has no decimal form, so the 5.17 node never applies there. Its syntax node is 5.15.
- For name selections (`-J`, `job_name`), the width, marked and alignment nodes do not apply. The part before `[` is not checked against the submission array-name rule (2.2): a name no array can have simply selects no job (5.2).
## Requirements Traceability
| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1 | `name[1-N]` creates 1..N | ApiSubmitExpr, ApiCreateJob, IdgenArray, ControllerValidate | `ParseSubmitIndexExpr`, `GenerateArrayJobID` | Admission |
| 1.2 | `name[1]` makes a one-element array | ApiSubmitExpr, ApiCreateJob, IdgenArray | same | Admission |
| 1.3 | distinct element identities | IdgenArray, ControllerValidate | `idgen_array`, `first + i` | Admission |
| 1.4 | names without brackets unchanged | ApiCreateJob (`preCheckJobRequest`) | `createArrayName` path unchanged; `isArray=false` → `GenerateJobID(..., 1)` | — |
| 1.5 | leading zeros | ApiSubmitExpr, ApiSel, CliSel, LibJobsel | number rule | Selection |
| 1.6 | configured limit used | ApiConfig | `MaxArraySize` | — |
| 1.7 | clamp to 32767 with warning | ApiConfig | startup `utils.Warn` | — |
| 1.8 | unset or 0 → 1000 | ApiConfig | — | — |
| 1.9 | negative refuses start | ApiConfig | `InitConf` error | — |
| 1.10 | recovery keeps numbering | (unchanged WAL/apply; option B) | — | — |
| 2.1 | interactive arrays rejected | ApiSubmitExpr | interactive check before name branch | — |
| 2.2 | malformed expression or bad name | ApiSubmitExpr | `ErrArrayName`, `ErrArrayMalformed` | — |
| 2.3 | lists or steps rejected | ApiSubmitExpr | `ErrArraySingleRange` | — |
| 2.4 | `%N` rejected | ApiSubmitExpr | `ErrArrayRunLimit` | — |
| 2.5 | start 0 rejected | ApiSubmitExpr | `ErrArrayStartZero` | — |
| 2.6 | reversed range | ApiSubmitExpr | `ErrArrayReversed` | — |
| 2.7 | start > 1 rejected | ApiSubmitExpr | `ErrArrayStartNotOne` | — |
| 2.8 | above limit | ApiSubmitExpr | `ErrArrayTooLarge` | — |
| 2.9 | lowest-numbered rule reported | ApiSubmitExpr | ordered evaluation | — |
| 2.10 | nothing created on rejection | ApiCreateJob | early `ReturnError` before `GenerateArrayJobID` | — |
| 3.1 | response names the array | ApiCreateJob, CliSubmit | `JobSubmitResponse.jobid` | — |
| 3.2 | element count shown | ApiCreateJob, CliSubmit | `array_size` | — |
| 3.3 | ordinary submit unchanged | ApiCreateJob | — | — |
| 4.1 | `jobsub -a N` makes 1..N | HarnessJobsub, IdgenArray | `cmdutil_parse_array_size`, `idgen_array` | Admission |
| 4.2 | `-a 1` stays ordinary | HarnessJobsub | `idgen` size 1 | — |
| 4.3 | N > 32767 refused | HarnessJobsub | `-ERANGE` message | — |
| 4.4 | bad, missing or 0 refused | HarnessJobsub | `-EINVAL` message | — |
| 4.5 | success line and `$jobid` are the array ID | HarnessJobsub | `array_jobid` | — |
| 4.6 | ID failure reported | HarnessJobsub | `idgen_array` rc | — |
| 4.7 | not-found hint | HarnessJobArg | `cmdutil_job_not_found` | — |
| 4.8 | ranges refused | HarnessJobArg | `cmdutil_job_arg` | Selection |
| 4.9 | exact ID acts as before | HarnessJobArg, HarnessExpectJob | numeric match | — |
| 4.10 | `$jobid[...]` substitution | HarnessAutovars | `autovar_substitute` | — |
| 4.11 | unset `$jobid` fails | HarnessAutovars | same | — |
| 5.1 | `<array-id>[k]` display | LibJobsel (display unchanged), CliControl, ApiControl | `unique_job_id`, `fmt_jobid` | — |
| 5.2 | element selection acts on elements only | ApiSel, ApiMatch, CliSel | `Selection`, element-only filter | Selection |
| 5.3 | missing numbers reported | ApiControl | per-number not-found items | — |
| 5.4 | missing `[0]` reported, no fallback | ApiControl | `foundJob(explicitIndex=true)` | — |
| 5.5 | reversed selection range | ApiSel, CliSel | `ErrSelReversed` | Selection |
| 5.6 | bare ID acts on whole array and ordinary job | ApiMatch | bare form | — |
| 5.7 | number > 32767 rejected | ApiSel, CliSel, LibJobsel | `*_EINDEX` | Selection |
| 5.8 | non-array-capable `<array-id>` rejected | ApiSel, CliSel, LibJobsel | `*_EALIGN` | Selection |
| 5.9 | array logs use the array ID | SchedGuard, ApiDispatchLog (dispatch + `CreateJob` log lines) | log lines | — |
| 5.10 | most-recent rule for one-element array | ApiControl | `isSingleElementArray` | — |
| 5.11 | query of missing `[0]` lists nothing | ApiMatch | explicit-index filter | — |
| 5.12 | first failed check reported | ApiSel, CliSel, LibJobsel | ordered evaluation | Selection |
| 5.13 | harness `[k]` acts on that element | HarnessJobArg | exact element ID | — |
| 5.14 | field-width rejection | ApiSel, CliSel, LibJobsel | `*_EWIDTH` | Selection |
| 5.15 | harness syntax rejection | LibJobsel, HarnessJobArg | `JOBSEL_ESYNTAX` | Selection |
| 5.16 | CLI/REST syntax rejection | ApiSel, CliSel, CliSplit | `ErrSelSyntax`, bracket-aware split | Selection |
| 5.17 | marked decimal without `[k]` rejected | ApiSel | `ErrSelMarked` | Selection |
| 6.1 | `XCE_JOBINDEX` is the element number | WorkerEnv | `array_idx` | — |
| 6.2 | ordinary jobs get 0 | WorkerEnv | tag test | — |
| 6.3 | user value cannot override | WorkerEnv | skip in user loop | — |
| 6.4 | warning per host | WorkerEnv | `XCECD_WORKER_LOG` WARN | — |
| 6.5 | `XCE_JOBID` unchanged | WorkerEnv | — | — |
| 7.1 | start-0 array → distinct error and log | ControllerValidate | `E_XCE_ARRAY_START0` | Admission |
| 7.2 | last > 32767 rejected | ControllerValidate | `INVALID` | Admission |
| 7.3 | untagged size > 1 rejected | ControllerValidate | `INVALID` | Admission |
| 7.4 | size 0 rejected | ControllerValidate | `INVALID` | Admission |
| 7.5 | other tags rejected | ControllerValidate | `INVALID` | Admission |
| 7.6 | nothing stored; service continues | ControllerValidate | early reply, no registry | Admission |
| 7.7 | valid array created from its first number | ControllerValidate, ControllerCollision | classify OK, then commit | Admission |
| 7.8 | one-element array behaves as array | ControllerValidate, WorkerEnv, ApiMatch | tagged single path | — |
| 7.9 | error precedence | ControllerValidate | classify order | Admission |
| 7.10 | collisions → `EEXIST` | ControllerCollision | `array_identity_exists`, registry `array_id` | Admission |
| 7.11 | concurrent same array ID: first wins | ControllerCollision | registry before table | Admission |
| 8.1 | pre-upgrade element 0 kept | (unchanged storage), LibJobsel accepts `[0]` | — | — |
| 8.2 | `<array-id>[0]` selects that element only | ApiMatch, ApiControl, HarnessJobArg | explicit-index flag | Selection |
| 8.3 | element 0 shown as `[0]` | CliControl, ApiControl | `unique_job_id`, `FormatJobID` | — |
| 8.4 | recovery after upgrade | (unchanged WAL/apply) | — | — |
| 9.1 | submit help | CliSubmit | help text | — |
| 9.2 | control help | CliControl | help text | — |
| 9.3 | harness help | HarnessJobsub | usage strings | — |
| 9.4 | user documentation | Docs | `src/docs/xce.md` | — |
## Components and Interfaces
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|--------------|--------------------------|-----------|
| LibJobsel | libxce | Bounded identity/selection parser | 1.5, 5.1, 5.7, 5.8, 5.12, 5.14, 5.15, 8.1 | config.h layout (P0) | Service |
| IdgenArray | libxce | Allocate an array's first element ID | 1.1-1.3, 4.1 | idgen ctx (P0) | Service |
| ControllerValidate | controller | Classify array submissions | 7.1-7.9 | LibJobsel `is_array_tag` (P0) | Service |
| ControllerCollision | controller | Element and array-ID collision checks | 7.7, 7.10, 7.11 | jobmgr, registry (P0) | Service, State |
| SchedGuard | scheduler | Guard expansion; log the array ID | 5.9 | driver translation (P1) | Service |
| WorkerEnv | worker | `XCE_JOBINDEX` | 6.1-6.5, 7.8 | `array_idx` (P0) | Service |
| HarnessJobsub | harness | `jobsub -a` and `$jobid` | 4.1-4.6, 9.3 | IdgenArray (P0) | Service |
| HarnessJobArg | harness | Single-job argument parsing and not-found reporting | 4.7-4.9, 5.13, 5.15, 8.2 | LibJobsel (P0) | Service |
| HarnessExpectJob | harness | Numeric `expect-job` | 4.9 | HarnessJobArg (P0) | Service |
| HarnessAutovars | harness | `$jobid` and `$jobid[...]` | 4.10, 4.11 | — | Service |
| ApiSubmitExpr | apiserver | Submit index expression | 1.1, 1.2, 1.5, 2.1-2.9 | ApiConfig (P0) | Service |
| ApiCreateJob | apiserver | IDs, submit, response | 1.1-1.4, 2.10, 3.1-3.3 | IdgenArray via cgo (P0) | API |
| ApiConfig | apiserver | Array-size limit | 1.6-1.9 | zap logger (P1) | State |
| ApiSel | apiserver | REST selection parser and key validator | 1.5, 5.2, 5.5, 5.7, 5.8, 5.12, 5.14, 5.16, 5.17 | backend ID helpers (P0) | Service |
| ApiMatch | apiserver | Selection → store filter | 5.2, 5.6, 5.11, 7.8, 8.2 | ApiSel (P0) | Service |
| ApiControl | apiserver | Control results, missing numbers, most-recent rule | 5.1, 5.3, 5.4, 5.10, 8.2, 8.3 | ApiMatch (P0) | API |
| ApiDispatchLog | apiserver | Array log line | 5.9 | — | — |
| CliSel | xcecli | Client selection parser | 1.5, 5.2, 5.5, 5.7, 5.8, 5.12, 5.14, 5.16 | — | Service |
| CliSplit | xcecli | Bracket-aware ID-list split | 5.16 | — | Service |
| CliSubmit | xcecli | Submit output and help | 3.1, 3.2, 9.1 | ApiCreateJob (P0) | — |
| CliControl | xcecli | Control by ID; result display; help | 5.1, 8.3, 9.2 | CliSel, ApiControl (P0) | — |
| Docs | docs | Syntax and LSF differences | 9.4 | — | — |
### libxce
#### LibJobsel
| Field | Detail |
|-------|--------|
| Intent | Parse one written job identity, optionally with `[k]` or `[a-b]`, under fixed bounds and check order |
| Requirements | 1.5, 5.1, 5.7, 5.8, 5.12, 5.14, 5.15, 8.1 |
**Responsibilities & Constraints**
- **Grammar:** optional leading/trailing spaces or tabs, `num-num-num-num`, optionally `[num]` or `[num-num]`. `num` is one or more decimal digits. There is no other whitespace and no sign.
- **Checks, in order:** syntax (5.15), field widths (5.14: cluster <= 63, API server <= 15, timestamp < 2^32, sequence < 2^20), alignment when `[...]` is present (5.8: `seq % 32768 == 0`), index bound (5.7: each number <= 32767), reversed range (`JOBSEL_EREVERSED`). The harness maps `JOBSEL_EREVERSED` to the 4.8 single-element refusal, because 5.5 applies only to the CLI and REST.
- **Overflow:** digits are accumulated with an overflow guard (`strtoull` with errno, or a digit-count bound), so oversized numbers fail 5.14 or 5.7 rather than wrapping.
- **Display unchanged:** `fmt_jobid`, `array_jobid` and `array_idx` keep their behaviour.
**Contracts**: Service [x]
##### Service Interface
```c
typedef enum {
JOBSEL_OK = 0,
JOBSEL_ESYNTAX, /* 5.15 */
JOBSEL_EWIDTH, /* 5.14 */
JOBSEL_EALIGN, /* 5.8 */
JOBSEL_EINDEX, /* 5.7 */
JOBSEL_EREVERSED /* 5.5 */
} jobsel_err_t;
typedef enum { JOBSEL_BARE = 0, JOBSEL_SINGLE, JOBSEL_RANGE } jobsel_form_t;
typedef struct {
uint64_t key; /* BARE: the exact (untagged) ID; else the array ID */
jobsel_form_t form;
uint32_t lo; /* SINGLE: lo == hi == k */
uint32_t hi;
} jobsel_t;
jobsel_err_t parse_jobsel(const char* text, jobsel_t* out);
const char* jobsel_strerror(jobsel_err_t err);
int is_array_tag(uint64_t jobid); /* TAG(jobid) == TAG_JOB_ARRAY */
uint64_t jobsel_element_id(const jobsel_t* s, uint32_t k); /* MAKE_JOBID(.., TAG_JOB_ARRAY, key|k) */
/* unchanged signature; now bounded, NULL-safe, and returns -EINVAL for a
* range, a rejected selection, or NULL; 1 on success */
int parse_jobid(uint64_t* jobid, char* buf);
```
- **Preconditions:** `out != NULL`. A NULL `text` returns `JOBSEL_ESYNTAX`.
- **Postconditions:** on `JOBSEL_OK`, `key` satisfies the field widths. For `SINGLE` and `RANGE`, `key` has tag 0 and its low 15 bits clear.
- **Invariants:** fields are masked to their widths before composition. `MAKE_JOBID` itself is unchanged.
**Implementation Notes**
- **Validation:** `test_jobsel.c` runs the `selection-cases` table (Testing Strategy).
- **Callers of `parse_jobid`:** the nine harness single-job commands (`test.c:3327, 3517, 4140, 4276, 4921, 5073, 5268, 5377, 5481`), which move to `cmdutil_job_arg`, and `scheduler/src/repl_info.c:114`. The REPL `info job` argument is out of scope, and its behaviour does not change: on any `parse_jobid` failure it falls back to `strtoull` (`repl_info.c:114-117`), which still yields the decimal value, 0, or the leading digits as today.
#### IdgenArray
| Field | Detail |
|-------|--------|
| Intent | Return the first element ID of a new array of `n` elements, numbered from 1 |
| Requirements | 1.1, 1.2, 1.3, 4.1 |
**Contracts**: Service [x]
```c
/* 1 <= n <= MAX_ARRAY_IDX (32767). On success *first = MAKE_JOBID(c, a, ts,
* TAG_JOB_ARRAY, (block << 15) | 1) and returns n. -EINVAL for bad args or n;
* -EOVERFLOW if the next block would leave the 20-bit sequence (the context
* is left unchanged). Advances the context past the whole block. */
int idgen_array(idgen_ctx_t* ctx, size_t clusterId, size_t apiserverId,
size_t n, uint64_t* first);
/* Ordinary IDs only: arraySize must be 1 (else -EINVAL). Returns 1. */
int idgen(idgen_ctx_t* ctx, size_t clusterId, size_t apiserverId,
size_t arraySize, uint64_t* pidx);
```
- **Postconditions:** `first + n - 1` has the same array ID and index `n`.
- **Go wrapper:** `GenerateArrayJobID(clusterID, apiserverID int, n int) (uint64, error)` in `backend/clib.go`, plus a matching `stub.go` stub that returns an error.
- **`GenerateJobID`:** its only callers are `CreateJob` for ordinary jobs and `RestartJobByID`, so it is always called with size 1; the `arraySize` parameter stays for signature stability.
- **`idgen` callers:** `clib.go:341` (through `GenerateJobID`) and the harness `test.c:3721`, which switches to `idgen_array` for N >= 2.
### Controller
#### ControllerValidate and ControllerCollision
| Field | Detail |
|-------|--------|
| Intent | Admit only representable array submissions; detect identity collisions without a window |
| Requirements | 7.1-7.11 |
**Responsibilities & Constraints**
- **Pure classification:** `jobsubmit_classify` runs first -- as the first statement after `init_reply_by_req` in `handle_jobsubmit` (there is no NULL check on `submit` today, and the existing field checks dereference it the same way), before the existing field checks (`NULLSTR` user/queue, `start_info`, empty `cmdline`, `job.c:2445-2454`), the `HOOK_BEFORE_EVENT` hook and any registry or table access. It reads only `submit->jobid` and `submit->array_size`, so this is safe, and it makes 7.9 hold for every submission: an index-0 array is answered `E_XCE_ARRAY_START0` with its log line even when a field is also empty, and a 7.2-7.5 break is answered `-EINVAL` before any field check. Consequence: `HOOK_BEFORE_EVENT` no longer fires for submissions rejected under 7.1-7.5 (today it fires for every submission that passes the field checks, including ones a later check rejects); such submissions create no job, so no hook consumer loses an event for a job that exists.
- **Registry and table:** registry add first, then the table checks (7.10, 7.11). Answering a registered submission and removing its registry entry are one atomic step: a new `inflight_reply_and_remove(endpoint_t* ep, xce_peerinfo_t* from, xce_reply_t* reply, uint64_t base, uint64_t size)` (the synchronous paths build `reply` with `init_reply_by_req`; the async paths build it from `actx->seq`/`actx->opcode` and pass `actx->ep`, `actx->c->myhost`, as `jobsubmit_reply_*_once` do today, `job.c:2234-2288`) takes `s_inflight_submits_lock`, calls `xce_reply` (always non-blocking, so allowed under that lock by `ha-async-io.md`), unlinks the entry, and unlocks. Every path after registration uses it -- the new table-hit path, the existing payload-build and allocation-failure early-outs (today `inflight_range_remove` then `xce_reply`), commit failure or timeout and apply success (today `jobsubmit_reply_*_once` then `jobsubmit_async_ctx_release`). It is the only place an entry is removed: `jobsubmit_async_ctx_release` stops calling `inflight_range_remove` (`job.c:2224`) and only frees the async context, because a removal by exact `(base, size)` after the reply could delete a retry of the same submission that registered in between (7.7 allows that retry). On the async paths, `jobsubmit_reply_*_once` calls `inflight_reply_and_remove` inside its one-shot guard, so reply and removal happen together exactly once; a suppressed second reply removes nothing because the first already did. So another submission's collision check (which takes the same lock) sees an entry exactly while its submission is unanswered, as the Terms' "array submission still being accepted" requires: neither a just-answered submission blocking a later one (7.7) nor an unanswered one failing to block it (7.10, 7.11).
- **Array-ID check:** for a tag-1 submission, one scan takes `lock(c->jobmgr, RTTI_LOCK_READ)` once and looks up keys `MAKE_JOBID(.., TAG_JOB_ARRAY, A|k)` for k = 0..32767. That one scan answers both the element-range check (keys inside `[first, first + size)`) and the array-identity check (any key), replacing `jobid_range_exists` for tag-1 submissions; tag-0 submissions keep `jobid_range_exists` unchanged. It is bounded CPU work (32768 hash lookups under one read-lock hold, no I/O, no blocking), so it is compatible with the vtask handler (`ha-async-io.md`, audit dimension 1). An array-ID index maintained at apply and snapshot load was considered and rejected: O(1), but it adds a derived structure that every apply, replay and snapshot-load seam must keep complete (dimension 7), for a cost paid only by array submissions.
- **No exits:** no exit or abort path is added (dimension 2). Errors become reply statuses (dimension 5).
**Contracts**: Service [x] / State [x]
```c
typedef enum { JOBSUBMIT_OK, JOBSUBMIT_INDEX0, JOBSUBMIT_INVALID } jobsubmit_class_t;
/* 7.1 first; then 7.4, 7.5, 7.3, 7.2; pure, no locks. size is the wire
* UInt64 (controller.capnp arraySize). 7.2 is evaluated without addition,
* as size > 32768 - idx (idx >= 1 here), so neither 2^32+2 nor UINT64_MAX
* can wrap past it. */
jobsubmit_class_t jobsubmit_classify(uint64_t first_jobid, uint64_t size);
/* One read-locked scan of the 32768 element keys of array_id's block.
* Returns 1 if any of them exists (array identity taken -- this covers the
* element range [first, first+size) too), 0 otherwise. */
int array_identity_exists(cluster_t* c, uint64_t array_id);
/* registry entry: { base_jobid, array_size, is_array, array_id }. Overlap if
* element ranges intersect, or both entries have is_array and equal array_id. */
```
- **Replies:**
| Class | Reply status | Log |
|---|---|---|
| `INDEX0` | `-E_XCE_ARRAY_START0` | ERROR naming the job; says array element numbers start at 1 and the submitting client must be upgraded |
| `INVALID` | `-EINVAL` | — |
| Collision | `-EEXIST` (unchanged) | — |
| Success | reply `jobid` = first element ID (unchanged) | — |
- **State:** the registry keeps its mutex. Entries gain one field. Removal is by exact `(base, size)`, now only inside `inflight_reply_and_remove`.
- **Invariant cross-check:**
| Dimension | Outcome |
|---|---|
| 1: async I/O | Pass: bounded CPU only |
| 2: lifecycle | Pass |
| 4: commit/apply | Pass: unchanged commit path; registry ordering tightened |
| 5: errors | Pass: every rejection replies |
| 7: field-complete apply | Not affected: no WAL field change |
| 9: redirect | Not affected: no new primary-only operation |
### Scheduler
#### SchedGuard
- **Expansion guard:** for `XCE_SCHED_OP_UPDATE_JOB_ARRAY`, if `!is_array_tag(first)` or `size > 32768 - array_idx(first)` (no addition, so a `UInt64` size cannot wrap), log ERROR (naming `array_jobid(first)` and the size) and expand nothing. The controller never sends such an array, so this guard is defensive only.
- **INFO log:** prints `array_jobid(first)` as `array_jobid=%s` (5.9).
- **Scheduler invariants:** expansion stays in `driver.c` translation before core events, so the recording and the simulator see expanded jobs and driver state-equivalence (scheduler dimension 1) holds. Dimensions 2-4 are not touched.
### Worker
#### WorkerEnv
- **System value:** `XCE_JOBINDEX` is `array_idx(jobid)` when `is_array_tag(jobid)`, and `0` otherwise (6.1, 6.2, 7.8).
- **User value:** in the user-variable loop, a variable whose name is exactly `XCE_JOBINDEX` is not applied, and `XCECD_WORKER_LOG` WARN names the job ID and the ignored variable (6.3, 6.4).
- **`XCE_JOBID`:** handling is unchanged (6.5).
- **Unchanged structure:** the function's other caller (the JSON-parse failure path) is unchanged.
### Harness
#### HarnessJobsub, HarnessJobArg, HarnessExpectJob, HarnessAutovars
```c
/* digits only (leading zeros ok); -EINVAL missing/non-digit/0, -ERANGE > 32767 */
int cmdutil_parse_array_size(const char* text, uint32_t* n);
/* Trims spaces/tabs, parse_jobsel(). RANGE -> refuse (4.8). Writes a message
* into errbuf on rejection. On success *jobid is the exact ID (element ID for
* [k]), *had_index says whether [k] was given. Returns 0 or -EINVAL. */
int cmdutil_job_arg(const char* arg, uint64_t* jobid, int* had_index,
char* errbuf, size_t errlen);
/* Prints "job <arg> not found" and, when !had_index, "array elements must be
* named as <array-id>[k]" (4.7). */
void cmdutil_job_not_found(FILE* out, const char* arg, int had_index);
/* Values today's substitute_autovars() reads from test.c globals. */
typedef struct {
const char* jobid; /* NULL when unset */
int last_status;/* $? */
int jobcount; /* $jobcount, -1 when unset */
} autovar_ctx_t;
/* $?, $jobcount, $jobid and (new) $jobid[...] token substitution (4.10,
* 4.11); same token rule as today; '!' and '#' lines untouched. Returns 0,
* -EINVAL for an unset $jobid / $jobcount with today's messages, or -ENOSPC
* when out is too small (as today). The caller passes jobid = NULL when
* g_last_jobid is empty. */
int autovar_substitute(const char* line, const autovar_ctx_t* ctx, char* out,
size_t outlen, char* errbuf, size_t errlen);
```
- **`jobsub -a N`:**
- N = 1 → `idgen` (ordinary).
- N >= 2 → `idgen_array`; a failure prints the error and submits nothing (4.6).
- The ack check compares against the first element ID.
- **Sequence seed:** `test.c:6896-6899` seeds `idctx.sequence` randomly in `[0, 2^20)`. Any seed above `31 * 32768` would make the first `idgen_array` of that second return `-EOVERFLOW` (about 1 run in 32). The seed is bounded to `[0, 30 * 32768)`, which leaves room for at least one whole block. (The apiserver's context starts at 0, `clib.go:332`.)
- The success line and `$jobid` use `fmt_jobid(array_jobid(first))` (4.5).
- **Single-job commands:** every command listed in the requirements' Harness boundary note calls `cmdutil_job_arg` and stops on its error. The controller's not-found replies go through `cmdutil_job_not_found`. For `jobinfo`, each `-j` is judged on its own.
- **`expect-job`:** parses its argument with `cmdutil_job_arg`. `joblist_query` matches `jobs[i]->jobid == id`. "Not found" and "refused" fail the assertion.
### Apiserver
#### ApiSubmitExpr
```go
type ArrayExprError struct {
Rule string // "2.1" ... "2.8"
Msg string // user-facing, quotes the name where 2.2 requires it
}
func (e *ArrayExprError) Error() string
// Called only for names containing '[' or ']' (names without them keep the
// existing createArrayName path, 1.4).
// Checks in requirement order 2.1 .. 2.8 and returns the first failure (2.9).
func ParseSubmitIndexExpr(name string, interactive bool, limit int) (
arrayName string, n int, isArray bool, err *ArrayExprError)
```
- **Grammar:** exactly 2.2's. Numbers are read digit-wise and saturate at a sentinel above any limit instead of overflowing. Rules 2.5-2.8 are then evaluated in order on the saturated values, so `name[99999999999999999999]` reports 2.7, `name[99999999999999999999-5]` reports 2.6, and `name[0-99999999999999999999]` reports 2.5 (2.9). Leading zeros are accepted (1.5).
- **2.3 / 2.4 / 2.5 / 2.6 / 2.7 / 2.8:** 2.3 lists or steps; 2.4 `%N`; 2.5 first number 0; 2.6 last < first; 2.7 first > 1; 2.8 last > limit.
- **Contract with `CreateJob`:** `preCheckJobRequest` becomes `preCheckJobRequest(req) (n int, isArray bool, err error)`. `CreateJob` branches on `isArray`, not on a count: `isArray == true` → `GenerateArrayJobID(c, a, n)` (n may be 1, so `name[1]` is a one-element array, 1.2, 7.8); `isArray == false` → `GenerateJobID(..., 1)` as today. Every existing error return in `preCheckJobRequest` (`jobs.go:1546`, `:1553`, `:1558`, `:1565`, `:1571`, `:1597`, `:1735`, ...) becomes `0, false, err`; the plain-name success path (the named-job early return, `:1578`) and the unnamed-job success path (the final return, `:1746`) return `1, false, nil`. On success for a bracketed name, `preCheckJobRequest` sets `req.JobName = arrayName` (as today at `jobs.go:1574`, so the stored name is `arr`, not `arr[1-3]`, and `-J 'arr[2]'` matches) and returns `(n, true, nil)` at the same point as today's named-job early return (`jobs.go:1578`), so the checks it skips are the same as today. The old sizing rule (`arraySize := 1; if len(parsedIndices) > 1`, `jobs.go:1419-1423`) is removed.
- **Names without `[` or `]` (1.4):** keep today's path unchanged -- `createArrayName` stays and is called only for such names, so `isValidArrayName` on the whole name, the slot-limit errors for a `%` suffix (`jobs.go:1840-1851`), the "job array list format is invalid: " prefix and their 500 status are exactly as before (they are not `*ArrayExprError`). `ParseSubmitIndexExpr` is called only for names containing `[` or `]`.
- **Caller:** `preCheckJobRequest` (`jobs.go:1537`, no gin context) calls `ParseSubmitIndexExpr` first for a bracketed name, before the existing early return for named jobs, so the interactive check (2.1) is reached. It passes `interactive = (req.Interactive != 0)` and `limit = utils.JobYamlConfig.MaxArraySize` (the value `resolveMaxArraySize` produced at startup; today read at `jobs.go:1901`) (xcecli sends 1, 2 or 3, `submit.go:245-249`). When the returned `*ArrayExprError` is non-nil it returns `0, false, ae` unwrapped (no "job array list format is invalid: " prefix, `jobs.go:1565, 1571`); a nil `*ArrayExprError` is never converted into a non-nil `error` (typed-nil).
- **HTTP mapping:** `CreateJob` (`jobs.go:1394-1398`, today `ReturnError(ctx, common.ERROR_CODE, ...)` = 500 for every preCheck error) first checks `var ae *ArrayExprError; if errors.As(err, &ae) { ReturnError(ctx, common.INVALID_PARAMS_CODE, ae.Msg, nil); return }`, then falls through to the existing 500 path for other errors.
#### ApiCreateJob
| Method | Endpoint | Request | Response | Errors |
|--------|----------|---------|----------|--------|
| POST | /xce/v1/jobs | `JobSubmitRequest` (unchanged) | `JobSubmitResponse{jobid, queue_name, message, array_size}` | 400 (2.x), 500 (ID allocation/submit) |
- **Test seams:** `controllers/jobs.go` adds package variables in the existing `validateResreqSyntax` hook pattern -- `var generateJobID = backend.GenerateJobID`, `var generateArrayJobID = backend.GenerateArrayJobID`, `var submitJob = services.SubmitJob`, `var controlJob = backend.ControlJob` -- and `CreateJob` and both control dispatch sites (`ControlJobByID`, `jobs.go:2611`; `ControlJob`, `:2847`) call them, so non-CGO handler tests can replace them and observe dispatch.
- **Arrays:** `GenerateArrayJobID(c, a, n)` returns the first element.
- The model has `Unique_JobID` = first, `JobID` = the array ID, `ArrayIdx` = 1.
- `primaryJobID` (used by `processJobRequest` and `postJobRequest` for the submit-time `%J`/`%I` substitution) stays the first element ID, so a new array's file names get `%I` = 1 and `%J` = `<array-id>[1]`, as 9.4 documents.
- It submits with `SubmitJob(&job, n)` (unchanged).
- The response `jobid` is the array ID's decimal, `array_size` is n, and `message` is `Job <array-id-decimal> is submitted to queue <q> (n elements).`
- **Ordinary jobs:** unchanged, with `array_size` omitted (3.3).
#### ApiConfig
- `job.max_array_size` absent or 0 → 1000 (1.8).
- Negative → startup error (1.9).
- 1..32767 → used as is (1.6).
- Above 32767 → 32767, plus `utils.Warn("job.max_array_size above supported maximum", zap.Int("configured", v), zap.Int("effective", 32767))` (1.7). The existing `> 2147483646` startup rejection (`utils/config.go:221`) is deleted; only a negative value is an error.
#### ApiSel and ApiMatch
```go
type SelForm int
const ( SelBare SelForm = iota; SelSingle; SelRange )
type Selection struct {
Raw string // as the user wrote it (trimmed)
Name string // name selections only (job_name / -J)
Key uint64 // identity selections: exact ID (bare) or array ID
Form SelForm
Lo, Hi int // element numbers, Lo == Hi for SelSingle
}
type SelError struct {
Rule string // "5.16", "5.14", "5.17", "5.8", "5.7", "5.5"
Msg string // names the selection as written
}
// Identity: written c-a-ts-seq (strict widths) or decimal uint64 (ParseUint, base 10).
// Ranges are always allowed at the server.
// An identity range in a query is evaluated by its predicate (today it was
// accepted and silently dropped, bug-1898); only xcecli's own ParseSelection
// refuses query ranges, after 5.16, 5.14, 5.8 and 5.7 pass.
func ParseIdentitySelection(raw string) (Selection, *SelError)
// Name: 5.16 syntax, 5.7 and 5.5 only; no array-name rule check. The name
// part is zero or more characters, none of them whitespace, '[' or ']'. An
// empty name part is not a 5.16 violation: with contains/startwith/endwith it
// matches every name (today's semantics of an empty pattern); with eq/in/notin
// the name predicate requires a non-empty name, so it selects no job (eq/in)
// -- jobs submitted without a name (e.g. by the harness) are stored with
// job_name "" (backend/convert.go goString) and must not be matched by
// -J '[1]'.
func ParseNameSelection(raw string) (Selection, *SelError)
```
- **Triggers:** any value containing `[` or `]` goes through these parsers before the existing plain-match fallback (5.16). Error messages pass through to `ReturnError` unchanged, so they are no longer replaced by `"bad job array index: "` (5.5).
- **Callers:** `GetJobsInfo` (`job_id` and `job_name`), `GetJobByIDs`, `ControlJobByID`, `FetchJobList` (and through it filter-based `ControlJob`).
- **Validate-all-first:** each of these handlers parses and validates every selection and key in the request before any store query or control dispatch. In `GetJobsInfo` this is a separate pass over all filter entries -- `ParseIdentitySelection` / `ParseNameSelection` on every value that needs it, plus the operator checks above -- run before the existing filter loop, which can query the store (`exec_hosts` resolution via `FetchJobIdList`, `jobs.go:465-480`) and return early with `count: 0`. Filter-based `ControlJob` validates `req.Filter.JobName` with `ParseNameSelection` itself -- only when it contains `[` or `]`, since unnamed jobs carry their command (with spaces) as their name (`jobs.go:1582-1584`) and `kill -J 'sleep 100'` must keep working -- before calling `FetchJobList`, so a rejection returns 400 with the specific message (today every `FetchJobList` error is mapped to 500 `ERROR_QUERY_JOBINFO_MSG`, `jobs.go:2734-2738`). The first rejection (in request order) is returned with `ReturnError`, and the command acts on no job and reports no "not found" results (Terms).
- **Matching (ApiMatch):** selections become row predicates passed in `QueryOptions.Predicates`, evaluated by `store.ExecuteQuery` before sorting and pagination (bug-1898). The bracketed `job_id`/`job_name` values no longer produce `CompFilterCondition`s.
- **Shape:** `Predicates []func(OrderMap) bool`, one composed predicate per filter field. Within one field (one `job_id` or `job_name` filter), the values' predicates are OR'ed; across fields, and with the plain `Conditions`, they are AND'ed, matching today's per-field filter semantics.
- **Whole-filter conversion:** when any value of a filter entry has `[` or `]`, every value of that entry becomes a predicate (a bare value becomes a `SelBare` predicate) and that entry, and only that entry, is removed from `Conditions`. Other entries on the same field with another operator (`utils.ParseFilters` allows them, `utils/utils.go:230-241`) stay in `Conditions` and are AND'ed. Today's removal is keyed by field name and drops them too (`jobs.go:494-512`, `:3420-3436`). So `job_id:in:(100,X[1])` matches job 100 or element `X[1]`.
- **Bare identity keys, one rule on both paths:** every value of a `job_id` filter entry whose operator is an identity operator (`eq`, `in`, `notin`, `ne`), and every value of a converted bracketed entry, goes through `ParseIdentitySelection`. Values of other operators (`between` bounds, `lt`/`lte`/`gt`/`gte` operands, string operators) are numbers or text, not identities, and stay on the plain path exactly as today when they carry no brackets. A written-form bare key (`c-a-ts-seq`) is normalised to its decimal value both on the plain `job_id in (...)` path and inside a converted filter, so the same key matches the same job whether or not a bracketed value shares its filter (5.6).
- **Operators:** the predicate honours the filter's operator instead of dropping it (bug-1898). It adds exactly one new rejection, for operator/value combinations that have no meaning with an element selection and were silently dropped until now: bracketed `job_id` values with comparison or null operators and bracketed `job_name` values with null operators (last bullets). They reuse today's "operator doesn't support" error:
- `eq` and `in` (both fields), and `between` on `job_id`: the row matches any of the values' selections. (`job_id:between` with bracketed values had no defined meaning and was dropped; it is evaluated like `in`. `job_name:between` is not in the string table and stays rejected, below.)
- `notin`: the row matches none of them.
- `job_name` with `contains`, `notcontains`, `startwith` or `endwith`: the name part is evaluated with today's condition semantics -- built with `utils.BuildFilterConditions` for that operator and evaluated with `store.MatchesCondition` (today `contains`/`startwith`/`endwith` all map to a case-insensitive substring `OpLIKE`, `utils/types.go:143-148`), so a bracketed and a plain value with the same operator match names identically -- and the row must also be an array element within the range (e.g. REST `job_name:startwith:foo[1]` selects element 1 of every array whose name matches `foo` under that operator). For `notcontains` the negation covers the name part only: the row is an array element within the range whose name does not contain the value. (`notin`, by contrast, negates whole selections, because its values are complete selections rather than one name pattern.) `job_name:between` stays rejected with today's "unsupported operator" error, as for plain values. xcecli's list path sends these operators for a leading and/or trailing `*` (`generateSearchRule`, `filters/job_filter.go:191-205`); for a bracketed `-J` value it checks the raw text first (CliSel `ParseNameSelection`, so `*` can only appear in the name part) and then takes the rule from, and strips `*` from, the name part only.
- `ne`: rejected with today's "operator doesn't support" error (`jobs.go:254-261` for `job_name`, `:357-364` for `job_id`), unchanged.
- Any operator not in the field type's existing mapping table (`IntOperatorMappingTable` for `job_id`, `StringOperatorMappingTable` for `job_name`) gets that table's existing "unsupported operator" error, as a plain value does today.
- `job_id` with `lt`, `lte`, `gt`, `gte`, `null`, `notnull` (in the int table, `utils/types.go:117-128`), and `job_name` with `null` or `notnull` (in the string table, but they ignore the value, `utils/types.go:382-388`), with a bracketed value: rejected with the existing "operator doesn't support" error -- the error class `ne` already gets today, not a new rejection category. These combinations had no meaning and were silently dropped (bug-1898); xcecli never sends them.
- Control by name (`getFilters`, `jobs.go:2965-2970`) always sends `eq` with the raw value, so `kill -J 'foo*[1]'` matches the literal name `foo*`. Wildcards are a list-only feature today, with or without brackets; that difference is pre-existing and out of scope.
- `SelBare` → `job_id == Key` (all elements and any ordinary job; 5.6).
- `SelSingle` / `SelRange` → `job_id == Key` and `IsArrayJob(unique_job_id)` and `array_idx` in `[Lo, Hi]` (5.2, 5.11, 8.2).
- Name selections → `job_name == Name` and `array_idx` in `[Lo, Hi]` with `IsArrayJob` (5.2). For the exact-match operators (`eq`, `in`, `notin`) an empty `Name` matches no row, even though unnamed jobs are stored with `job_name == ""` (`backend/convert.go:28-31`); so `kill -J '[1]'` (control always sends `eq`) acts on no job.
- No special case for element 0: `<array-id>[0]` / `[0-0]` get the identity predicate with `Lo = Hi = 0` (8.2, 5.11); `-J 'name[0]'` and `<array-id>[0-3]` (out of scope in the requirements) get the general name or range predicate, which matches array elements only.
- **5.17:** a decimal key with tag 1 and no `[...]` is rejected with "array elements must be named as `<array-id>[k]`". The written form never carries the tag.
- **Pre-store key checks:** `GetJobByIDs` (every key) and the `GetJobsInfo` `job_id` branch (values of identity operators, and every value of a converted bracketed entry including `job_id:between`, see Matching), including the `job_id in (...)` path, apply 5.14 and 5.17 to every such key, and 5.8 only to keys written with `[...]`, before querying the store. A bare ordinary job ID is therefore never rejected for not being array-capable (5.6).
#### ApiControl
- **Dispatch once (bug-1899):** `ControlJobByID` today dispatches the accumulated `successJobs` inside its per-selection loop (`jobs.go:2464`, `:2575-2617`), so earlier selections are re-sent for every later one. After validate-all-first and matching, the selection loop collects (a) the jobs to act on, de-duplicated by `unique_job_id`, and (b) the not-found maps, one per missing number per selection, in selection order (not de-duplicated; 5.3, 5.4). It does not dispatch inside the loop. After the loop, for each job in (a), in order, it calls `validateJobControl(job, action, signal)` once (signature unchanged, `jobs.go:3015`; it checks one row) and appends its accepted and failed items; it then appends every not-found map from (b). It then dispatches each accepted job once through `controlJob`; a dispatch error still returns 500 as today (`jobs.go:2611-2616`), it is not turned into a failed item.
- **`foundJob`:** becomes `foundJob(jobID int64, idx int, explicitIndex bool, rows []utils.OrderMap) ([]utils.OrderMap, error)` (today: `foundJob(int64, int, rows) ([]utils.OrderMap, error)`). When `explicitIndex` is true, only rows whose `unique_job_id` is an array element (`IsArrayJob`) with `array_idx == idx` match (5.4, 8.2). When it is false, today's whole-`job_id` match applies (5.6).
- **Not-found items:** a missing number in a `SINGLE` or `RANGE` selection yields one "job not found" map, whose `unique_job_id` is the would-be element ID (5.3, 5.4). Accepted and failed result maps already carry `unique_job_id` (`jobs.go:2575-2588`, `:2813-2826`).
- **Result items:** every result map therefore carries `unique_job_id` = the element's own (or would-be) ID, so clients render `<array-id>[k]`, including `[0]` (5.1, 8.3).
- **Most-recent rule (5.10):** applies only when the control filter contains no element number (`[...]`) and the all-jobs option is not set. With an element number, the command acts on exactly the selected elements (5.2), whatever the newest row is. Without one, when the last matching row has `array_idx == 0`, or is the only stored element of its array, act on it alone.
- **`isSingleElementArray(row)`:** counts the stored rows with that `job_id` that are array elements. The API server's store records no array size, and the controller pushes a one-element array as a single job, so the stored row count is the only available signal. Deliberate approximation: a multi-element array whose other elements have all been purged from the store is treated as one-element. The multi-element rule is out of scope (requirements), so this only affects arrays already reduced to one stored element.
#### ApiDispatchLog
- `dispatchJobArray` replaces its `zap.Uint64("base_jobid", baseJobID)` field (`handler.go:273`) with `zap.String("array_jobid", FormatJobID(int64(ExtractArrayBaseID(baseJobID))))` (5.9).
- `CreateJob`'s two log entries (`jobs.go:1434` "Generated job ID" and `:1515` "Job submission successfully"), for an array, log `array_jobid` (formatted) and `array_size` instead of the raw `jobID`; ordinary jobs log as today (5.9).
### xcecli
#### CliSel and CliSplit
```go
// Same grammar and order as ApiSel for identities (5.16, 5.14, 5.8, 5.7, 5.5).
// 5.8 and 5.7 apply to decimal keys with [...] too (tag 0, low 15 bits clear),
// so an upgraded client is protected against an un-upgraded server. Only 5.17
// (a marked decimal key without [k]) is left to the server.
func ParseSelection(raw string, allowRange bool) (Selection, error)
// Splits at commas outside [...] (as extractJobNames does).
func SplitSelections(list string) []string
// 5.16 / 5.7 / 5.5 for a raw -J value that contains '[' or ']', run before the
// wildcard mapping and '*' removal: the name part is zero or more characters,
// none whitespace, '[' or ']' ('*' allowed there only); after it exactly one
// [num] or [num-num] and nothing else. So -J 'foo[1]*' and -J 'foo[1*]' are
// rejected (5.16). A name part made only of '*' is accepted: -J '*[1]' is
// sent as job_name:contains:[1] (empty name part), i.e. element 1 of every
// array, as -J '*' already means every name today. For an accepted bracketed value the list path picks the
// search rule (generateSearchRule) from the name part alone and strips '*'
// from the name part only: -J 'foo*[1]' is sent as job_name:startwith:foo[1],
// -J '*foo*[1]' as job_name:contains:foo[1]. Values without brackets keep
// today's mapping. Used by list and control -J (control sends eq as today).
func ParseNameSelection(raw string) (Selection, error)
// "<decimal>" or "<decimal>[k]" or "<decimal>[a-b]" for the REST path.
func (s Selection) APIKey() string
// allowRange=false refuses a range ("a range of elements is accepted only by
// control commands") only after 5.16, 5.14, 5.8 and 5.7 have passed, so a
// query range breaking one of those reports that rule (5.12).
```
- **`ToAPIJobID(s, allowRange)`:** becomes `ParseSelection(s, allowRange).APIKey()`. Control passes `true`, queries pass `false`. Text that is neither a written ID nor a decimal key is rejected rather than passed through.
- **Callers:** `controlJobsByIDs` and `filters/job_filter.go` both validate every selection before any request. The error message quotes the trimmed selection.
#### CliSubmit and CliControl
- **Submit output:** xcecli prints the server's `message` after replacing `<jobid-decimal>` with `<FormatJobID(jobid)>`, as today. The count comes from the server message's `(n elements)` (3.2); the client does not append it again. `array_size` is mirrored in the client model for scripted use.
- **Control result lines:** `FormatJobID(UniqueJobID)` when it is non-zero, otherwise today's `FormatArrayChildID` (compatibility with an older server).
- **Help (9.1):** `name[1-N]`, `name[1]`, numbering starts at 1, the limit is set by the API server (default 1000, at most 32767).
- **Help (9.2):** kill, stop, resume and checkpoint show `<array-id>[k]` and `<array-id>[a-b]`.
## Data Models
### Domain Model
- **Job ID** (uint64) = tag (2) | cluster (6) | API server (4) | timestamp (32) | sequence (20).
- **Array ID** = tag 0, with the low 15 sequence bits 0.
- **Element ID** = tag 1 | array ID | k, with 1 <= k <= 32767 for new arrays and k = 0 only for arrays submitted before this change.
- **Invariants:**
- An array ID never equals an element ID.
- An array occupies element IDs `first .. first + n - 1` within one 32768 block.
- The controller accepts tag 0 (size 1) and tag 1 only.
### Data Contracts & Integration
- `JobSubmitResponse` (REST): `jobid` (int64, the array ID for arrays), `queue_name`, `message`, `array_size` (int, omitted for ordinary jobs).
- `JobControlResponseItem` (REST): adds `unique_job_id` (int64, omitempty).
- **Controller `OPCode_jobsub`:** wire format unchanged. New possible reply status `-E_XCE_ARRAY_START0`.
- **Persistence:** no field is added to, or removed from, any WAL record, snapshot or recording. The save/restore completeness review does not apply: the per-field round trip is untouched, and 1.10 and 8.4 rely on the existing `JOB_ARRAY_SUBMIT` apply, which stores `(first, size, submit)` and expands `first + i`.
## Error Handling
### Error Strategy
- **C:** negative errno or `-E_XCE_*` reply statuses. Harness messages are written to stderr, and the command returns non-zero. Worker WARN goes through `XCECD_WORKER_LOG`.
- **Go apiserver:** `ReturnError(ctx, common.INVALID_PARAMS_CODE, msg, nil); return` for 2.x, 5.x and 4xx-class rejections. ID allocation or submit failures use `common.ERROR_CODE` as today. Each rejection is logged at `Debug` with `zap.String("selection", raw)`. Startup clamping is logged at `Warn`.
- **xcecli:** a client-side rejection exits non-zero before any HTTP request.
### Error Categories and Responses
| Condition | Surface | Response |
|---|---|---|
| 2.1-2.8 | REST/CLI submit | 400 with the specific message |
| 5.5, 5.7, 5.8, 5.14, 5.16, 5.17 | REST/CLI query or control | 400; whole command rejected; no job acted on |
| 5.3, 5.4 | REST/CLI control | 200 with per-number "job not found" items |
| 7.1 | controller | `-E_XCE_ARRAY_START0` + controller ERROR log |
| 7.2-7.5 | controller | `-EINVAL` |
| 7.10 | controller | `-EEXIST` |
| 4.3, 4.4, 4.6, 4.8, 4.11, 5.7, 5.8, 5.14, 5.15 | harness | stderr message; non-zero; nothing submitted or acted on |
| identity range in a query (client-side, as today) | xcecli list | non-zero exit before any HTTP request |
| bracketed value with `ne`, a comparison or null operator, or an operator outside the field's mapping table | REST/CLI query or control | 400 with the existing "operator doesn't support" / "unsupported operator" message; whole command rejected |
### Monitoring
- Controller: one ERROR line per 7.1 rejection.
- Scheduler: an ERROR on a guarded expansion. Expected never; it would indicate a broken controller.
- API server: a startup WARN when the limit is clamped.
## Testing Strategy
### Shared conformance table `selection-cases`
One table of `(input, expected outcome)` rows is copied verbatim into `test_jobsel.c`, `controllers/jobsel_test.go` and `common/format/jobsel_test.go`. Each implementation runs the rows its surface supports.
| Group | Cases | Expected outcome |
|---|---|---|
| Valid | `X`, `X[1]`, `X[0]`, `X[01]`, `X[1-3]`, `X[3-3]`, `X[32767]` | accepted |
| Syntax (5.15/5.16) | `X [1]`, `X[a]`, `X[]`, `X[1`, `X]`, `X[+1]`, `X[-1]`, `X[ 1]`, `X[1-3:2]`, `X[1]x`, `X[1-2-3]` | rejected |
| Widths (5.14) | cluster 64, API server 16, timestamp 2^32, sequence 2^20 (e.g. `0-0-T-1048576[1]`) | rejected |
| Alignment (5.8) | `c-a-ts-5[3]` | rejected |
| Index (5.7) | `X[32768]`, `X[1-40000]`, `X[99999999999999999999]` | rejected |
| Reversed (5.5) | `X[3-1]` | rejected |
| Names (5.16; name surfaces only) | `foo[1]`; `[1]` (empty name: `eq`/`in` match no job, including unnamed jobs stored with `job_name ""`; `contains` matches every array); xcecli raw `-J 'foo*[1]'` (list sends `job_name:startwith:foo[1]`) and `-J '*[1]'` (list sends `job_name:contains:[1]`) | accepted |
| Names (5.16) | `foo [1]`, `my job[1]`, `foo[1]x`; xcecli raw `-J 'foo[1]*'`, `-J 'foo[1*]'` | rejected |
| Precedence (5.12) | `0-0-T-1048576[40000]` → 5.14, not 5.7; `c-a-ts-5[3-1]` → 5.8 | first failing check reported |
### Unit Tests
- **libxce:**
- `test_jobsel.c` runs the table.
- `parse_jobid` NULL safety and range rejection.
- `test_idgen.c`: `idgen_array` from a seed in `(31 * 32768, 2^20)` returns `-EOVERFLOW` with the context unchanged; a seed of exactly `31 * 32768` and seeds below `30 * 32768` succeed. Because `idgen` resets the sequence when the second changes (`idgen.c:36-41`), each seeded case sets `ctx->timestamp = time(NULL)` immediately before the call and retries if the second changed in between; `idgen_array` returns index 1; the last element has index n; n = 32767 is accepted and 0 or 32768 is rejected; block overflow returns `-EOVERFLOW` and leaves the context unchanged; `idgen` with size 2 returns `-EINVAL`.
- **Controller (`test_jobsubmit_validate.c`):**
- `handle_jobsubmit` with an index-0 tagged array and an empty queue is answered `-E_XCE_ARRAY_START0` and writes the 7.1 ERROR line (classification precedes the field checks, 7.9).
- `jobsubmit_classify` over all 7.1-7.5 combinations, including the 7.9 precedence, a size of 2^32+2 and `idx = 2, size = UINT64_MAX` (both rejected under 7.2, neither truncated nor wrapped).
- `handle_jobsubmit` with an existing array ID (an existing element `X|3` blocks a new `X|5`), giving 7.10.
- Two registered concurrent submissions with the same array ID; the second gets `-EEXIST` (7.11).
- Reply and release are atomic. Seam (project convention for test-only symbols, `testing.md`): under `#ifdef XCE_TESTING`, `job.c` exposes `void (*jobsubmit_test_reply_hook)(void)` -- called inside `inflight_reply_and_remove` after `xce_reply` and before the unlink, with the registry lock held -- plus `int jobsubmit_inflight_overlaps_for_test(uint64_t base, uint64_t size, int is_array, uint64_t array_id)` (takes the lock), `jobsubmit_inflight_overlaps_locked_for_test(...)` (same check, caller already holds the lock) and `int jobsubmit_inflight_trylock_for_test(void)` (returns `pthread_mutex_trylock`'s result, unlocking on success); declared under `#ifdef XCE_TESTING` as `cluster.h:110-118` does for other test-only symbols. (There is no `xce_reply` test hook in `src/lib`, `test_commit_barrier_reply.c:27`, and replies are read from `ep->outq` only after the handler returns, so neither order is observable without this seam.)
- Deterministic checks inside the hook (no second thread, since a thread would just block on the lock and pass even against a split implementation): (1) `jobsubmit_inflight_trylock_for_test()` returns `EBUSY` (the registry lock is held across the reply); (2) A's reply is already queued on the endpoint's `outq`; (3) `jobsubmit_inflight_overlaps_locked_for_test()` (lock-free variant) still finds A's entry. After `inflight_reply_and_remove` returns, `jobsubmit_inflight_overlaps_for_test()` finds no entry.
- A retry of the same `(base, size)` registered after A's reply is not removed by A's async-context release (`jobsubmit_async_ctx_release` no longer touches the registry).
- Once A has been answered with a failure, a same-array-ID submission B is accepted (7.7); while A is registered and unanswered, B gets `-EEXIST` (7.11).
- A size-0 submission is answered `-EINVAL` (7.4).
- **Apiserver:**
- `jobs_handler_test.go` (new, `//go:build !cgo`): every handler-level case (`CreateJob`, `GetJobsInfo`, `GetJobByIDs`, `ControlJobByID`, filter-based `ControlJob`). It is `!cgo` because the JWT/RBAC fixtures `signTestJWT`/`setupRBACTest` live in the `!cgo`-only `controllers/integration_test.go:54, :83`. It replaces the seams `generateJobID`, `generateArrayJobID`, `submitJob` and `controlJob` (the last records each dispatched `unique_job_id`, so "acts on" is observable), seeds `store.Jobs` (with a controllable `job_status`, since `validateJobControl` checks it, `jobs.go:3045`) and `store.Queues` for every seeded job's queue (the role's allowed queues come from `store.GetAllQueueNames()`, otherwise control returns 403 before dispatch; pattern `rbac_convention_test.go:86`), and sets `utils.JobYamlConfig = &utils.JobConfig{MaxArraySize: 1000}`, restoring every replaced global and store in `t.Cleanup`. Order matters: `setupRBACTest` calls `resetIntegrationState()` → `store.InitStores()` (`integration_test.go:33-36, :83-86`), which replaces `store.Jobs` and `store.Queues` with empty stores; so each test saves the original store pointers, calls `setupRBACTest`, then seeds, and restores the saved pointers in `t.Cleanup`. Every control test also asserts at least one expected dispatched `unique_job_id` (or, for the act-on-no-job cases, that the fixture rows are present and nothing was dispatched), so an empty store cannot make it pass.
- Submit: `arr[1-3]` leaves `req.JobName == "arr"`; one 2.x row returns 400 with the exact message; a valid ordinary submit is not rejected (typed-nil guard); `name[1]` goes through `generateArrayJobID` with n = 1 and the response has `array_size: 1`; plain names `my job`, `foo%5` and `foo%` get today's errors and 500 status; submit-time `%I` = 1 and `%J` = `<array-id>[1]` for a new array (9.4).
- Query: `GetJobsInfo?filter=job_id:eq:X[2]` returns only element 2 (bug-1898); `GET /jobs/X[1-3]` returns elements 1-3; element-only matching for `<ordinary-id>[0]` (5.11); with an unnamed (`job_name ""`) array element in the store, `job_name:eq:[1]` lists nothing while `job_name:contains:[1]` lists it; an `exec_hosts` filter matching no host together with `job_id:eq:X[40000]` returns 400 (5.7), not `count: 0`; `job_name:notnull:foo[1]` gets the existing "operator doesn't support" error; `job_name:notcontains:foo[1]` selects element 1 of arrays whose name does not contain `foo`; `{job_id:in:(X[1])},{job_id:ne:100}` keeps the `ne` entry; `job_id:in:(X[1]);job_name:eq:foo[2]` is an intersection; `job_id:in:(100,X[1])` matches both; `job_id:notin:(X[1])` excludes element `X[1]`; `job_name:startwith:foo[1]` selects element 1 of arrays named `foo...`; `job_id:ne:X[1]` keeps today's rejection; the same written-form bare key matches alone and mixed with a bracketed value.
- Control: two selections in one request (`X[1],Y[2]`) dispatch each element exactly once and list each once (bug-1899); `X[1-3]` dispatches to the existing elements and reports missing numbers as not-found items carrying `unique_job_id` (5.3); `X[0]` with no element 0 dispatches nothing and reports it not found (5.4); a pre-upgrade element 0 is dispatched alone (8.2); a request with a valid selection followed by an invalid one dispatches nothing (Terms); filter-based `-J 'foo[40000]'` returns 400 with the 5.7 message; filter-based `-J '[1]'` with an unnamed array in the store dispatches nothing; the most-recent rule with a one-element array dispatches that element alone (5.10); `-J 'foo[1]'` with two one-element arrays named `foo` dispatches both (5.2).
- `jobs_test.go` (untagged, pure functions only): `foundJob` with `explicitIndex` (`[0]`, 5.4, 8.2); the predicate builder for each operator rule in ApiMatch, evaluated over `store.Jobs`-shaped rows; `isSingleElementArray`.
- **xcecli:**
- `jobsel_test.go`: the table plus `SplitSelections("X[1,3],Y")`.
- `jobid_test.go`: passthrough cases `"123456"` and `"32768[3]"`; `"123456[3]"` rejected under 5.8; `ToAPIJobID("32768[1-3]", true)` passes through while `ToAPIJobID("32768[1-3]", false)` is refused.
- `cmd/jobs/controller_test.go` (new): `controlJobsByIDs` with `request.NewAPIClient(srv.URL)` pointed at an `httptest` server; `X[1-3]` reaches the request path as `<decimal>[1-3]` (guards `allowRange=true` at `controller.go:52`).
- **Harness:**
- `test_cmdutil.c`: `cmdutil_parse_array_size` (`1`, `01`, `0`, `-3`, `+5`, ` 5`, `5.0`, `0x5`, `abc`, `32767`, `32768`) and `cmdutil_job_arg` (ranges refused including `X[3-1]` with the 4.8 message, `[0]` accepted).
- `test_autovars.c`: `$jobid[3]`, `$jobid[3]x`, unset `$jobid`, `$?`, `$jobcount` (set and unset), and `!` / `#` lines.
- **Worker (`test_exdata_jobindex.c`):** element index, ordinary 0, a user `XCE_JOBINDEX` ignored with a WARN, `XCE_JOBID` unchanged.
- **Scheduler (`test_driver_translate.c`):** tagged `block|1` arrays expand to indices 1..n; untagged arrays, overflowing arrays and `idx = 2, size = UINT64_MAX` are not expanded.
### Integration Tests
- **Controller suite:** existing commit-barrier tests pass with tagged first IDs (1.10 and 8.4 rely on the existing WAL replay tests, which still pass).
- **Apiserver:** CGO `--suite apiserver` submit through `GenerateArrayJobID` (real libxce), asserting the response `jobid` equals `array_jobid(first)` and `array_size`.
- **Harness pty (`test_test_cli.c`):** `jobsub -a 0` and `jobsub -a 40000` are refused; `jobkill -j 1-1-1-0[1-3]` is refused with the 4.8 message before any RPC.
### E2E Tests
- **`/xce-test-multihost`:** `jobsub -a 2` elements `[1]` and `[2]` reach DONE on both hosts.
- **`/xce-test-ha`:** a 6-round soak verifies elements 1..N survive failover (1.10).
- **Manual REST flow** (recorded in validation):
- `xcecli jobs submit -J 'arr[1-3]'` lists `[1]..[3]`.
- `xcecli jobs kill X[2]` acts on element 2 only.
- A job script prints `XCE_JOBINDEX`.
## Migration Strategy
- **Recommended order for clusters that use arrays:**
1. Upgrade the API servers and the harness. An un-upgraded controller accepts their `array-id | 1` submissions unchanged (it does not validate tags), and those arrays are numbered from 1 (Adjacent expectations).
2. Upgrade the controllers.
3. Upgrade the workers, schedulers and xcecli in any order. The scheduler's expansion guard accepts arrays from un-upgraded clients (tagged, starting at index 0).
- **Controllers first is also supported:** until the API servers and harness are upgraded, their multi-element array submissions are rejected with `E_XCE_ARRAY_START0`, and the controller log explains why (7.1). Ordinary jobs are unaffected. Operators choosing this order accept that rejection window.
- **No data migration:** pre-upgrade arrays keep element 0 (8.1-8.4).

215
.kiro/specs/job-array-one-based-index/requirements.md Normal file
View file

@ -0,0 +1,215 @@
# Requirements Document
## Project Description (Input)
**Who has the problem.** Users who submit job arrays through `xcecli`/REST (`-J 'name[...]'`) or the test harness (`xce_test jobsub -a N`), and anyone porting LSF job-array scripts: those scripts read the element index (`LSB_JOBINDEX`, `%I`) and expect indices that start at 1.
**Current situation.** XCE silently ignores the indices the user writes. The apiserver parses `arr[1-3]` or `arr[1,5,9]` correctly, but sends only the element count to the controller, and the controller numbers elements 0..N-1 -- so both become `[0]`, `[1]`, `[2]` (bug-1896). The harness's `jobsub -a N` also produces 0..N-1. The worker sets `XCE_JOBINDEX` to a hardcoded 0 for every job, so a running element cannot learn its index. LSF defines array indices as positive integers (index 0 is invalid) and sets `LSB_JOBINDEX` to 0 only for non-array jobs (LSF 10.1 Admin Guide, "Chunk Jobs and Job Arrays", p.519).
**What should change.** Array elements are numbered 1..N, as in LSF, everywhere they are created, shown, parsed and exposed to the running job; anything this spec does not support is rejected with a clear message instead of being silently renumbered.
### Decisions already made (with the user, 2026-09-30)
1. **Option B -- the index lives in the job ID.** A job ID stays one `uint64_t`; an element's index is the existing 15-bit array-index field. The first element's ID is submitted as `base | 1`, and the existing contiguous expansion (`first_jobid + i`) in the controller and scheduler yields elements 1..N. No new start field and no wire, WAL or snapshot format change. `fmt_jobid`, `parse_jobid`, `array_jobid` and `array_idx` keep their signatures.
2. **Start is fixed at 1 in this spec.** Only `name[1-N]` is accepted. Any other start (`[1001-2000]`, a single `[n]` with n > 1), index 0, comma lists and steps are rejected with a clear message. Start > 1 is deferred to a future spec; this spec is a strict subset of LSF behavior so that extension will not change anything accepted here.
3. **The submit reply identifies the whole array** (`array_jobid()`, as LSF's `Job <123> is submitted`), not the first element; the harness's `$jobid` after `jobsub -a N` likewise names the array. (Round 3: the controller keeps replying with the first element's ID -- the apiserver's `clib.go:653-657` and `RecordAcceptedJob` rely on it -- and the clients derive the array identity with `array_jobid()` for what they show.)
4. **A one-element array `name[1-1]` is accepted** (as in LSF) and behaves like any other array.
5. **Index 0 is rejected only at submission.** Arrays submitted before this change keep their element `[0]`, which stays selectable and controllable.
6. **`XCE_JOBINDEX` is set by the system and cannot be overridden** by a user-supplied environment variable of the same name.
Resolved during requirements review (round 1, 2026-09-30), following the decisions above:
7. **The cluster rejects a new array whose first element number is 0**, and accepts any first element number from 1 to 32767, so a future start > 1 needs no cluster-side change; "start = 1" is enforced where users submit (command-line client, REST, harness). Consequence for a rolling upgrade: an un-upgraded client's array submissions are rejected with an error until it is upgraded; ordinary jobs are unaffected. An un-upgraded command-line client is not protected by 5.7, 5.8 and 5.14: selections it resolves before sending them to the REST API reach the API server as ordinary job numbers, until the client is upgraded.
8. **Whole-array control by the array ID stays a command-line-client/REST feature** (the apiserver fans it out; the controller and harness act on exact IDs only). The harness's job commands refuse a bare array ID with a message pointing to `<array-id>[k]`.
9. **Pre-upgrade element `[0]` is covered for single-element references** (`<array-id>[0]`), including listing and control results; ranges or lists that include 0 stay with the separately tracked duplicate-signal bug.
10. **Syntax details:** bounds are plain decimal digits (no sign); whitespace inside the brackets is rejected; leading zeros are accepted as the same number (LSF reads indices as numbers); interactive jobs cannot be arrays (as in LSF). Harness `jobsub -a 1` stays an ordinary job; `-a 0`, negative or non-numeric values are refused.
### Boundary (from a four-area investigation, 2026-09-30)
- **libxce:** `lib/idgen.c` (a size-1 array currently gets a plain untagged ID; bound size so that indices 1..N fit in 15 bits, i.e. N <= 32767); `lib/utils.c` `parse_jobid` and `include/config.h` `REGEX_JOBARRAY_ID` (bound the index to 32767, reject a misaligned array block, replace the unchecked `atoi` -- and must reject any ID field wider than its bit width (cluster <= 63, API server <= 15, timestamp < 2^32, sequence < 2^20), since `MAKE_JOBID` ORs fields unmasked (`include/utils.h:20-24`) and e.g. `0-0-T-1048576[1]` would otherwise resolve to `0-0-(T+1)-0[1]`).
- **Controller:** `controller/job.c` `handle_jobsubmit` (~:2461) needs pre-commit validation -- array tag set, size >= 1, `array_idx(jobid) + size - 1 <= 32767` -- because the controller trusts the client today; it also needs a check that no existing array already has the submission's array identity (7.7, 7.10), since today's duplicate check compares element identities only (`jobid_range_exists` `job.c:214-229`); the in-flight submission registry (`inflight_range_overlaps_locked`/`inflight_range_try_add`, `job.c:117-158`, bug-1419) compares element ranges only and must also treat a pending array submission with the same array identity as an overlap, or two concurrent submissions such as `X|1` (2 elements) and `X|5` (1 element) are both accepted; today the job-table check (`job.c:2463`) runs before registration (`job.c:2482`), which leaves a window in which a submission applied and answered between the two steps is seen by neither, so the job-table checks (element and array identity) must run after the submission is registered (and release it on a hit) — a submission that commits in time is in the job table before it leaves the registry (`persist.c:~5930` apply, then `on_applied` → `jobsubmit_apply_complete` → release at `job.c:2321`); one answered with a commit failure or timeout leaves the registry at `job.c:2356`, and any later apply of it is the out-of-scope bug-1419 class; the array-identity check is bounded work (up to 32768 element keys of the block, element 0 included for pre-upgrade arrays); the controller's submit reply keeps carrying the first element's ID (~:2284), and the clients derive the array identity from it.
- **Scheduler:** `scheduler/src/driver.c` :477-478 overflow/tag guard on expansion; :509-512 the whole-array INFO log must print `array_jobid()` instead of the first element's ID.
- **Worker:** `worker/src/xcecd_exdata.c` :419-420 `XCE_JOBINDEX` becomes `array_idx(jobid)` (0 for non-array jobs); user environment variables must not override it (:431-457) -- only `XCE_JOBINDEX` becomes non-overridable; a user-supplied `XCE_JOBID` keeps winning as today (:408, :431-436; out of scope); `XCE_JOBID` stays the raw element ID (`xrun.c:70` parses it).
- **Apiserver:** `controllers/jobs.go` `createArrayName` (:1824-1979: accept only `[1-N]`, explicit rejection messages, cap at 32767); `CreateJob` sizing and ID (:1419-1434, :1454-1457: always generate an array ID, submit `base + 1`); the submit response (:1517-1520: report the array); `utils/config.go:214-225` and `config/app.yaml:71` (`max_array_size` capped at 32767); `parseArrayIndexSection` (:3290-3291: allow `[k-k]`, fix the reversed error text).
- **xcecli:** help text (`cmd/jobs/submit.go:626-684`, `controller.go:133-240`, including checkpoint's help at :236-240, `list.go:329` vs :360); `common/format/jobid.go` `ParseJobID`/`ToAPIJobID` (index unbounded: `X[40000]` resolves into the next array block; no alignment check: `ParseJobID` adds `idJob+idIdx` (:102) and `ToAPIJobID` clears the low 15 bits (:141, via `ExtractBaseJobID` :116-118), so a misaligned `c-a-ts-5[3]` becomes element 8 of another array before REST ever sees it -- both must reject an index above 32767 and an `<array-id>` whose sequence has any low 15 bit set -- and must reject any ID field wider than its bit width (cluster <= 63, API server <= 15, timestamp < 2^32, sequence < 2^20), since `MAKE_JOBID` ORs fields unmasked (`include/utils.h:20-24`) and e.g. `0-0-T-1048576[1]` would otherwise resolve to `0-0-(T+1)-0[1]`) and `FormatArrayChildID` (:124-127, prints the bare array ID for index 0); `cmd/jobs/list.go:257` (`array_idx:ne:0`) is dead code -- its `-W` flag registration is commented out (`list.go:360`, used at :90) -- so it hides nothing today and does not affect 8.3; interactive submit waits on `jobResp.Data.JobID` (`submit.go:505-507`). The ID-list splits at `cmd/jobs/controller.go:114` and `filters/job_filter.go:43` cut at every comma, even inside `[...]`; they must split only at commas outside brackets, as the API server's `extractJobNames` (`jobs.go:1043-1068`) does, so that a rejected selection such as `X[1,3]` is reported as the user wrote it (Terms, 5.12, 5.16).
- **Name-based element selection:** `xcecli jobs list -J` (`list.go:355`) and kill/stop/resume `-J` (`controller.go:154,190,225`) reach the REST `job_name` filter, which maps `name[a-b]` to `array_idx between (a,b)` (`jobs.go:265-300`, `:3362-3390`) -- the same selection rules apply.
- **"Most recent job" control:** the filter-based `ControlJob` (`jobs.go:2730-2830`) acts on only the last matching job when `array_idx == 0` and on all matching jobs otherwise (`:2753-2757`); "last" is by `unique_job_id` ascending (`FetchJobList` sort, `:3458`), not submit time. `foo[1]` used to create an ordinary job and now creates a one-element array element, so this rule must keep acting on that element alone (5.10).
- **Explicit element 0 in `ControlJobByID`:** the query already narrows a single `<array-id>[0]` to rows with index 0 (`jobs.go:2380-2409`), but `foundJob`'s index-0 branch (`jobs.go:3111-3127`) matches every row with that `job_id`, so when the same request also brings in other rows of that array (e.g. bare `X` plus `X[0]`, or `X[0]` plus `X[2]`) it acts on the whole array; that branch must fire only for a selection with no `[...]` (5.4, 8.2).
- **Reversed-range message:** every caller of `parseArrayIndexSection` replaces its error with `"bad job array index: ..."` (`jobs.go:300, :407, :844, :2392, :3389`), so the fix must pass the specific message through (5.5).
- **Element-only matching:** an ordinary job is stored with `job_id` = its full ID and `array_idx` 0 (`backend/convert_pure.go:50-52`, `jobs.go:1455-1456`), so `<ordinary-id>[0]` matches it on REST (`jobs.go:2380-2409`); whenever a selection has `[...]`, `GetJobByIDs`, `GetJobsInfo`'s `job_id` branch (`jobs.go:366-420`, the path `xcecli jobs list <id>[...]` uses via `xcecli/filters/job_filter.go:55`), `ControlJobByID` and `foundJob`'s explicit-index path must match only array elements (e.g. `IsArrayJob(unique_job_id)`) (5.2, 5.4, 5.11).
- **REST identity alignment:** `ControlJobByID` accepts a numeric or `ParseHumanJobID` key (`jobs.go:2369-2384`; `backend/convert_pure.go:120-133`); `GetJobByIDs` (`jobs.go:799`) and `GetJobsInfo`'s `job_id` branch (`jobs.go:377`) accept numeric keys only; a `job_id` filter value without both `[` and `]` never reaches `:377` — it `continue`s at `jobs.go:354-356` into the generic `job_id in (...)` filter, which is the path `xcecli jobs list <decimal>` takes (`filters/job_filter.go:49-58`), so the checks below must also cover every key in that branch before the `continue`. All three pass the key to `job_id eq` without an array-alignment check, so the 5.8 check (and, for `ControlJobByID`'s written form, the field-width check) is needed there, and all three must also reject a numeric key that carries the array marking and has no `[...]` (5.17).
- **Job-name rule:** the part before `[` must satisfy the existing `isValidArrayName` rule (at most 4094 characters, `^[a-zA-Z0-9._-]+$`, `jobs.go:1981-1987`); the apiserver already enforces it (`jobs.go:1568-1572`).
- **Controller rejection text:** error replies carry only a status (`job.c:2451-2454`) and older clients print only an rc or errno text (`lib/client.c:352-355`, `clib.go:629-631`, `test.c:3825-3835`), so the "client must be upgraded" explanation goes in a distinct error status plus a controller log entry.
- **Apiserver parsers and logs:** the element-number bound for REST selections belongs in `controllers/jobs.go` `parseArrayName`/`parseArrayIndexSection` (they strip and parse `[...]`; `ParseHumanJobID` never sees the index); those parsers are reached only when a value contains both `[` and `]` (`job_name` in `GetJobsInfo` `jobs.go:250-252` and `FetchJobList` `jobs.go:3356-3358`, the latter reached by filter-based `ControlJob`; `job_id` at `jobs.go:354`), so a name or identity containing only one of `[` or `]` falls through to a plain match today and the 5.16 syntax check must run on any value containing `[` or `]` before that fallback; `backend/handler.go:272-275` `dispatchJobArray` logs the raw first-element ID where it means the array.
- **One-element arrays vs ordinary jobs:** `idgen(..., 1, ...)` is how ordinary jobs get IDs (harness `jobsub -a 1`, `RestartJobByID` via `GenerateJobID(...,1)` at `jobs.go:1185`) and must keep returning an untagged ID; a one-element array needs a distinct request. C `array_idx`/`array_jobid` (any non-zero tag, `lib/utils.c:694,705`), C `fmt_jobid` and xcecli `IsArrayJob`/`FormatJobID` (tag & 1, `lib/utils.c:591`, `xcecli/common/format/jobid.go:46,60-62`) and apiserver `IsArrayJob`/`FormatJobID` (tag == 1, `backend/convert_pure.go:78-80,88`) disagree on what marks an array; the cluster must accept only tag value 1 as an array.
- **Harness:** `substitute_autovars` (`test.c:1283-1360`) expands only a whole token named exactly `jobid` (`:1306-1308`, `:1315`) and copies `$jobid[3]` verbatim (`:1268-1272`, `:1344-1350`), so it must also expand `$jobid` followed by a `[...]` suffix, keeping the suffix, and fail as for an unset `$jobid` when it is unset (4.10, 4.11); single-job commands look up exact IDs only and ignore `parse_jobid`'s return code (`jobkill`, `jobinfo`, `jobckpt` :4276, `jobrun` :3517, `taskrun` :4921, `setpend` :5073, `jobstart` :5268, `jobfinish` :5377, `jobfail` :5481, and others at :3327, :4140), so a rejected ID silently becomes job 0; `expect-job` compares the text against the job list (`cmd_expect_job` :2787-2817 → `joblist_query` `fmt_jobid` + `strcmp`), so it must instead parse its argument with the bounded `parse_jobid` and match the job number exactly, which makes it a harness command that parses job identities (Terms); `-a` is read with `atoi` and silently turns 0/negative/non-numeric into a plain job (`test.c:3670-3674`). `src/harness/test.c` `jobsub -a N` yields elements 1..N (:3721 submit `base | 1` and check the `idgen` return code; :3670-3675 cap `-a` at 32767; :3840 the ack comparison must match the controller's reply; :3866 `$jobid` = the array ID; help text :556-562, :603, :3680).
- **Skill scripts and docs:** `.claude/skills/xce-test-ha/run-ha-soak.sh` (:2601, :2731-2798, :2808-2815, :2975-2976 verify elements 0..N-1) and `.claude/skills/xce-test-multihost/run-multihost-verify.sh:792` (verifies element 0), plus both `SKILL.md` files.
- **Tests:** `controller/testsuite/test_commit_barrier_rollout.c:2323-2370` (32768 -> 32767 elements); `controller/testsuite/test_commit_barrier_reply.c` (`cbrp_arrange_jobsubmit` :1025-1040, tests at :1640ff and :1875ff) submits untagged index-0 arrays straight to `handle_jobsubmit`, which the new validation (7.1, 7.3) rejects -- its IDs must become array-tagged `base|1`; new tests for `createArrayName`/`CreateJob` sizing, harness `-a` bounds and `$jobid`, controller submit validation, and `parse_jobid` bounds. Existing `base + i` expansion tests stay valid under option B.
- **Already index-agnostic (no change expected):** WAL `JOB_ARRAY_SUBMIT` append/apply/replay, standby apply, HA/raft replication, snapshots, controller-to-scheduler/apiserver pushes, scheduler snapshot/replay/REPL formatter (expansion arithmetic apart from the guard in the Scheduler bullet), `fmt_jobid` and the Go `FormatJobID`, the apiserver's `array_idx between` filter construction itself, `RecordAcceptedJob`.
- **Docs:** the XCE user documentation is `src/docs/xce.md` (Requirement 9.4); this spec supersedes `job-array-single-wal-record` Requirement 5 and the `$jobid`/`expect-job` definitions in `xce-test-e2e-primitives/requirements.md:43-44, :66-67, :83` (which make `$jobid` the acknowledged element ID) and `design.md:77` (whole-token-only substitution, relaxed by 4.10) (no client-visible identity change; indices `[0, size)`); update `.kiro/specs/controller-job-cleanup/brief.md:86-87` and `.kiro/specs/job-dependency/brief.md:45-46`; add a roadmap entry.
### Out of scope (future spec or separate bugs)
- Start > 1, comma lists and steps (rejected with a clear message in this spec).
- Setting LSF's own `LSB_JOBINDEX`/`LSB_JOBID` variables (scripts read `XCE_JOBINDEX`).
- Whole-array control by the bare array ID through the harness or directly at the controller.
- A user-supplied `XCE_JOBID` overriding the system value (e.g. `xcecli jobs submit --env` from inside a running job copies the parent's `XCE_JOBID`, and `xrun.c:70` then starts remote tasks for the parent job) -- a separate defect to log.
- Missing-number reporting for query (as opposed to control) commands (`GetJobByIDs` returns only matching rows).
- The array statistics view (`xce jobs list -A`), which is empty because `store.ArrayJobs` is never written.
- Job output / peek by identity (`GET /jobs/:jobids/output`, `xcecli jobs peek`; `controllers/output.go:224-244` `parseJobIDParam` ignores `[k]` and looks up the raw key): element peek works today only with the element's decimal ID and is left as is; 5.2, 5.7, 5.8, 5.14, 5.16 and 5.17 do not apply to it.
- Job restart by identity (`POST /jobs/:jobids/restart`, `xcecli jobs restart`; `jobs.go:1074-1175`): left as is; 5.2, 5.6, 5.7, 5.8, 5.14, 5.16 and 5.17 do not apply to it.
- The scheduler REPL's `info job` identity argument (`scheduler/src/repl_info.c:114-117`, falls back to `strtoull` and reports job 0) -- a developer diagnostic, not a user surface.
- Selecting a range of elements by identity in queries (`xcecli jobs list <array-id>[a-b]`): rejected by the client today (`xcecli/filters/job_filter.go:175-187` `isValidJobId`; `ToAPIJobID` passes ranges through and `GetJobsInfo`'s `job_id` branch requires a numeric key, `jobs.go:~372`).
- Log entries that print a job ID as a raw number rather than in its written form (e.g. `controller/resync.c:591`, `controller/job.c:3250`, `scheduler/src/core_events.c:314`, `snapshot_load.c:341,398`, `snapshot_wire.c:185`, `repl_info.c:139`).
- Missing-number reporting for name-based (`-J 'name[...]'`) control selections: the filter-based `ControlJob` returns only matching rows (`jobs.go:2734-2748`).
- Selecting element 0 by job name (`-J 'name[0]'`, `-J 'name[0-0]'`), including its also matching ordinary jobs of that name (stored with `array_idx` 0, `jobs.go:1456`, `:3380-3395`).
- Selections that combine element 0 with other element numbers (e.g. `<array-id>[0-3]`), for new and pre-upgrade arrays alike.
- The "most recent job" rule widening to all matching jobs when the latest job is an element of a multi-element array (`jobs.go:2753-2757`); only the one-element case this feature creates is covered (5.10).
- The "most recent job" rule ordering by job identity rather than submit time, so any array element (tagged ID) outranks a newer ordinary job (`jobs.go:3458`).
- `job.max_array_size` values that are not whole numbers: `viper.GetInt` silently turns them into 0 (then 1000) or truncates them (`utils/config.go:217`) -- a separate config-parsing defect.
- The `%N` slot limit (parsed, then silently dropped today -- reject it for now).
- Per-element `%I`/`%J` expansion: blocked on the `out_file`/`err_file` decision (bug-1508) -- those fields never reach the controller or worker today. The apiserver's existing substitution is left as it is: at submit time `processJobRequest` (`jobs.go:2065-2096`, `%J`/`%EJ` from `FormatJobID(primaryJobID)`, `:1434-1435`) and `postJobRequest` (`jobs.go:2217-2234`, `%I` from the first element's `ArrayIdx`) substitute once for the whole array, and `resolveOutputPath` (`output.go:209-221`) substitutes per element when serving output. For a new array the submit-time values therefore become the first element's (`1` / `<array-id>[1]`) instead of `0` / `<array-id>[0]`. (LSF expands `%I`/`%J` at execution time; `%J` is the array ID.)
- Pre-existing problems found during the investigation, to be logged separately: `preCheckJobRequest` returning early and skipping validation for named jobs (`jobs.go:1578`), element restart and peek broken, `store.ArrayJobs` never written, duplicate signal on `X[0],X[3]`, the `transfer_files` key mismatch, `idgen` sequence overflow; duplicate identities from a submission whose commit timed out or that an earlier primary admitted (the in-flight registry forgets it before it is applied, bug-1419 class), and a resent submission rejected as a duplicate being reported to the client as success (`lib/client_failover.c:51-53`); control commands silently ignoring extra operands after the first (`xcecli/cmd/jobs/controller.go:108-116`, harness `cmd_jobkill` getopt loop `test.c:4121-4151` and the other single-job commands).
Touches the controller submit path and job-identity semantics, so it follows the full three-phase workflow with human review.
## Introduction
XCE lets users submit a job array: one submission that creates many element jobs sharing a command, each distinguished by an element number. XCE follows LSF's `-J 'name[...]'` syntax, but today it discards the numbers the user writes and numbers every array's elements from 0, and a running element cannot learn its own number. This feature numbers array elements from 1, as in LSF, consistently for every way of submitting, referring to, displaying and running them. It supports exactly one form, `name[1-N]` (and its one-element spelling `name[1]`); every other index form is rejected with an explanation rather than silently renumbered, so that later support for other starting numbers, lists or steps can be added without changing anything this feature accepts.
Terms used below:
- An **array identity** names a whole array; an **element identity** is an array identity followed by an element number in brackets, `<array-id>[k]`. An array identity never equals any element's identity, including element 0 of an array submitted before this change: every element carries the array marking and the array identity does not, so an array identity is never the identity of any array element. (An ordinary job may happen to share that number; 5.2, 5.4, 5.6 and 5.11 cover that case.)
- A **harness command that acts on one job** is any test-harness command that takes one job identity, or one per repeated `-j` (`jobinfo`), including `expect-job`; every such command parses job identities, so 4.7-4.9, 5.7, 5.8 and 5.13-5.15 apply to it. For `jobinfo`, each `-j` is judged on its own, as today: a rejection or refusal under 4.7, 4.8, 5.7, 5.8, 5.14 or 5.15 applies to that `-j` only ("act on no job" means no job for that argument), and the other `-j` arguments are still looked up. For `expect-job`, "act on the job" means the assertion passes, and "report not found", "refuse", "reject" and "act on no job" mean the assertion fails and the harness records the run as failed.
- An **array submission still being accepted**, relative to a submission being judged, is one that declares itself an array, that the current primary controller found to break none of 7.1-7.5, whose collision check it began before it began the collision check of the submission being judged, and that it had not yet answered (success or failure) to its client at the moment the collision check of the submission being judged began; whether it is answered later does not change this. Of two such submissions, the one whose collision check began first therefore blocks the other, even if it later fails (for example a failed submission check or a commit failure); the blocked one is not retried. A submission already answered with a commit timeout, or admitted by an earlier primary before a failover, is not one; collisions with such submissions are the separately tracked defect listed under Out of scope.
- A submission **declares itself an array** when its identity carries the "array" marking (see 7.5); a submission with any other non-ordinary marking is neither an ordinary job nor an array.
- An **index expression** is the job-name form defined in 2.2.
- An array **submitted before this change** is any array accepted by a controller without this feature.
- When one command is given several identities or element selections, each is judged on its own under Requirements 4, 5 and 8, and the command acts on the union of the jobs they select; every "only", "exactly", "never", "no job", "list no job" and "not the whole array" clause in Requirements 4, 5 and 8 (e.g. 5.2, 5.3, 5.4, 5.11, 5.13, 8.2) limits only what that one selection contributes (no job *for that selection*), and the other selections in the same command are still acted on -- except that, in a query or control command, a selection rejected under 5.5, 5.7, 5.8, 5.14, 5.16 or 5.17 makes "act on no job" apply to the whole command (for `jobinfo`, the Terms bullet on harness commands that act on one job applies instead), as the Terms bullet beginning "If any identity or element selection in a query or control command is rejected" states. Whether an element selected more than once receives the action once is out of scope (duplicate-signal defect).
- A **query command** lists jobs or shows job details: `xcecli jobs list` and REST `GET /jobs` and `GET /jobs/:jobids`. Job output (`xcecli jobs peek`, `GET /jobs/:jobids/output`) and job restart are not query commands here.
- If any identity or element selection in a query or control command is rejected under 5.5, 5.7, 5.8, 5.14, 5.16 or 5.17, XCE rejects the whole command, reports the rejection of at least one rejected selection, naming that selection (with the rejection chosen for it by 5.12; reporting the others as well is allowed but not required, as today's command-line client and API server stop at the first error), and acts on no job; for such a command XCE reports no "not found" result (5.3 and 5.4 do not apply to its other selections); "not found" results (5.3, 5.4) are reported per selection, and a selection under 5.11 contributes no job without a report; in both cases the command's other selections are still acted on.
- A **control command** is kill, stop, resume or checkpoint (`xcecli jobs kill|stop|resume|checkpoint`, REST `/jobs/control` and `/jobs/:jobids/control`, and the harness commands that kill, stop, resume or checkpoint a job, such as `jobkill` and `jobckpt`); `jobinfo`, `expect-job` and job restart are not control commands here.
- An **element selection** refers to array elements either by identity (`<array-id>[...]`, the elements of that one array) or by job name (`name[...]`, e.g. `-J 'name[2-5]'` on query and control commands: the elements with those numbers in every array with that name).
## Boundary Context
- **In scope**: submitting arrays as `name[1-N]` or `name[1]` through the command-line client and REST API; the test harness's `jobsub -a N`; rejection of every other index form, of malformed index expressions and of interactive array submissions, each with a specific message; the identity reported for a submitted array; how elements are shown, selected and controlled; the element number a running element sees in its environment, and that the submission's own environment cannot override it; the cluster refusing array submissions it cannot represent and accepting any start from 1 to 32767; continued handling of arrays submitted before this change; help text and user documentation of the supported syntax and its differences from LSF.
- **Out of scope**: *support for* starting numbers other than 1 at the command-line client, REST API and harness, comma-separated index lists, index steps and per-array running limits (`%N`) -- these are rejected (Requirement 2), not implemented; per-element `%I`/`%J` substitution in file names (depends on per-job input/output file support, which XCE does not provide today; the existing submit-time substitution is unchanged and, for a new array, uses the first element's values (`%I` = 1, `%J` = `<array-id>[1]`) for every element); setting LSF's own `LSB_JOBINDEX`/`LSB_JOBID` variables; whole-array control by the bare array identity through the test harness or directly at the cluster; maximum-array-size settings that are not whole numbers; selections that combine element 0 with other elements (except that they are still rejected under 5.5, 5.7, 5.8 and 5.14-5.16, and refused by the test harness under 4.8); reporting of missing element numbers by query commands and by name-based control selections; selecting a range of elements by identity in queries (except that such selections are still rejected under 5.7, 5.8, 5.14 and 5.16); the scheduler REPL's job-identity argument; job output (peek) and job restart by identity; selecting element 0 by job name; log entries that print raw job numbers; the "most recent job" control rule for multi-element arrays and its ordering by job identity rather than submit time; the array statistics view; a user-supplied `XCE_JOBID` overriding the system value; extra operands after the first to single-job control commands (e.g. unquoted `kill X [1]`, which acts on `X` today because the extra `[1]` operand is ignored: `xcecli/cmd/jobs/controller.go:108-116`, harness getopt loops such as `test.c:4121-4151`); identity collisions with a submission that is no longer an array submission still being accepted (see Terms). Other defects found while scoping this feature are tracked on their own and are not changed here.
- **Adjacent expectations**: the stored format of jobs, their replication between controllers and their recovery after restart are unchanged and are relied on as they are. During a rolling upgrade, once the controller is upgraded, multi-element array submissions from an un-upgraded API server or test harness are rejected with an error status, and the controller log explains that the client must be upgraded (Requirement 7); one-element submissions written `name[k]` through an un-upgraded API server still create an ordinary job, and `name[k-k]` is still rejected, as before, until it is upgraded; ordinary jobs are unaffected. Arrays that a not-yet-upgraded controller accepts are treated as arrays submitted before this change (Requirement 8); they are numbered from 0 when an un-upgraded client submitted them and from 1 when an upgraded client did. An element that starts on a worker not yet upgraded sees `XCE_JOBINDEX` = 0 (or a user-supplied value) until that worker is upgraded; 6.1-6.4 apply to upgraded workers. Until the API server is upgraded, query and control commands sent through it by an un-upgraded command-line client or directly by a REST caller behave as before this feature; once the API server is upgraded, 5.2-5.8, 5.10-5.12, 5.14, 5.16, 5.17 and 8.2 apply to them, except that for an un-upgraded command-line client the checks it resolves itself (5.7, 5.8, 5.14) are not enforced, as stated below. An upgraded command-line client applies its own checks (5.7, 5.8, 5.14) and its element display whatever the API server's version; the other criteria for its commands apply once the API server is upgraded. An un-upgraded command-line client is not bound by 5.7, 5.8 and 5.14: an element selection it resolves itself (e.g. `X[40000]`, a misaligned `<array-id>[k]`) can reach the API server as a valid selection of another array until the client is upgraded. This feature replaces the earlier commitment in `job-array-single-wal-record` (Requirement 5) that array element identities would not change for clients and would always be numbered from 0, and the `xce-test-e2e-primitives` definition of the harness's `$jobid` after an array submission.
## Requirements
### Requirement 1: Submitting a job array numbered from 1
**Objective:** As a user submitting a job array, I want `name[1-N]` to create elements numbered 1 through N, so that the element numbers match what I wrote and what LSF would create.
#### Acceptance Criteria
1. When a user submits through the command-line client or REST API a non-interactive job whose name is an index expression (2.2) with exactly one item, `1-N`, and no `%` suffix, where 1 <= N <= the array size limit (both inclusive), XCE shall create one array with exactly N elements, numbered 1 through N.
2. When a user submits through the command-line client or REST API a non-interactive job whose name is an index expression with exactly one item, `1`, and no `%` suffix, XCE shall create an array with one element numbered 1.
3. When XCE creates an array, XCE shall give each of its elements a distinct element identity.
4. When a user submits a job through the command-line client or REST API whose name contains no `[` or `]`, XCE shall handle it exactly as before this feature, including the existing job-name validation, and shall never create an array for it.
5. When a number in an index expression or in an element selection (`<array-id>[...]` or `name[...]`) has leading zeros, XCE shall read it as the same number without the zeros.
6. While the API server's configured maximum array size is a whole number from 1 to 32767, XCE shall use it as the array size limit.
7. If the API server's configured maximum array size is greater than 32767, then XCE shall use 32767 as the array size limit and write a warning to the API server's startup log stating the configured and the effective value.
8. If the API server's configured maximum array size is not set or is 0, then XCE shall use 1000 as the array size limit, as before this feature.
9. If the API server's configured maximum array size is negative, then XCE shall refuse to start the API server and report the invalid setting, as before this feature.
10. When a controller restarts or fails over, XCE shall recover every array accepted under this feature with the same array identity, the same element numbers it was created with (1 through N for arrays submitted as `name[1-N]`, `name[1]` or harness `jobsub -a N`), and the state each element had when it was last stored.
### Requirement 2: Rejecting index forms this feature does not support
**Objective:** As a user, I want any array syntax XCE cannot honor to be rejected with a specific explanation, so that my jobs are never silently renumbered, resized or turned into ordinary jobs.
#### Acceptance Criteria
1. If a submission through the command-line client or REST API requests an interactive job whose name contains `[` or `]`, then XCE shall reject the submission with a message stating that interactive jobs cannot be arrays.
2. If a job name submitted through the command-line client or REST API contains `[` or `]` and is not an index expression, then XCE shall reject the submission with a message that quotes the name and states that the array name is invalid when the part before the first `[` (the whole name if it contains no `[`) breaks the array-name rule, or otherwise that the array index expression is malformed. An index expression is: a name of 1 to 4094 characters, each an ASCII letter (`a`-`z`, `A`-`Z`), digit (`0`-`9`), `.`, `_` or `-` (the existing array-name rule, which may differ from the rule for ordinary job names); then `[`; then one or more items separated by `,`; then `]`; then optionally `%` followed by a number; and nothing after that. An item is a number, or number`-`number, or number`-`number`:`number. A number is one or more decimal digits with no sign. The index expression contains no whitespace anywhere (including before the name and after `]` or the `%` number).
3. If a submitted index expression contains more than one item or an item with a step (`:`), then XCE shall reject the submission with a message stating that only a single range `name[1-N]` is supported.
4. If a submitted index expression includes a per-array running limit (`%N`), then XCE shall reject the submission with a message stating that running limits are not supported.
5. If a submitted index expression's first element number is 0, then XCE shall reject the submission with a message stating that element numbers start at 1.
6. If a submitted index expression's last element number is less than its first, then XCE shall reject the submission with a message stating that the range is reversed.
7. If a submitted index expression's first element number is greater than 1 (for example `name[1001-2000]` or `name[5]`), then XCE shall reject the submission with a message stating that only `name[1-N]` and `name[1]` are currently supported.
8. If a submitted index expression's last element number is greater than the array size limit (including numbers too large to represent), then XCE shall reject the submission with a message stating the effective array size limit.
9. If a submission breaks more than one of 2.1-2.8, then XCE shall report only the broken rule with the lowest number.
10. If XCE rejects a submission under any criterion of this requirement, then XCE shall create no job and no array element for it. (Raw cluster submissions, which carry no job-name index expression to check, are governed by Requirement 7 only.)
### Requirement 3: Identity reported for a submitted array
**Objective:** As a user, I want the submission result to identify the whole array and say how many elements it created, so that I can refer to the array and its elements the way LSF users do.
#### Acceptance Criteria
1. When an array submission is accepted, XCE shall report the array identity, without an element number, as the identity of the submitted job, both in the command-line client's output and in the REST response.
2. When an array submission is accepted, XCE shall include the number of elements created in the command-line client's output and in the REST response.
3. When an ordinary (non-array) job submission is accepted, XCE shall report its identity exactly as before this feature.
### Requirement 4: Submitting arrays from the test harness
**Objective:** As a developer or tester using the XCE test harness, I want `jobsub -a N` to create elements 1 through N, so that harness-based tests exercise the same numbering users get.
#### Acceptance Criteria
1. When a harness user runs `jobsub -a N`, with N written as one or more decimal digits (leading zeros allowed) and 2 <= N <= 32767, the XCE test harness shall submit one array whose elements are numbered 1 through N, regardless of any configured maximum array size.
2. When a harness user runs `jobsub -a N` where N is one or more decimal digits whose value is 1 (for example `1` or `01`), the XCE test harness shall submit an ordinary non-array job, as it does today.
3. If a harness user runs `jobsub -a N` with N greater than 32767, then the XCE test harness shall refuse the command with a message stating the limit, and shall submit nothing.
4. If a harness user runs `jobsub -a` with a value that is missing, or that is not one or more decimal digits (for example `-3`, `+5`, ` 5`, `5.0`, `0x5`, `abc`), or whose value is 0, then the XCE test harness shall refuse the command with a message stating the accepted range (1 to 32767), and shall submit nothing.
5. When an array submitted with `jobsub -a N` is accepted, the XCE test harness shall print a success line containing the array identity without an element number and shall set its `$jobid` variable to that identity (so `expect-job $jobid` after `jobsub -a N` is handled by 4.7, or by 4.9 if an ordinary job has that identity; elements are checked as `$jobid[k]`).
6. If the XCE test harness fails to obtain an identity for an array submission, then the XCE test harness shall print an error naming the failure and shall submit nothing.
7. If a harness command that acts on one job is given an identity without a `[k]` suffix that 5.14 and 5.15 do not reject, and no job has that identity, then the XCE test harness shall report that the job was not found and state that array elements must be named as `<array-id>[k]`.
8. If a harness command that acts on one job is given an element selection of a range (`<array-id>[a-b]`) that 5.7, 5.8, 5.14 and 5.15 do not reject, then the XCE test harness shall refuse the command with a message stating that it takes a single element `<array-id>[k]`, and shall act on no job.
9. When a harness command that acts on one job is given an identity without a `[k]` suffix that 5.14 and 5.15 do not reject, and a job has exactly that identity, the XCE test harness shall act on that job only, as before this feature (for `expect-job`, the assertion now compares job numbers rather than text, so any written form of the identity that 5.14 and 5.15 accept passes).
10. When a harness command line other than a `!` shell-escape line or a `#` comment line (both handled as today: the shell-escape line is passed to the shell unchanged, the comment line is ignored) contains a token (a run of non-whitespace characters preceded by the line start or by a space or tab, the same token rule as for `$jobid` today) that is `$jobid` followed immediately by `[`, and `$jobid` is set, the XCE test harness shall replace `$jobid` with its current value and copy the rest of the token unchanged, so that `$jobid[3]` after `jobsub -a N` names element 3 of that array (an invalid remainder such as `$jobid[3]x` is then handled by 5.15).
11. If a harness command line other than a `!` shell-escape line or a `#` comment line contains such a token (4.10) while `$jobid` is not set, then the XCE test harness shall fail the command with the same message as for an unset `$jobid`, and shall act on no job.
### Requirement 5: Referring to, showing and controlling array elements
**Objective:** As a user or operator, I want element numbers to mean the same thing in every command, listing and log, so that the element I see is the element I act on.
#### Acceptance Criteria
1. XCE shall display each array element as `<array-id>[k]` in job listings, job detail output, control command results, and those controller, scheduler and worker log entries that print a job identity in its standard written form.
2. When a user makes, in a query or control command, an element selection of a single element (`[k]`) or a range of elements (`[a-b]`, `a` equal to `b` allowed; a range by identity only in control commands) through the command-line client or REST API, and the selection's numbers are all at most 32767, its last number is not less than its first, it neither combines element 0 with other element numbers nor selects element 0 by job name, it does not select element 0 by identity (`<array-id>[0]` or `<array-id>[0-0]`; see 5.4, 5.11 and 8.2), and any `<array-id>` in it is valid (5.8), XCE shall act on exactly the existing array elements with those numbers (and never on an ordinary job with identity `<array-id>`; when no array has identity `<array-id>`, on no job), including when the most recent matching job is the element of a one-element array (5.10 applies only to filters without an element number).
3. If a user's control command, through the command-line client or REST API, makes an element selection by an identity that is valid (5.8) whose numbers are all at most 32767, whose last number is not less than its first, and which does not combine element 0 with other element numbers, and some selected numbers from 1 to 32767 are not elements of an array with that identity (all of them when no array has that identity), then XCE shall report each missing number as not found and shall act only on the existing elements in the selection.
4. If a user's control command, through the command-line client or the REST API, selects `<array-id>[0]` (or `[0-0]`), where `<array-id>` is valid (5.8), and no array with identity `<array-id>` has an element numbered 0 (whether or not an ordinary job has identity `<array-id>`), then XCE shall report element 0 as not found and shall act on no job, in particular not on the whole array or on an ordinary job with identity `<array-id>`.
5. If a user makes, in a query or control command, an element selection of a range through the command-line client or the REST API (by job name, or by identity in a control command) whose first and last numbers are both at most 32767 and whose last number is less than its first, then XCE shall reject the selection with a message stating that the range is reversed.
6. When a user gives, through the command-line client or REST API, a query or control command an identity written without any `[k]` suffix that 5.14, 5.16 and 5.17 do not reject, XCE shall act on every element of the array with that identity and on any ordinary job with that identity, as before this feature.
7. If a user makes an element selection of a single element whose number is greater than 32767, or of a range whose first or last number is greater than 32767, then XCE shall reject the selection as invalid, instead of resolving it to any other job, in query and control commands on the command-line client and the REST API and in every harness command that parses job identities.
8. If a user makes an element selection by identity (`<array-id>[k]` or `<array-id>[a-b]`) where `<array-id>` is not an identity XCE could have assigned to an array -- for example, it lies inside another array's range of element identities, or it is an ordinary job's identity that no array could have (most ordinary identities); an `<array-id>` written with any field wider than 5.14 allows is never an identity XCE could have assigned, even if its value would equal one -- then XCE shall reject the selection as an invalid job identity, instead of resolving it to elements of another array, in query and control commands on the command-line client and the REST API and in every harness command that parses job identities.
9. When the scheduler or the API server logs that it received a new array, XCE shall identify the array in that log entry by its array identity rather than by an element identity or raw number.
10. When a user applies a control command whose filter (for example `-J foo`, `-u`, `-q`) contains no element number (`[...]`) and does not use the all-jobs option, and the job that command's "most recent job" rule selects (the matching job with the highest job identity) is the element of a one-element array, XCE shall act on that element alone, as it does when the selected job is an ordinary job.
11. When a user's query selects `<array-id>[0]`, where `<array-id>` is valid (5.8), and no array with identity `<array-id>` has an element numbered 0 (whether or not an ordinary job has identity `<array-id>`), XCE shall list no job, in particular not the elements of the whole array or an ordinary job with identity `<array-id>`.
12. If a job identity or element selection breaks more than one of 5.5, 5.7, 5.8, 5.14, 5.15, 5.16 and 5.17, then XCE shall report only the rejection of the criterion checked first in the order 5.15, 5.16, 5.14, 5.17, 5.8, 5.7, 5.5.
13. When a harness user gives a command that acts on one job an element identity `<array-id>[k]`, where 0 <= k <= 32767 and `<array-id>` is valid (5.8), the XCE test harness shall act on that element if an element with that number exists in that array (8.2 for a pre-upgrade element 0), and otherwise shall report that the job was not found and act on no job, in particular not on the whole array or on an ordinary job with identity `<array-id>`.
14. If a user gives a job identity (with or without `[...]`) in which any field is wider than it can be (cluster above 63, API server above 15, timestamp of 2^32 or more, sequence of 2^20 or more), then XCE shall reject it as an invalid job identity, instead of resolving it to any other job, in query and control commands on the command-line client and the REST API and in every harness command that parses job identities.
15. If a harness command that parses job identities is given an argument that, after removing any spaces or tabs before or after it, is not a valid written job identity optionally followed by `[k]` or `[a-b]` (k, a and b each one or more decimal digits, with no sign and no whitespace) (for example `'X [1]'` given as one argument, `X[a]`, `X[-1]`, `X[+1]`, `X[ 1]`, `X[1`, `X[]`), then the XCE test harness shall reject it as an invalid job identity and shall act on no job.
16. If a user gives, through the command-line client or REST API, to a query or control command a job identity or element selection whose `[...]` part is not one number or two numbers joined by `-` (a number: one or more decimal digits, with no sign and no whitespace), or that has an unmatched `[` or `]` or any non-whitespace character after the closing `]` (whitespace before the identity or name, or after the closing `]`, is ignored, as before this feature; whitespace anywhere else, including between the identity or name and `[`, makes it invalid) (for example `'X [1]'` given as one argument, `-J 'foo [1]'`, `X[a]`, `X[]`, `X[1`, `X[+1]`, `X[1-3:2]`, `-J 'name[ 1-3]'`, `-J 'name[a]'`, `-J 'name[1-3]x'`), then XCE shall reject it as an invalid job identity or element selection and shall act on no job.
17. If a user gives, through the command-line client or REST API, to a query or control command a decimal job number without a `[k]` suffix that carries the array marking (for example the value of `XCE_JOBID`), whether or not an element with that number exists, then XCE shall reject it with a message stating that array elements must be named as `<array-id>[k]`, and shall act on no job.
### Requirement 6: Element number visible to a running element
**Objective:** As a user whose job script depends on the element number, I want a running element to know its own number, so that each element can select its own input and parameters, as LSF scripts do with `LSB_JOBINDEX`.
#### Acceptance Criteria
1. When an array element starts running, XCE shall set the environment variable `XCE_JOBINDEX` to that element's number in the environment XCE sets up for that element on each host it runs on.
2. When an ordinary (non-array) job starts running, XCE shall set `XCE_JOBINDEX` to 0 in the environment XCE sets up for that job on each host it runs on.
3. If a job's submission supplies its own environment variable named `XCE_JOBINDEX`, then XCE shall still set `XCE_JOBINDEX` to the value defined in 6.1 or 6.2.
4. If a job's submission supplies an environment variable named `XCE_JOBINDEX`, whatever its value, then XCE shall write a warning naming the job and the ignored variable to the log of each host on which XCE sets up that job's environment.
5. XCE shall set `XCE_JOBID` for every running job to the decimal number of that job's identity, as before this feature, unless the submission supplies its own `XCE_JOBID` (unchanged and out of scope).
### Requirement 7: Cluster-side protection against unrepresentable arrays
**Objective:** As an operator, I want the cluster itself to refuse array submissions that it cannot represent, so that no client, including a faulty or outdated one, can create elements that collide with other jobs.
#### Acceptance Criteria
1. If any client makes a submission that declares itself an array and whose first element number is 0, then XCE shall reject the submission with an error status distinct from other submission errors, and shall write a controller log entry stating that array element numbers start at 1 and that the submitting client must be upgraded.
2. If any client makes a submission that declares itself an array, whose first element number is not 0 and whose last element number would be greater than 32767, then XCE shall reject the submission and return an error to that client.
3. If any client submits more than one element in a submission that does not declare itself an array, then XCE shall reject the submission and return an error to that client.
4. If any client submits a job with an element count of 0, then XCE shall reject the submission and return an error to that client.
5. If any client submits a job whose identity carries a marking other than "ordinary job" or "array", then XCE shall reject the submission and return an error to that client.
6. If XCE rejects a submission under 7.1-7.5, then XCE shall store no job for it and shall continue to accept valid submissions.
7. When any client makes a submission that declares itself an array, whose first element number is between 1 and 32767, whose element count is at least 1 and whose last element number is at most 32767, and the submission breaks none of 7.1-7.5, and no existing job has any of the element identities the submission would create, and no existing array, and no other array submission still being accepted (judged when this submission's collision check began; see Terms), has its array identity, and the submission passes the controller's other submission checks and is committed, XCE shall create it with elements numbered consecutively from that first number.
8. When XCE accepts (7.7) a one-element submission that declares itself an array, XCE shall treat it as a one-element array whose element behaves like an element of any larger array, including its listing and its `XCE_JOBINDEX`.
9. If a submission breaks 7.1 and also breaks any of 7.2-7.5 or collides (an existing job already has any of the element identities it would create, or it declares itself an array and an existing array or another array submission still being accepted already has its array identity), then XCE shall respond only as 7.1 requires. If a submission does not break 7.1 but breaks more than one of 7.2-7.5, or one of them and also collides (an existing job already has any of the element identities it would create, or it declares itself an array and an existing array or another array submission still being accepted already has its array identity), then XCE shall reject it with a single error that is not the duplicate-submission error of 7.10, and store no job for it.
10. If a submission breaks none of 7.1-7.5 and an existing job already has any of the element identities it would create, or it declares itself an array and an existing array or another array submission still being accepted (judged when this submission's collision check began; see Terms) already has the array identity it would have, then XCE shall reject it with the same duplicate-submission error as before this feature, or, if it also fails another of the controller's submission checks (7.7) and that check is reported first, with that check's error; XCE shall store no job for it and leave the existing jobs unchanged.
11. If XCE receives two or more submissions that declare themselves arrays, break none of 7.1-7.5 and have the same array identity, and each of them begins its collision check while every one of them whose collision check began earlier is still an array submission still being accepted (see Terms), then XCE shall create at most one of them: the one whose collision check began first, unless that one is rejected under 7.10 (because an existing job or array, or another array submission still being accepted, already has one of its identities), fails another submission check or fails to commit, in which case XCE shall create none of them; XCE shall reject each of the others with the duplicate-submission error of 7.10, or with the error of its own failed check if that check is reported first.
### Requirement 8: Arrays submitted before this change
**Objective:** As an operator upgrading a running cluster, I want arrays that already exist to keep working unchanged, so that the upgrade does not disturb jobs already in the system.
#### Acceptance Criteria
1. While a cluster holds arrays submitted before this change, XCE shall keep those arrays' elements, including any element numbered 0, with their original numbers after the upgrade.
2. While an array submitted before this change contains an element numbered 0, when a user selects `<array-id>[0]` in a query command or controls `<array-id>[0]` (or, in a control command through the command-line client or the REST API, `<array-id>[0-0]`) as a single element through the command-line client, the REST API or the test harness, where `<array-id>` is that array's identity and is valid (5.8), XCE shall act on that element only and not on the whole array.
3. While an array submitted before this change contains an element numbered 0, XCE shall list that element and report control results for it as `<array-id>[0]`.
4. When a controller restarts or fails over after the upgrade, XCE shall recover every job and array stored before the upgrade with the same identities and element numbers, and with the state each had when it was last stored.
### Requirement 9: Documentation of the supported syntax
**Objective:** As a user, I want the help text and documentation to state exactly which array syntax is supported and how it differs from LSF, so that I know what to write and what to expect.
#### Acceptance Criteria
1. The XCE command-line help for job submission shall describe the `name[1-N]` and `name[1]` forms, state that element numbers start at 1, and state that the maximum array size is set by the API server's configuration (default 1000, never above 32767).
2. The XCE command-line help for job control commands shall show how to refer to one element and to a range of elements of an array.
3. The XCE test harness help for `jobsub -a` shall state that elements are numbered 1 through N and that N may be at most 32767.
4. The XCE user documentation shall list the LSF array features this release does not support: starting numbers other than 1, index lists, index steps, `%N` running limits, per-element `%I`/`%J` file-name substitution (at submission every element's file names get the first element's values, `1` and `<array-id>[1]`), interactive arrays, and the `LSB_JOBINDEX`/`LSB_JOBID` variables (scripts read `XCE_JOBINDEX` instead).

220
.kiro/specs/job-array-one-based-index/research.md Normal file
View file

@ -0,0 +1,220 @@
# Research & Design Decisions
## Summary
- **Feature**: `job-array-one-based-index`
- **Discovery Scope**: Extension (light discovery; four parallel codebase investigations on 2026-10-01 at `8f5ba635`: libxce + controller, apiserver, xcecli, harness + worker + scheduler + skill scripts)
- **Key Findings**:
- The requirements' Boundary notes are accurate line for line. Discovery adds a set of defects in the same code that the design must absorb because the acceptance criteria depend on them (listed under "Discovery corrections" below).
- Every identity parser in the system (C `parse_jobid`, Go apiserver `ParseHumanJobID`/`parseArrayName`/`parseArrayIndexSection`/`IsNumeric`, Go xcecli `ParseJobID`/`ToAPIJobID`/`isValidJobId`) is unbounded and lenient in its own way. Requirements 5.7, 5.8 and 5.12-5.17 reduce to one grammar plus one set of range checks. The design therefore has exactly one bounded parser per language, with every caller routed through it.
- Nothing on the persistence, replication or scheduler-recording side depends on where element numbering starts: WAL `JOB_ARRAY_SUBMIT`, snapshots, HA apply and scheduler recording carry `(first element ID, size)` and expand with `first + i`. Scheduler expansion happens in `driver.c` translation, before core events, so the recording and the simulator see expanded jobs and the driver state-equivalence invariant is untouched.
## Research Log
### Job identity bit layout and helpers
- **Context**: Option B (decided in requirements) puts the element number in the existing 15-bit index field and submits the first element as `base | 1`.
- **Sources Consulted**: `src/include/config.h:37-72`, `src/include/utils.h:20-48`, `src/lib/utils.c:578-710`, `src/frontend/apiserver/backend/convert_pure.go:12-133`, `src/frontend/xcecli/common/format/jobid.go:9-148`.
- **Findings**:
- Layout: tag (2 bits, 62-63), cluster (6, 56-61), API server (4, 52-55), timestamp (32, 20-51), sequence (20, 0-19). For arrays, sequence = block (5 bits, 15-19) + index (15 bits, 0-14). So there are 32 array blocks per second per API server.
- `MAKE_JOBID` and both Go `MakeJobID`s OR fields without masking. An oversized field silently spills into its neighbour.
- The array identity (`EXTRACT_ARRAY_BASEID`, C `array_jobid`, Go `ExtractArrayBaseID`) is untagged with the low 15 bits clear.
- Array marking disagrees: `fmt_jobid` and xcecli test `tag & 1`; `array_jobid`/`array_idx` and apiserver `ExtractArrayBaseID` treat any non-zero tag as an array; apiserver `IsArrayJob`/`FormatJobID` test `tag == 1`.
- **Implications**: The array test becomes `tag == 1` wherever a new decision is made (controller 7.5, element-only matching, worker). The display helpers keep their behaviour, because only tags 0 and 1 have ever been issued.
### ID generation
- **Sources**: `src/lib/idgen.c:22-55`, `backend/clib.go:338-347`, `controllers/jobs.go:1185,1426`, `src/harness/test.c:3721,6893-6898`.
- **Findings**:
- `idgen(ctx, c, a, size, *out)` returns `size` on success.
- Size 1 returns an untagged ordinary ID. Size > 1 returns a tagged index-0 ID after rounding the sequence up to a 32768 block.
- Size 32768 is accepted.
- There is no sequence-overflow guard. The harness seeds the sequence randomly in [0, 2^20).
- The harness ignores the return code.
- **Implications**: A one-element array needs a distinct request (decision D2). The overflow guard in general is out of scope (separately tracked). The new array request refuses a block that would leave the 20-bit sequence, because otherwise the 1-based first ID could land in the timestamp.
### Controller submission path
- **Sources**: `src/controller/job.c:108-229, 2224-2356, 2389-2552`, `persist.c:5930-5938`, `src/include/xce/errnox.h:11-26`, `cluster.c:1685`.
- **Findings**:
- Order today: field checks → `HOOK_BEFORE_EVENT` → `jobid_range_exists` → `inflight_range_try_add` → array/ordinary build.
- Release happens on success (`:2321`), on commit failure (`:2356`) and on early errors.
- Duplicate submission replies `-EEXIST`.
- Size 0 reaches payload building and is likely never answered.
- `jobmgr` is a uint64-keyed hash with per-lookup read locks.
- **Implications**:
- Validation (7.1-7.5) becomes a pure function called before any registry or table work.
- The table checks move after registration (Boundary note).
- The array-identity check is at most 32768 hash lookups (CPU-bound, no blocking), which is acceptable in the vtask handler under the async-I/O invariant.
- 7.1 needs a new `errnox.h` value.
### API server
- **Sources**: `controllers/jobs.go` (createArrayName :1824-1979, preCheckJobRequest :1565-1578, :1734, CreateJob :1375-1523, parseArrayName :1761-1821, parseArrayIndexSection :3271-3295 and 5 callers, GetJobsInfo :227-437, GetJobByIDs :725-860, ControlJobByID :2261-2635, foundJob :3111-3131, ControlJob :2652-2830, FetchJobList :3299-3480), `utils/config.go:211-227`, `cmd/main.go:92-101`, `backend/handler.go:272-275`, `models/jobs.go:709-713`.
- **Findings**:
- `createArrayName` trims whitespace, accepts `+1`, rejects `[k-k]` and parses `%N`.
- The named-job branch returns early, so the interactive check never runs for named jobs.
- `parseArrayName` slices to `len-1`, so text after `]` is lost.
- `parseArrayIndexSection` reads `1-2-3` as `(1,1)`. All its callers replace its error with a generic one.
- `IsNumeric` is `ParseFloat`.
- `foundJob` cannot distinguish explicit `[0]` from "no index".
- The submit response has no count.
- The config rejects values above 2147483646 and does not clamp at 32767. The logger is initialised before the config, so a startup warning is possible.
- The in-memory `store.Jobs` and the existing hook-variable pattern (`validateResreqSyntax`, `fetchFile`) make all of this unit-testable without CGO.
- **Implications**: There is one selection parser and one key validator (D4), and they replace the scattered parsing. `foundJob` gains an explicit-index flag. The response gains a count field.
### xcecli
- **Sources**: `common/format/jobid.go`, `cmd/jobs/{submit,controller,list}.go`, `filters/job_filter.go`, `models/jobs.go:365-376,454-458`.
- **Findings**:
- `ParseJobID` discards `ParseUint` errors and checks neither the index nor alignment.
- `ToAPIJobID` passes anything unrecognised through, and `TestToAPIJobIDPassthrough` locks that in for `"123456[3]"`.
- Control-by-ID never validates.
- Both ID lists split at every comma.
- `FormatArrayChildID(base, 0)` prints the bare array ID.
- Checkpoint has no filter flags.
- **Implications**:
- There is one client-side identity parser that both paths use (D6).
- Numeric keys (`123456[3]`) still pass through, because only the API server can judge them (5.17, 5.8 for numeric forms). The passthrough test keeps its numeric case.
- Control results print from the element's own ID.
### Harness, worker, scheduler, skills
- **Sources**: `src/harness/test.c` (jobsub :3615-3866, substitute_autovars :1283-1360, cmd_expect_job :2787-2817, joblist_query :2543-2608, single-job commands :3327/3517/4140/4276/4921/5073/5268/5377/5481), `src/harness/cmdutil.c` + `testsuite/test_cmdutil.c`, `testsuite/test_test_cli.c`, `src/worker/src/xcecd_exdata.c:396-457`, `src/scheduler/src/driver.c:440-512`, `scheduler/testsuite/test_driver_translate.c:720-826`, both skill scripts and SKILL.md files, `src/docs/xce.md`.
- **Findings**:
- `arsize` is `uint32_t`, so a negative `-a` wraps to about 4.29e9 rather than becoming a plain job.
- `taskrun` sends job 0 for a rejected ID.
- `test.c` is not linked into any unit test. Logic must move into a paired file (as `cmdutil.c` is) to be unit-tested; otherwise it can only be tested over a pty with `test_test_cli.c`.
- The worker sets `XCE_JOBINDEX=0` and then lets user variables override it.
- The scheduler expansion has no tag or overflow guard, and the expansion tests use untagged IDs.
- `xce.md` has no array section.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| A: start field | Add an explicit start index to the submission | Supports any start | Wire, WAL and snapshot format change; HA replay compatibility | Rejected in requirements (decision 1) |
| B: index in ID (selected) | First element ID is `base\|k0`; elements are `first + i` | No format change; existing expansion and apply unchanged | Index range is 1..32767 (one slot lost) | Decided with the user in requirements |
| Per-caller fixes | Patch each parser where it is used | Smallest diffs | Ten-plus divergent copies of the same grammar; 5.12 precedence impossible to keep consistent | Rejected (synthesis: generalization) |
| One bounded parser per language (selected) | C `parse_jobsel`, Go apiserver `jobsel` and Go xcecli `jobsel` share one grammar and one check order | One place enforces 5.7/5.8/5.12-5.17; callers stay thin | Three implementations of one grammar (C, two Go modules) | The two Go modules cannot share code today (separate modules); a shared conformance case table keeps them aligned |
## Design Decisions
### Decision: one selection grammar and check order, implemented once per language
- **Context**: 5.7, 5.8, 5.12, 5.14, 5.15, 5.16 and 5.17 apply to the CLI, REST and harness alike, with one fixed precedence (5.15, 5.16, 5.14, 5.17, 5.8, 5.7, 5.5).
- **Alternatives Considered**: 1) fix each caller; 2) one parser per language.
- **Selected Approach**: C `parse_jobsel()` in libxce (the harness and `parse_jobid` use it); Go `jobsel` in the apiserver (all REST selection paths); Go `jobsel` in xcecli. Each returns a typed result `{key, form, lo, hi}` or the first failed check.
- **Rationale**: The precedence rule (5.12) can only hold if one function evaluates the checks in order.
- **Trade-offs**: Three implementations of one grammar, mitigated by a single conformance table (`selection-cases` in design.md's Testing Strategy) copied into each test suite.
- **Follow-up**: Keep the three case tables identical in review.
### Decision: a distinct ID request for one-element arrays
- **Context**: `idgen(..., 1, ...)` must keep returning an ordinary ID (harness `-a 1`, restart). `name[1]` must create a one-element array (1.2, 7.8).
- **Selected Approach**: A new `idgen_array(ctx, c, a, n, *first)` for 1 <= n <= 32767 returns the tagged first element `block|1`. `idgen` keeps size 1 only for ordinary IDs; its size > 1 path is removed, because both callers switch to `idgen_array`. The apiserver gets a matching cgo wrapper `GenerateArrayJobID`.
- **Rationale**: Callers name their intent; size 1 stays unambiguous.
- **Trade-offs**: A new libxce symbol; the non-cgo stub needs a matching stub.
### Decision: controller validation before collision checks, collision checks after registration
- **Context**: 7.1-7.11 and the Boundary note on the registry window.
- **Selected Approach**:
1. Pure `jobsubmit_classify(jobid, size)` → `OK | INDEX0 | INVALID`, evaluated first (7.9: INDEX0 wins over everything; INVALID wins over collisions).
2. Then `inflight_range_try_add` extended to also compare array identities among array entries.
3. Then the job-table element-range and array-identity checks; a hit releases the registration.
- **Rationale**: This closes the window between the table check and registration; the precedence matches 7.9 and 7.10.
- **Trade-offs**: A rejected collision now costs one registry add and release.
### Decision: whole-array identity in replies is derived by clients
- **Context**: Decision 3 / round 3 in the requirements. The controller reply keeps the first element ID; `clib.go` and `RecordAcceptedJob` rely on it.
- **Selected Approach**: The apiserver response `jobid` becomes `array_jobid(first)` with a new `array_size` field. The harness prints and stores `array_jobid`. The controller is unchanged.
### Decision: the worker skips a user-supplied `XCE_JOBINDEX`
- **Selected Approach**: In the user-variable loop, a variable named `XCE_JOBINDEX` is skipped with a WARN naming the job. The system value is `array_idx` for tag-1 IDs and 0 otherwise. `XCE_JOBID` behaviour is unchanged (6.5).
### Synthesis outcomes
- **Generalization**:
- All identity and selection parsing collapses into one grammar per language.
- The five apiserver `parseArrayIndexSection` callers and the `foundJob` index-0 branch become consumers of one parsed selection type.
- The harness's ten single-job commands share one argument helper.
- **Build vs adopt**: Nothing external fits. Go `strconv.ParseUint` with explicit bit-size and C `strtoull` with `errno` replace `atoi` and `ParseFloat`. No new dependencies.
- **Simplification**:
- The `idgen` size > 1 path is removed rather than kept beside `idgen_array`.
- No start field and no format change (option B).
- `%N` and steps are rejected rather than parsed and dropped.
- Start > 1 is not modelled anywhere in the clients. The controller accepts starts from 1 to 32767 only because 7.7 requires it.
## Discovery corrections to the requirements' Boundary notes (absorbed by the design)
1. `parse_jobid` dereferences its buffer before the NULL check (`utils.c:625`); the rewrite fixes this.
2. The harness `-a` value is `uint32_t`, so a negative value wraps to about 4.29e9 rather than turning into a plain job (4.4 covers it).
3. `taskrun` has no `jobid == 0` guard; the shared harness helper replaces every unchecked `parse_jobid` call.
4. `parseArrayName` has six callers: the sixth, `output.go:226`, is peek, which is out of scope and left unchanged. `FetchJobList` also replaces `parseArrayName` errors with a generic message.
5. `parseArrayIndexSection` silently reads `a-b-c` as `(a,a)`; the new parser rejects it (5.16).
6. `IsNumeric` (`ParseFloat`) accepts `1e3`, `-1`, `NaN`; the new key validator uses `ParseUint`.
7. The interactive check (`jobs.go:1734`) is unreachable for named jobs because of the early return at :1578. The 2.1 check runs before the name branch. The early return itself (a separately listed defect) is otherwise left as it is.
8. Scheduler expansion tests (`test_driver_translate.c:720-826`) use untagged IDs. The new tag guard requires them to become tagged `block|1` IDs.
9. The jobsub help is at `test.c:600-613`; :556-562 is `expect-job`'s.
10. `list.go`'s `-W` flag is commented out at :345, not :360.
11. Submissions of size 0 are probably never answered today (empty payload). 7.4 now rejects them up front.
12. **bug-1898 (found in design review round 1):** the apiserver builds bracketed `job_id`/`job_name` selections as `CompConditions` that `store.ExecuteQuery` never evaluates, so `GetJobsInfo`, `GetJobByIDs` and `FetchJobList` (hence filter-based `ControlJob`) ignore them. The design replaces them with evaluated row predicates (`QueryOptions.Predicates`), which this feature's element-only matching needs anyway.
13. The apiserver has no control-result struct; results are `map[string]interface{}`, and accepted/failed maps already carry `unique_job_id`. Only not-found maps need it.
14. Today's `substitute_autovars` also expands `$?` and `$jobcount`; the extracted `autovar_substitute` takes a context struct for them.
15. libxce and controller tests are registered in `src/lib/meson.build` / `src/controller/meson.build` plus their runners (`lib/testsuite/test.c`, `controller/testsuite/controller_test.c`), not in per-testsuite `meson.build` files.
## Design review log
- **Round 1 (2026-10-01):** two independent reviewers (requirements fidelity; code feasibility) reported 18 issues, all fixed in design.md:
- Requirements fidelity: most-recent rule scoped to filters without `[...]`; 5.8 only for keys with `[...]`; harness reversed range → 4.8; submit-number saturation for 2.9; client 5.7/5.8 on decimal keys; validate-all-first on REST; no array-name rule on name selections; 5.10 approximation documented; `primaryJobID` stays the first element for `%I`/`%J`.
- Code feasibility: unevaluated `CompConditions` (bug-1898) → predicates; no apiserver control struct; autovar context; test registration paths; peek keeps a legacy `ToAPIJobID`; one read-locked scan for the collision check; `foundJob` signature and its existing test; a dispatch-log type conversion; the harness pty command form.
- **Round 2 (2026-10-01):** 16 issues, all fixed:
- Requirements fidelity (7): control keeps identity ranges (`ToAPIJobID(s, allowRange)`); range refusal ordered after 5.16/5.14/5.17/5.8/5.7; `CreateJob` log lines name the array (5.9); the `> 2147483646` config rejection removed; element 0 by name has an explicit predicate; the count is printed once; one test bullet relocated; one upgrade order stated in both documents.
- Code feasibility (9, one already fixed by round-2 edits): 64-bit size in `jobsubmit_classify`; per-field OR / cross-field AND predicates; bracketed filters only with `eq`/`in`; whole-filter conversion; two scheduler tests and `peek_test.go` added to the plan; `cluster_t`; harness header paths; the autovar `-ENOSPC` return.
- **Round 3 (2026-10-01):** 12 issues (one already resolved by earlier round-3 edits), all fixed:
- Requirements fidelity (6): bracketed filters honour their operator instead of a new eq/in-only rejection (`notin` negates, `contains`/`startwith`/`endwith` match the name part, `between` as `in`, `ne` keeps today's rejection); REST queries evaluate identity ranges (only the xcecli list path refuses them); written-form bare keys normalised on every path; 5.9 log lines drop the raw first-element number; name-part grammar and test rows; schedulers in the upgrade order.
- Code feasibility (6): harness sequence seed bounded below `30 * 32768`; 7.2 and the scheduler guard written without addition (`size > 32768 - idx`); filter-based `ControlJob` validates names itself so rejections are 400; per-entry (not per-field) filter conversion; `resolveMaxArraySize` pure function for config tests.
- **Round 4 (2026-10-01):** 10 issues (two reviewers overlapped on the xcecli wildcard case), all fixed: xcecli checks the raw `-J` text (5.16) before wildcard mapping and picks the search rule from the name part of a bracketed value; bracketed `job_name` operators reuse today's condition semantics (`BuildFilterConditions` + `MatchesCondition`); operators outside a field's mapping table keep the existing "unsupported operator" error, and `job_id` comparison operators with brackets get the existing `ne`-class error; identity parsing limited to identity operators and converted entries; server `ParseIdentitySelection` has no range flag; `ControlJob` name pre-validation only for bracketed names; `test_idgen` seed cases pin the timestamp; idgen overflow boundary stated exactly; line references corrected.
- **Round 5 (2026-10-01):** 8 issues (one overlap), all fixed: line references; xcecli rejects an all-`*` name part; bracketed `job_name` `null`/`notnull` rejected like the `job_id` comparison operators, with `notcontains` negation scope stated; `GetJobsInfo` validates all filter entries in a pre-pass before `exec_hosts` resolution; the one new operator rejection stated honestly and added to the Error table; Allowed Dependencies split between parsers and predicate building; harness and xcecli rows completed in the Error table; pre-store key checks aligned with Matching for converted `between` entries.
- **Round 6 (2026-10-01):** requirements review found 1 issue, fixed: the round-5 client rejection of an all-`*` name part (`-J '*[1]'`) had no authorising criterion; the name part may now be empty (`contains` → every array, `eq` → no job), so no rejection is added.
Code review found 1 issue, fixed: submit-expression errors pass through `preCheckJobRequest` unwrapped and `CreateJob` maps `*ArrayExprError` to 400 via `errors.As` (it mapped every preCheck error to 500), with a typed-nil guard.
- **Round 7 (2026-10-01):** requirements review found 2 issues, fixed: `preCheckJobRequest` returns `(n, isArray, err)` and `CreateJob` branches on `isArray` (so `name[1]` is a one-element array); `createArrayName` is kept for names without brackets so their validation and errors are unchanged (1.4).
Code review found 3 issues: 2 duplicated the requirements findings (already fixed); 1 new, fixed: unnamed jobs are stored with `job_name ""`, so an empty name part with `eq`/`in`/`notin` now matches no row (otherwise `kill -J '[1]'` would hit every unnamed array).
- **Round 8 (2026-10-01):** code review found 3 low issues, fixed: the four `preCheckJobRequest` test call sites listed; error returns written for the new `(n, isArray, err)` shape; interactive passed as `req.Interactive != 0`.
Requirements review found 2 issues: 1 duplicated the code finding (fixed); 1 new, fixed: 1.4 traced to ApiCreateJob's unchanged `createArrayName` path instead of the parser.
- **Round 9 (2026-10-01):** requirements review found 2 issues, fixed: `CreateJob` test seams (`generateJobID`, `generateArrayJobID`, `submitJob` package variables) with the submit tests in a new `!cgo` file next to the JWT/RBAC fixtures; the bracketed-name success path stated (`req.JobName = arrayName`, `(n, true, nil)` at today's early return). Non-Goals wording on the interactive check corrected.
Code review found 5 issues: 1 duplicated (test seams, fixed); 4 new, fixed: line numbers corrected; the unnamed-job success return (`:1746`) stated; the REPL claim corrected (its `strtoull` fallback keeps behaviour unchanged); `ToAPIJobID` described as a signature change with all call sites listed.
- **Round 10 (2026-10-01):** code review found 1 issue, fixed: the submit parser's limit comes from `utils.JobYamlConfig.MaxArraySize`, and the `!cgo` submit tests set and restore `JobYamlConfig` (nil in tests otherwise).
Requirements review found 1 issue, fixed: a `controlJob` seam at both control dispatch sites; every handler-level submit/query/control test moved into the new `!cgo` `jobs_handler_test.go` (fixtures and dispatch observable); `jobs_test.go` keeps pure-function cases only.
- **Round 11 (2026-10-01):** code review found 2 issues, fixed: handler tests seed `store.Queues` and a controllable `job_status` (else control returns 403 before dispatch); `ControlJobByID` re-dispatches earlier selections per later selection -- logged as bug-1899, and the design now dispatches a de-duplicated list once after the selection loop.
- **Round 12 (2026-10-01):** requirements review found 1 issue, fixed: the bug-1899 "dispatch once" rule now keeps the per-selection not-found maps in the response (collected separately from the de-duplicated act-on list).
Code review found 2 issues, fixed: handler tests seed after `setupRBACTest` (which resets the stores) and assert real dispatch; `validateJobControl` runs once on the de-duplicated list after the loop, and a dispatch error stays a 500.
- **Round 13 (2026-10-01):** code review found 1 low issue, fixed: `validateJobControl` is per row, so it is called once per de-duplicated job after the loop (signature unchanged).
- **Round 14 (2026-10-01):** code review found 1 issue, fixed: `TestToAPIJobIDPassthrough`'s `"123456[3]"` is misaligned under the client's 5.8 check; replaced by `"32768[3]"` plus a rejection case; `jobid_test.go:109` added to the call-site list with `allowRange=false`.
- **Round 15 (2026-10-01):** requirements review found 1 issue (introduced by the round-14 edit), fixed: the `ToAPIJobID` call-site list now states `controller.go:52` passes `allowRange=true` and only the list/test sites pass `false`; tests added that catch a wrong flag.
- **Round 16 (2026-10-01):** code review: no issues. Requirements review found 2 issues, fixed: every rejection after registration replies before releasing the registry entry (table-hit path and the existing payload/OOM early-outs), so a submission blocks others until answered (7.11, Terms); the xcecli control test placed in a new `cmd/jobs/controller_test.go` with its httptest mechanism.
- **Round 17 (2026-10-01):** requirements review found 1 issue, fixed: reply and registry removal are now one atomic step under the registry lock (`inflight_reply_and_remove`, `xce_reply` being non-blocking) on every post-registration path, closing both ordering windows; the System Flows diagram matches.
Code review found 1 issue (the undefined "reply hook"), fixed together with the atomic step: an `XCE_TESTING` hook inside `inflight_reply_and_remove` plus a test-only overlap accessor; the concurrent check runs on a second thread because the hook holds the registry lock.
- **Round 18 (2026-10-01):** code review found 2 issues (no deadlock risk found), fixed: `inflight_reply_and_remove` is the only registry removal -- `jobsubmit_async_ctx_release` no longer calls `inflight_range_remove`, and the async paths call it inside their one-shot reply guard; the atomicity test uses deterministic in-hook checks (trylock EBUSY, reply queued, entry still present) instead of a second thread.
Requirements review found 1 issue, fixed: classification runs before the existing field checks and hook, so 7.1/7.9 hold even when an index-0 array also has an empty field.
- **Round 19 (2026-10-01):** code review found 2 low issues (and confirmed no registry entry can leak and no hook consumer acts on the moved before-event), fixed: `inflight_reply_and_remove` takes `(ep, from, reply, base, size)` so the async paths can call it; classification is anchored as the first statement after `init_reply_by_req`.
- **Round 20 (2026-10-01):** both reviewers (requirements fidelity; code feasibility) reported NO ISSUES. Review loop closed.
## Risks & Mitigations
- **Rolling upgrade: an old API server or harness sends index-0 arrays to an upgraded controller.** Rejected with the distinct `E_XCE_ARRAY_START0` status and an explanatory log line (7.1). Recommended order for clusters using arrays: API servers and harness first, then controllers (an old controller accepts `array-id | 1` submissions); controllers first is supported with that rejection window accepted (design.md, Migration Strategy).
- **Three parser implementations drift.** Shared conformance case table, plus a review checklist item in tasks.
- **Control-result display of pre-upgrade element 0 needs the element's own ID.** Accepted and failed result maps already carry `unique_job_id`; the apiserver adds it to not-found maps. xcecli prints `FormatJobID(unique_job_id)` when it is present and falls back to today's rendering otherwise (an older server).
- **The array-identity collision check walks up to 32768 hash keys inside a vtask handler.** One read-lock hold for the whole scan, which also answers the element-range check; bounded CPU, no I/O, permitted by `ha-async-io.md`. The cost is paid only by array submissions. An apply-maintained array-ID index was rejected (it would need completeness across apply, replay and snapshot load, audit dimension 7).
- **`idgen_array` refusing a block past the 20-bit sequence surfaces as a submit error after about 31 arrays in one second from one API server.** That is the same limit as today, except it is now an error instead of a corrupted ID. It is recorded as a known limit. The harness's random sequence seed is bounded below `30 * 32768` so that its first array cannot hit the limit.
## References
- `.kiro/specs/job-array-one-based-index/requirements.md`: decisions 1-10 and the Boundary notes.
- `.kiro/specs/job-array-single-wal-record/`: the array WAL record and expansion this design relies on unchanged.
- `.kiro/steering/ha-async-io.md`, `error-handling.md`, `api-standards.md`, `testing.md`, `persistence.md`.
- LSF 10.1 Administration Guide, "Chunk Jobs and Job Arrays" (p.519): index 0 is invalid; `LSB_JOBINDEX` is 0 only for non-array jobs.

22
.kiro/specs/job-array-one-based-index/spec.json Normal file
View file

@ -0,0 +1,22 @@
{
"feature_name": "job-array-one-based-index",
"created_at": "2026-09-30T07:09:58Z",
"updated_at": "2026-10-01T07:50:49Z",
"language": "en",
"phase": "tasks-generated",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true
}

238
.kiro/specs/job-array-one-based-index/tasks.md Normal file
View file

@ -0,0 +1,238 @@
# Implementation Plan
- [ ] 1. Foundation: bounded job identities and array ID allocation in libxce
- [ ] 1.1 Add the bounded job-identity and element-selection parser
- Parse a written identity with an optional single element or element range under the fixed grammar (no sign, no inner whitespace, optional surrounding spaces/tabs), accumulating digits with an overflow guard
- Evaluate the checks in the fixed order: syntax, field widths, alignment of the array identity when an element part is present, element-number bound, reversed range; report only the first failure
- Provide the array-marking test (tag equals the array tag only), the element-identity composition helper and a message for each rejection
- Add the shared `selection-cases` conformance table as a libxce unit test registered in the libxce test sources and runner
- Done when the libxce suite runs the conformance table with every row passing, including `X[0]`, `X[01]`, the width cases (`0-0-T-1048576[1]`) and the precedence rows
- _Requirements: 1.5, 5.7, 5.8, 5.12, 5.14, 5.15, 8.1_
- [ ] 1.2 Re-implement the existing identity parse on top of the new parser
- Keep its signature; make it NULL-safe, bounded, and reject element ranges and every rejected selection with an invalid-argument result
- Keep the existing format/parse round-trip tests passing unchanged
- Callers outside libxce (the harness single-job commands and the scheduler REPL, whose fall-back to plain number parsing is accepted unchanged) see only the stricter acceptance
- Done when the libxce suite passes with new cases for a NULL buffer, a range, an oversized field and an index above 32767 all returning invalid-argument, the existing round trips still succeed, and the scheduler and harness suites still pass
- _Requirements: 5.7, 5.14, 5.15_
- [ ] 1.3 (P) Add array ID allocation numbered from 1
- Allocate the first element of an n-element array (1 to 32767) as the next whole 32768 block with element number 1, advancing the context past the block
- Refuse a block that would leave the 20-bit sequence with an overflow result and leave the context unchanged; refuse n of 0 or above 32767
- Leave the existing ordinary-ID allocation untouched in this task (its multi-element path is removed in 9.1, after its callers move)
- Done when the libxce suite shows: index 1 for the first element and index n for the last; 32767 accepted, 0 and 32768 rejected; a seed above 31 blocks overflows while exactly 31 blocks and below 30 blocks succeed (each seeded case pins the context timestamp and retries if the second changed)
- _Requirements: 1.1, 1.2, 1.3, 4.1_
- _Boundary: IdgenArray_
- [ ] 2. Controller: array admission and collision detection
- [ ] 2.1 Classify every submission before any other check
- Add the distinct "array numbering starts at 1" error status and its message text to the shared error table
- Classify a submission from its first ID and 64-bit size only: start-at-0 array first; then size 0, an unknown marking, an unmarked multi-element submission, and a last element above 32767 (computed without addition so no size can wrap)
- Run the classification as the first step after the reply is initialised, before the existing field checks and the before-event hook; answer start-at-0 with the new status plus an ERROR log saying numbering starts at 1 and the client must be upgraded, and the other classes with invalid-argument; store nothing
- Create the controller validation test file and register it in the controller test sources and runner; move the existing commit-barrier reply and rollout tests to tagged first IDs and at most 32767 elements
- Caveat: from this task until 4.1, 6.2 and 9.2 land, live array submissions from the current harness and API server (which still number from 0) are rejected by design; unit suites are unaffected; do not run the multihost or HA-soak skills in between
- Done when the controller suite passes with cases for every class, the 7.9 precedence, sizes of 2^32+2 and the maximum 64-bit value with index 2, a size-0 submission answered, and an index-0 array with an empty queue answered with the new status and its log line
- _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.9_
- _Depends: 1.1_
- [ ] 2.2 Make answering a registered submission and releasing it one atomic step
- Extend in-flight registry entries with an array flag and array identity; two array entries with the same array identity overlap
- Add one reply-and-remove operation that sends the reply and unlinks the entry while holding the registry lock (the reply is non-blocking), and use it on every path after registration: the payload-build and allocation-failure early-outs, commit failure or timeout, and apply success (inside the existing one-shot reply guard)
- Stop the async-context release from removing registry entries, so it is the only removal point
- Add the test-only hook and registry accessors (trylock, locked and unlocked overlap checks) under the testing build flag
- Done when the controller suite shows, from inside the hook, that the registry lock is held, the reply is already queued and the entry is still present; that the entry is gone afterwards; and that a retry of the same submission registered after the reply survives the original's context release
- _Requirements: 7.7, 7.10, 7.11_
- [ ] 2.3 Check element and array-identity collisions after registration
- Move the job-table collision checks after registration; on a hit, reply duplicate-submission and release through the atomic step
- For array submissions, one scan under a single job-table read lock over the 32768 element keys of the array's block answers both the element-range and the array-identity check; unmarked submissions keep the existing range check
- Done when the controller suite shows: an existing element `X|3` blocks a new `X|5` with duplicate-submission; two concurrent submissions with the same array identity, the second gets duplicate-submission while the first is unanswered; once the first is answered with a failure, a new one with the same identity is accepted
- _Requirements: 7.7, 7.10, 7.11_
- [ ] 3. Scheduler and worker
- [ ] 3.1 (P) Guard scheduler array expansion and log the array identity
- Expand an array push only when the first ID carries the array marking and its size fits within the block (checked without addition); otherwise log an ERROR naming the array identity and size and expand nothing
- Log the received array by its array identity
- Move the existing expansion tests (translation, array-expand and wire end-to-end tests) to tagged block-plus-one first IDs
- Done when the scheduler suite passes, with tagged arrays expanding to elements 1..n and untagged, overflowing and maximum-64-bit-size arrays not expanded
- _Requirements: 5.9_
- _Boundary: SchedGuard_
- _Depends: 1.1_
- [ ] 3.2 (P) Expose the element number to running jobs
- Set `XCE_JOBINDEX` to the element number for jobs carrying the array marking and to 0 otherwise
- Skip a submission-supplied `XCE_JOBINDEX`, logging a WARN that names the job and the ignored variable; leave `XCE_JOBID` handling unchanged
- Add the worker test and register it in the worker test build
- Done when the worker suite shows the element number for an array element (including a one-element array), 0 for an ordinary job, a user value ignored with its WARN, and `XCE_JOBID` unchanged
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 7.8_
- _Boundary: WorkerEnv_
- _Depends: 1.1_
- [ ] 4. Test harness
- [ ] 4.1 Submit harness arrays numbered from 1 (fixes bug-1896 for the harness)
- Parse `-a` strictly (digits only, leading zeros allowed): 1 submits an ordinary job; 2..32767 allocate with the new array allocator and check its result; missing, non-digit, 0 and above-32767 values are refused with the accepted range or limit, submitting nothing
- Bound the random sequence seed so the first array of a run cannot overflow
- Print the array identity on success and set `$jobid` to it; update the jobsub help text
- Add strict-size cases to the command-utility unit test and pty cases for `-a 0` and `-a 40000`
- Done when the harness unit and pty tests pass and `jobsub -a 3` against a running controller (upgraded or not) creates elements `[1]`, `[2]`, `[3]` with `$jobid` naming the array
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 9.3_
- _Depends: 1.3_
- [ ] 4.2 Substitute `$jobid[...]` in harness command lines
- Move the existing autovariable substitution into its own unit-testable module with an explicit context for `$jobid`, `$?` and `$jobcount`, keeping today's token rule, messages and buffer-size result
- Expand `$jobid` followed by `[` keeping the rest of the token; fail as for an unset `$jobid` when it is unset; leave `!` and `#` lines untouched
- Register the module in the harness build and its new unit test
- Done when the autovariable unit test passes for `$jobid[3]`, `$jobid[3]x`, unset `$jobid`, `$?`, `$jobcount` set and unset, and `!`/`#` lines, and the existing harness tests still pass
- _Requirements: 4.10, 4.11_
- [ ] 4.3 Parse single-job harness arguments strictly
- Add one argument helper that trims spaces/tabs, applies the libxce selection parser, refuses element ranges (including reversed ones) with the single-element message, and reports whether an element number was given
- Add a not-found reporter that adds the `<array-id>[k]` hint for identities given without an element number
- Use both in every single-job command (including `jobinfo` per `-j`) and make `expect-job` parse its argument the same way; the job-list lookup behind it compares job numbers instead of formatted text
- Done when the command-utility tests pass (ranges refused including `X[3-1]`, `[0]` accepted), a pty test shows `jobkill -j 1-1-1-0[1-3]` refused with the single-element message before any RPC, and no single-job command can send job 0 for a rejected argument
- _Requirements: 4.7, 4.8, 4.9, 5.7, 5.8, 5.13, 5.14, 5.15, 8.2_
- _Depends: 1.1, 1.2_
- [ ] 5. API server foundation
- [ ] 5.1 Add backend ID helpers and the array ID wrapper
- Wrap the libxce array allocator for the API server, with a non-cgo stub that returns an error
- Add strict written-identity parsing with field-width checks, an array-capable key test and element-ID composition as pure helpers
- Done when the backend unit tests pass for the strict parser (width rejections), the array-capable test and element composition, and both the cgo and non-cgo builds compile
- _Requirements: 1.1, 1.2, 5.8, 5.14_
- _Depends: 1.3_
- [ ] 5.2 (P) Clamp the array size limit setting
- Move the limit rule into a pure resolver: unset or 0 gives 1000, 1..32767 used as is, above 32767 clamped to 32767, negative refuses startup; drop the old upper-bound rejection
- Log a startup warning with the configured and effective values when clamped; update the shipped config's comment for the setting (default 1000, at most 32767)
- Done when the config tests pass for 0, 1, 32767, 32768, 3000000000 and -1, and a clamped startup logs the warning
- _Requirements: 1.6, 1.7, 1.8, 1.9_
- _Boundary: ApiConfig_
- [ ] 5.3 (P) Evaluate row predicates in the query pipeline
- Add row predicates to the query options and apply them after the plain conditions and before sorting and pagination
- Done when the store query tests show predicates filtering before pagination, with page boundaries and bookmarks consistent
- _Requirements: 5.2, 5.11_
- _Boundary: store query pipeline_
- [ ] 5.4 Add handler test seams and the non-cgo handler test fixture
- Route ID generation, array ID generation, job submission and control dispatch through replaceable package variables
- Create the non-cgo handler test file with a fixture that saves the stores, sets up auth/RBAC, then seeds jobs (controllable status) and their queues and the array-size config, and restores everything on cleanup; the dispatch replacement records each dispatched job
- Done when a smoke handler test in the new file submits an ordinary job and controls one seeded job through the seams with the dispatch recorded, and the existing apiserver tests still pass
- _Depends: 5.1_
- [ ] 6. API server: array submission
- [ ] 6.1 (P) Parse submit index expressions
- Accept only `name[1-N]` and `name[1]`; reject interactive arrays, malformed expressions or invalid names (quoting the name), lists and steps, `%N`, start 0, reversed ranges, start above 1 and sizes above the limit, each with its specific message, reporting only the lowest-numbered broken rule
- Saturate oversized numbers so the rule order still decides; accept leading zeros
- Done when the index-expression unit tests pass for every rule, the precedence rows (`[0-0:2]`, `[99999999999999999999]`, `[99999999999999999999-5]`, `[0-99999999999999999999]`), leading zeros, `[1]` vs `[1-1]` vs `[5]` and the limit edge
- _Requirements: 1.5, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9_
- _Boundary: ApiSubmitExpr_
- [ ] 6.2 Create arrays from bracketed submissions and report the array (fixes bug-1896 for REST and the client)
- Make the job pre-check return the element count and whether the job is an array; call the expression parser only for bracketed names (passing the interactive flag and the configured limit), store the name without its index part, and keep the existing path for names without brackets
- Map expression errors to 400 with their message unchanged and keep every other pre-check error as today
- Allocate array IDs through the array wrapper (one-element arrays included); keep the first element for the submit-time `%I`/`%J` substitution
- Report the array identity, the element count and a message naming both; log new arrays by array identity in the submit path and in the array dispatch log
- Update the existing pre-check test call sites to the new result shape
- Done when the handler tests show `arr[1-3]` creating three elements with the stored name `arr`, `name[1]` creating a one-element array with `array_size` 1, a 2.x rejection returning 400 with its exact message, plain names (`my job`, `foo%5`, `foo%`) failing exactly as today, and `%I` = 1 and `%J` = `<array-id>[1]`
- _Requirements: 1.1, 1.2, 1.3, 1.4, 2.10, 3.1, 3.2, 3.3, 5.9, 7.8_
- _Depends: 5.2, 5.4, 6.1_
- [ ] 7. API server: element selection in queries and control
- [ ] 7.1 (P) Parse REST identity and name selections
- Parse identity selections (strict written form or base-10 decimal) and name selections (name part may be empty, no whitespace or brackets) under the shared grammar and check order, including the marked-decimal-without-index rejection
- Return a rule tag and a message that names the selection as written
- Done when the selection unit tests pass for the shared conformance table plus decimal-key cases (marked decimal without index, `1e3`, `-1`) and the name rows
- _Requirements: 1.5, 5.5, 5.7, 5.8, 5.12, 5.14, 5.16, 5.17_
- _Boundary: ApiSel_
- _Depends: 5.1_
- [ ] 7.2 Build selection predicates
- Turn each filter entry containing a bracketed value into one composed predicate (values OR'ed within the entry, entries AND'ed with each other and with plain conditions), converting the whole entry and only that entry
- Honour the operator: `eq`/`in` (and `between` on job identities) select, `notin` excludes, name pattern operators reuse today's condition semantics on the name part, `ne` and comparison/null operators keep today's "operator doesn't support" error, operators outside the field's table keep today's error
- Match identity elements only among array-marked rows, match empty names under exact operators to no row, and normalise written bare keys to decimal on both the plain and converted paths
- Done when the pure predicate tests pass for every operator rule, empty-name semantics with an unnamed array row, and the same bare key matching alone and mixed with a bracketed value
- _Requirements: 5.2, 5.6, 5.11, 7.8, 8.2_
- _Depends: 5.3, 7.1_
- [ ] 7.3 Route query handlers through selections (fixes bug-1898)
- Validate every filter entry and key before any store access (a pre-pass ahead of the existing filter loop, including before host resolution), returning the first rejection as 400 with its specific message
- Apply width and marked-decimal checks to identity-operator keys and converted entries, and the alignment check only to keys with an element part
- Evaluate selections through the predicates on the job listing and job-detail paths; remove the unevaluated compound conditions and the old index-section parsing on these paths (the filter-based control path follows in 7.5)
- Done when the handler tests show `job_id:eq:X[2]` returning only element 2, `GET /jobs/X[1-3]` returning elements 1-3, `<ordinary-id>[0]` listing nothing, an invalid selection combined with a host filter returning 400, and the operator, intersection and mixed-value rows from the design
- _Requirements: 5.2, 5.5, 5.6, 5.7, 5.8, 5.11, 5.12, 5.14, 5.16, 5.17_
- _Depends: 5.4, 7.2_
- [ ] 7.4 Control jobs by identity: validate first, dispatch once (fixes bug-1899)
- Replace the index-section parsing in this handler with the REST selection parser; validate all selections before matching; give the element matcher an explicit-index flag so `<array-id>[0]` never falls back to the whole array or an ordinary job
- Collect the de-duplicated jobs to act on and, separately, one not-found result per missing number per selection carrying the would-be element ID; validate each job once after the loop, then dispatch each accepted job once (a dispatch error stays a 500)
- Update the existing element-matcher test to the new signature with a tagged fixture
- Done when the handler tests show two selections dispatching each element exactly once, `X[1-3]` acting on existing elements and reporting missing numbers with their IDs, `X[0]` with no element 0 reporting not found and dispatching nothing, a pre-upgrade element 0 dispatched alone, and a valid-then-invalid request dispatching nothing
- _Requirements: 5.1, 5.3, 5.4, 8.2, 8.3_
- _Depends: 5.4, 7.2_
- [ ] 7.5 Control jobs by filter with element selections
- Validate a bracketed name filter before fetching (unbracketed names, including command-line names with spaces, unchanged), returning 400 with the specific message
- Route the filter-based job fetch's bracketed name handling through the name parser and the selection predicates; with its last use gone, delete the old index-section parser
- Apply the most-recent-job rule only when the filter has no element number and all-jobs is not set, acting on the newest row alone when it is ordinary or the only stored element of its array
- Add the would-be element ID to not-found results
- Done when no use of the old index-section parser remains and the handler tests show `-J 'foo[40000]'` returning 400 with the 5.7 message, `-J '[1]'` with an unnamed array dispatching nothing, a one-element array as newest row dispatched alone, and `-J 'foo[1]'` with two one-element arrays named `foo` dispatching both
- _Requirements: 5.1, 5.2, 5.10, 8.3_
- _Depends: 5.4, 7.2, 7.4_
- [ ] 8. Command-line client
- [ ] 8.1 (P) Parse identities and selections in the client
- Add the client selection parser (written and decimal keys, the shared order, alignment and element bound on decimal keys with an element part, range refusal for queries only after the other checks), the bracket-aware list splitter and the raw `-J` name check
- Make identity parsing bounded and change the API-key conversion to take whether ranges are allowed; keep a legacy conversion for peek; update its two command callers mechanically (control allows ranges, the list filter does not) so the client keeps building
- Update the conversion tests: `"123456"` and `"32768[3]"` pass through, `"123456[3]"` is rejected under 5.8, a range passes only when allowed; move peek and its test to the legacy conversion
- Done when the client builds and its format, selection and peek tests pass, including the shared conformance table and `SplitSelections("X[1,3],Y")`
- _Requirements: 1.5, 5.5, 5.7, 5.8, 5.12, 5.14, 5.16_
- _Boundary: CliSel, CliSplit_
- [ ] 8.2 Use the client parser in list and control commands
- Split ID lists only at commas outside brackets; validate every selection before any request; control by identity allows ranges, listing does not
- Check raw `-J` values containing brackets before wildcard mapping, taking the search rule from the name part only for bracketed values
- Print control results from each result's own job ID when present, falling back to today's rendering; add the control result's job ID field to the client model
- Update control and list help to show `<array-id>[k]` and `<array-id>[a-b]`
- Done when a new control-command test against a local test server sees `X[1-3]` arrive as `<decimal>[1-3]` in the request path, `-J 'foo[1]*'` is refused before any request, and a pre-upgrade element 0 result prints as `<array-id>[0]`
- _Requirements: 5.1, 5.2, 5.16, 8.2, 8.3, 9.2_
- _Depends: 8.1_
- [ ] 8.3 Show the array in submit output and help
- Add the element count to the client's submit response model; print the server's submit message with the array identity formatted (the count comes from the message); describe `name[1-N]`, `name[1]`, numbering from 1 and the limit (default 1000, at most 32767) in submit help
- Done when a submit-output test shows `Job <array-id> is submitted ... (3 elements)` with the formatted array identity and the help text lists the supported forms
- _Requirements: 3.1, 3.2, 9.1_
- _Depends: 8.2_
- [ ] 9. Integration and cleanup
- [ ] 9.1 Restrict ordinary ID allocation to single jobs
- Remove the multi-element path from ordinary ID allocation now that the harness and the API server use the array allocator
- Migrate the existing ID-allocation test loop that requests sizes above 1 to the array allocator
- Done when the old multi-element test loop is gone or migrated, ordinary allocation with size 2 returns invalid-argument in the libxce suite, and the harness and API server builds and suites still pass
- _Requirements: 1.3, 4.2_
- _Depends: 4.1, 6.2_
- [ ] 9.2 Update test skills for elements numbered from 1
- Make the HA soak and multihost verification scripts and their skill descriptions verify elements 1..N (first, middle, last) instead of 0..N-1
- Done when both scripts pass a syntax check and every element check in them refers to indices 1..N
- _Requirements: 4.1, 4.5_
- [ ] 9.3 Document the supported array syntax
- Add a "Job arrays" section to the user documentation listing the supported forms and the LSF features not supported (other starts, lists, steps, `%N`, per-element `%I`/`%J`, interactive arrays, `LSB_JOBINDEX`/`LSB_JOBID`)
- Mark the superseded statements in the job-array-single-wal-record and xce-test-e2e-primitives specs, update the numbering notes in the controller-job-cleanup and job-dependency briefs, and add a roadmap entry
- Done when the user documentation has the section with every item 9.4 lists and each referenced spec document carries its note
- _Requirements: 9.4_
- [ ] 10. Validation
- [ ] 10.1 Run every affected suite
- Build with tests and run the libxce, controller, scheduler, worker, harness, apiserver (non-cgo and the cgo suite, including a real array submit asserting the reported array identity and `array_size`) and client suites
- Done when every suite passes with zero failures under the ASAN profile and the cgo apiserver submit test passes
- _Requirements: 1.1, 1.10, 3.1, 3.2, 8.1, 8.4_
- [ ] 10.2 Verify end to end on a live cluster
- Run the multihost verification (elements `[1]` and `[2]` reach DONE across both hosts) and a six-round HA soak confirming elements 1..N survive failover
- Through the client: submit `arr[1-3]` and list `[1]`..`[3]`; kill `<array-id>[2]` and see only element 2 act; run a job script that prints `XCE_JOBINDEX`
- Close bug-1896, bug-1898 and bug-1899 through the buglog with their fix commits
- Done when both skills report PASS, the manual flow shows the expected elements and `XCE_JOBINDEX` values, and the three bug records are fixed
- _Requirements: 1.1, 1.10, 5.1, 5.2, 6.1, 8.4_
- _Depends: 3.2, 7.4, 8.2, 8.3, 9.1, 9.2_

54
.wolf/buglog.json
View file

@ -16693,6 +16693,60 @@
"0fe27d4b"
],
"record": ".wolf/bugs/bug-1897.md"
},
{
"id": "bug-1898",
"status": "open",
"status_updated": "2026-10-01",
"status_note": "newly logged",
"component": "apiserver",
"severity": "wrong-result",
"timestamp": "2026-10-01T04:26:54Z",
"last_seen": "2026-10-01T04:26:54Z",
"occurrences": 1,
"error_message": "apiserver ignores [..] job_id/job_name filters: CompConditions never evaluated\n\n**Observed:** found by code review, no failing run captured yet. Every REST selection value containing `[` and `]` is turned into a `CompFilterCondition`, removed from the plain filters and passed to `utils.BuildQueryOptions` as `CompConditions`, but `store.ExecuteQuery` evaluates only `opts.Conditions`:\n```go\n// src/frontend/apiserver/store/query.go:19-22\nif len(opts.Conditions) > 0 {\n rows = FilterRows(rows, opts.Conditions)\n}\n// opts.CompConditions (utils/types.go:96, set at :454) is never read anywhere\n```\nSo the bracketed selection is silently dropped: `GetJobsInfo` (`job_id`/`job_name` with `[...]`, `controllers/jobs.go:186,494-510`), `GetJobByIDs` (`:780,961`) and `FetchJobList` (`:3311,3420-3436,3463`, used by filter-based `ControlJob`) return every row that matches the remaining filters.\n\n**Expected:** a selection such as `job_id:eq:X[2]` or `job_name:eq:foo[1-3]` returns only the matching array elements; a control command `xcecli jobs kill -J 'foo[1]'` acts only on element 1 of arrays named `foo`.\n**Impact:** wrong result, and for control a wrong target: `xcecli jobs list X[2]` lists unrelated jobs; filter-based control with `-J 'name[k]'` and no other filter matches all jobs, and its \"most recent job\" rule (`jobs.go:2752-2757`) then kills/stops/resumes the newest job in the cluster instead of the named element. `ControlJobByID` is not affected (it post-filters with `foundJob`).\n**Found by:** review (design review of spec `job-array-one-based-index`, 2026-10-01)",
"file": "src/frontend/apiserver/store/query.go, src/frontend/apiserver/utils/types.go, src/frontend/apiserver/controllers/jobs.go",
"root_cause": "TODO: required for fixed/partial",
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
"tags": [
"apiserver",
"job-array",
"query",
"control",
"code-review"
],
"related_bugs": [
"bug-1896"
],
"fix_commits": [],
"record": ".wolf/bugs/bug-1898.md"
},
{
"id": "bug-1899",
"status": "open",
"status_updated": "2026-10-01",
"status_note": "newly logged",
"component": "apiserver",
"severity": "wrong-result",
"timestamp": "2026-10-01T05:10:35Z",
"last_seen": "2026-10-01T05:10:35Z",
"occurrences": 1,
"error_message": "ControlJobByID re-dispatches earlier selections on every later selection of one request\n\n**Observed:** found by code review, no failing run captured yet. In `ControlJobByID` the slice `successJobs` is declared once before the loop over the request's selections (`src/frontend/apiserver/controllers/jobs.go:2464`), is appended to inside that loop (`:2575-2583`), and the dispatch loop over the whole accumulated slice also sits inside it (`:2594-2617`):\n```go\nsuccessJobs := []map[string]interface{}{} // :2464, outside\nfor _, ujobFilterCdArray := range allFiltCdArrays { // one per selection\n ...\n successJobs = append(successJobs, ...) // :2575\n if len(successJobs) > 0 {\n for _, successJob := range successJobs { // re-sends all earlier ones\n err = backend.ControlJob(...) // :2611\n```\n**Expected:** each selected job receives the control action once per request; the response lists it once.\n**Impact:** wrong result: `xcecli jobs kill X[1],Y[2]` (or `kill A,B`) signals the first selection's jobs once per later selection (N selections -> the first job is sent N times); a repeated kill of an already-killed job can make the backend call fail and the request report an error; `BuildBatchOperationResponse` receives the accumulated slice, so the response lists earlier jobs repeatedly. Distinct from the out-of-scope duplicate-signal case of one element selected twice (job-array-one-based-index requirements).\n**Found by:** review (design review round 11 of spec `job-array-one-based-index`, 2026-10-01)",
"file": "src/frontend/apiserver/controllers/jobs.go",
"root_cause": "TODO: required for fixed/partial",
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
"tags": [
"apiserver",
"control",
"job-array",
"code-review"
],
"related_bugs": [
"bug-1898",
"bug-1896"
],
"fix_commits": [],
"record": ".wolf/bugs/bug-1899.md"
}
]
}

60
.wolf/bugs/bug-1898.md Normal file
View file

@ -0,0 +1,60 @@
---
id: "bug-1898"
status: "open"
status_updated: "2026-10-01"
status_note: "newly logged"
component: "apiserver"
severity: "wrong-result"
timestamp: "2026-10-01T04:26:54Z"
last_seen: "2026-10-01T04:26:54Z"
occurrences: 1
files: ["src/frontend/apiserver/store/query.go", "src/frontend/apiserver/utils/types.go", "src/frontend/apiserver/controllers/jobs.go"]
tags: ["apiserver", "job-array", "query", "control", "code-review"]
related: ["bug-1896"]
---
# bug-1898: apiserver ignores [..] job_id/job_name filters: CompConditions never evaluated
## Description
**Observed:** found by code review, no failing run captured yet. Every REST selection value containing `[` and `]` is turned into a `CompFilterCondition`, removed from the plain filters and passed to `utils.BuildQueryOptions` as `CompConditions`, but `store.ExecuteQuery` evaluates only `opts.Conditions`:
```go
// src/frontend/apiserver/store/query.go:19-22
if len(opts.Conditions) > 0 {
rows = FilterRows(rows, opts.Conditions)
}
// opts.CompConditions (utils/types.go:96, set at :454) is never read anywhere
```
So the bracketed selection is silently dropped: `GetJobsInfo` (`job_id`/`job_name` with `[...]`, `controllers/jobs.go:186,494-510`), `GetJobByIDs` (`:780,961`) and `FetchJobList` (`:3311,3420-3436,3463`, used by filter-based `ControlJob`) return every row that matches the remaining filters.
**Expected:** a selection such as `job_id:eq:X[2]` or `job_name:eq:foo[1-3]` returns only the matching array elements; a control command `xcecli jobs kill -J 'foo[1]'` acts only on element 1 of arrays named `foo`.
**Impact:** wrong result, and for control a wrong target: `xcecli jobs list X[2]` lists unrelated jobs; filter-based control with `-J 'name[k]'` and no other filter matches all jobs, and its "most recent job" rule (`jobs.go:2752-2757`) then kills/stops/resumes the newest job in the cluster instead of the named element. `ControlJobByID` is not affected (it post-filters with `foundJob`).
**Found by:** review (design review of spec `job-array-one-based-index`, 2026-10-01)
## Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: n/a (Go apiserver, pure-Go query path); host glibc 2.40
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (xce.build); the defect is in pure Go and independent of the profile
- Code: scheduler-framework @ 8f5ba635-dirty
- Deployment: any (single-node or HA) with at least one job array and one other job
## Reproduction
1. Read `src/frontend/apiserver/store/query.go:15-42` (`ExecuteQuery`) and `grep -rn CompConditions src/frontend/apiserver --include=*.go` (only `utils/types.go:96,454` -- never read).
2. Unit-test sketch: put an array element `X[1]`, `X[2]` and an ordinary job `Y` into `store.Jobs`; call `GetJobsInfo` with `filter=job_id:eq:<X>[2]`; expected only `X[2]`, today all three rows.
3. Live: submit `xcecli jobs submit -J 'foo[1-2]' sleep 300` and an ordinary `sleep 300`; run `xcecli jobs kill -J 'foo[1]'` -- the newest job is signalled, not `foo[1]`.
- Frequency: deterministic (by code reading; not yet run)
- Evidence: not kept (code reading only)
## Root cause
TODO: required for fixed/partial
## Fix
TODO: required for fixed/partial
## Verification
- Test: TODO: name + file; RED before the fix, GREEN after
- Suites: TODO: suite results and build profile
- Live: TODO: soak/deploy run, or n/a and why
## References
- Related: bug-1896 (found while designing its fix, spec job-array-one-based-index)

59
.wolf/bugs/bug-1899.md Normal file
View file

@ -0,0 +1,59 @@
---
id: "bug-1899"
status: "open"
status_updated: "2026-10-01"
status_note: "newly logged"
component: "apiserver"
severity: "wrong-result"
timestamp: "2026-10-01T05:10:35Z"
last_seen: "2026-10-01T05:10:35Z"
occurrences: 1
files: ["src/frontend/apiserver/controllers/jobs.go"]
tags: ["apiserver", "control", "job-array", "code-review"]
related: ["bug-1898", "bug-1896"]
---
# bug-1899: ControlJobByID re-dispatches earlier selections on every later selection of one request
## Description
**Observed:** found by code review, no failing run captured yet. In `ControlJobByID` the slice `successJobs` is declared once before the loop over the request's selections (`src/frontend/apiserver/controllers/jobs.go:2464`), is appended to inside that loop (`:2575-2583`), and the dispatch loop over the whole accumulated slice also sits inside it (`:2594-2617`):
```go
successJobs := []map[string]interface{}{} // :2464, outside
for _, ujobFilterCdArray := range allFiltCdArrays { // one per selection
...
successJobs = append(successJobs, ...) // :2575
if len(successJobs) > 0 {
for _, successJob := range successJobs { // re-sends all earlier ones
err = backend.ControlJob(...) // :2611
```
**Expected:** each selected job receives the control action once per request; the response lists it once.
**Impact:** wrong result: `xcecli jobs kill X[1],Y[2]` (or `kill A,B`) signals the first selection's jobs once per later selection (N selections -> the first job is sent N times); a repeated kill of an already-killed job can make the backend call fail and the request report an error; `BuildBatchOperationResponse` receives the accumulated slice, so the response lists earlier jobs repeatedly. Distinct from the out-of-scope duplicate-signal case of one element selected twice (job-array-one-based-index requirements).
**Found by:** review (design review round 11 of spec `job-array-one-based-index`, 2026-10-01)
## Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: n/a (pure-Go control handler); host glibc 2.40
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (xce.build); defect is in pure Go, independent of profile
- Code: scheduler-framework @ 8f5ba635-dirty
- Deployment: any (single-node or HA) with at least two controllable jobs
## Reproduction
1. Read `src/frontend/apiserver/controllers/jobs.go:2464-2620` (`ControlJobByID`): the `backend.ControlJob` loop over `successJobs` is inside the per-selection loop.
2. Live: submit two `sleep 300` jobs A and B; `xcecli jobs stop A,B`; the controller log shows the stop for A twice.
- Frequency: deterministic (by code reading; not yet run)
- Evidence: not kept (code reading only)
## Root cause
TODO: required for fixed/partial
## Fix
TODO: required for fixed/partial
## Verification
- Test: TODO: name + file; RED before the fix, GREEN after
- Suites: TODO: suite results and build profile
- Live: TODO: soak/deploy run, or n/a and why
## References
- Related: bug-1898 (same review), bug-1896 (spec job-array-one-based-index restructures this handler)