myxce/.kiro/specs/job-array-one-based-index/tasks.md

Implementation Plan

• 1. Foundation: bounded job identities and array ID allocation in libxce

• 1.1 Add the bounded job-identity and element-selection parser

  • Parse a written identity with an optional single element or element range under the fixed grammar (no sign, no inner whitespace, optional surrounding spaces/tabs), accumulating digits with an overflow guard
  • Evaluate the checks in the fixed order: syntax, field widths, alignment of the array identity when an element part is present, element-number bound, reversed range; report only the first failure
  • Provide the array-marking test (tag equals the array tag only), the element-identity composition helper and a message for each rejection
  • Add the shared selection-cases conformance table as a libxce unit test registered in the libxce test sources and runner
  • Done when the libxce suite runs the conformance table with every row passing, including X[0], X[01], the width cases (0-0-T-1048576[1]) and the precedence rows
  • Requirements: 1.5, 5.7, 5.8, 5.12, 5.14, 5.15, 8.1

• 1.2 Re-implement the existing identity parse on top of the new parser

  • Keep its signature; make it NULL-safe, bounded, and reject element ranges and every rejected selection with an invalid-argument result
  • Keep the existing format/parse round-trip tests passing unchanged
  • Callers outside libxce (the harness single-job commands and the scheduler REPL, whose fall-back to plain number parsing is accepted unchanged) see only the stricter acceptance
  • Done when the libxce suite passes with new cases for a NULL buffer, a range, an oversized field and an index above 32767 all returning invalid-argument, the existing round trips still succeed, and the scheduler and harness suites still pass
  • Requirements: 5.7, 5.14, 5.15

• 1.3 (P) Add array ID allocation numbered from 1

  • Allocate the first element of an n-element array (1 to 32767) as the next whole 32768 block with element number 1, advancing the context past the block
  • Refuse a block that would leave the 20-bit sequence with an overflow result and leave the context unchanged; refuse n of 0 or above 32767
  • Leave the existing ordinary-ID allocation untouched in this task (its multi-element path is removed in 9.1, after its callers move)
  • Done when the libxce suite shows: index 1 for the first element and index n for the last; 32767 accepted, 0 and 32768 rejected; a seed above 31 blocks overflows while exactly 31 blocks and below 30 blocks succeed (each seeded case pins the context timestamp and retries if the second changed)
  • Requirements: 1.1, 1.2, 1.3, 4.1
  • Boundary: IdgenArray

• 2. Controller: array admission and collision detection

• 2.1 Classify every submission before any other check

  • Add the distinct "array numbering starts at 1" error status and its message text to the shared error table
  • Classify a submission from its first ID and 64-bit size only: start-at-0 array first; then size 0, an unknown marking, an unmarked multi-element submission, and a last element above 32767 (computed without addition so no size can wrap)
  • Run the classification as the first step after the reply is initialised, before the existing field checks and the before-event hook; answer start-at-0 with the new status plus an ERROR log saying numbering starts at 1 and the client must be upgraded, and the other classes with invalid-argument; store nothing
  • Create the controller validation test file and register it in the controller test sources and runner; move the existing commit-barrier reply and rollout tests to tagged first IDs and at most 32767 elements
  • Caveat: from this task until 4.1, 6.2 and 9.2 land, live array submissions from the current harness and API server (which still number from 0) are rejected by design; unit suites are unaffected; do not run the multihost or HA-soak skills in between
  • Done when the controller suite passes with cases for every class, the 7.9 precedence, sizes of 2^32+2 and the maximum 64-bit value with index 2, a size-0 submission answered, and an index-0 array with an empty queue answered with the new status and its log line
  • Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.9
  • Depends: 1.1

• 2.2 Make answering a registered submission and releasing it one atomic step

  • Extend in-flight registry entries with an array flag and array identity; two array entries with the same array identity overlap
  • Add one reply-and-remove operation that sends the reply and unlinks the entry while holding the registry lock (the reply is non-blocking), and use it on every path after registration: the payload-build and allocation-failure early-outs, commit failure or timeout, and apply success (inside the existing one-shot reply guard)
  • Stop the async-context release from removing registry entries, so it is the only removal point
  • Add the test-only hook and registry accessors (trylock, locked and unlocked overlap checks) under the testing build flag
  • Done when the controller suite shows, from inside the hook, that the registry lock is held, the reply is already queued and the entry is still present; that the entry is gone afterwards; and that a retry of the same submission registered after the reply survives the original's context release
  • Requirements: 7.7, 7.10, 7.11

• 2.3 Check element and array-identity collisions after registration

  • Move the job-table collision checks after registration; on a hit, reply duplicate-submission and release through the atomic step
  • For array submissions, one scan under a single job-table read lock over the 32768 element keys of the array's block answers both the element-range and the array-identity check; unmarked submissions keep the existing range check
  • Done when the controller suite shows: an existing element X|3 blocks a new X|5 with duplicate-submission; two concurrent submissions with the same array identity, the second gets duplicate-submission while the first is unanswered; once the first is answered with a failure, a new one with the same identity is accepted
  • Requirements: 7.7, 7.10, 7.11

• 3. Scheduler and worker

• 3.1 (P) Guard scheduler array expansion and log the array identity

  • Expand an array push only when the first ID carries the array marking and its size fits within the block (checked without addition); otherwise log an ERROR naming the array identity and size and expand nothing
  • Log the received array by its array identity
  • Move the existing expansion tests (translation, array-expand and wire end-to-end tests) to tagged block-plus-one first IDs
  • Done when the scheduler suite passes, with tagged arrays expanding to elements 1..n and untagged, overflowing and maximum-64-bit-size arrays not expanded
  • Requirements: 5.9
  • Boundary: SchedGuard
  • Depends: 1.1

• 3.2 (P) Expose the element number to running jobs

  • Set XCE_JOBINDEX to the element number for jobs carrying the array marking and to 0 otherwise
  • Skip a submission-supplied XCE_JOBINDEX, logging a WARN that names the job and the ignored variable; leave XCE_JOBID handling unchanged
  • Add the worker test and register it in the worker test build
  • Done when the worker suite shows the element number for an array element (including a one-element array), 0 for an ordinary job, a user value ignored with its WARN, and XCE_JOBID unchanged
  • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 7.8
  • Boundary: WorkerEnv
  • Depends: 1.1

• 4. Test harness

• 4.1 Submit harness arrays numbered from 1 (fixes bug-1896 for the harness)

  • Parse -a strictly (digits only, leading zeros allowed): 1 submits an ordinary job; 2..32767 allocate with the new array allocator and check its result; missing, non-digit, 0 and above-32767 values are refused with the accepted range or limit, submitting nothing
  • Bound the random sequence seed so the first array of a run cannot overflow
  • Print the array identity on success and set $jobid to it; update the jobsub help text
  • Add strict-size cases to the command-utility unit test and pty cases for -a 0 and -a 40000
  • Done when the harness unit and pty tests pass and jobsub -a 3 against a running controller (upgraded or not) creates elements [1], [2], [3] with $jobid naming the array
  • Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 9.3
  • Depends: 1.3

• 4.2 Substitute $jobid[...] in harness command lines

  • Move the existing autovariable substitution into its own unit-testable module with an explicit context for $jobid, $? and $jobcount, keeping today's token rule, messages and buffer-size result
  • Expand $jobid followed by [ keeping the rest of the token; fail as for an unset $jobid when it is unset; leave ! and # lines untouched
  • Register the module in the harness build and its new unit test
  • Done when the autovariable unit test passes for $jobid[3], $jobid[3]x, unset $jobid, $?, $jobcount set and unset, and !/# lines, and the existing harness tests still pass
  • Requirements: 4.10, 4.11

• 4.3 Parse single-job harness arguments strictly

  • Add one argument helper that trims spaces/tabs, applies the libxce selection parser, refuses element ranges (including reversed ones) with the single-element message, and reports whether an element number was given
  • Add a not-found reporter that adds the <array-id>[k] hint for identities given without an element number
  • Use both in every single-job command (including jobinfo per -j) and make expect-job parse its argument the same way; the job-list lookup behind it compares job numbers instead of formatted text
  • Done when the command-utility tests pass (ranges refused including X[3-1], [0] accepted), a pty test shows jobkill -j 1-1-1-0[1-3] refused with the single-element message before any RPC, and no single-job command can send job 0 for a rejected argument
  • Requirements: 4.7, 4.8, 4.9, 5.7, 5.8, 5.13, 5.14, 5.15, 8.2
  • Depends: 1.1, 1.2

• 5. API server foundation

• 5.1 Add backend ID helpers and the array ID wrapper

  • Wrap the libxce array allocator for the API server, with a non-cgo stub that returns an error
  • Add strict written-identity parsing with field-width checks, an array-capable key test and element-ID composition as pure helpers
  • Done when the backend unit tests pass for the strict parser (width rejections), the array-capable test and element composition, and both the cgo and non-cgo builds compile
  • Requirements: 1.1, 1.2, 5.8, 5.14
  • Depends: 1.3

• 5.2 (P) Clamp the array size limit setting

  • Move the limit rule into a pure resolver: unset or 0 gives 1000, 1..32767 used as is, above 32767 clamped to 32767, negative refuses startup; drop the old upper-bound rejection
  • Log a startup warning with the configured and effective values when clamped; update the shipped config's comment for the setting (default 1000, at most 32767)
  • Done when the config tests pass for 0, 1, 32767, 32768, 3000000000 and -1, and a clamped startup logs the warning
  • Requirements: 1.6, 1.7, 1.8, 1.9
  • Boundary: ApiConfig

• 5.3 (P) Evaluate row predicates in the query pipeline

  • Add row predicates to the query options and apply them after the plain conditions and before sorting and pagination
  • Done when the store query tests show predicates filtering before pagination, with page boundaries and bookmarks consistent
  • Requirements: 5.2, 5.11
  • Boundary: store query pipeline

• 5.4 Add handler test seams and the non-cgo handler test fixture

  • Route ID generation, array ID generation, job submission and control dispatch through replaceable package variables
  • Create the non-cgo handler test file with a fixture that saves the stores, sets up auth/RBAC, then seeds jobs (controllable status) and their queues and the array-size config, and restores everything on cleanup; the dispatch replacement records each dispatched job
  • Done when a smoke handler test in the new file submits an ordinary job and controls one seeded job through the seams with the dispatch recorded, and the existing apiserver tests still pass
  • Depends: 5.1

• 6. API server: array submission

• 6.1 (P) Parse submit index expressions

  • Accept only name[1-N] and name[1]; reject interactive arrays, malformed expressions or invalid names (quoting the name), lists and steps, %N, start 0, reversed ranges, start above 1 and sizes above the limit, each with its specific message, reporting only the lowest-numbered broken rule
  • Saturate oversized numbers so the rule order still decides; accept leading zeros
  • Done when the index-expression unit tests pass for every rule, the precedence rows ([0-0:2], [99999999999999999999], [99999999999999999999-5], [0-99999999999999999999]), leading zeros, [1] vs [1-1] vs [5] and the limit edge
  • Requirements: 1.5, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9
  • Boundary: ApiSubmitExpr

• 6.2 Create arrays from bracketed submissions and report the array (fixes bug-1896 for REST and the client)

  • Make the job pre-check return the element count and whether the job is an array; call the expression parser only for bracketed names (passing the interactive flag and the configured limit), store the name without its index part, and keep the existing path for names without brackets
  • Map expression errors to 400 with their message unchanged and keep every other pre-check error as today
  • Allocate array IDs through the array wrapper (one-element arrays included); keep the first element for the submit-time %I/%J substitution
  • Report the array identity, the element count and a message naming both; log new arrays by array identity in the submit path and in the array dispatch log
  • Update the existing pre-check test call sites to the new result shape
  • Done when the handler tests show arr[1-3] creating three elements with the stored name arr, name[1] creating a one-element array with array_size 1, a 2.x rejection returning 400 with its exact message, plain names (my job, foo%5, foo%) failing exactly as today, and %I = 1 and %J = <array-id>[1]
  • Requirements: 1.1, 1.2, 1.3, 1.4, 2.10, 3.1, 3.2, 3.3, 5.9, 7.8
  • Depends: 5.2, 5.4, 6.1

• 7. API server: element selection in queries and control

• 7.1 (P) Parse REST identity and name selections

  • Parse identity selections (strict written form or base-10 decimal) and name selections (name part may be empty, no whitespace or brackets) under the shared grammar and check order, including the marked-decimal-without-index rejection
  • Return a rule tag and a message that names the selection as written
  • Done when the selection unit tests pass for the shared conformance table plus decimal-key cases (marked decimal without index, 1e3, -1) and the name rows
  • Requirements: 1.5, 5.5, 5.7, 5.8, 5.12, 5.14, 5.16, 5.17
  • Boundary: ApiSel
  • Depends: 5.1

• 7.2 Build selection predicates

  • Turn each filter entry containing a bracketed value into one composed predicate (values OR'ed within the entry, entries AND'ed with each other and with plain conditions), converting the whole entry and only that entry
  • Honour the operator: eq/in (and between on job identities) select, notin excludes, name pattern operators reuse today's condition semantics on the name part, ne and comparison/null operators keep today's "operator doesn't support" error, operators outside the field's table keep today's error
  • Match identity elements only among array-marked rows, match empty names under exact operators to no row, and normalise written bare keys to decimal on both the plain and converted paths
  • Done when the pure predicate tests pass for every operator rule, empty-name semantics with an unnamed array row, and the same bare key matching alone and mixed with a bracketed value
  • Requirements: 5.2, 5.6, 5.11, 7.8, 8.2
  • Depends: 5.3, 7.1

• 7.3 Route query handlers through selections (fixes bug-1898)

  • Validate every filter entry and key before any store access (a pre-pass ahead of the existing filter loop, including before host resolution), returning the first rejection as 400 with its specific message
  • Apply width and marked-decimal checks to identity-operator keys and converted entries, and the alignment check only to keys with an element part
  • Evaluate selections through the predicates on the job listing and job-detail paths; remove the unevaluated compound conditions and the old index-section parsing on these paths (the filter-based control path follows in 7.5)
  • Done when the handler tests show job_id:eq:X[2] returning only element 2, GET /jobs/X[1-3] returning elements 1-3, <ordinary-id>[0] listing nothing, an invalid selection combined with a host filter returning 400, and the operator, intersection and mixed-value rows from the design
  • Requirements: 5.2, 5.5, 5.6, 5.7, 5.8, 5.11, 5.12, 5.14, 5.16, 5.17
  • Depends: 5.4, 7.2

• 7.4 Control jobs by identity: validate first, dispatch once (fixes bug-1899)

  • Replace the index-section parsing in this handler with the REST selection parser; validate all selections before matching; give the element matcher an explicit-index flag so <array-id>[0] never falls back to the whole array or an ordinary job
  • Collect the de-duplicated jobs to act on and, separately, one not-found result per missing number per selection carrying the would-be element ID; validate each job once after the loop, then dispatch each accepted job once (a dispatch error stays a 500)
  • Update the existing element-matcher test to the new signature with a tagged fixture
  • Done when the handler tests show two selections dispatching each element exactly once, X[1-3] acting on existing elements and reporting missing numbers with their IDs, X[0] with no element 0 reporting not found and dispatching nothing, a pre-upgrade element 0 dispatched alone, and a valid-then-invalid request dispatching nothing
  • Requirements: 5.1, 5.3, 5.4, 8.2, 8.3
  • Depends: 5.4, 7.2

• 7.5 Control jobs by filter with element selections

  • Validate a bracketed name filter before fetching (unbracketed names, including command-line names with spaces, unchanged), returning 400 with the specific message
  • Route the filter-based job fetch's bracketed name handling through the name parser and the selection predicates; with its last use gone, delete the old index-section parser
  • Apply the most-recent-job rule only when the filter has no element number and all-jobs is not set, acting on the newest row alone when it is ordinary or the only stored element of its array
  • Add the would-be element ID to not-found results
  • Done when no use of the old index-section parser remains and the handler tests show -J 'foo[40000]' returning 400 with the 5.7 message, -J '[1]' with an unnamed array dispatching nothing, a one-element array as newest row dispatched alone, and -J 'foo[1]' with two one-element arrays named foo dispatching both
  • Requirements: 5.1, 5.2, 5.10, 8.3
  • Depends: 5.4, 7.2, 7.4

• 8. Command-line client

• 8.1 (P) Parse identities and selections in the client

  • Add the client selection parser (written and decimal keys, the shared order, alignment and element bound on decimal keys with an element part, range refusal for queries only after the other checks), the bracket-aware list splitter and the raw -J name check
  • Make identity parsing bounded and change the API-key conversion to take whether ranges are allowed; keep a legacy conversion for peek; update its two command callers mechanically (control allows ranges, the list filter does not) so the client keeps building
  • Update the conversion tests: "123456" and "32768[3]" pass through, "123456[3]" is rejected under 5.8, a range passes only when allowed; move peek and its test to the legacy conversion
  • Done when the client builds and its format, selection and peek tests pass, including the shared conformance table and SplitSelections("X[1,3],Y")
  • Requirements: 1.5, 5.5, 5.7, 5.8, 5.12, 5.14, 5.16
  • Boundary: CliSel, CliSplit

• 8.2 Use the client parser in list and control commands

  • Split ID lists only at commas outside brackets; validate every selection before any request; control by identity allows ranges, listing does not
  • Check raw -J values containing brackets before wildcard mapping, taking the search rule from the name part only for bracketed values
  • Print control results from each result's own job ID when present, falling back to today's rendering; add the control result's job ID field to the client model
  • Update control and list help to show <array-id>[k] and <array-id>[a-b]
  • Done when a new control-command test against a local test server sees X[1-3] arrive as <decimal>[1-3] in the request path, -J 'foo[1]*' is refused before any request, and a pre-upgrade element 0 result prints as <array-id>[0]
  • Requirements: 5.1, 5.2, 5.16, 8.2, 8.3, 9.2
  • Depends: 8.1

• 8.3 Show the array in submit output and help

  • Add the element count to the client's submit response model; print the server's submit message with the array identity formatted (the count comes from the message); describe name[1-N], name[1], numbering from 1 and the limit (default 1000, at most 32767) in submit help
  • Done when a submit-output test shows Job <array-id> is submitted ... (3 elements) with the formatted array identity and the help text lists the supported forms
  • Requirements: 3.1, 3.2, 9.1
  • Depends: 8.2

• 9. Integration and cleanup

• 9.1 Restrict ordinary ID allocation to single jobs

  • Remove the multi-element path from ordinary ID allocation now that the harness and the API server use the array allocator
  • Migrate the existing ID-allocation test loop that requests sizes above 1 to the array allocator
  • Done when the old multi-element test loop is gone or migrated, ordinary allocation with size 2 returns invalid-argument in the libxce suite, and the harness and API server builds and suites still pass
  • Requirements: 1.3, 4.2
  • Depends: 4.1, 6.2

• 9.2 Update test skills for elements numbered from 1

  • Make the HA soak and multihost verification scripts and their skill descriptions verify elements 1..N (first, middle, last) instead of 0..N-1
  • Done when both scripts pass a syntax check and every element check in them refers to indices 1..N
  • Requirements: 4.1, 4.5

• 9.3 Document the supported array syntax

  • Add a "Job arrays" section to the user documentation listing the supported forms and the LSF features not supported (other starts, lists, steps, %N, per-element %I/%J, interactive arrays, LSB_JOBINDEX/LSB_JOBID)
  • Mark the superseded statements in the job-array-single-wal-record and xce-test-e2e-primitives specs, update the numbering notes in the controller-job-cleanup and job-dependency briefs, and add a roadmap entry
  • Done when the user documentation has the section with every item 9.4 lists and each referenced spec document carries its note
  • Requirements: 9.4

• 10. Validation

• 10.1 Run every affected suite

  • Build with tests and run the libxce, controller, scheduler, worker, harness, apiserver (non-cgo and the cgo suite, including a real array submit asserting the reported array identity and array_size) and client suites
  • Done when every suite passes with zero failures under the ASAN profile and the cgo apiserver submit test passes
  • Requirements: 1.1, 1.10, 3.1, 3.2, 8.1, 8.4

• 10.2 Verify end to end on a live cluster

  • Run the multihost verification (elements [1] and [2] reach DONE across both hosts) and a six-round HA soak confirming elements 1..N survive failover
  • Through the client: submit arr[1-3] and list [1]..[3]; kill <array-id>[2] and see only element 2 act; run a job script that prints XCE_JOBINDEX
  • Close bug-1896, bug-1898 and bug-1899 through the buglog with their fix commits
  • Done when both skills report PASS, the manual flow shows the expected elements and XCE_JOBINDEX values, and the three bug records are fixed
  • Requirements: 1.1, 1.10, 5.1, 5.2, 6.1, 8.4
  • Depends: 3.2, 7.4, 8.2, 8.3, 9.1, 9.2

Implementation Notes

• 1.2: REGEX_JOBID, REGEX_JOBARRAY_ID and N_MAX_FIELDS_JOBID (src/include/config.h) are now unused, and comments still describe regex-based parse_jobid in src/scheduler/src/repl.c:143, src/harness/testsuite/test_test_cli.c:2324-2326 and src/frontend/xcecli/common/format/jobid.go:86-93; clean these up in 4.3 / 8.1 / 9.x where those files are touched. A direct parse_jobid("...[0]") accept test is a cheap addition when next touching test_jobsel.c.
• 1.3: plain idgen() still has no sequence-overflow guard; after idgen_array takes block 31 the context sequence is 2^20, so a plain idgen() in the same second would spill into the timestamp (pre-existing class). Add an -EOVERFLOW check to plain idgen() in 9.1.
• 2.2: two extra XCE_TESTING seams exist (jobsubmit_test_release_hook, jobsubmit_inflight_try_add_for_test); the controller test fixture has no WAL, so async commit-failure runs synchronously inside handle_jobsubmit — reuse it in 2.3. Stale comment at src/controller/testsuite/test_commit_barrier_reply.c:1458 (claims release via jobsubmit_async_ctx_release) — fix in 2.3.
• 4.1: after jobsub -a N, $jobid is the array identity, so expect-job $jobid currently reports not found; 4.3 must make it behave per 4.7/4.9 (not-found hint naming <array-id>[k], or exact-ID match). Nit for 4.3: the refused early return in cmd_jobsub sits between the bug-786 ack comment and the code it describes — move it above the comment.