From 6439e5852f5890b40c1f00ef21864527f50827cc Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Thu, 1 Oct 2026 13:46:07 +0000 Subject: [PATCH] docs(job-array-one-based-index): document the supported array syntax (task 9.3) src/docs/xce.md gains a 'Job arrays' section: name[1-N] and name[1], numbering from 1, the API-server limit (default 1000, at most 32767), [k] and [a-b] selection (with the peek/restart caveat), XCE_JOBINDEX, the harness jobsub -a N / $jobid[k], and every LSF array feature this release does not support. The superseded statements in job-array-single-wal-record and xce-test-e2e-primitives are annotated (history kept), the controller-job-cleanup and job-dependency briefs describe 1-based numbering, and the roadmap gains an entry. Co-Authored-By: Claude Opus 5.5 (1M context) --- .kiro/specs/controller-job-cleanup/brief.md | 6 +- .../specs/job-array-one-based-index/tasks.md | 2 +- .../requirements.md | 2 + .kiro/specs/job-dependency/brief.md | 6 +- .kiro/specs/xce-test-e2e-primitives/design.md | 1 + .../xce-test-e2e-primitives/requirements.md | 6 ++ .kiro/steering/roadmap.md | 1 + src/docs/xce.md | 77 +++++++++++++++++++ 8 files changed, 98 insertions(+), 3 deletions(-) diff --git a/.kiro/specs/controller-job-cleanup/brief.md b/.kiro/specs/controller-job-cleanup/brief.md index b6f5697e..d2560232 100644 --- a/.kiro/specs/controller-job-cleanup/brief.md +++ b/.kiro/specs/controller-job-cleanup/brief.md @@ -84,7 +84,11 @@ Codebase exploration, 2026-09-29. (`opget.c:159-176`, `xceapi.c:787`). Control operations return `-ENOENT`. - Worker resync ignores unknown jobids (`resync.c:638-647`), which is safe. - Array jobs: there is no parent object. Each element is an independent job with - a jobid in `[base, base+array_size)`. + its own jobid. Elements are numbered 1..N per `job-array-one-based-index` + (2026-10-01): element k of array `` is `[k]`, k = 1..N + (arrays submitted before that change may still have elements 0..N-1). The + earlier note here, jobid in `[base, base+array_size)`, described numbering + from 0. - There is no accounting or history file, so a removed job leaves no record. **Job dependencies** diff --git a/.kiro/specs/job-array-one-based-index/tasks.md b/.kiro/specs/job-array-one-based-index/tasks.md index 74a2edf6..d94cac5b 100644 --- a/.kiro/specs/job-array-one-based-index/tasks.md +++ b/.kiro/specs/job-array-one-based-index/tasks.md @@ -217,7 +217,7 @@ - Done when both scripts pass a syntax check and every element check in them refers to indices 1..N - _Requirements: 4.1, 4.5_ -- [ ] 9.3 Document the supported array syntax +- [x] 9.3 Document the supported array syntax - Add a "Job arrays" section to the user documentation listing the supported forms and the LSF features not supported (other starts, lists, steps, `%N`, per-element `%I`/`%J`, interactive arrays, `LSB_JOBINDEX`/`LSB_JOBID`) - Mark the superseded statements in the job-array-single-wal-record and xce-test-e2e-primitives specs, update the numbering notes in the controller-job-cleanup and job-dependency briefs, and add a roadmap entry - Done when the user documentation has the section with every item 9.4 lists and each referenced spec document carries its note diff --git a/.kiro/specs/job-array-single-wal-record/requirements.md b/.kiro/specs/job-array-single-wal-record/requirements.md index 4536fe44..1227708d 100644 --- a/.kiro/specs/job-array-single-wal-record/requirements.md +++ b/.kiro/specs/job-array-single-wal-record/requirements.md @@ -64,6 +64,8 @@ This feature builds on the apply seam introduced by `raft-apply-decoupling` (now 2. The Controller shall support querying, cancelling, and completing an individual array element by its per-element identity exactly as it does for job elements created via the current per-element path. 3. This feature shall not change any client-visible identity, status, or addressing behavior for an individual array element. +> **Superseded by job-array-one-based-index (2026-10-01):** the commitment above that element identities stay unchanged for clients, with elements numbered from 0 (indices `[0, size)`), no longer holds. Since `job-array-one-based-index`, a submitted array's elements are numbered 1 through N (`name[1-N]`), each element is shown and addressed as `[k]`, and submission reports the array identity rather than an element identity (see that spec's Requirements 1, 3 and 5). The single-WAL-record storage, replication and recovery of arrays specified here are unchanged. Kept for history. + ### Requirement 6: Single push to scheduler and API server, with local expansion **Objective:** As a maintainer of scheduler and API-server load, I want both the scheduler and the API server to receive one push per committed array, so that push volume to either consumer no longer scales linearly with array size. diff --git a/.kiro/specs/job-dependency/brief.md b/.kiro/specs/job-dependency/brief.md index 657740c5..c9158564 100644 --- a/.kiro/specs/job-dependency/brief.md +++ b/.kiro/specs/job-dependency/brief.md @@ -43,7 +43,11 @@ Codebase exploration, 2026-09-29. - **Controller job table.** `c->jobmgr` holds jobs. The scheduler's `core->jobs` also keeps every job forever today, which `controller-job-cleanup` changes. - **Array jobs.** There is no parent object. Each element is an independent job - with jobid `base+i`, so an element can be referenced by its own ID. + with its own jobid. Elements are numbered 1..N per `job-array-one-based-index` + (2026-10-01) and are referenced as `[k]`, k = 1..N; a bare element + jobid is rejected by the CLI/REST listing and control commands (use + `[k]`; peek and restart still take the element's own jobid). The earlier note here, + jobid `base+i` with i from 0, described numbering from 0. - **Submit paths:** REST (apiserver, fed every job by controller push, cgo into libxce), `xcecli jobs submit`, the harness `xce_test jobsub` (`harness/test.c:3575`), and libxce directly. diff --git a/.kiro/specs/xce-test-e2e-primitives/design.md b/.kiro/specs/xce-test-e2e-primitives/design.md index c77aa5d2..17649664 100644 --- a/.kiro/specs/xce-test-e2e-primitives/design.md +++ b/.kiro/specs/xce-test-e2e-primitives/design.md @@ -75,6 +75,7 @@ graph TD - **Session/verdict state** (file-scope): `g_session_failed` (int), `g_continue_on_error` (bool), and auto-vars `g_last_status` (int), `g_last_jobid[XCE_MAXNAME_LEN]` (string), `g_last_jobcount` (int). - **`preprocess_and_run(char* line)`** (new): handle blank/`#`; leading `!` → `system(rest)`, set `g_last_status` from `WEXITSTATUS`; else substitute auto-vars then call `execute_line`. Update `g_session_failed`/`g_last_status` from the result; honor abort-vs-continue. Called from the main loop in place of the direct `execute_line`. - **`substitute_autovars(line) → expanded`** (new): replace whole-token `$?`, `$jobid`, `$jobcount` with current values; a reference with no current value yields a command failure (Req 3.6). + > **Superseded by job-array-one-based-index (2026-10-01):** whole-token-only substitution is relaxed: a token that is `$jobid` followed immediately by `[` (e.g. `$jobid[3]`) now has `$jobid` replaced and the rest of the token copied unchanged, and fails like an unset `$jobid` when it is unset (job-array-one-based-index Requirement 4.10, 4.11). Kept for history. - **New command handlers**: `cmd_wait_primary`, `cmd_expect_jobs`, `cmd_expect_job`, `cmd_echo`, registered in the command table(s). - **`cmd_set`**: add `continue-on-error` toggle. - **`cmd_jobsub`**: gate the "ok" report on a successful reply status + valid JobId; on success set `g_last_jobid`; otherwise print a typed error and return `<0` (bug-786). diff --git a/.kiro/specs/xce-test-e2e-primitives/requirements.md b/.kiro/specs/xce-test-e2e-primitives/requirements.md index 9a88b465..a687c405 100644 --- a/.kiro/specs/xce-test-e2e-primitives/requirements.md +++ b/.kiro/specs/xce-test-e2e-primitives/requirements.md @@ -42,6 +42,8 @@ This feature adds a small set of scripting primitives to the `xce_test` client s 1. When any command completes, the xce_test client shall set `$?` to that command's status (zero on success, non-zero on failure). 2. When a `jobsub` command receives a successful acknowledgment, the xce_test client shall set `$jobid` to the acknowledged JobId. 3. When a subsequent command references `$jobid` (e.g. `expect-job $jobid`), the xce_test client shall substitute the most recently acknowledged JobId. + +> **Superseded by job-array-one-based-index (2026-10-01):** 3.2/3.3 no longer hold for array submissions. After `jobsub -a N` (N >= 2) the acknowledged identity is the array identity (no element number), `$jobid` is set to it, and elements are named `$jobid[k]`, k = 1..N; `$jobid` followed immediately by `[` is now substituted too (see job-array-one-based-index Requirement 4.5, 4.10, 4.11). Ordinary `jobsub` is unchanged. Kept for history. 4. Where the `$jobcount` auto-variable is provided, when a `joblist` command completes, the xce_test client shall set `$jobcount` to the number of jobs returned. 5. The xce_test client shall not provide user-assignable variables, arithmetic, or expression evaluation beyond substituting this fixed set of auto-variables. 6. If a command references an auto-variable that has no current value, then the xce_test client shall fail that command rather than substitute an undefined or garbage value. @@ -66,6 +68,8 @@ This feature adds a small set of scripting primitives to the `xce_test` client s 3. When `expect-job ` is issued and the given JobId is present in the job list, the xce_test client shall treat the assertion as passed. 4. If `expect-job ` is issued and the given JobId is absent, then the xce_test client shall fail the assertion and report the missing JobId. +> **Superseded by job-array-one-based-index (2026-10-01):** for arrays, `expect-job ` no longer matches text: it compares job numbers, an identity without `[k]` that no job has fails with a hint that elements must be named `[k]`, `[k]` checks element k, and a range `[a-b]` is refused (job-array-one-based-index Requirement 4.7-4.9, 5.13, 5.15). Kept for history. + ### Requirement 6: Script ergonomics **Objective:** As an E2E script author, I want comments, blank lines, and progress messages, so that scenario scripts are readable and self-documenting. @@ -81,6 +85,8 @@ This feature adds a small set of scripting primitives to the `xce_test` client s #### Acceptance Criteria 1. When `jobsub` receives a successful acknowledgment carrying a valid JobId, the xce_test client shall report success and set `$jobid` to that JobId. + +> **Superseded by job-array-one-based-index (2026-10-01):** for `jobsub -a N` (N >= 2) the reported identity and `$jobid` are the array identity, not an element JobId; elements are `$jobid[1]`..`$jobid[N]` (job-array-one-based-index Requirement 4.1, 4.5). Kept for history. 2. If `jobsub` does not receive a successful acknowledgment with a valid JobId, then the xce_test client shall report a typed failure (e.g. redirect, no-master, not-ready, or timeout) and set a non-zero status. 3. The xce_test client shall never report `jobsub` success with a malformed or uninitialized JobId. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index aff8f3e2..e8a17b9f 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -84,6 +84,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han - [ ] **component-rejoin-resync** — **PROMOTED to spec 2026-08-10** (`/kiro-discovery`; brief at `.kiro/specs/component-rejoin-resync/brief.md`) from the "lightweight rejoin re-attach" item below. **apiserver + scheduler** reconnect/rejoin resync efficiency — today each does a full re-join and receives full state again (whole job list, queue/resource/config state), not a delta; for a large cluster this is sizable but not yet a measured problem. Sibling of `worker-rejoin-resync` — disjoint data (list/config state vs. job-execution facts), NOT merged into that spec. Not started: first step is investigation (is reconnect cost actually a measured problem at realistic cluster sizes?) before committing to any delta/session-token mechanism — original YAGNI discipline still applies, this spec is the promoted vehicle for that investigation, not a rubber-stamped design. Original decision (libxce-failover design review) retained below for provenance. Dependencies: libxce-failover (rejoin hook — implemented), component-failover-adoption (daemon rejoin content — in progress, sequence alongside/after). Next: `/kiro-spec-requirements component-rejoin-resync` when ready to begin (not immediate). - [x] ~~lightweight rejoin re-attach (failover optimization)~~ — **recorded 2026-06-16; deferred (YAGNI)**; apiserver/scheduler half **PROMOTED to `component-rejoin-resync` 2026-08-10** (see above); worker half already promoted to `worker-rejoin-resync`. No longer tracked as a bare backlog item — provenance only. Original text: on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive **full** state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is **measured** as a problem at scale. - [x] job-array-single-wal-record — **IMPLEMENTED, validate-impl GO 2026-07-17** (reopened once 2026-07-16 for three user-directed residual gaps — bug-1443 bulk-insert partial-visibility, task 4.2's boot-replay fence gap, bug-1444 replay_wal array-blind tally — all closed, re-validated fresh GO). **PROMOTED to a spec 2026-06-24** (`/kiro-discovery`; brief at `.kiro/specs/job-array-single-wal-record/brief.md`; now tracked under the *Raft Apply-Ordering & Hook Decoupling* phase above, sequenced after raft-apply-decoupling). Original note retained for provenance: Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a **single WAL record** rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via `/kiro-spec-init "job-array-single-wal-record"` when ready. +- [ ] **job-array-one-based-index** — **NEW 2026-09-30** (spec `.kiro/specs/job-array-one-based-index/`); **IMPLEMENTED per spec 2026-10-01 (tasks 1.1–9.3 done), validation pending in tasks 10.1 (affected suites) and 10.2 (live cluster)**. LSF-compatible array numbering: `xcecli jobs submit -J 'name[1-N]'` / `'name[1]'` create elements numbered 1..N (previously 0..N-1, and the indices in `-J` were ignored — bug-1896); submit reports the array identity and element count; elements are shown and addressed as `[k]`, control commands also take `[a-b]`; unsupported LSF forms (other starts, index lists, steps, `%N`, interactive arrays) are rejected with specific messages; API-server `max_array_size` default 1000, clamped to 32767; running elements see `XCE_JOBINDEX` (0 for ordinary jobs); harness `jobsub -a N` numbers 1..N and `$jobid[k]` names elements; the controller rejects multi-element arrays starting at 0 from un-upgraded clients (`E_XCE_ARRAY_START0`). Supersedes `job-array-single-wal-record` Requirement 5 (identities unchanged, numbered from 0) and the `xce-test-e2e-primitives` `$jobid`/`expect-job` definitions (superseded notes added in those specs). User documentation: `src/docs/xce.md` "Job arrays" section, including the LSF features not supported. Bugs: bug-1896 (indices ignored, numbered from 0) fixed by this spec (record still `open` pending `buglog.py status` after validation); bug-1898 `partial` (query + filter-control paths fixed; `RestartJobByID` still builds unevaluated `CompFilterCondition`s, restart out of scope); bug-1899, bug-1900, bug-1901 fixed. - [ ] bulk job-array query/cancel — **recorded 2026-07-13, deferred out of `job-array-single-wal-record` scope by design.** A single client operation to query or cancel an entire array (not one element at a time) is explicitly out of scope for `job-array-single-wal-record` — see that spec's Boundary Context. Reason: array elements can be in heterogeneous states at request time (done/exited elements need no action; running elements need a worker kill signal; pending elements need a direct state transition to cancelled without ever running), which is a materially different bulk-operation problem from atomic array *submission* and needs its own requirements (partial-failure semantics, atomicity vs best-effort per element, etc.) rather than being folded into the submission spec. Dependencies: `job-array-single-wal-record` (per-element identity/addressing this would operate over). Promote via `/kiro-discovery` once job-array-single-wal-record ships. - [x] ~~WAL group-commit batching~~ — **recorded 2026-06-17 as deferred; ABSORBED INTO ha-commit-durability 2026-06-18 (design-review reversal).** The `/kiro-validate-design` review surfaced that the highest-volume WAL traffic is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage) — not array-collapsible and not covered by `job-array-single-wal-record` — so a pure per-entry synchronous design would penalize the dominant write path. Decision (with user): **do group-commit now** — repurpose `fsync_worker` from a timer into a demand-driven flusher (batch one `fdatasync` across pending appends, advance `durable_lsn`, wake commit-eval) as part of ha-commit-durability rather than deferring. No longer a separate backlog item. - [ ] primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization). diff --git a/src/docs/xce.md b/src/docs/xce.md index 42316679..4dbd9a70 100644 --- a/src/docs/xce.md +++ b/src/docs/xce.md @@ -54,6 +54,83 @@ cluster. Common operations: Built as a standalone Go binary; communicates with the API server over its REST interface. +## Job arrays + +A job array is a set of identical jobs (elements) submitted at once. XCE +supports a subset of the LSF array syntax. + +### Supported + +- **Submitting.** `xcecli jobs submit -J 'name[1-N]' ` submits an + array of N elements, numbered 1 through N. `-J 'name[1]'` submits a + one-element array. Element numbers always start at 1. +- **Array size limit.** N may be at most the API server's + `max_array_size` setting (`config/app.yaml`; default 1000 when unset or 0, + never above 32767 — larger values are clamped to 32767 with a startup + warning). A larger N is rejected with a message stating the effective limit. +- **Array name.** The part before `[` is 1 to 4094 characters, each an ASCII + letter, digit, `.`, `_` or `-`. No whitespace is allowed anywhere in the + expression. +- **Submit result.** Submission reports the array identity, without an + element number, and the number of elements created, e.g. + `Job is submitted to queue (N elements).` The REST + response carries the same identity and an `array_size` field. +- **Referring to elements.** Each element is shown and addressed as + `[k]`, in listings, job detail output, control command results and + in log entries that print a job identity in its written form (some log + entries still print the raw job number). Element numbers in a selection are + at most 32767. + - Listing (`xcecli jobs list`) and control commands (`kill`, `stop`, + `resume`, `checkpoint`) accept one element, `[k]`. + - Control commands (`kill`, `stop`, `resume`, ...) also accept a range, + `[a-b]` (`a` equal to `b` allowed). Queries accept a range only + by job name, e.g. `-J 'name[a-b]'`. + - `-J 'name[k]'` selects element k of every array named `name`. + - An identity without `[...]` selects every element of that array (and any + ordinary job with that identity), as before. + - In listing and control commands, a bare element job number (for example + the value of `XCE_JOBID`) is rejected: name the element as + `[k]`. + - `xcecli jobs peek` and `xcecli jobs restart` do not accept + `[k]` yet: peek an element by its own job number + (`XCE_JOBID`); restart takes job IDs as before. +- **Element number in the job's environment.** A running element finds its + own number in `XCE_JOBINDEX` (1..N). An ordinary job gets + `XCE_JOBINDEX=0`. A user-supplied `XCE_JOBINDEX` is overridden, with a + warning in the worker log. `XCE_JOBID` is the element's own job number, as + before. +- **Test harness.** `xce_test jobsub -a N` submits elements 1 through N + (2 <= N <= 32767, regardless of `max_array_size`; `-a 1` submits an + ordinary job). `$jobid` is then the array identity and `$jobid[k]` names + element k (for example `expect-job $jobid[3]`). + +### LSF array features not supported + +Each of these is rejected at submission with a message naming the problem; no +job or element is created. + +- **Starting numbers other than 1**, e.g. `name[0-9]` ("element numbers start + at 1") or `name[1001-2000]` / `name[5]` ("only name[1-N] and name[1] are + currently supported"). +- **Index lists**, e.g. `name[1-5,10]` or `name[1,3,5]` ("only a single range + name[1-N] is supported"). +- **Index steps**, e.g. `name[1-10:2]` (same message). +- **`%N` running limits**, e.g. `name[1-100]%5` ("running limits are not + supported"). +- **Interactive arrays**: an interactive job (`-I`) whose name contains `[` + or `]` is rejected ("interactive jobs cannot be arrays"). + +These LSF features are not available either: + +- **Per-element `%I`/`%J` file-name substitution.** `%I` and `%J` in file + names (`-i`, `-o`, `-e`, output directory, working directory) are replaced + once, at submission, with the first element's values: `%I` becomes `1` and + `%J` becomes `[1]` for every element. All elements therefore write + to the same files unless the job script chooses its own file names from + `XCE_JOBINDEX`. +- **`LSB_JOBINDEX` / `LSB_JOBID` variables** are not set. Scripts read + `XCE_JOBINDEX` (element number) and `XCE_JOBID` (job number) instead. + ## shell An interactive bash shell used for troubleshooting and inspection. Not