spec(job-array-single-wal-record): scaffold requirements

.kiro/specs/job-array-single-wal-record/requirements.md

@@ -0,0 +1,91 @@
+# Requirements Document
+
+## Introduction
+
+A job-array submission today fans out into one Raft log record per array element: `job_submit()` loops over every element calling `create_job()`, and the persistence layer commits each resulting job through its own independent majority-commit barrier. A 10,000-element array therefore produces 10,000 WAL appends, 10,000 sequential majority-commit round trips, and 10,000 scheduler pushes — amplifying commit-engine load, replication bandwidth, scheduler push volume, and failover replay time linearly with array size, while holding the submitting thread across every per-element barrier in sequence.
+
+This feature makes a job-array submission persist, replicate, and push as a single atomic unit: one WAL record describing the whole array (base job specification, index range/count, and any sparse per-index overrides), committed through exactly one commit barrier. Standbys reconstruct the entire array from that one committed log position. The scheduler receives the array as a single push and expands it locally. After expansion, every array element retains its existing per-element identity and is addressable exactly as it is today for status, cancellation, and completion.
+
+This feature builds on the apply seam introduced by `raft-apply-decoupling` (now complete): the single array record is expanded into individual in-memory job entries inside `apply_committed_entry()`, the same seam that already handles single-job submission, so online apply and restart-replay expand identically with no divergence between the two paths.
+
+> Full discovery context (problem, current state, desired outcome, approach, boundary, upstream/downstream) is in `.kiro/specs/job-array-single-wal-record/brief.md`.
+
+## Boundary Context
+
+- **In scope**: a new WAL record format for a job-array submission (base spec + index range/count + sparse per-index overrides); atomic single-barrier commit of that one record; standby reconstruction of the full array from one committed LSN; the expand-on-apply step shared by online apply and restart replay; a single scheduler push carrying the array descriptor with local scheduler-side expansion; preservation of per-element identity and state addressing after expansion; idempotent re-application of a committed array record (replay or retry must not double-create elements); a bounded record size or chunking policy so a pathologically large array does not produce a single oversized WAL entry.
+- **Out of scope**: the apply-ordering/hook-decoupling refactor itself (delivered by `raft-apply-decoupling`, consumed here unchanged); any change to the non-array single-job submission path; scheduler dispatch/allocation policy; per-element override semantics beyond what single-job submission already supports; the commit engine's or commit-barrier's internal mechanics (consumed unchanged from `ha-commit-engine` / `ha-commit-barrier-rollout`).
+- **Adjacent expectations**: depends on `raft-apply-decoupling`'s `apply_committed_entry()` seam existing and being the single point where a committed WAL entry becomes in-memory state (now complete); depends on `ha-commit-engine` / `ha-commit-barrier-rollout` for one-record atomic commit and idempotency guards, consumed unchanged; the scheduler-side push path is the one `scheduler-controller-msg-gap` already established, extended here to carry an array descriptor instead of only single-job pushes.
+
+## Requirements
+
+### Requirement 1: Single WAL record for a job-array submission
+
+**Objective:** As a cluster operator submitting a large job array, I want the whole array persisted as one Raft log record, so that submission cost no longer scales linearly with array size.
+
+#### Acceptance Criteria
+1. When a job-array submission is accepted, the Controller shall encode the entire array — base job specification, index range/count, and any sparse per-index overrides — into a single WAL record.
+2. The Controller shall commit that single WAL record through exactly one majority-commit barrier, regardless of the array's element count.
+3. The Controller shall not append a separate WAL record per array element for a job-array submission.
+4. When array elements share the same base specification with no per-index overrides, the encoded record shall not need to enumerate each element individually.
+
+### Requirement 2: Atomic, all-or-nothing commit
+
+**Objective:** As a cluster operator, I want a job-array submission to commit as a single unit, so that a standby can never observe a partial array.
+
+#### Acceptance Criteria
+1. When the array's WAL record is committed on a majority, the Controller shall consider the entire array committed as one unit — there shall be no state in which some array elements are committed and others are not.
+2. If the array's WAL record fails to reach majority commit, then the Controller shall treat the whole array submission as not committed and shall not construct any array elements from it.
+3. A standby reconstructing committed state shall never observe a job-array submission in a partially-applied form.
+
+### Requirement 3: Shared expand-on-apply for online and replay
+
+**Objective:** As a maintainer, I want array expansion to happen in one shared code path, so that a live apply and a restart replay can never diverge in how they expand the same committed array record.
+
+#### Acceptance Criteria
+1. When a committed job-array WAL record is applied — whether during live/online apply or during restart replay — the Controller shall expand it into individual job entries through the same `apply_committed_entry()` seam used for single-job submission.
+2. The Controller shall use identical expansion logic for the online-apply path and the restart-replay path, so that replaying a committed array record produces byte-for-byte the same set of job elements as the original online apply produced.
+3. When re-applying an already-applied array record (e.g. during replay after a restart, or a retried commit), the Controller shall not create duplicate job elements for any array index.
+
+### Requirement 4: Standby reconstruction from one committed position
+
+**Objective:** As a cluster operator relying on HA failover, I want a standby to reconstruct a whole job array from one committed log position, so that replication bandwidth and failover replay time no longer scale linearly with array size.
+
+#### Acceptance Criteria
+1. When a standby receives and applies the array's single committed WAL record, the standby shall reconstruct the complete set of array elements from that one record.
+2. Replication of a job-array submission shall carry the single array WAL record, not one record per element.
+3. Failover replay time for a committed job array shall be governed by the size of the one array record, not by a per-element replay cost proportional to array size.
+
+### Requirement 5: Preserved per-element identity and addressing
+
+**Objective:** As a client of the job API, I want every array element to remain individually addressable after expansion, so that existing status, cancellation, and completion behavior is unaffected by this change.
+
+#### Acceptance Criteria
+1. After a job-array WAL record is applied, each resulting job element shall have the same per-element identity scheme (base id + index) that single-per-element WAL records produce today.
+2. The Controller shall support querying, cancelling, and completing an individual array element by its per-element identity exactly as it does for job elements created via the current per-element path.
+3. This feature shall not change any client-visible identity, status, or addressing behavior for an individual array element.
+
+### Requirement 6: Single scheduler push with local expansion
+
+**Objective:** As a maintainer of scheduler load, I want the scheduler to receive one push per committed array, so that scheduler push volume no longer scales linearly with array size.
+
+#### Acceptance Criteria
+1. When a job-array submission commits, the Controller shall deliver it to the scheduler as a single push carrying the array descriptor.
+2. The Scheduler shall expand the array descriptor into individual schedulable tasks locally, without requiring one push per element from the Controller.
+3. The Controller shall not send a separate scheduler push per array element for a job-array submission.
+
+### Requirement 7: Bounded record size for very large arrays
+
+**Objective:** As a maintainer of WAL and replication capacity, I want an upper bound on a single array record's size, so that an extremely large array cannot produce a pathologically oversized WAL entry.
+
+#### Acceptance Criteria
+1. The Controller shall define a maximum size or chunking policy for a job-array WAL record so that no single record exceeds a bounded size regardless of array element count.
+2. If a job-array submission would exceed the bounded record size, then the Controller shall apply its chunking policy (e.g. multiple atomically-committed records) rather than produce one oversized record, while Requirement 2's atomicity guarantee continues to hold for each committed unit.
+
+### Requirement 8: No regression to non-array submission or existing tests
+
+**Objective:** As a maintainer, I want this feature scoped strictly to array submissions, so that ordinary single-job submission and existing HA guarantees are unaffected.
+
+#### Acceptance Criteria
+1. The Controller shall leave the non-array, single-job submission WAL path unchanged by this feature.
+2. A full HA soak (sequential and chaotic `/xce-test-ha` regression), including job submission and failover, shall show no new job loss, divergence, or liveness stall attributable to this feature.
+3. Regression tests shall cover: WAL/commit-barrier count for a large-array submission (one record/one barrier, not N), failover replay time for a committed array, and correct per-element status/cancel/completion after array expansion.

.kiro/specs/job-array-single-wal-record/spec.json

@@ -0,0 +1,23 @@
+{
+  "feature_name": "job-array-single-wal-record",
+  "created_at": "2026-07-09T17:20:00Z",
+  "updated_at": "2026-07-09T17:20:00Z",
+  "language": "en",
+  "phase": "requirements-generated",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": false
+    },
+    "design": {
+      "generated": false,
+      "approved": false
+    },
+    "tasks": {
+      "generated": false,
+      "approved": false
+    }
+  },
+  "ready_for_implementation": false,
+  "blocked_on": []
+}