From 00b19d997b56d887966bfc7e6700dd04608ed56a Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sun, 12 Jul 2026 04:25:50 +0000 Subject: [PATCH] spec(job-array-single-wal-record): scaffold requirements --- .../requirements.md | 91 +++++++++++++++++++ .../job-array-single-wal-record/spec.json | 23 +++++ 2 files changed, 114 insertions(+) create mode 100644 .kiro/specs/job-array-single-wal-record/requirements.md create mode 100644 .kiro/specs/job-array-single-wal-record/spec.json diff --git a/.kiro/specs/job-array-single-wal-record/requirements.md b/.kiro/specs/job-array-single-wal-record/requirements.md new file mode 100644 index 0000000..c556b1d --- /dev/null +++ b/.kiro/specs/job-array-single-wal-record/requirements.md @@ -0,0 +1,91 @@ +# Requirements Document + +## Introduction + +A job-array submission today fans out into one Raft log record per array element: `job_submit()` loops over every element calling `create_job()`, and the persistence layer commits each resulting job through its own independent majority-commit barrier. A 10,000-element array therefore produces 10,000 WAL appends, 10,000 sequential majority-commit round trips, and 10,000 scheduler pushes — amplifying commit-engine load, replication bandwidth, scheduler push volume, and failover replay time linearly with array size, while holding the submitting thread across every per-element barrier in sequence. + +This feature makes a job-array submission persist, replicate, and push as a single atomic unit: one WAL record describing the whole array (base job specification, index range/count, and any sparse per-index overrides), committed through exactly one commit barrier. Standbys reconstruct the entire array from that one committed log position. The scheduler receives the array as a single push and expands it locally. After expansion, every array element retains its existing per-element identity and is addressable exactly as it is today for status, cancellation, and completion. + +This feature builds on the apply seam introduced by `raft-apply-decoupling` (now complete): the single array record is expanded into individual in-memory job entries inside `apply_committed_entry()`, the same seam that already handles single-job submission, so online apply and restart-replay expand identically with no divergence between the two paths. + +> Full discovery context (problem, current state, desired outcome, approach, boundary, upstream/downstream) is in `.kiro/specs/job-array-single-wal-record/brief.md`. + +## Boundary Context + +- **In scope**: a new WAL record format for a job-array submission (base spec + index range/count + sparse per-index overrides); atomic single-barrier commit of that one record; standby reconstruction of the full array from one committed LSN; the expand-on-apply step shared by online apply and restart replay; a single scheduler push carrying the array descriptor with local scheduler-side expansion; preservation of per-element identity and state addressing after expansion; idempotent re-application of a committed array record (replay or retry must not double-create elements); a bounded record size or chunking policy so a pathologically large array does not produce a single oversized WAL entry. +- **Out of scope**: the apply-ordering/hook-decoupling refactor itself (delivered by `raft-apply-decoupling`, consumed here unchanged); any change to the non-array single-job submission path; scheduler dispatch/allocation policy; per-element override semantics beyond what single-job submission already supports; the commit engine's or commit-barrier's internal mechanics (consumed unchanged from `ha-commit-engine` / `ha-commit-barrier-rollout`). +- **Adjacent expectations**: depends on `raft-apply-decoupling`'s `apply_committed_entry()` seam existing and being the single point where a committed WAL entry becomes in-memory state (now complete); depends on `ha-commit-engine` / `ha-commit-barrier-rollout` for one-record atomic commit and idempotency guards, consumed unchanged; the scheduler-side push path is the one `scheduler-controller-msg-gap` already established, extended here to carry an array descriptor instead of only single-job pushes. + +## Requirements + +### Requirement 1: Single WAL record for a job-array submission + +**Objective:** As a cluster operator submitting a large job array, I want the whole array persisted as one Raft log record, so that submission cost no longer scales linearly with array size. + +#### Acceptance Criteria +1. When a job-array submission is accepted, the Controller shall encode the entire array — base job specification, index range/count, and any sparse per-index overrides — into a single WAL record. +2. The Controller shall commit that single WAL record through exactly one majority-commit barrier, regardless of the array's element count. +3. The Controller shall not append a separate WAL record per array element for a job-array submission. +4. When array elements share the same base specification with no per-index overrides, the encoded record shall not need to enumerate each element individually. + +### Requirement 2: Atomic, all-or-nothing commit + +**Objective:** As a cluster operator, I want a job-array submission to commit as a single unit, so that a standby can never observe a partial array. + +#### Acceptance Criteria +1. When the array's WAL record is committed on a majority, the Controller shall consider the entire array committed as one unit — there shall be no state in which some array elements are committed and others are not. +2. If the array's WAL record fails to reach majority commit, then the Controller shall treat the whole array submission as not committed and shall not construct any array elements from it. +3. A standby reconstructing committed state shall never observe a job-array submission in a partially-applied form. + +### Requirement 3: Shared expand-on-apply for online and replay + +**Objective:** As a maintainer, I want array expansion to happen in one shared code path, so that a live apply and a restart replay can never diverge in how they expand the same committed array record. + +#### Acceptance Criteria +1. When a committed job-array WAL record is applied — whether during live/online apply or during restart replay — the Controller shall expand it into individual job entries through the same `apply_committed_entry()` seam used for single-job submission. +2. The Controller shall use identical expansion logic for the online-apply path and the restart-replay path, so that replaying a committed array record produces byte-for-byte the same set of job elements as the original online apply produced. +3. When re-applying an already-applied array record (e.g. during replay after a restart, or a retried commit), the Controller shall not create duplicate job elements for any array index. + +### Requirement 4: Standby reconstruction from one committed position + +**Objective:** As a cluster operator relying on HA failover, I want a standby to reconstruct a whole job array from one committed log position, so that replication bandwidth and failover replay time no longer scale linearly with array size. + +#### Acceptance Criteria +1. When a standby receives and applies the array's single committed WAL record, the standby shall reconstruct the complete set of array elements from that one record. +2. Replication of a job-array submission shall carry the single array WAL record, not one record per element. +3. Failover replay time for a committed job array shall be governed by the size of the one array record, not by a per-element replay cost proportional to array size. + +### Requirement 5: Preserved per-element identity and addressing + +**Objective:** As a client of the job API, I want every array element to remain individually addressable after expansion, so that existing status, cancellation, and completion behavior is unaffected by this change. + +#### Acceptance Criteria +1. After a job-array WAL record is applied, each resulting job element shall have the same per-element identity scheme (base id + index) that single-per-element WAL records produce today. +2. The Controller shall support querying, cancelling, and completing an individual array element by its per-element identity exactly as it does for job elements created via the current per-element path. +3. This feature shall not change any client-visible identity, status, or addressing behavior for an individual array element. + +### Requirement 6: Single scheduler push with local expansion + +**Objective:** As a maintainer of scheduler load, I want the scheduler to receive one push per committed array, so that scheduler push volume no longer scales linearly with array size. + +#### Acceptance Criteria +1. When a job-array submission commits, the Controller shall deliver it to the scheduler as a single push carrying the array descriptor. +2. The Scheduler shall expand the array descriptor into individual schedulable tasks locally, without requiring one push per element from the Controller. +3. The Controller shall not send a separate scheduler push per array element for a job-array submission. + +### Requirement 7: Bounded record size for very large arrays + +**Objective:** As a maintainer of WAL and replication capacity, I want an upper bound on a single array record's size, so that an extremely large array cannot produce a pathologically oversized WAL entry. + +#### Acceptance Criteria +1. The Controller shall define a maximum size or chunking policy for a job-array WAL record so that no single record exceeds a bounded size regardless of array element count. +2. If a job-array submission would exceed the bounded record size, then the Controller shall apply its chunking policy (e.g. multiple atomically-committed records) rather than produce one oversized record, while Requirement 2's atomicity guarantee continues to hold for each committed unit. + +### Requirement 8: No regression to non-array submission or existing tests + +**Objective:** As a maintainer, I want this feature scoped strictly to array submissions, so that ordinary single-job submission and existing HA guarantees are unaffected. + +#### Acceptance Criteria +1. The Controller shall leave the non-array, single-job submission WAL path unchanged by this feature. +2. A full HA soak (sequential and chaotic `/xce-test-ha` regression), including job submission and failover, shall show no new job loss, divergence, or liveness stall attributable to this feature. +3. Regression tests shall cover: WAL/commit-barrier count for a large-array submission (one record/one barrier, not N), failover replay time for a committed array, and correct per-element status/cancel/completion after array expansion. diff --git a/.kiro/specs/job-array-single-wal-record/spec.json b/.kiro/specs/job-array-single-wal-record/spec.json new file mode 100644 index 0000000..5946069 --- /dev/null +++ b/.kiro/specs/job-array-single-wal-record/spec.json @@ -0,0 +1,23 @@ +{ + "feature_name": "job-array-single-wal-record", + "created_at": "2026-07-09T17:20:00Z", + "updated_at": "2026-07-09T17:20:00Z", + "language": "en", + "phase": "requirements-generated", + "approvals": { + "requirements": { + "generated": true, + "approved": false + }, + "design": { + "generated": false, + "approved": false + }, + "tasks": { + "generated": false, + "approved": false + } + }, + "ready_for_implementation": false, + "blocked_on": [] +}