lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/job-array-single-wal-record/requirements.md

9.7 KiB
Raw Blame History

Requirements Document

Introduction

A job-array submission today fans out into one Raft log record per array element: job_submit() loops over every element calling create_job(), and the persistence layer commits each resulting job through its own independent majority-commit barrier. A 10,000-element array therefore produces 10,000 WAL appends, 10,000 sequential majority-commit round trips, and 10,000 scheduler pushes — amplifying commit-engine load, replication bandwidth, scheduler push volume, and failover replay time linearly with array size, while holding the submitting thread across every per-element barrier in sequence.

This feature makes a job-array submission persist, replicate, and push as a single atomic unit: one WAL record describing the whole array (base job specification, index range/count, and any sparse per-index overrides), committed through exactly one commit barrier. Standbys reconstruct the entire array from that one committed log position. The scheduler receives the array as a single push and expands it locally. After expansion, every array element retains its existing per-element identity and is addressable exactly as it is today for status, cancellation, and completion.

This feature builds on the apply seam introduced by raft-apply-decoupling (now complete): the single array record is expanded into individual in-memory job entries inside apply_committed_entry(), the same seam that already handles single-job submission, so online apply and restart-replay expand identically with no divergence between the two paths.

Full discovery context (problem, current state, desired outcome, approach, boundary, upstream/downstream) is in .kiro/specs/job-array-single-wal-record/brief.md.

Boundary Context

  • In scope: a new WAL record format for a job-array submission (base spec + index range/count + sparse per-index overrides); atomic single-barrier commit of that one record; standby reconstruction of the full array from one committed LSN; the expand-on-apply step shared by online apply and restart replay; a single scheduler push carrying the array descriptor with local scheduler-side expansion; preservation of per-element identity and state addressing after expansion; idempotent re-application of a committed array record (replay or retry must not double-create elements); a bounded record size or chunking policy so a pathologically large array does not produce a single oversized WAL entry.
  • Out of scope: the apply-ordering/hook-decoupling refactor itself (delivered by raft-apply-decoupling, consumed here unchanged); any change to the non-array single-job submission path; scheduler dispatch/allocation policy; per-element override semantics beyond what single-job submission already supports; the commit engine's or commit-barrier's internal mechanics (consumed unchanged from ha-commit-engine / ha-commit-barrier-rollout).
  • Adjacent expectations: depends on raft-apply-decoupling's apply_committed_entry() seam existing and being the single point where a committed WAL entry becomes in-memory state (now complete); depends on ha-commit-engine / ha-commit-barrier-rollout for one-record atomic commit and idempotency guards, consumed unchanged; the scheduler-side push path is the one scheduler-controller-msg-gap already established, extended here to carry an array descriptor instead of only single-job pushes.

Requirements

Requirement 1: Single WAL record for a job-array submission

Objective: As a cluster operator submitting a large job array, I want the whole array persisted as one Raft log record, so that submission cost no longer scales linearly with array size.

Acceptance Criteria

  1. When a job-array submission is accepted, the Controller shall encode the entire array — base job specification, index range/count, and any sparse per-index overrides — into a single WAL record.
  2. The Controller shall commit that single WAL record through exactly one majority-commit barrier, regardless of the array's element count.
  3. The Controller shall not append a separate WAL record per array element for a job-array submission.
  4. When array elements share the same base specification with no per-index overrides, the encoded record shall not need to enumerate each element individually.

Requirement 2: Atomic, all-or-nothing commit

Objective: As a cluster operator, I want a job-array submission to commit as a single unit, so that a standby can never observe a partial array.

Acceptance Criteria

  1. When the array's WAL record is committed on a majority, the Controller shall consider the entire array committed as one unit — there shall be no state in which some array elements are committed and others are not.
  2. If the array's WAL record fails to reach majority commit, then the Controller shall treat the whole array submission as not committed and shall not construct any array elements from it.
  3. A standby reconstructing committed state shall never observe a job-array submission in a partially-applied form.

Requirement 3: Shared expand-on-apply for online and replay

Objective: As a maintainer, I want array expansion to happen in one shared code path, so that a live apply and a restart replay can never diverge in how they expand the same committed array record.

Acceptance Criteria

  1. When a committed job-array WAL record is applied — whether during live/online apply or during restart replay — the Controller shall expand it into individual job entries through the same apply_committed_entry() seam used for single-job submission.
  2. The Controller shall use identical expansion logic for the online-apply path and the restart-replay path, so that replaying a committed array record produces byte-for-byte the same set of job elements as the original online apply produced.
  3. When re-applying an already-applied array record (e.g. during replay after a restart, or a retried commit), the Controller shall not create duplicate job elements for any array index.

Requirement 4: Standby reconstruction from one committed position

Objective: As a cluster operator relying on HA failover, I want a standby to reconstruct a whole job array from one committed log position, so that replication bandwidth and failover replay time no longer scale linearly with array size.

Acceptance Criteria

  1. When a standby receives and applies the array's single committed WAL record, the standby shall reconstruct the complete set of array elements from that one record.
  2. Replication of a job-array submission shall carry the single array WAL record, not one record per element.
  3. Failover replay time for a committed job array shall be governed by the size of the one array record, not by a per-element replay cost proportional to array size.

Requirement 5: Preserved per-element identity and addressing

Objective: As a client of the job API, I want every array element to remain individually addressable after expansion, so that existing status, cancellation, and completion behavior is unaffected by this change.

Acceptance Criteria

  1. After a job-array WAL record is applied, each resulting job element shall have the same per-element identity scheme (base id + index) that single-per-element WAL records produce today.
  2. The Controller shall support querying, cancelling, and completing an individual array element by its per-element identity exactly as it does for job elements created via the current per-element path.
  3. This feature shall not change any client-visible identity, status, or addressing behavior for an individual array element.

Requirement 6: Single scheduler push with local expansion

Objective: As a maintainer of scheduler load, I want the scheduler to receive one push per committed array, so that scheduler push volume no longer scales linearly with array size.

Acceptance Criteria

  1. When a job-array submission commits, the Controller shall deliver it to the scheduler as a single push carrying the array descriptor.
  2. The Scheduler shall expand the array descriptor into individual schedulable tasks locally, without requiring one push per element from the Controller.
  3. The Controller shall not send a separate scheduler push per array element for a job-array submission.

Requirement 7: Bounded record size for very large arrays

Objective: As a maintainer of WAL and replication capacity, I want an upper bound on a single array record's size, so that an extremely large array cannot produce a pathologically oversized WAL entry.

Acceptance Criteria

  1. The Controller shall define a maximum size or chunking policy for a job-array WAL record so that no single record exceeds a bounded size regardless of array element count.
  2. If a job-array submission would exceed the bounded record size, then the Controller shall apply its chunking policy (e.g. multiple atomically-committed records) rather than produce one oversized record, while Requirement 2's atomicity guarantee continues to hold for each committed unit.

Requirement 8: No regression to non-array submission or existing tests

Objective: As a maintainer, I want this feature scoped strictly to array submissions, so that ordinary single-job submission and existing HA guarantees are unaffected.

Acceptance Criteria

  1. The Controller shall leave the non-array, single-job submission WAL path unchanged by this feature.
  2. A full HA soak (sequential and chaotic /xce-test-ha regression), including job submission and failover, shall show no new job loss, divergence, or liveness stall attributable to this feature.
  3. Regression tests shall cover: WAL/commit-barrier count for a large-array submission (one record/one barrier, not N), failover replay time for a committed array, and correct per-element status/cancel/completion after array expansion.