myxce/.kiro/specs/job-array-single-wal-record/requirements.md

Requirements Document

Introduction

A job-array submission today fans out into one Raft log record per array element: job_submit() loops over every element calling create_job(), and the persistence layer commits each resulting job through its own independent majority-commit barrier. A 10,000-element array therefore produces 10,000 WAL appends, 10,000 sequential majority-commit round trips, and 10,000 scheduler pushes — amplifying commit-engine load, replication bandwidth, scheduler push volume, and failover replay time linearly with array size, while holding the submitting thread across every per-element barrier in sequence.

This feature makes a job-array submission persist, replicate, and push as a single atomic unit: one WAL record describing the whole array (base job specification plus index range/count — every element shares the base spec, distinguished only by jobid), committed through exactly one commit barrier. Standbys reconstruct the entire array from that one committed log position. The scheduler receives the array as a single push and expands it locally. After expansion, every array element retains its existing per-element identity and is addressable exactly as it is today for status, cancellation, and completion.

This feature builds on the apply seam introduced by raft-apply-decoupling (now complete): the single array record is expanded into individual in-memory job entries inside apply_committed_entry(), the same seam that already handles single-job submission, so online apply and restart-replay expand identically with no divergence between the two paths.

Full discovery context (problem, current state, desired outcome, approach, boundary, upstream/downstream) is in .kiro/specs/job-array-single-wal-record/brief.md.

Boundary Context

• In scope: a new WAL record format for a job-array submission (base spec + index range/count — every element shares the base spec, distinguished only by jobid); atomic single-barrier commit of that one record; standby reconstruction of the full array from one committed LSN; the expand-on-apply step shared by online apply and restart replay; a single push carrying the full array to each HOOK_DETAIL_JOB_NEW subscriber — scheduler (with local scheduler-side expansion) and API server; preservation of per-element identity and state addressing after expansion; idempotent re-application of a committed array record (replay or retry must not double-create elements); a bounded record size or chunking policy so a pathologically large array does not produce a single oversized WAL entry; log-based confirmation on the scheduler and API server that all array elements were received, for harness verification; hardening src/lib/jobj.c's job/job-submit construction primitives (job_init, makeJobsub) so they are reliable and leak-free under N-element array construction, since Requirement 9's deterministic-apply guarantee depends on them.
• Out of scope: the apply-ordering/hook-decoupling refactor itself (delivered by raft-apply-decoupling, consumed here unchanged) — persist_job_hook_fun's WAL_JOB_SUBMIT branch is already a no-op (task 3.1/6.1: commit is invoked explicitly by handle_jobsubmit, never from this hook, for either single-job or today's per-element array submission); this spec replaces the explicit per-element commit loop (submit_job_wal_entry_and_commit, called once per array element today) with one array-shaped commit call — the hook plays no role in submission before or after this change, and stays untouched for its other live purpose (job-state facts originating from job_ctrl's direct synchronous path); any change to the non-array single-job submission path; scheduler dispatch/allocation policy; per-element override semantics beyond what single-job submission already supports; the commit engine's or commit-barrier's internal mechanics (consumed unchanged from ha-commit-engine / ha-commit-barrier-rollout); bulk query/cancel of a whole array as one client operation — deferred to a future spec because array elements can be in heterogeneous states at request time (done/exited elements need no action, running elements need a worker kill signal, pending elements need a direct state transition without ever running), which is a materially different bulk-operation problem from atomic array submission and deserves its own requirements rather than scope creep here.
• Adjacent expectations: depends on raft-apply-decoupling's apply_committed_entry() seam existing and being the single point where a committed WAL entry becomes in-memory state (now complete); depends on ha-commit-engine / ha-commit-barrier-rollout for one-record atomic commit and idempotency guards, consumed unchanged; the scheduler-side push path is the one scheduler-controller-msg-gap already established, extended here to carry an array descriptor instead of only single-job pushes.

Requirements

Requirement 1: Single WAL record for a job-array submission

Objective: As a cluster operator submitting a large job array, I want the whole array persisted as one Raft log record, so that submission cost no longer scales linearly with array size.

Acceptance Criteria

1. When a job-array submission is accepted, the Controller shall encode the entire array — base job specification plus index range/count — into a single WAL record. Every array element shares the same base job specification; the only per-element distinction is its jobid (base id + index).
2. The Controller shall commit that single WAL record through exactly one majority-commit barrier, regardless of the array's element count.
3. The Controller shall not append a separate WAL record per array element for a job-array submission.

Requirement 2: Atomic, all-or-nothing commit

Objective: As a cluster operator, I want a job-array submission to commit as a single unit, so that a standby can never observe a partial array.

Acceptance Criteria

1. When a single array WAL record is committed on a majority, the Controller shall consider every element described by that record committed as one unit — there shall be no state in which some elements of that record are committed and others are not. (When Requirement 7's chunking policy splits a pathologically large array into multiple records, this atomicity guarantee applies per committed record/chunk, not across the whole array — Requirement 7.2 defines the atomicity unit in that case.)
2. If an array WAL record fails to reach majority commit, then the Controller shall treat that record's elements as not committed and shall not construct any array elements from it.
3. A standby reconstructing committed state shall never observe a single array WAL record in a partially-applied form.

Requirement 3: Shared expand-on-apply for online and replay

Objective: As a maintainer, I want array expansion to happen in one shared code path, so that a live apply and a restart replay can never diverge in how they expand the same committed array record.

Acceptance Criteria

1. When a committed job-array WAL record is applied — whether during live/online apply or during restart replay — the Controller shall expand it into individual job entries through the same apply_committed_entry() seam used for single-job submission.
2. The Controller shall use identical expansion logic for the online-apply path and the restart-replay path, so that replaying a committed array record produces byte-for-byte the same set of job elements as the original online apply produced.
3. When re-applying an already-applied array record (e.g. during replay after a restart, or a retried commit), the Controller shall not create duplicate job elements for any array index.
4. The array-expand step shall fire HOOK_DETAIL_JOB_NEW exactly once per committed array record, carrying all of that record's constructed job entries in one list — not once per element. (This is what Requirement 6's single-push guarantee to the scheduler and API server rests on — both are subscribers of this same hook fire.)

Requirement 4: Standby reconstruction from one committed position

Objective: As a cluster operator relying on HA failover, I want a standby to reconstruct a whole job array from one committed log position, so that replication bandwidth and failover replay time no longer scale linearly with array size.

Acceptance Criteria

1. When a standby receives and applies the array's single committed WAL record, the standby shall reconstruct the complete set of array elements from that one record.
2. Replication of a job-array submission shall carry the single array WAL record, not one record per element.
3. Failover replay time for a committed job array shall be governed by the size of the one array record, not by a per-element replay cost proportional to array size.

Requirement 5: Preserved per-element identity and addressing

Objective: As a client of the job API, I want every array element to remain individually addressable after expansion, so that existing status, cancellation, and completion behavior is unaffected by this change.

Acceptance Criteria

1. After a job-array WAL record is applied, each resulting job element shall have the same per-element identity scheme (base id + index) that single-per-element WAL records produce today.
2. The Controller shall support querying, cancelling, and completing an individual array element by its per-element identity exactly as it does for job elements created via the current per-element path.
3. This feature shall not change any client-visible identity, status, or addressing behavior for an individual array element.

Requirement 6: Single push to scheduler and API server, with local expansion

Objective: As a maintainer of scheduler and API-server load, I want both the scheduler and the API server to receive one push per committed array, so that push volume to either consumer no longer scales linearly with array size.

Context: HOOK_DETAIL_JOB_NEW (HOOK_AFTER_EVENT) is a single hook fired once per constructed job entry, with two independent subscribers: sched.c (xcesched_push_joblist) and apiserver.c (apiserver_push_joblist). Today both fire once per array element, N times per array. Batching the hook fire to once per array (Requirement 3's shared apply step) fixes both consumers by construction — this requirement holds both to the same bar explicitly, so a fix that batches only the scheduler side and leaves the API server side unbatched does not satisfy it.

Wire shape note: "single push" means the push carries the array descriptor (base spec + index range/count + jobid basis) — the same compact shape as the WAL record (Requirement 1) — not N individually-encoded full job structs bundled into one message. Encoding N full duplicate structs would satisfy "one push" by message count while silently reintroducing N-scaling in payload size, since the wire encoder does not know two Text fields share the same interned source and will encode each independently. Each receiver decodes one base spec and expands it locally into N entries (Requirement 6.3 / Requirement 6.5), so sharing happens by construction on the receiving side too, with no separate interning step needed there.

Acceptance Criteria

1. When a single array WAL record commits, the Controller shall deliver it to the scheduler as a single push carrying that record's array descriptor (base spec + index range/count), not N individually-encoded job structs. (When Requirement 7's chunking policy applies, this is one push per committed record/chunk, not one push for the whole array — mirroring Requirement 3.4's per-record hook fire.)
2. When a single array WAL record commits, the Controller shall deliver it to the API server as a single push carrying that record's array descriptor (base spec + index range/count), not N individually-encoded job structs. (Same per-record scoping as 6.1 when chunking applies.)
3. The Scheduler shall expand the array descriptor into individual schedulable tasks locally, without requiring one push per element from the Controller. The scheduler shall treat each array element as an ordinary single job for scheduling purposes; scheduling/dispatch outcome for array elements is out of scope for this feature.
4. The Controller shall not send a separate scheduler push, nor a separate API-server push, per array element for a job-array submission.
5. The API server shall expand the array descriptor into individual per-element records locally (mirroring the scheduler's local expansion in 6.3), rather than the Controller sending or the API server requiring one full job struct per element on the wire.

Requirement 7: Bounded record size for very large arrays

Objective: As a maintainer of WAL and replication capacity, I want an upper bound on a single array record's size, so that an extremely large array cannot produce a pathologically oversized WAL entry.

Acceptance Criteria

1. The Controller shall define a maximum size or chunking policy for a job-array WAL record so that no single record exceeds a bounded size regardless of array element count.
2. If a job-array submission would exceed the bounded record size, then the Controller shall apply its chunking policy (e.g. multiple atomically-committed records) rather than produce one oversized record, while Requirement 2's atomicity guarantee continues to hold for each committed unit.

Requirement 8: No regression to non-array submission or existing tests

Objective: As a maintainer, I want this feature scoped strictly to array submissions, so that ordinary single-job submission and existing HA guarantees are unaffected.

Acceptance Criteria

1. The Controller shall leave the non-array, single-job submission WAL record format and commit-barrier count unchanged by this feature. (This does not prohibit Requirement 10's fixes to jobj.c's shared construction primitives — job_init/makeJobsub are used by single-job construction too, and Requirement 10's bugfixes are expected to also improve, not regress, that path; "unchanged" here means the WAL wire format and commit-barrier behavior, not a blanket freeze on shared-library code this feature must touch for Requirement 9's correctness.)
2. A full HA soak (sequential and chaotic /xce-test-ha regression), including job submission and failover, shall show no new job loss, divergence, or liveness stall attributable to this feature.
3. Regression tests shall cover: WAL/commit-barrier count for a large-array submission (one record/one barrier, not N), failover replay time for a committed array, correct per-element status/cancel/completion after array expansion, single-push delivery to both the scheduler and the API server, and a client retrying an identical array submission across a primary failover — the retry shall follow the same cross-primary dedup mechanism as single-job submission (bug-1419), extended to the array's full index range as one unit, and shall not double-submit or partially-duplicate the array.
4. The Scheduler and the API server shall each log a single, identifiable confirmation that all N elements of a committed array were received and handled correctly (e.g. base jobid + element count), so that push-completeness for the whole array can be verified by log inspection without a new query API.

Requirement 9: Deterministic, total apply — content failures rejected pre-commit, resource failures fenced not rejected

Objective: As a maintainer of cluster correctness, I want array expansion at apply time to never fail for a reason the commit already promised was safe, so that a committed array can never be un-appliable and no node can silently diverge from its peers.

Context: once an entry commits, Raft guarantees it is eventually applied on every correct server — but the apply function itself must be deterministic (every replica computes the same outcome from the same entry). Two distinct failure classes require two distinct handlings: (A) content-derived failures (jobid-range collision, malformed spec) are computable identically by every replica from the entry's own bytes plus prior state — these must be caught before the entry is ever appended/committed, never left to fail at apply time. (B) node-local resource exhaustion (e.g. OOM while constructing element 7000 of 10000) cannot be predicted at submit time and must not be allowed to vary apply's outcome across replicas — it is not a commit failure and must never cause element rejection or divergence.

Acceptance Criteria

1. Before appending a job-array WAL record, the Controller shall validate everything that could deterministically prevent successful expansion of every element (jobid-range collision with existing or in-flight jobs, base-spec validity) — reusing the existing in-flight jobid-range claim mechanism (bug-1419), extended to cover the array's full index range in one check.
2. If pre-commit validation fails, the Controller shall reject the submission before the WAL append — the array shall never be committed and shall never reach the apply stage.
3. Once a job-array WAL record is committed, expansion at apply time shall not fail for any content-derived reason — pre-commit validation (9.1) shall make apply total for this class.
4. If a node encounters a resource-exhaustion failure (e.g. allocation failure) while expanding a committed array record, that node shall not reject the array, shall not construct a partial/divergent set of elements, and shall not treat this as a reason to fail the commit. The node shall log the condition loudly, fence itself as behind on that LSN (non-promotable while fenced, per the existing apply_divergent/fence pattern), and keep retrying the apply until it succeeds — consistent with the "controller never self-terminates" invariant.
5. A resource-exhaustion apply failure on one node shall not block other nodes that can successfully apply the entry from proceeding; the fenced node shall self-heal (clear its fence) once the apply succeeds.
6. A fault-injection test shall force a resource-exhaustion failure (e.g. an injected allocation failure) during array expansion on one node and verify: the node fences itself as behind on that LSN, constructs no partial or divergent set of elements, other nodes proceed unaffected, and the fenced node self-heals (clears its fence, completes the apply) once the injected failure condition clears.

Requirement 10: jobj.c hardening — the shared construction primitive must be reliable under array-scale reuse

Objective: As a maintainer of cluster correctness, I want src/lib/jobj.c's job/job-submit construction primitives to be reliable and leak-free under N-element array construction, so that Requirement 9's "deterministic, total apply" guarantee rests on a sound foundation rather than a primitive with known gaps.

Context: job_init (jobj.c) already interns/shares job_submit_data_t across identical submissions via makeJobsub() (hash+refcount) — for N uniform array elements this collapses to one real allocation plus N cheap refcount bumps, which is the right foundation for Requirement 9. But inspection of the current implementation found gaps that predate this spec and become higher-stakes at array scale.

Acceptance Criteria

1. job_init shall check makeJobsub()'s return value; on NULL (allocation failure or interning-table error), job_init shall fail cleanly (propagate an error) rather than store a NULL job.submit and report success. The same check shall apply to makeString(src->ckptinfo).
2. The root cause of the previously observed makeJobsub() NULL-with-EINVAL failure (returned even for a valid non-NULL src, traced to a likely jobsub_table initialization-ordering issue) shall be identified and fixed, not merely papered over by 10.1's error propagation.
3. The array-expansion construction path shall not rely on a pattern that triggers job_init's destructive MOVE_PTR fields (pidinfo, allocs, accounts, run_usage, exit_info, pend_reasons) more than once against a shared source struct. Either confirm these fields are always NULL for a freshly-submitted (not-yet-started) job — making repeated MOVE_PTR from a shared template benign — or construct each array element from its own independent source struct.
4. The job_destroy leak on accounts (marked /* FIXME??? */, never freed) shall be fixed. Array submission multiplies this leak by element count; it shall not remain unfixed while this spec optimizes everything else for array scale.
5. Whether decode_Job ever populates ckptinfo for a freshly-submitted job shall be verified; if it does, free_Job's coverage shall be extended to include it (currently undocumented/unclear whether it is freed).