# Brief: job-array-single-wal-record

## Problem

A job-array submission fans out into **one Raft log record per array element**. Confirmed in the controller:

- `job_submit()` loops `for (i = 0; i < submit->array_size; i++)` calling `create_job()` once per element (`job.c:74`).
- `persist_job_hook_fun` then loops over the resulting job list calling `persist_wal_sync_append(p, WAL_JOB_SUBMIT, ...)` once per element (`persist.c:2556`), and **each** call runs its own `repl_wait_majority_ack` commit barrier (`persist.c:2248`).
- The scheduler receives one push per committed element (`sched.c`).

So a 10k-element array becomes 10k WAL appends + 10k sequential majority-commit barriers + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, scheduler push volume, and failover replay time **linearly in array size**, while holding the submit thread across every per-element barrier.

## Current State

- No batched/atomic array record exists; the array is a loop over the single-job path.
- The commit engine (`ha-commit-engine`) and barrier rollout (`ha-commit-barrier-rollout`) provide per-LSN majority commit and idempotency guards — the primitives needed to commit one larger record atomically.
- Per-element identity (jobid = base_id + i) and per-element addressing (status / cancel / completion) are assumed throughout the controller and scheduler.

## Desired Outcome

A job-array submission persists, replicates, and pushes as a **single** Raft log record:

- The controller writes **one** WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through one commit barrier.
- Replication carries the single record; standbys reconstruct the whole array from one committed LSN.
- The scheduler receives the array as one push and expands locally.
- Per-element identity and state addressing are preserved after expansion (status, cancel, completion still operate per task id).

## Approach

Single atomic array record on top of the post-decoupling apply seam:

- Define a `WAL_JOB_ARRAY_SUBMIT` record encoding base job spec + index range/count (no per-index overrides — array elements are uniform clones today, distinguished only by jobid; the client submit API has no per-index override field).
- Command handler appends + commits this one record through a single barrier; on commit, the shared `apply_committed_entry()` seam expands it into N in-memory `job_t` entries (reusing `create_job` per element) under one apply step.
- Restart replay decodes the single record and expands identically (same apply seam → no online/replay divergence).
- Scheduler push delivers one array descriptor; scheduler expands locally.
- Preserve idempotency: re-applying a committed array record (replay or retry) must not double-create elements.

## Scope

- **In**: single WAL record format for arrays; atomic single-barrier commit; standby reconstruction from one LSN; single scheduler push + local expansion; preserved per-element identity/addressing; replay expansion; idempotency on re-apply; requirement + regression tests (large-array commit count, failover replay time, per-element status/cancel still correct).
- **Out**: the apply-ordering/hook-decoupling refactor itself (sibling spec `raft-apply-decoupling`); non-array single-job path changes; scheduler policy changes.

## Boundary Candidates

- The `WAL_JOB_ARRAY_SUBMIT` record format + encode/decode.
- The single-barrier commit of the array record.
- The expand-on-apply step (online + replay, shared).
- The single scheduler push + scheduler-side expansion.

## Out of Boundary

- Per-element override semantics beyond what single-job submit already supports.
- The commit engine / barrier internals (consumed unchanged).

## Upstream / Downstream

- **Upstream**: `raft-apply-decoupling` (HARD — builds the array expand step on the clean `apply_committed_entry()` seam rather than the old optimistic-before-commit path); `ha-commit-engine` + `ha-commit-barrier-rollout` (single atomic commit + idempotency); `ha-consistency-invariants` (replicated scheduling state).
- **Downstream**: failover replay time and replication bandwidth improve for array-heavy workloads; lightweight rejoin re-attach optimization benefits.

## Existing Spec Touchpoints

- **Extends**: none (new record type).
- **Adjacent**: `raft-apply-decoupling` (shared apply seam — must land first); `ha-commit-barrier-rollout` (idempotency guards reused for re-apply safety); `scheduler-controller-msg-gap` (scheduler push path).

## Constraints

- Must preserve per-element identity/state addressing after expansion (status, cancel, completion per task id).
- Atomic commit: either the whole array commits or none of it (no partial array on a standby).
- Idempotent re-apply (replay/retry must not duplicate elements).
- Bounded record size / encoding for very large arrays (avoid a single pathological WAL entry; define a max or chunking policy in design).

## Provenance

Promoted 2026-06-24 from the roadmap backlog item `job-array-single-wal-record` (recorded 2026-06-16) via `/kiro-discovery`.