myxce/.kiro/specs/job-array-single-wal-record/brief.md

Brief: job-array-single-wal-record

Problem

A job-array submission fans out into one Raft log record per array element. Confirmed in the controller:

• job_submit() loops for (i = 0; i < submit->array_size; i++) calling create_job() once per element (job.c:74).
• persist_job_hook_fun then loops over the resulting job list calling persist_wal_sync_append(p, WAL_JOB_SUBMIT, ...) once per element (persist.c:2556), and each call runs its own repl_wait_majority_ack commit barrier (persist.c:2248).
• The scheduler receives one push per committed element (sched.c).

So a 10k-element array becomes 10k WAL appends + 10k sequential majority-commit barriers + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, scheduler push volume, and failover replay time linearly in array size, while holding the submit thread across every per-element barrier.

Current State

• No batched/atomic array record exists; the array is a loop over the single-job path.
• The commit engine (ha-commit-engine) and barrier rollout (ha-commit-barrier-rollout) provide per-LSN majority commit and idempotency guards — the primitives needed to commit one larger record atomically.
• Per-element identity (jobid = base_id + i) and per-element addressing (status / cancel / completion) are assumed throughout the controller and scheduler.

Desired Outcome

A job-array submission persists, replicates, and pushes as a single Raft log record:

• The controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through one commit barrier.
• Replication carries the single record; standbys reconstruct the whole array from one committed LSN.
• The scheduler receives the array as one push and expands locally.
• Per-element identity and state addressing are preserved after expansion (status, cancel, completion still operate per task id).

Approach

Single atomic array record on top of the post-decoupling apply seam:

• Define a WAL_JOB_ARRAY_SUBMIT record encoding base job spec + index range/count (no per-index overrides — array elements are uniform clones today, distinguished only by jobid; the client submit API has no per-index override field).
• Command handler appends + commits this one record through a single barrier; on commit, the shared apply_committed_entry() seam expands it into N in-memory job_t entries (reusing create_job per element) under one apply step.
• Restart replay decodes the single record and expands identically (same apply seam → no online/replay divergence).
• Scheduler push delivers one array descriptor; scheduler expands locally.
• Preserve idempotency: re-applying a committed array record (replay or retry) must not double-create elements.

Scope

• In: single WAL record format for arrays; atomic single-barrier commit; standby reconstruction from one LSN; single scheduler push + local expansion; preserved per-element identity/addressing; replay expansion; idempotency on re-apply; requirement + regression tests (large-array commit count, failover replay time, per-element status/cancel still correct).
• Out: the apply-ordering/hook-decoupling refactor itself (sibling spec raft-apply-decoupling); non-array single-job path changes; scheduler policy changes.

Boundary Candidates

• The WAL_JOB_ARRAY_SUBMIT record format + encode/decode.
• The single-barrier commit of the array record.
• The expand-on-apply step (online + replay, shared).
• The single scheduler push + scheduler-side expansion.

Out of Boundary

• Per-element override semantics beyond what single-job submit already supports.
• The commit engine / barrier internals (consumed unchanged).

Upstream / Downstream

• Upstream: raft-apply-decoupling (HARD — builds the array expand step on the clean apply_committed_entry() seam rather than the old optimistic-before-commit path); ha-commit-engine + ha-commit-barrier-rollout (single atomic commit + idempotency); ha-consistency-invariants (replicated scheduling state).
• Downstream: failover replay time and replication bandwidth improve for array-heavy workloads; lightweight rejoin re-attach optimization benefits.

Existing Spec Touchpoints

• Extends: none (new record type).
• Adjacent: raft-apply-decoupling (shared apply seam — must land first); ha-commit-barrier-rollout (idempotency guards reused for re-apply safety); scheduler-controller-msg-gap (scheduler push path).

Constraints

• Must preserve per-element identity/state addressing after expansion (status, cancel, completion per task id).
• Atomic commit: either the whole array commits or none of it (no partial array on a standby).
• Idempotent re-apply (replay/retry must not duplicate elements).
• Bounded record size / encoding for very large arrays (avoid a single pathological WAL entry; define a max or chunking policy in design).

Provenance

Promoted 2026-06-24 from the roadmap backlog item job-array-single-wal-record (recorded 2026-06-16) via /kiro-discovery.