myxce/.kiro/specs/job-array-single-wal-record/design.md

# Design Document: job-array-single-wal-record

## Overview

**Purpose**: This feature collapses a job-array submission's persistence, replication, and downstream-notification cost from O(array size) to O(1). Today, `handle_jobsubmit()` builds one WAL payload per array element and appends them through `persist_submit_job_payloads_async()`, registering one commit continuation per element (job.c:1055). A 10,000-element array therefore produces 10,000 WAL appends and 10,000 commit registrations before a single client reply is sent. This design replaces that per-element loop with one `WAL_JOB_ARRAY_SUBMIT` record — base job spec plus `(base_jobid, array_size)` — committed through exactly one majority-commit barrier, and expanded into individual `job_t` entries inside the existing `apply_committed_entry()`/`apply_wal_entry()` seam shared by live apply, restart replay, and standby catch-up.

**Users**: Cluster operators submitting large job arrays (commit-engine load, replication bandwidth, and failover replay time no longer scale linearly with array size); maintainers of the controller's submit/apply/push code (one array-shaped record instead of N single-job records to reason about).

**Impact**: Adds one new `wal_entry_type` (`WAL_JOB_ARRAY_SUBMIT`) alongside the unchanged single-job `WAL_JOB_SUBMIT`; adds one new shared expansion primitive (`restore_job_array()`) consumed by both live dispatchers of committed WAL entries; adds a compact push variant to the existing scheduler/API-server notification path; closes a live apply-pump gap (a resource-exhaustion apply failure is currently silently skipped rather than retried); hardens five specific, pre-existing gaps in `src/lib/jobj.c`'s job-construction primitives that this feature's N-way apply-time loop amplifies.

### Goals
- One WAL record, one majority-commit barrier, one scheduler push, one API-server push per job-array submission — regardless of array size.
- Online apply, restart replay, and standby catch-up expand a committed array record identically, through one shared primitive.
- A committed array is applied deterministically: content-derived failures are impossible post-commit (caught pre-commit); resource-exhaustion failures fence the affected node without rejecting the array or diverging from peers, and self-heal automatically.
- Zero behavior change to non-array (single-job) submission: same WAL format, same commit-barrier count, same wire shape.

### Non-Goals
- Per-element override semantics (array elements remain uniform clones of one base spec, distinguished only by jobid — no client-facing API change).
- Bulk query/cancel of a whole array as one operation (deferred; array elements can be in heterogeneous states at request time).
- Any change to the commit engine's or commit barrier's internal mechanics (consumed unchanged from `ha-commit-engine`/`ha-commit-barrier-rollout`).
- Scheduler dispatch/allocation policy for array elements (scheduled as ordinary single jobs once expanded).

## Boundary Commitments

### This Spec Owns
- The `WAL_JOB_ARRAY_SUBMIT` wire format (reusing the existing `JobSubmit` capnp struct) and its encode/append/commit path in `handle_jobsubmit()`.
- The shared array-expansion primitive (`restore_job_array()`) and its wiring into both live dispatchers of committed WAL entries (`apply_committed_entry()`, `apply_wal_entry()`).
- Atomicity of one array record's apply: all-or-nothing per attempt, idempotent re-apply, no partial/divergent element sets ever observable.
- The compact scheduler/API-server push variant for a committed array and the corresponding local-expansion logic on both receivers.
- The live apply-pump return-value fix (currently discarded) and the new live, self-healing, non-promotable fence for a resource-exhaustion apply failure.
- Hardening `src/lib/jobj.c`'s `job_init`/`makeJobsub`/`job_destroy`/`decode_Job`/`free_Job` for the five specific gaps requirements.md names, since the apply-time N-way loop amplifies each by array size.

### Out of Boundary
- The apply-ordering/hook-decoupling refactor itself (delivered by `raft-apply-decoupling`, consumed unchanged) — `persist_job_hook_fun`'s `WAL_JOB_SUBMIT` branch is already a no-op and stays untouched.
- The non-array single-job submission WAL format, commit path, or reply plumbing — `array_size == 1` continues through today's exact code, unmodified.
- The commit engine's/commit barrier's internal mechanics — consumed via `commit_submit_async()` unchanged.
- Bulk array cancel/query as one client operation (deferred to a future spec — heterogeneous per-element state at request time is a materially different problem).
- Scheduler placement/dispatch policy for array elements — out of scope once elements are ordinary schedulable jobs.

### Allowed Dependencies
- `raft-apply-decoupling` (complete): `apply_committed_entry()` and its dispatch contract, `persist_apply_pump_run()`'s LSN-ordered walk, `persist_register_pending_apply()`/`persist_take_pending_apply()` hand-off, `jobsubmit_async_ctx_t`'s batched-reply mechanism (consumed as-is, `pending` set to 1 for the array path instead of N).
- `ha-commit-engine`/`ha-commit-barrier-rollout` (complete): `commit_submit_async()`, per-LSN idempotency guards — consumed unchanged.
- bug-1419's in-flight jobid-range registry (`inflight_range_try_add`/`inflight_range_remove`/`jobid_range_exists`, job.c) — already range-shaped; consumed unchanged, called exactly once per array submission as it is today.
- `idgen()`'s existing `MAX_ARRAY_IDX` (32767) cap on `array_size`, enforced at jobid-minting time in the Go apiserver before a request ever reaches the controller — relied on as the structural bound satisfying Requirement 7, not re-implemented.
- `scheduler-controller-msg-gap` (complete): the controller→scheduler push protocol and its established pattern for adding a new `Data` union variant.

### Revalidation Triggers
- Any change to `apply_committed_entry()`'s or `apply_wal_entry()`'s dispatch signature or contract (per `raft-apply-decoupling`'s own trigger: a new WAL entry type not added to both switches silently fails to apply).
- Any change to `struct JobSubmit`'s capnp shape (`controller.capnp:533`) — it is now load-bearing for three wire uses (client request, WAL payload, push payload), not one.
- Any change to `jobsub_table`/`makeJobsub()`'s interning contract in `src/lib/jobj.c` — every array element's `job_t` construction depends on it.
- Any change to `ha.c`'s promotion-gate eligibility conditions — the new Live Apply Resource Fence check must stay wired at both existing call sites (periodic release, election-time).
- Any change to `MAX_ARRAY_IDX`/`N_ARRAY_IDX_BITS` (`src/include/config.h`) or to `idgen()`'s `arraySize` validation — Requirement 7's "no chunking needed" conclusion depends on this cap continuing to hold upstream of the controller.

## Architecture

### Existing Architecture Analysis

The brief's original framing (`job_submit()` looping `create_job()` per element) predates `raft-apply-decoupling` task 3.2 and is stale. Verified current state:

| Aspect | Current state | Gap vs. this design |
|---|---|---|
| Array submission WAL cost | `handle_jobsubmit()` → `build_job_submit_payloads()` builds N plain `xce_job_t` payloads (job.c:221) → `persist_submit_job_payloads_async()` appends N `WAL_JOB_SUBMIT` entries, registers N commit continuations (persist.c:4846) | Requirement 1: N WAL appends / N commit barriers must become 1 |
| `create_job()` call site | Only inside the apply seam, via `restore_job()` (job.c:307) — never in `handle_jobsubmit` (removed by task 3.2) | None — this is already the correct location for this design's N-way expansion loop |
| Client reply batching | Already one reply per request regardless of `array_size`, via `jobsubmit_async_ctx_t` (job.c:754) and `pending` refcounting | None — reused as-is; the array path sets `pending = 1` instead of `plist.count` |
| Live apply dispatch | **Correction (found during task 3.3 review, 2026-07-14):** `apply_wal_entry()` (persist.c:2574) has **zero production call sites** — a 2026-07-10 refactor (`ha_standby_advance_committed_and_apply`, ha.c:8024) moved standby entity materialization onto the SAME `apply_committed_entry()`/`persist_apply_pump_kick` pump the primary uses; the persist.c:3722 comment claiming `apply_wal_entry()` "remains the live standby catch-up dispatch" predates that refactor and was never updated. `apply_committed_entry()` (persist.c:2721) is the **only** live dispatcher — for primary live apply, boot replay, *and* standby catch-up. | Requirement 3/4.1 is fully delivered by task 3.2's `apply_committed_entry()` wiring alone. Task 3.3's parallel wiring into `apply_wal_entry()` is kept as defensive parity with `WAL_JOB_SUBMIT`'s existing (also-dead) case in the same switch, not as a requirement-delivery path. |
| Live apply-pump failure handling | `persist_apply_pump_run()` discards `apply_committed_entry()`'s return value (`(void)apply_committed_entry(...)`, persist.c:3281) and unconditionally advances `last_applied_lsn` | Requirement 9.4/9.5: a resource-exhaustion apply failure would silently be skipped forever, never retried, never fenced |
| Promotion-gate conditions | `ha.c` gates on `truncated`, `durable_floor` vs HWM, `apply_verdict == APPLY_DIVERGENT`, `self_restart_pending` — never on `last_applied_lsn` vs `committed_lsn` | Requirement 9.4: a node stuck on a resource-exhaustion apply failure looks fully promotable today |
| Client-facing array wire format | `struct JobSubmit` (`controller.capnp:533`) already carries `(jobid, arraySize, submit: JobSubmitData)` | None — this is exactly the compact shape Requirement 1/6 need; reused verbatim, not redesigned |
| Array-size bound | `idgen()` already rejects `arraySize > MAX_ARRAY_IDX + 1` (32768) at jobid-minting time, upstream of the controller (`clib.go:165`) | None — Requirement 7 is satisfied by this pre-existing cap plus the record's O(1) payload; no chunking policy needed |
| Scheduler/API-server push | `sched_job_new_fun`/`apiserver_job_new_fun` unconditionally call `xcesched_push_joblist`/`apiserver_push_joblist` with whatever `xlist_t` the hook fire carries — already N-capable by call count | Requirement 6: the wire payload itself still enumerates N full `Job`/`SchedJob` structs; a compact descriptor push is a new wire path, not a byproduct of the existing one |

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    Client[Client array submit]
    Handler[handle_jobsubmit]
    ArrayAppend[Single array WAL append]
    CommitEngine[Commit engine one barrier]
    ApplyPumpLive[Apply pump live]
    ApplyPumpReplay[Boot replay]
    StandbyCatchup[Standby catch up]
    RestoreArray[restore_job_array shared helper]
    Fence[Live apply resource fence]
    HookFire[HOOK_DETAIL_JOB_NEW one fire N jobs]
    SchedPush[Compact push to scheduler]
    ApiPush[Compact push to apiserver]
    SchedExpand[Scheduler local expansion]
    ApiExpand[Apiserver local expansion]

    Client --> Handler
    Handler --> ArrayAppend
    ArrayAppend --> CommitEngine
    CommitEngine -->|majority ack| ApplyPumpLive
    ApplyPumpLive --> RestoreArray
    ApplyPumpReplay --> RestoreArray
    StandbyCatchup --> RestoreArray
    RestoreArray -->|resource exhaustion| Fence
    RestoreArray -->|success| HookFire
    HookFire --> SchedPush
    HookFire --> ApiPush
    SchedPush --> SchedExpand
    ApiPush --> ApiExpand
```

**Architecture Integration**:
- **Selected pattern**: extend the existing commit-then-apply pipeline (`raft-apply-decoupling`) with one new WAL entry type and one new shared expansion primitive, rather than introducing a parallel array-specific pipeline. The array record travels through the exact same append → majority-commit → apply-pump → hook-fire stages a single-job submission does; only the payload shape and the expansion fan-out differ.
- **Domain/feature boundaries**: `restore_job_array()` is the only place array-record bytes become N `job_t` entries — both live dispatchers call it, neither reimplements it, mirroring how `restore_job()` already anchors single-job construction for both dispatchers today.
- **Existing patterns preserved**: `jobsubmit_async_ctx_t`'s batched single-reply mechanism, `persist_register_pending_apply()`'s command hand-off, the in-flight jobid-range registry, the `t_apply_seam_notifying` thread-local guard, `wal_classify`'s WAIT/RETAIN table, the `Data` union's additive-variant extension pattern.
- **New components rationale**: `restore_job_array()` (no array-aware expansion primitive exists); the Live Apply Resource Fence (no live-path analog of the boot-only `recovery_unrecoverable`/`persist_recover_loud_fence_if_unrecoverable` fence exists); the compact push variant (existing push wire format is per-job, not descriptor-shaped).
- **Steering compliance**: preserves the synchronous-commit-as-policy invariant (`ha-async-io.md`) — the array append still runs off the iomgr thread through the unchanged commit-engine contract; the Live Apply Resource Fence follows the controller's Process Lifecycle invariant exactly (`src/controller/CLAUDE.md`: loud ERROR, non-promotable, stay alive, keep retrying, self-heal — never self-terminate).

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|------------------|-------|
| Controller core | C (gnu17), existing | All changes land in `src/controller/{job.c,persist.c,sched.c,apiserver.c,ha.c}` and `src/lib/jobj.c` | No new language/runtime |
| Serialization | Cap'n Proto (existing `controller.capnp`) | `WAL_JOB_ARRAY_SUBMIT` payload and the new compact push payload both reuse the existing `struct JobSubmit`; one new `Data` union variant added for the push path | No new struct; `encode_JobSubmit`/`decode_JobSubmit` already generated |
| Scheduler daemon | C (gnu17), existing | `src/scheduler/src/handler.c` gains a sibling case to `Data_data_schedjobs` for local array expansion | Follows `scheduler-controller-msg-gap`'s established extension pattern |
| API server | Go 1.23.4, existing | `src/frontend/apiserver/backend/` gains a sibling decode path to `cJobToModel` for local array expansion | CGO bridge, consistent with existing job-push consumption |
| Test harness | `controller_unit_test`, `xcelib_test`, `/xce-test-ha` | Unit coverage for the new apply path and fence; live soak for non-regression | Extends existing harnesses, no new framework |

## File Structure Plan

### Modified Files
- `src/include/wal_types.h` — add `WAL_JOB_ARRAY_SUBMIT` to `wal_entry_type` (a new value distinct from `WAL_JOB_SUBMIT`, keeping the single-job format untouched per Requirement 8.1).
- `src/controller/job.c` — `handle_jobsubmit()`: branch on `submit->array_size == 1` (today's exact code path, unmodified) vs. `> 1` (new single-append path: encode once via a new `capnp_encode_job_array()`-equivalent using `encode_JobSubmit`, append via a new `append_job_array_wal_entry_local()`, register one continuation via a new `persist_submit_job_array_async()`, set `actx->pending = 1`). `jobsubmit_commit_continuation`/`jobsubmit_apply_complete`/`jobsubmit_async_ctx_t` are reused unmodified — both are already parametric on `pending` and keyed by LSN, not by element count. New shared helper `restore_job_array()` (sibling to `restore_job()`, same file) implements atomic N-element construction: scratch-builds all elements first, bulk-inserts into `c->jobmgr` only once every element succeeds, and discards the scratch list directly (never touching `c->jobmgr`) on failure.
- `src/controller/persist.c` — `wal_classify()`: add `WAL_JOB_ARRAY_SUBMIT → WAL_MODE_WAIT` (same classification as `WAL_JOB_SUBMIT`). `apply_committed_entry()` and `apply_wal_entry()`: each gains one new `case WAL_JOB_ARRAY_SUBMIT` calling the shared `restore_job_array()`; `apply_committed_entry()`'s case additionally builds the N-element `xlist_t` and fires `HOOK_DETAIL_JOB_NEW` once (mirroring its existing single-job case); `apply_wal_entry()`'s case stays hook-free (mirroring its existing single-job case). `persist_apply_pump_run()`: capture `apply_committed_entry()`'s return value; on negative rc, stop the walk without advancing `last_applied_lsn`, log loud, engage the Live Apply Resource Fence (mirrors the existing `apply_pump_read_entry` failure branch immediately above it); on success following a prior failure at this LSN, clear the fence. New small primitives: `persist_apply_resource_fenced()`, `persist_mark_apply_resource_fenced()`, `persist_clear_apply_resource_fence()`.
- `src/controller/ha.c` — both promotion-gate eligibility call sites (periodic fence-release check, election-time check) gain one additional required condition: `!persist_apply_resource_fenced(&c->persist)`, threaded alongside the existing `apply_divergent`/`self_restart_pending` conditions.
- `src/controller/sched.c` — `sched_job_new_fun()`: branch on `l->count > 1` (the only way a `HOOK_DETAIL_JOB_NEW` fire ever carries more than one element, per Requirement 3.4) to call a new `xcesched_push_jobarray()` (encodes the compact `Data` union array variant) instead of `xcesched_push_joblist()`.
- `src/controller/apiserver.c` — `apiserver_job_new_fun()`: same `l->count > 1` branch, calling a new `apiserver_push_jobarray()`.
- `src/schema/controller.capnp` — `struct Data`'s union gains one new variant reusing `JobSubmit` (e.g. `jobArray @6: JobSubmit $C.mapname("job_array")`), following the existing additive-variant pattern (`schedjobs`/`jobs`/`workers`/`resources`/`resvals`).
- `src/scheduler/src/handler.c` — sibling case to the existing `Data_data_schedjobs` (line 112) decoding the new array variant and expanding it locally into N schedulable tasks via the same `base_jobid + i` arithmetic used by the controller, logging one confirmation line per array (Requirement 6.4).
- `src/frontend/apiserver/backend/` — sibling decode path to `cJobToModel` (`convert.go:31`) for the new array variant, expanding N `models.JobInfo` entries locally, logging one confirmation line per array (Requirement 6.4).
- `src/lib/jobj.c` — `job_init()`: check `makeJobsub()`'s (and `makeString(src->ckptinfo)`'s) return value; propagate failure instead of storing NULL with a success return (Requirement 10.1). Root-cause and fix the `jobsub_table` initialization-ordering gap behind the previously observed NULL-with-EINVAL failure (Requirement 10.2). `job_destroy()`: fix the `accounts` leak (currently `/* FIXME??? */`, jobj.c:584) by adding the missing free call, following the existing pattern used for `allocs`/`pidinfo`/`run_usage` (Requirement 10.4).
- **`src/lib/request.c` (added during implementation, task 5.3b, discovered 2026-07-15):** `extract_data()` — the shared inbound-push reassembly dispatcher every receiver (scheduler, API server) passes through inside `vtask_entry()` (`src/lib/xcectx.c`) before its own handler ever runs. Missed by this component's original architecture analysis, which examined `sched.c`/`apiserver.c`/`handler.c` but not the transport dispatch layer beneath them. Its switch has no case for `Data_data_jobArray` and falls into `default: rc = -EINVAL`; because `vtask_entry()`'s `if (callnext)` treats any nonzero return (including a negative errno) as "reassembly complete, proceed," this silently frees the real payload and hands the receiver an empty `dataset_t`, with no log anywhere in the chain. Add a `case Data_data_jobArray:` passing the single, unchunked descriptor straight through — no pagination loop needed, unlike the existing LIST-shaped cases.

### New Test Files
- `src/controller/testsuite/test_job_array_apply.c` — unit tests: single-record N-way expansion, idempotent re-apply (no duplicate elements), all-or-nothing scratch-discard on a simulated construction failure (never touches `c->jobmgr`), identical results from `apply_committed_entry()` vs. `apply_wal_entry()` for the same record.
- `src/controller/testsuite/test_job_array_fence.c` — unit tests: `persist_apply_pump_run()` halts and does not advance `last_applied_lsn` on a negative `apply_committed_entry()` return; Live Apply Resource Fence engages, blocks promotion eligibility, and self-clears on a successful retry (Requirement 9.6's fault-injection scenario).
- `src/lib/testsuite/test_jobj_hardening.c` — unit tests for the five Requirement 10 gaps: `job_init` propagates a `makeJobsub()`/`makeString()` NULL; `jobsub_table` is initialized before first use under the previously-observed ordering; MOVE_PTR fields are confirmed NULL on a freshly-built array-element payload; `job_destroy` no longer leaks `accounts`; `decode_Job`/`free_Job` `ckptinfo` coverage is confirmed correct as-is (no wire field exists, so no leak).
- Extend `/xce-test-ha`'s soak scenario set with a large-array submission case (commit/push count assertions) and a chaotic-failover case submitting an array immediately before killing the primary (Requirement 8.2, 8.3).

## System Flows

### Array submission: append, commit, apply, push

```mermaid
sequenceDiagram
    participant C as Client
    participant H as handle_jobsubmit
    participant CE as Commit engine
    participant AP as Apply pump
    participant RA as restore_job_array
    participant N as HOOK_DETAIL_JOB_NEW
    participant S as Scheduler push
    participant A as Apiserver push

    C->>H: array submit base spec plus array_size
    H->>H: validate, jobid_range_exists, inflight_range_try_add (unchanged, range-shaped)
    H->>CE: append ONE WAL_JOB_ARRAY_SUBMIT, register ONE continuation
    CE-->>CE: majority ack (one barrier)
    CE->>AP: resolve committed, hand off to apply pump
    AP->>RA: restore_job_array (base_jobid, array_size, base spec, lsn)
    alt all N constructed
        RA->>N: fire once, N-element list
        N->>S: compact push (base spec + range)
        N->>A: compact push (base spec + range)
        AP->>H: reply success (base jobid)
    else resource exhaustion mid-loop
        RA->>RA: discard scratch list (c->jobmgr never touched)
        RA-->>AP: negative rc
        Note over AP: last_applied_lsn NOT advanced; fence engaged; retried next kick
    end
```

Key decision not obvious from the diagram: `restore_job_array()` builds every element in a scratch list and only bulk-inserts into `c->jobmgr` once the whole array has successfully constructed — so the array is never observably partial to a concurrent reader on the *success* path either, not only when a retry starts over from index 0 after a failure (Requirement 2.3, 9.4).

### Live apply resource fence: engage and self-heal

```mermaid
flowchart LR
    Apply[apply_committed_entry fails resource exhaustion] --> Log[Loud ERROR logged]
    Log --> Fence[Live Apply Resource Fence engaged]
    Fence --> Gate[Promotion gate refuses this node]
    Gate --> Retry[Apply pump retries same LSN on next kick]
    Retry -->|still failing| Log
    Retry -->|succeeds| Clear[Fence cleared by apply pump itself]
    Clear --> Eligible[Node eligible for promotion again]
```

Key decision: the fence clears from inside the apply pump the moment the retried LSN succeeds — it does not depend on an election or on the unrelated WAL-durability catch-up check ever running, so a long-lived primary that never loses leadership still self-heals (Requirement 9.5). This is deliberately a *new*, narrowly-scoped fence rather than a reuse of `recovery_unrecoverable`, whose existing periodic clear is gated on WAL durability catch-up alone and would wrongly clear before the stuck apply actually resolves (see `research.md`, Design Decisions).

## Requirements Traceability

| Requirement | Summary | Components |
|---|---|---|
| 1.1-1.3 | Single WAL record, single commit barrier, no per-element append | WAL Array Submission |
| 2.1-2.3 | Atomic, all-or-nothing commit per record | WAL Array Submission (commit), Array Apply Expansion (no partial construction from an uncommitted record) |
| 3.1-3.4 | Shared expand-on-apply, online = replay = standby catch-up, idempotent re-apply, one hook fire | Array Apply Expansion |
| 4.1-4.3 | Standby reconstruction from one committed position | WAL Array Submission (replication carries one record, unchanged replication mechanics), Array Apply Expansion |
| 5.1-5.3 | Preserved per-element identity/addressing | Array Apply Expansion (unchanged `base_jobid + i` arithmetic — verification-only, no new mechanism) |
| 6.1-6.5 | Single compact push to scheduler and API server, local expansion | Compact Scheduler/API-Server Push |
| 7.1-7.2 | Bounded record size | WAL Array Submission (satisfied by construction — see Architecture table) |
| 8.1-8.4 | No regression to non-array submission or existing tests | Non-Regression & Testing (cross-cutting; `array_size == 1` path untouched) |
| 9.1-9.6 | Deterministic, total apply — pre-commit rejection, live resource fence | WAL Array Submission (9.1, 9.2 — pre-commit validation), Array Apply Expansion (9.3 — total apply post-commit), Live Apply Resource Fence (9.4, 9.5, 9.6) |
| 10.1-10.5 | jobj.c hardening | jobj.c Hardening |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|---|---|---|---|---|---|
| WAL Array Submission | job.c, persist.c, wal_types.h | Encode and commit one array record through one barrier | 1, 4.1-4.2, 7, 9.1, 9.2 | Commit Engine (P0), in-flight range registry (P0) | Service, Batch |
| Array Apply Expansion | persist.c, job.c (`restore_job_array`) | Expand a committed array record into N `job_t` entries, identically for live apply / replay / standby catch-up | 2, 3, 4.3, 5, 9.3 | `apply_committed_entry`/`apply_wal_entry` (P0), jobj.c primitives (P0) | Service, Batch |
| Live Apply Resource Fence | persist.c, ha.c | Fence a node non-promotable on a resource-exhaustion apply failure, self-heal on success | 9.4, 9.5, 9.6 | Apply pump (P0), promotion gate (P0) | State |
| Compact Scheduler/API-Server Push | sched.c, apiserver.c, controller.capnp, scheduler `handler.c`, apiserver `backend/` | One compact push per committed array, expanded locally by each receiver | 6 | `HOOK_DETAIL_JOB_NEW` (P0), `Data` union (P0) | Event, API |
| jobj.c Hardening | src/lib/jobj.c | Reliable, leak-free job construction under N-element reuse | 10 | RTTI object system (P0) | Service |

### WAL Array Submission

#### Single array record append and commit

| Field | Detail |
|-------|--------|
| Intent | Encode the whole array as one `WAL_JOB_ARRAY_SUBMIT` record and commit it through one barrier |
| Requirements | 1.1, 1.2, 1.3, 4.1, 4.2, 7.1, 7.2, 9.1, 9.2 |

**Responsibilities & Constraints**
- `handle_jobsubmit()` branches on `array_size`: `== 1` runs today's unmodified single-job path (Requirement 8.1); `> 1` runs the new single-append path.
- Pre-commit validation (`jobid_range_exists`, `inflight_range_try_add`) is unchanged and already range-shaped — no new range-claim logic is needed (Requirement 9.1).
- The record's encoded payload is `struct JobSubmit` (`jobid` = base, `arraySize`, one `submit: JobSubmitData`) — O(1) in `array_size` by construction, and `array_size` is already hard-capped at 32768 upstream by `idgen()` (Requirement 7 satisfied without a chunking policy).
- Rejection on failed pre-commit validation happens strictly before any WAL append (Requirement 9.2), identical ordering to today's single-job path.

**Dependencies**
- Outbound: `commit_submit_async()` via a new `persist_submit_job_array_async()` (Criticality P0)
- Outbound: `inflight_range_try_add`/`jobid_range_exists` (Criticality P0, unchanged)
- Inbound: `handle_jobsubmit()` (Criticality P0)

**Contracts**: Service [x] / Batch [x]

##### Service Interface
```c
/* Encodes and appends ONE array-shaped WAL entry (WAL_JOB_ARRAY_SUBMIT,
 * payload = encoded JobSubmit), registers ONE commit continuation. Mirrors
 * persist_submit_job_payloads_async's per-entry contract but for a single
 * array-shaped entry -- no loop over elements at this layer. */
int persist_submit_job_array_async(persist_t* p, uint64_t base_jobid,
                                   uint64_t array_size,
                                   const xce_job_submit_t* base_spec,
                                   uint64_t* out_lsn,
                                   persist_job_commit_notify_fn fn, void* ctx);
```
- Preconditions: `array_size > 1`; the `[base_jobid, base_jobid + array_size)` range has already been claimed via `inflight_range_try_add`.
- Postconditions: on success, exactly one WAL entry is durably appended locally and exactly one continuation is registered; `fn` is invoked exactly once with the outcome (mirrors `persist_submit_job_payloads_async`'s per-payload contract, specialized to one payload).
- Invariants: no `job_t` object exists in `c->jobmgr` for any array index until the commit succeeds and the apply seam runs (same non-negotiable ordering as the single-job path).

##### Batch / Job Contract
- Trigger: `handle_jobsubmit()`'s array-size-branch, after pre-commit validation passes.
- Input / validation: `(base_jobid, array_size, base_spec)`, already range-claimed.
- Output / destination: one WAL entry; one registered commit continuation feeding `jobsubmit_commit_continuation`/`jobsubmit_apply_complete` unmodified (`actx->pending = 1`).
- Idempotency & recovery: a client retry after a commit timeout produces a second, independent `WAL_JOB_ARRAY_SUBMIT` entry for the same range — the array-scoped analog of bug-1419's existing single-job dedup story (see Testing Strategy, Requirement 8.3).

### Array Apply Expansion

#### restore_job_array — shared N-way expansion primitive

| Field | Detail |
|-------|--------|
| Intent | Expand one committed array record into N `job_t` entries, atomically (scratch-build then bulk-insert — never partially visible), identically for every caller |
| Requirements | 2.1, 2.2, 2.3, 3.1, 3.2, 3.3, 3.4, 4.3, 5.1, 5.2, 5.3, 9.3 |

**Responsibilities & Constraints**
- Decodes `struct JobSubmit` from the committed payload; for `i` in `[0, array_size)`, computes `jobid = base_jobid + i` (the exact arithmetic `build_job_submit_payloads` uses today, Requirement 5.1) and constructs a `job_t` via `create_job()`, sharing the one decoded `submit` pointer across elements (mirroring `restore_job()`'s existing `submit`/`ckptinfo` ownership discipline for a single job).
- Idempotent per index: if `jobid` already exists in `c->jobmgr`, that index is skipped, not treated as an error (handles a re-apply of an already-fully-applied record). If every index already exists, returns `-EEXIST` (mirrors `restore_job()`'s convention), and the caller fires no hook (Requirement 3.3).
- Scratch-construct-then-bulk-insert, not construct-then-rollback: all `array_size` elements are built via `create_job()`-equivalent construction into a local scratch list first, **never touching `c->jobmgr` while the construction loop is running**. Only after every element in the scratch list succeeds does one final pass insert all of them into `c->jobmgr` under a single write-lock acquisition (one atomic visibility step, not per-element lock/unlock cycles — this is what makes the array invisible-then-visible rather than visible-then-rolled-back on the success path). This means there is no window during *construction*, success or failure, where a concurrent reader of `c->jobmgr` (a status query, a cancel request) can observe some array indices present and others absent. If construction fails at index `k` (resource exhaustion only — content-derived failures are impossible here because Requirement 9.1's pre-commit validation already ran), the `0..k-1` scratch objects are destroyed directly (no `c->jobmgr` mutation ever occurred, so no `key_del`/rollback race is possible) and the function returns a negative rc. A retry re-enters the loop from index 0 against a fresh scratch list, not from a saved offset.
  - **Residual gap in the bulk-insert step itself** (found during implementation, task 3.1 review round 1): the final insert-into-`c->jobmgr` pass is *not* actually failure-proof — `key_add()` can itself fail on allocation/growth exhaustion (`c->jobmgr`'s underlying table grows via `realloc()`, and no capacity-reservation primitive exists anywhere in this codebase's `kv.c`/`kvmgr.c`/`zarray.c` stack to pre-check this before mutating). Because the underlying table has no `key_del`, an already-inserted prefix `[0, inserted)` cannot be cleanly un-inserted if a later element in the same bulk pass fails. The implemented contract is therefore: `restore_job_array()` never *silently* claims success on a bulk-insert failure (it returns a negative errno, honestly), but a prefix of elements may be left genuinely, permanently visible in `c->jobmgr` while the overall call reports failure — and critically, that prefix is never included in `out_new_jobs`, so `HOOK_DETAIL_JOB_NEW` never fires for it, on this attempt or any later successful retry of the same LSN. Tasks 3.2/4.x must account for this: a bulk-insert-phase failure leaves elements that are individually addressable (status/cancel/completion work) but permanently un-pushed to the scheduler/API server unless something later re-derives and re-notifies them. This is a narrower, real exception to "atomically invisible-then-visible," not the general case — it fires only on late-stage resource exhaustion, not on ordinary construction failures (which remain fully atomic per the paragraph above).
- Called identically from `apply_committed_entry()` (which additionally builds the N-element list and fires `HOOK_DETAIL_JOB_NEW` once, Requirement 3.4) and from `apply_wal_entry()` (hook-free, matching its existing single-job case) — this is the one place expansion logic exists (Requirement 3.1, 3.2). **Correction (2026-07-14, see the Existing Architecture Analysis table's "Live apply dispatch" row):** `apply_wal_entry()` has zero production call sites — `apply_committed_entry()` alone delivers Requirement 3.1/3.2/4.1 for live apply, boot replay, and standby catch-up. The `apply_wal_entry()` wiring is retained as defensive parity with `WAL_JOB_SUBMIT`'s existing (also-dead) case in the same switch, not as a requirement-delivery path.
- Every constructed element is tagged with `submit_lsn = entry_lsn` (mirrors `restore_job()`, feeds the existing egress-watermark predicate unchanged).
- Exposes an `XCE_TESTING`-guarded fault-injection seam (a forceable construction failure at a caller-chosen element index, compiled out in production builds, matching this codebase's existing `#ifdef XCE_TESTING`-guarded test-only-symbol convention per `structure.md`) — the concrete unit `test_job_array_apply.c` drives directly to satisfy Requirement 9.6's fault-injection scenario.

**Dependencies**
- Outbound: `create_job()` (job.c, unchanged), `jobj.c`'s hardened primitives (Criticality P0)
- Inbound: `apply_committed_entry()`, `apply_wal_entry()` (Criticality P0, both)

**Contracts**: Service [x] / Batch [x]

##### Service Interface
```c
/* Expands one committed WAL_JOB_ARRAY_SUBMIT payload into job_t entries.
 * Builds every element into a local scratch list first -- c->jobmgr is
 * never mutated until every element has successfully constructed, then one
 * final pass inserts them all (a pass that cannot fail). Returns the count
 * of NEWLY created elements (0..array_size) on success, -EEXIST if every
 * index already existed (idempotent no-op, no hook should fire), or a
 * negative errno on a genuine construction failure -- in which case the
 * scratch list is discarded directly and c->jobmgr was never touched (Req
 * 9.4: no partial set ever observable, on the failure path OR the success
 * path). */
int restore_job_array(cluster_t* c, const void* capnp_payload, size_t len,
                      uint64_t entry_lsn, xlist_t* out_new_jobs /* optional */);
```
- Preconditions: `lsn` is committed; payload decodes to a valid `JobSubmit` (guaranteed by Requirement 9.1/9.3 — pre-commit validation makes a content-derived decode failure here impossible).
- Postconditions: either all `array_size` elements exist in `c->jobmgr` tagged with `entry_lsn`, or none of this attempt's elements do — and this holds at every instant, not only at return: `c->jobmgr` is untouched until the entire array is ready to insert as one bulk pass, so no concurrent reader can ever observe a partially-constructed array (Requirement 2.3).
- Invariants: applying the same `(WAL_JOB_ARRAY_SUBMIT, lsn, payload)` twice produces the same observable state as applying it once (Requirement 3.3).

##### Batch / Job Contract
- Trigger: a new `case WAL_JOB_ARRAY_SUBMIT` in `apply_committed_entry()` (live apply pump, boot replay) and in `apply_wal_entry()` (standby catch-up).
- Input / validation: committed WAL payload, already CRC-verified by the point apply is reached (unchanged from the single-job path).
- Output / destination: N `job_t` entries in `c->jobmgr`; one `HOOK_DETAIL_JOB_NEW` fire (live-apply path only) carrying all N.
- Idempotency & recovery: guarded per-index existence check; `last_applied_lsn` is the resumable cursor for restart (unchanged mechanism, now also gated on this function's return value — see Live Apply Resource Fence).

### Live Apply Resource Fence

#### Apply-pump return-value gating and live promotion fence

| Field | Detail |
|-------|--------|
| Intent | Never silently skip a resource-exhaustion apply failure; fence the node non-promotable while retrying; self-heal on success |
| Requirements | 9.4, 9.5, 9.6 |

**Responsibilities & Constraints**
- `persist_apply_pump_run()` starts checking `apply_committed_entry()`'s return value (currently discarded) — on a negative result, the walk stops at that LSN without advancing `last_applied_lsn`, mirroring the existing `apply_pump_read_entry` failure branch immediately above it in the same function. This is a general fix (any WAL type's apply failure, not array-specific) that this feature's array-expansion failure is the first live trigger for.
- On engaging, logs a loud, operator-visible ERROR (LSN, reason) and sets a new, narrow fence flag — deliberately **not** `recovery_unrecoverable` (its existing periodic clear is gated on WAL-durability catch-up alone and would wrongly clear before the stuck apply actually resolves; see `research.md` Design Decisions).
- The fence clears itself, from inside the apply pump, the instant a retry of the same LSN succeeds — no dependency on an election or on any other periodic check running (Requirement 9.5).
- A fence on one node never blocks any other node's independent apply pump — each node's fence state is local (Requirement 9.5, satisfied structurally: nothing here is cross-node).
- Both existing `ha.c` promotion-gate call sites (periodic fence-release check, election-time eligibility check) gain one additional required condition alongside the existing `apply_divergent`/`self_restart_pending` checks.

**Dependencies**
- Outbound: none new — a local atomic flag on `persist_t`
- Inbound: `persist_apply_pump_run()` (sets/clears), `ha.c`'s two promotion-gate call sites (reads) (Criticality P0, both)

**Contracts**: State [x]

##### State Management
- State model: one new field on `persist_t` (an atomic LSN or bool) recording "an LSN is currently stuck failing to apply." Orthogonal to `recovery_unrecoverable` and `apply_verdict` — no shared state, no risk of one clearing the other's condition.
- Persistence & consistency: in-memory only, like `last_applied_lsn` itself — recomputed by construction (the pump re-derives "stuck" the next time it fails to apply the same LSN after a restart; it is not itself a correctness-critical persisted fact).
- Concurrency strategy: set/cleared only from `persist_apply_pump_run()`'s single-threaded walk (the existing `apply_pump_lock` already serializes this); read from `ha.c`'s promotion-gate paths under their existing locking.

## Data Models

No new persistent entity type. `job_t`, `xce_job_t`, and `job_submit_data_t` are unchanged — array elements are ordinary jobs once expanded, addressed exactly as single-submission jobs are today.

### Data Contracts & Integration

**WAL payload** (`WAL_JOB_ARRAY_SUBMIT`, new `wal_entry_type` value): the existing `struct JobSubmit` capnp struct, unmodified —

```capnp
struct JobSubmit $C.mapname("xce_job_submit_t") {
  jobid      @0: UInt64;              # base jobid
  arraySize  @1: UInt64 $C.mapname("array_size");
  submit     @2: JobSubmitData;       # one shared spec for every element
}
```

**Push payload** (new `Data` union variant, additive): reuses the same struct —

```capnp
struct Data $C.mapname("xce_data_t"){
  data :union $C.mapuniontag("table") {
    schedjobs     @0: SchedJobData $C.mapname("jobs");
    schedworkers  @1: WorkerData $C.mapname("workers");
    jobs          @2: JobData $C.mapname("jobs");
    workers       @3: WorkerData $C.mapname("workers");
    resources     @4: ResourceData $C.mapname("resources");
    resvals       @5: ResourceVals $C.mapname("resvals");
    jobArray      @6: JobSubmit $C.mapname("job_array");   # NEW
  }
}
```

Both reuses are deliberate (Requirement 6's own wire-shape note: "the same compact shape as the WAL record") — one capnp struct, three uses (client request, WAL payload, push payload), zero new encoder/decoder pairs to write or maintain.

### Compact Scheduler/API-Server Push

#### One push per committed array, expanded locally

| Field | Detail |
|-------|--------|
| Intent | Deliver a committed array to the scheduler and API server as one compact descriptor; each expands it locally into N entries |
| Requirements | 6.1, 6.2, 6.3, 6.4, 6.5 |

**Responsibilities & Constraints**
- `sched_job_new_fun()`/`apiserver_job_new_fun()` (the existing `HOOK_DETAIL_JOB_NEW` subscribers) branch on `l->count > 1` — the only way `apply_committed_entry()` ever fires this hook with more than one element (Requirement 3.4) — to call a new `xcesched_push_jobarray()`/`apiserver_push_jobarray()` instead of the existing per-job `..._push_joblist()`. The `l->count == 1` branch (ordinary single-job submission) is untouched (Requirement 8.1).
- The new push functions derive `(base_jobid, array_size)` from the already-constructed `xlist_t` (first element's jobid, list count) and encode one `Data.jobArray` message — never N individually-encoded `Job`/`SchedJob` structs.
- The scheduler (`handler.c`, sibling case to `Data_data_schedjobs`) and the Go apiserver (`backend/`, sibling path to `cJobToModel`) each expand the descriptor locally into N entries using the identical `base_jobid + i` arithmetic the controller itself uses — the same algorithm at three layers, not three different implementations.
- Redelivery on reconnect/rejoin must not double-expand: both receivers dedupe expanded entries by `jobid` using the exact same key space and lookup an ordinary single-job push already uses downstream (no new dedup mechanism) — a scheduler or API-server reconnect that re-receives an already-expanded array's compact push is a no-op per index, mirroring `restore_job_array()`'s own idempotent-per-index contract at the controller layer.
- Each receiver logs one identifiable confirmation line (base jobid + element count) once local expansion completes, satisfying Requirement 6.4's log-based verification without a new query API.
- Elements are treated as ordinary single jobs for scheduling purposes once expanded (Requirement 6.3) — no new scheduling logic.

**Dependencies**
- Inbound: `HOOK_DETAIL_JOB_NEW` fire from `apply_committed_entry()` (Criticality P0)
- Outbound scheduler push: direct `xce_request()`/`OPCode_push`/`Request_body_pushmsg` (Criticality P0). **Correction (found during task 5.1 review, 2026-07-15):** `xce_push_data_req()`'s dispatch switch has explicit cases only for jobs/workers/resources/resvals and hard-rejects (`-EINVAL`) anything else via its `default` case — it cannot carry a single non-list `Data.jobArray` descriptor, only chunk LIST-shaped payloads. `xcesched_push_jobarray()` therefore sends directly via `xce_request()`, mirroring `xce_push_req()`'s existing shape for a single non-list `xce_data_t` payload. This is a faithful, wire-compatible substitute but does NOT go through `xce_push_data_req()`'s `checkf`-gated `check_controller`/`c->running` liveness check that every other push path gets — a narrow, low-severity shutdown-race behavioral difference from sibling pushes, not a new safety hazard. Task 5.2's API-server push faces the identical fork and should follow the same direct-`xce_request()` pattern rather than re-deriving it.

**Contracts**: Event [x] / API [x]

##### Event Contract
- Published events: `HOOK_DETAIL_JOB_NEW` (unchanged detail, unchanged signature) — the branch on element count lives entirely inside the two existing subscribers, not in the hook system itself.
- Subscribed events: none new.
- Ordering/delivery guarantees: unchanged — one push per committed array, delivered after the array's own commit (same ordering guarantee single-job pushes already have).

##### API Contract
| Direction | Message | Payload | Notes |
|---|---|---|---|
| Controller → Scheduler | `Data.jobArray` push | `JobSubmit` (base jobid, array_size, one spec) | New variant; sent via direct `xce_request()` (not `xce_push_data_req()` — see Dependencies note above) |
| Controller → API server | `Data.jobArray` push | `JobSubmit` (base jobid, array_size, one spec) | Same variant, same direct-`xce_request()` transport, per connected API server (mirrors existing fan-out) |

**Implementation Notes**
- Integration: no new opcode — `Data`'s existing union discriminator (`table`) already routes to the new variant on both the scheduler and the API-server receive paths.
- Validation: confirm the scheduler's local expansion treats a partially-received/decode-failed array push the same way a malformed single-job push is handled today (existing error path, not new).
- Risks: the scheduler and API server must never assume `array_size == l->count` implies the array is contiguous starting at any index other than the pushed `jobid` — enforced by construction here since `base_jobid + i` is the only arithmetic either side uses.

### jobj.c Hardening

#### Reliable, leak-free construction under N-element reuse

| Field | Detail |
|-------|--------|
| Intent | Close five specific, pre-existing gaps in job/job-submit construction that this feature's N-way apply-time loop amplifies by array size |
| Requirements | 10.1, 10.2, 10.3, 10.4, 10.5 |

**Responsibilities & Constraints**
- **10.1** — `job_init()` (jobj.c:532) currently stores `makeJobsub()`'s return value into `dst->job.submit` unconditionally (jobj.c:543), including a NULL on allocation/interning failure, and still returns success. Fix: check the return, propagate failure. Same check for `makeString(src->ckptinfo)`.
- **10.2** — the previously observed `makeJobsub()` NULL-with-EINVAL-for-a-valid-`src`, traced to a likely `jobsub_table` (jobj.c:16) initialization-ordering issue: `job_obj_init()` must complete before the first `makeJob`/`makeJobsub` call on every path that can reach one, including this feature's new N-way `restore_job_array()` loop. Root cause to be confirmed and fixed at the actual ordering dependency, not papered over by 10.1's error propagation alone.
- **10.3** — `job_init()`'s `MOVE_PTR` fields (`pidinfo, allocs, accounts, run_usage, exit_info, pend_reasons`) are confirmed always NULL for a freshly-submitted job: `build_job_submit_payloads()` (job.c:229) and `restore_job_array()`'s equivalent per-element construction both `calloc()` a fresh `xce_job_t` per element and never set these six fields — repeated `MOVE_PTR` from a shared template is benign (NULL→NULL) and, for the array case specifically, each element gets its own independently-zeroed struct in the first place. No code change required; documented as a verified invariant.
- **10.4** — `job_destroy()`'s `accounts` leak (jobj.c:584, `/* FIXME??? */`) is fixed by adding the missing free call, following the existing pattern already used for `allocs`/`pidinfo`/`run_usage` in the same function. Array submission multiplies this leak by element count, making it a correctness requirement here, not a nice-to-have.
- **10.5** — verified: `struct Job`'s capnp schema (`controller.capnp:287`) has no `ckptinfo` field at all, so `decode_Job` never populates it and `free_Job` correctly never frees it — not a gap. Documented as a verified non-issue, no code change.

**Dependencies**
- Inbound: `create_job()`/`makeJob()` (job.c, jobj.c) — every caller, including this feature's `restore_job_array()` (Criticality P0)

**Contracts**: Service [x]

##### Service Interface
```c
/* job_init's contract changes: propagates a construction failure instead of
 * silently storing NULL and reporting success. */
static int job_init(meta_obj_t* obj, va_list ap); /* now fails on
   makeJobsub()/makeString(ckptinfo) returning NULL, not only on src==NULL */
```
- Preconditions: unchanged (`src != NULL`, `src->submit != NULL`).
- Postconditions: on success, `dst->job.submit` and `dst->job.ckptinfo` are never NULL-with-success; `job_obj_init()` has run before this call on every reachable path.
- Invariants: a `job_t` never exists in a half-initialized state visible to any caller.

**Implementation Notes**
- Integration: `restore_job_array()`'s all-or-nothing scratch-discard (Array Apply Expansion component) is the safety net if 10.1's now-propagated failure occurs mid-array — it is treated exactly like any other `create_job()` failure in that loop, discarded from the scratch list, never inserted into `c->jobmgr`.
- Validation: `src/lib/testsuite/test_jobj_hardening.c` (File Structure Plan) exercises all five items directly, independent of the array-expansion path, since these are general `jobj.c` fixes the single-job path benefits from too (Requirement 8.1's note explicitly permits this).
- Risks: 10.2's root cause is not yet confirmed at design time — flagged as a task-level investigation, not a pre-decided fix, consistent with Design vs. Implementation (state the contract: "`makeJobsub` must never observe an uninitialized `jobsub_table`," not the exact ordering fix). **Boundary note**: this item's File Structure Plan entry is `src/lib/jobj.c` only, on the assumption the fix is local to `jobj.c`'s own init/call-ordering. If root-causing 10.2 shows the actual gap is elsewhere (e.g. controller boot sequencing, a test-harness call path that skips `job_obj_init()`), that is a signal to split 10.2 into its own task with its own explicit file scope at task-generation time, not to silently widen this component's boundary.

## Error Handling

### Error Strategy
- **Pre-commit validation failure** (Requirement 9.2): `handle_jobsubmit()` rejects before any WAL append, identical to today's single-job path — no new error class.
- **Commit failure/timeout for the array record**: the single registered continuation resolves failed; `jobsubmit_reply_failure_once()` fires (unchanged); no job ever constructed (Requirement 2.2, 2.3 by construction — nothing to roll back).
- **Post-commit content-derived apply failure**: structurally impossible per Requirement 9.3 (9.1's pre-commit validation already ran) — if it somehow occurred, it is a programming-invariant violation, handled the same way the Live Apply Resource Fence handles any negative `apply_committed_entry()` return: loud ERROR, fence, retry forever. No `assert()`/abort (`src/controller/CLAUDE.md` Process Lifecycle invariant).
- **Post-commit resource-exhaustion apply failure**: `restore_job_array()` rolls back this attempt's partial construction; `persist_apply_pump_run()` halts the walk at this LSN; Live Apply Resource Fence engages; retried on the next apply-pump kick; self-heals on success (Requirement 9.4, 9.5).
- **`jobj.c` construction failure** (post-hardening): propagates as an ordinary `create_job()` failure into the same rollback path above — no new error class, just a previously-silent failure now made visible.

### Monitoring
- Loud `ERROR()` on Live Apply Resource Fence engagement, naming the stuck LSN and reason (mirrors `persist_recover_loud_fence_if_unrecoverable`'s existing log shape).
- `INFO()` on fence self-heal (LSN, retry count if tracked).
- One `INFO()` confirmation per array from the scheduler and one from the API server on successful local expansion (base jobid + element count), the log-based verification Requirement 8.4/6.4 rely on.

## Testing Strategy

- **Unit Tests**:
  - `persist_submit_job_array_async()` appends exactly one WAL entry and registers exactly one continuation for `array_size > 1` (Requirement 1.1-1.3).
  - `restore_job_array()`: N-way expansion produces `base_jobid..base_jobid+array_size-1`; idempotent re-apply (all-existing) is a no-op returning `-EEXIST`; a partial-existing edge case skips existing indices without duplicating (Requirement 2, 3.3, 5.1).
  - Simulated construction failure at element `k` of `N`, driven via `restore_job_array()`'s `XCE_TESTING`-guarded injection seam (forces a construction failure at a chosen index): elements `0..k-1` exist only in the discarded scratch list, never in `c->jobmgr` (nothing to tear down there); a subsequent successful retry builds a fresh scratch list from index 0 and bulk-inserts once, producing the full, non-duplicated set (Requirement 9.4, fault-injection scenario for 9.6).
  - `apply_committed_entry()` vs. `apply_wal_entry()` applied to the same committed array record produce byte-for-byte identical `c->jobmgr` state (Requirement 3.1, 3.2).
  - `persist_apply_pump_run()`: a negative `apply_committed_entry()` return halts the walk without advancing `last_applied_lsn`; a subsequent success advances it and clears the Live Apply Resource Fence (Requirement 9.4, 9.5).
  - Promotion-gate refusal while the Live Apply Resource Fence is set, at both existing `ha.c` call sites (Requirement 9.4).
  - `jobj.c`: all five Requirement 10 items, independent of array submission.
- **Integration Tests**:
  - End-to-end array submit: commit success → all N elements visible and individually addressable (status/cancel/completion) via existing per-element APIs, unchanged (Requirement 5.2, 5.3).
  - Single push received by both the scheduler and API server, each expanding N local entries and logging one confirmation (Requirement 6.1-6.5).
  - Non-array (`array_size == 1`) submission: byte-identical WAL record, identical commit-barrier count, identical reply, before and after this feature (Requirement 8.1).
- **Regression / Live Soak**:
  - `/xce-test-ha` extended with a large-array submission scenario asserting WAL/commit-barrier count (one record/one barrier, not N) and failover replay time for the committed array (Requirement 8.3, 4.3).
  - A client retrying an identical array submission across a primary failover follows the same cross-primary dedup mechanism as single-job submission (bug-1419), extended to the array's full index range as one unit — asserted not to double-submit or partially duplicate (Requirement 8.3).
  - Full sequential and chaotic `/xce-test-ha` soak, including array submission immediately before killing the primary, shows no new job loss, divergence, or liveness stall (Requirement 8.2).
  - Non-array (`array_size == 1`) submission WAL format, commit-barrier count, and reply are byte-identical before and after this feature (Requirement 8.1).

### Performance & Scalability
- WAL/replication cost for an array submission becomes O(1) in `array_size` (one record, one barrier, one replicated entry) versus today's O(N) — the target metric is commit-engine load and replication bandwidth for a large-array submission, measured via the `/xce-test-ha` large-array scenario.
- Apply-time cost (constructing N `job_t` entries inside one `apply_committed_entry()` call) is O(N) CPU inside a single, strictly-ordered apply-pump step — an accepted, bounded cost (N is hard-capped at 32767 by `idgen()`), not a new unbounded-latency risk; noted here as a deliberate trade-off, not re-litigated as a chunking requirement (see Requirement 7's Architecture-table resolution).
- Failover replay time for a committed array is governed by the size of the one array record (O(1)) rather than a per-element replay cost proportional to array size, since replay decodes and expands through the same `restore_job_array()` primitive as live apply (Requirement 4.3) — measured via the `/xce-test-ha` large-array scenario's failover-replay-time assertion.