myxce/.kiro/specs/job-array-single-wal-record/research.md

Research & Design Decisions: job-array-single-wal-record

---

Purpose: Capture discovery findings, architectural investigations, and rationale that inform design.md.

Summary

• Feature: job-array-single-wal-record
• Discovery Scope: Extension (existing system) — the controller's job-submit and apply-seam machinery already exists (raft-apply-decoupling, complete); this feature adds an array-shaped WAL record and expansion step on top of it.
• Key Findings:
  • The brief's premise ("job_submit() loops calling create_job() per element") is stale. raft-apply-decoupling task 3.2 already removed that loop: handle_jobsubmit() today builds N lightweight xce_job_t WAL payloads (build_job_submit_payloads, job.c:221) and appends N separate WAL_JOB_SUBMIT entries via persist_submit_job_payloads_async() (persist.c:4846), each with its own commit continuation, batched into one client reply (jobsubmit_async_ctx_t, job.c:754). create_job() itself now runs only inside the apply seam (restore_job(), job.c:271). This feature's real job is collapsing those N WAL appends into one, not removing a create_job() loop that no longer exists.
  • The client-facing JobSubmit capnp struct (controller.capnp:533) already carries exactly (jobid, arraySize, submitData) — the compact shape Requirement 1 needs for the new WAL record. No new capnp struct is needed for encoding; a new wal_entry_type (WAL_JOB_ARRAY_SUBMIT) is needed so the existing single-job WAL_JOB_SUBMIT/struct Job format stays byte-for-byte unchanged (Requirement 8.1).
  • Because array elements are uniform clones (locked earlier in requirements discovery — no per-index overrides), the new record's encoded size is O(1) in array size: one JobSubmitData regardless of whether array_size is 2 or 32767. Combined with the pre-existing hard cap arraySize <= MAX_ARRAY_IDX + 1 (32768) enforced at jobid-minting time (idgen(), idgen.c:31, called from the Go apiserver before a request ever reaches the controller), Requirement 7's "bounded record size" is satisfied by construction — no chunking policy needs to be built.
  • apply_committed_entry() (persist.c:2721) is not the only live dispatcher of WAL_JOB_SUBMIT. apply_wal_entry() (persist.c:2574) is a second, hook-free dispatcher that remains the live standby catch-up/replicated-apply path (confirmed by an explicit comment at persist.c:3722-3725: "apply_wal_entry() ... is NOT retired by this change; it remains the live standby catch-up/replicated-apply dispatch used elsewhere (ha.c)"). Today's codebase avoids divergence between these two dispatchers by sharing one primitive, restore_job(), called from both switch statements. This feature must follow the same pattern with a new restore_job_array() shared primitive, not add expansion logic to only one dispatcher.
  • The scheduler/apiserver push infrastructure already accepts an N-element xlist_t in one hook fire (sched_job_new_fun/apiserver_job_new_fun unconditionally call xcesched_push_joblist/apiserver_push_joblist with whatever list they receive) — so "one push call" is already achievable for free. But Requirement 6's wire-shape note explicitly demands more: the wire payload itself must stay compact (a descriptor), not N individually-encoded Job/SchedJob structs, because the existing List(Job)/List(SchedJob) wire shape re-introduces O(N) payload size even inside a single RPC. This requires a genuinely new compact push message and a new discriminator (xlist->count > 1) in the two existing hook handlers, plus new local-expansion logic on the scheduler (src/scheduler/src/handler.c:112, case Data_data_schedjobs:) and the Go apiserver (src/frontend/apiserver/backend/convert.go, sibling to cJobToModel).
  • persist_apply_pump_run() (persist.c:3244) currently discards apply_committed_entry()'s return value ((void)apply_committed_entry(...), persist.c:3281) and unconditionally advances last_applied_lsn. A resource-exhaustion failure mid-array-expansion would, under today's code, be silently skipped forever — the array would never be constructed and never be retried. This is a genuine, load-bearing gap this feature must close (not a hypothetical): the pump must start checking the return value and gate last_applied_lsn advancement on it, mirroring the existing apply_pump_read_entry failure branch immediately above it.
  • No existing promotion-gate condition considers last_applied_lsn vs committed_lsn at all (ha.c gates only on truncated, durable_floor, apply_verdict == APPLY_DIVERGENT, self_restart_pending). A node stuck retrying a resource-exhaustion apply failure would look fully promotable today. A new, narrowly-scoped live fence is required — deliberately not reusing recovery_unrecoverable directly, because that flag's existing periodic auto-clear (ha.c:1069-1071, gated on WAL durability catch-up, not apply progress) would wrongly clear a fence whose underlying condition (stuck apply) is unrelated to durability and has not actually resolved.
  • jobj.c's job_init (jobj.c:532) does not check makeJobsub()'s return value before storing it (jobj.c:543) — a real, pre-existing gap, N×-amplified by this feature's tight per-array create_job() loop. job_destroy's accounts leak (jobj.c:584, /* FIXME??? */) is likewise pre-existing and N×-amplified. Both are already in scope via Requirement 10.

Research Log

Job-array submission path: brief vs. live code

• Context: the brief (written before raft-apply-decoupling fully landed) describes a for loop over create_job() in job_submit(). Needed to confirm this is still accurate before designing an insertion point.
• Sources Consulted: src/controller/job.c (live read, lines 196-1076), git history reference in code comments (commit 6f1fa97).
• Findings: The loop was removed by task 3.2. Current path: handle_jobsubmit() → build_job_submit_payloads() (builds N plain xce_job_t payload structs, no RTTI objects) → persist_submit_job_payloads_async() (N WAL appends, N commit continuations sharing one jobsubmit_async_ctx_t) → each continuation hands off to persist_register_pending_apply() → the apply pump's apply_committed_entry() constructs the actual job_t via restore_job()/create_job(), one per committed entry.
• Implications: The N-scaling this feature must eliminate is in WAL appends/commit continuations (job.c/persist.c), not in a create_job() loop inside handle_jobsubmit — that loop is already gone. Design must replace build_job_submit_payloads + persist_submit_job_payloads_async's per-element loop with a single array-shaped append, and add the N-way expansion loop to the apply seam instead (where create_job() already lives today).

WAL wire format and capnp schema

• Context: Requirement 1 needs "base spec + index range/count" as one record; needed to know whether new capnp schema is required.
• Sources Consulted: src/schema/controller.capnp (struct Job line 287, struct JobSubmit line 533), src/include/wal_types.h.
• Findings: struct JobSubmit $C.mapname("xce_job_submit_t") already has exactly (jobid, arraySize, submit: JobSubmitData) — the client request format. struct Job (the current WAL_JOB_SUBMIT payload) has no array fields and is single-job only.
• Implications: Reuse struct JobSubmit's existing capnp encoder (encode_JobSubmit, already generated) as the new WAL_JOB_ARRAY_SUBMIT payload verbatim — zero schema changes for the WAL format itself. Gate on a new wal_entry_type value so the single-job WAL_JOB_SUBMIT/struct Job format is untouched (Requirement 8.1).

Record size bound (Requirement 7)

• Context: needed to determine whether a chunking policy must be built.
• Sources Consulted: src/include/config.h (N_ARRAY_IDX_BITS, MAX_ARRAY_IDX), src/lib/idgen.c (idgen()), src/frontend/apiserver/backend/clib.go (GenerateJobID).
• Findings: idgen() rejects arraySize > MAX_ARRAY_IDX + 1 (32768) at jobid-minting time, called from the Go apiserver before any submission reaches the controller (clib.go:165) — a request whose array_size exceeds this cap never gets a jobid and never reaches handle_jobsubmit. Combined with the record's O(1) payload (one JobSubmitData, not per-element), the WAL_JOB_ARRAY_SUBMIT record's byte size is already bounded and does not grow with array_size at all.
• Implications: No new chunking mechanism is needed. Requirement 7.1/7.2 are satisfied by (a) reusing the compact JobSubmit shape (Requirement 1's own design decision) and (b) citing the pre-existing MAX_ARRAY_IDX cap as the hard upper bound on element count. Building "multiple atomically-committed chunks" machinery for a scenario that cannot occur would be scope creep (design-synthesis Simplification lens).

Apply-seam dispatch: two live dispatchers, not one

• Context: Requirement 3 demands one shared expansion path for online apply and replay; needed to confirm apply_committed_entry() is really the only live dispatcher.
• Sources Consulted: src/controller/persist.c (apply_wal_entry, line 2574; apply_committed_entry, line 2721; persist_recover's replay loop, line ~3742), .kiro/specs/raft-apply-decoupling/design.md.
• Findings: apply_wal_entry() was NOT retired by raft-apply-decoupling — an explicit in-code comment (persist.c:3722-3725) confirms it "remains the live standby catch-up/replicated-apply dispatch used elsewhere (ha.c)". persist_recover()'s own replay loop was switched to call apply_committed_entry() (persist.c:3742), but apply_wal_entry() is a second, separate switch statement over the same WAL types, still reachable from ha.c for standby catch-up. Both dispatchers avoid divergence today only because they both delegate the actual single-job construction to one shared helper, restore_job().
• Implications: the array-expansion loop must live in one new shared helper (restore_job_array(), sibling to restore_job()), called from a new WAL_JOB_ARRAY_SUBMIT case added to both apply_committed_entry() and apply_wal_entry()'s switch statements — matching the codebase's existing (not the aspirational) architecture. Recorded as a correction to Requirement 3's framing: the shared seam is the helper function, not either dispatcher alone.

Live apply-pump failure handling gap

• Context: Requirement 9.4/9.5 need "keep retrying the apply until it succeeds" and "fence non-promotable" for a resource-exhaustion failure during array expansion.
• Sources Consulted: src/controller/persist.c (persist_apply_pump_run, line 3244), src/controller/ha.c (promotion-gate call sites, lines ~1069 and ~1344).
• Findings: persist_apply_pump_run()'s dispatch loop discards apply_committed_entry()'s return value ((void)apply_committed_entry(...), persist.c:3281) and unconditionally advances last_applied_lsn. The read-failure branch immediately above it (apply_pump_read_entry failing) DOES stop the walk and leave last_applied_lsn unadvanced for retry — but nothing analogous exists for an apply_committed_entry()-internal failure. Separately, ha.c's promotion-eligibility gate never inspects last_applied_lsn vs committed_lsn at all; it gates on WAL durability/truncation/apply-divergence/self-restart-pending only.
• Implications: two concrete, necessary fixes, both required for Requirement 9 to hold, neither optional: (1) the pump must check apply_committed_entry()'s return value and mirror the existing read-failure branch (stop, don't advance, retried next kick) on a negative result; (2) a new, narrow live fence must gate promotion on "no LSN is currently stuck failing to apply" — deliberately not reusing recovery_unrecoverable (see next entry).

Why not reuse recovery_unrecoverable for the live apply fence

• Context: recovery_unrecoverable + persist_recover_loud_fence_if_unrecoverable() already implement the exact "loud, non-promotable, stay alive, keep retrying, self-heal" shape Requirement 9.4/9.5 wants — first instinct was to reuse it directly.
• Sources Consulted: src/controller/ha.c (both promotion-gate call sites), src/include/persist.h (recovery_unrecoverable field doc comment, APPLY_DIVERGENT handling).
• Findings: recovery_unrecoverable already has two distinct existing consumers with different clear semantics: (a) a periodic/timer-driven clear (ha.c:1069-1071) gated purely on WAL durability catch-up (durable_floor >= target_hwm), unrelated to apply progress; (b) apply_verdict == APPLY_DIVERGENT, a separate field, explicitly documented as "only ever recomputed by a fresh boot-time persist_recover() replay" and never live-cleared. If the new live apply-resource fence reused recovery_unrecoverable directly, condition (a)'s periodic clear would fire as soon as WAL durability caught up — which happens independently of whether the stuck array LSN has actually finished applying — silently un-fencing a node that is still failing to construct the array.
• Implications: introduce a new, narrowly-scoped field (design.md calls it the Live Apply Resource Fence) instead of overloading recovery_unrecoverable. Set by the pump on an apply_committed_entry() failure; cleared by the pump itself the moment a retry of that same LSN succeeds — no dependency on the election/promotion-gate call sites ever running, so a long-lived primary that never loses leadership still self-heals (Requirement 9.5).

Scheduler/apiserver compact push wire shape

• Context: Requirement 6's wire-shape note explicitly rules out "N individually-encoded full job structs bundled into one message" — needed to confirm today's push path does exactly that despite already being list-shaped.
• Sources Consulted: src/controller/sched.c (sched_job_new_fun line 271, xcesched_push_joblist line 14), src/controller/apiserver.c (apiserver_job_new_fun line 127, apiserver_push_joblist line 97), src/schema/controller.capnp (struct Data union, line 355; SchedJobData/JobData, lines 306-323), src/scheduler/src/handler.c (case Data_data_schedjobs:, line 112), src/frontend/apiserver/backend/convert.go (cJobToModel, line 31).
• Findings: both sched_job_new_fun and apiserver_job_new_fun unconditionally call their respective ..._push_joblist(c, l) regardless of l->count — for an N-element list (which the new array apply case will produce) this would encode N full Job/SchedJob capnp structs on the wire, reintroducing exactly the O(N) payload growth Requirement 6 rules out. struct Data's data :union $C.mapuniontag("table") (controller.capnp:356) is an additive discriminated union already carrying 6 variants (schedjobs, schedworkers, jobs, workers, resources, resvals) — adding one more compact variant is the established, low-risk extension pattern (matches scheduler-controller-msg-gap's own precedent).
• Implications: sched_job_new_fun/apiserver_job_new_fun need a branch on l->count > 1 (the only way apply_committed_entry's HOOK_DETAIL_JOB_NEW fire ever produces a multi-element list, per Requirement 3.4) that pushes a new compact Data union variant (reusing JobSubmit's shape again — base jobid + array_size + one JobSubmitData) instead of the existing per-job list. The scheduler and Go apiserver each need one new local-expansion routine, mirroring the controller's own base_jobid + i arithmetic.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
New WAL_JOB_ARRAY_SUBMIT type + reused JobSubmit capnp shape	Add one new wal_entry_type, reuse the existing JobSubmit struct as its payload	Zero capnp schema change; keeps WAL_JOB_SUBMIT/struct Job untouched (Req 8.1) by construction; symmetric with client wire format	New type must be added to two live dispatch switches (apply_committed_entry, apply_wal_entry) and to wal_classify	Selected
Overload WAL_JOB_SUBMIT with a variable-shape payload (array flag + optional range)	Single WAL type handles both single-job and array	Fewer enum values	Directly violates Requirement 8.1 ("leave the non-array WAL record format... unchanged"); forces every existing single-job decode site to branch defensively	Rejected
Chunk large arrays into multiple committed records	Bounds record size explicitly via a chunking policy	Handles unbounded array_size	Unnecessary: array_size is already hard-capped at 32768 by idgen(), and the record is O(1) regardless of count — chunking machinery would have no scenario to ever trigger	Rejected (Requirement 7 satisfied by construction instead)

Design Decisions

Decision: Reuse JobSubmit as the WAL_JOB_ARRAY_SUBMIT payload and the compact push payload

• Context: Requirement 1 needs a compact WAL shape; Requirement 6 needs a compact push shape "the same compact shape as the WAL record."
• Alternatives Considered:
  1. Define a new JobArraySubmit capnp struct duplicating JobSubmit's fields.
  2. Reuse struct JobSubmit verbatim for both the WAL payload and the push payload.
• Selected Approach: (2) — one capnp struct, three uses (client request, WAL payload, push payload), one encoder/decoder pair already generated.
• Rationale: JobSubmit already has exactly the three fields every one of these three uses needs (jobid as base, arraySize, one submit: JobSubmitData); requirements.md's own wire-shape note for Requirement 6 explicitly asks for "the same compact shape as the WAL record," making this the requirement-mandated answer, not just a simplification.
• Trade-offs: none identified — no functionality is given up by reuse here.
• Follow-up: confirm at implementation time that encode_JobSubmit/decode_JobSubmit (capnp-generated) round-trip cleanly when submit is a shared/interned pointer across all three use sites.

Decision: Array expansion lives in a new shared helper, called from both live dispatchers

• Context: apply_wal_entry() (standby catch-up) and apply_committed_entry() (primary live apply + boot replay) are both live and both currently delegate single-job construction to restore_job().
• Alternatives Considered:
  1. Add the expansion loop only inside apply_committed_entry(), leaving apply_wal_entry() unchanged.
  2. Add a new shared restore_job_array() helper, called from a new case in both switches.
• Selected Approach: (2).
• Rationale: Option (1) would silently leave standby catch-up unable to expand an array record, reintroducing exactly the online/replay divergence Requirement 3 exists to prevent — for the array case specifically, on day one.
• Trade-offs: one more call site to keep in sync (mitigated: both call sites do nothing but forward to the shared helper, so there is no logic to keep in sync, only a switch-case line).
• Follow-up: none — this mirrors restore_job()'s existing, already-proven sharing pattern exactly.

Decision: A new, narrow live apply-resource fence, not a reuse of recovery_unrecoverable

• Context: Requirement 9.4/9.5 need a live, self-healing, non-promotable fence for a resource-exhaustion apply failure.
• Alternatives Considered:
  1. Set recovery_unrecoverable directly from the live apply-pump failure path.
  2. Introduce a new, dedicated field with its own set/clear triggers.
• Selected Approach: (2).
• Rationale: recovery_unrecoverable's existing periodic clear (ha.c:1069-1071) is gated on WAL durability catch-up, a condition orthogonal to whether the stuck array LSN has actually finished applying — reusing it would risk a premature, incorrect un-fence.
• Trade-offs: one more boolean/condition threaded through the two promotion-gate call sites in ha.c; justified by correctness.
• Follow-up: verify at implementation time that both ha.c promotion-gate call sites (periodic release, election-time) are updated together — a design.md Revalidation Trigger.

Risks & Mitigations

• Apply-pump return-value fix is a behavior change on an existing, heavily-tested path (every WAL type's apply failure, not just arrays, now halts the walk instead of silently skipping) — Mitigation: today's apply_committed_entry() implementations never actually return a negative value in practice for any existing type (verified during discovery), so this fix is a no-op for all pre-existing call paths and only activates for the new array-expansion failure case; full HA soak (Requirement 8.2) is the regression gate.
• Two live dispatchers (apply_committed_entry, apply_wal_entry) must both gain the new case — Mitigation: shared helper (restore_job_array) means the only per-dispatcher code is one switch-case line; covered explicitly in File Structure Plan and Revalidation Triggers.
• jobj.c hardening (Requirement 10) touches a shared primitive used by every job construction, not just arrays — Mitigation: scoped narrowly to the five specific gaps requirements.md names (NULL-check, table-init-ordering root cause, MOVE_PTR-reuse audit, accounts leak, ckptinfo coverage audit); no broader refactor.

References

• .kiro/specs/raft-apply-decoupling/design.md — the apply_committed_entry() seam contract this feature builds on; its own Revalidation Triggers already flag "any new WAL entry type must be added to the apply_committed_entry() dispatch table or it silently fails to apply" and "job-array-single-wal-record landing — builds its array-expand step directly on apply_committed_entry()."
• .kiro/specs/ha-commit-engine/, .kiro/specs/ha-commit-barrier-rollout/ — majority-commit and idempotency primitives consumed unchanged.
• .kiro/specs/scheduler-controller-msg-gap/ — established precedent for extending the controller→scheduler push protocol with a new Data union variant.
• src/controller/CLAUDE.md — Process Lifecycle invariant ("a controller process MUST NEVER self-terminate... fail loud, fence, stay alive, keep retrying") governing the Live Apply Resource Fence's behavior.

---

Gap Analysis: job-array-single-wal-record

Note: design.md and tasks.md-adjacent work for this spec already exist and have passed design review (GO, zero critical issues, 2026-07-13). This gap analysis is produced retroactively per explicit user request and is used here to confirm/cross-check the design's approach against the framework below, not to precede it.

Requirement-to-Asset Map

Requirement	Existing Asset	Gap
1.1-1.3 (single WAL record/barrier)	handle_jobsubmit() (job.c:926), build_job_submit_payloads() (job.c:221), persist_submit_job_payloads_async() (persist.c:4846) — today's N-append, N-continuation path	Missing — no single-array-append path; wal_entry_type (wal_types.h:102) has no array variant
2.1-2.3 (atomic commit)	commit_submit_async() (commit engine, unchanged), jobsubmit_async_ctx_t/pending refcount (job.c:754, already generalizes to pending=1)	Constraint — commit-barrier atomicity is free (existing engine); apply-side atomicity (no partial c->jobmgr visibility) is not, since c->jobmgr insertion isn't natively transactional across N constructor calls
3.1-3.4 (shared expand-on-apply)	apply_committed_entry() (persist.c:2721), apply_wal_entry() (persist.c:2574) — two live dispatchers, both call restore_job() for the single-job case today	Missing — no restore_job_array()-equivalent; neither switch has a WAL_JOB_ARRAY_SUBMIT case
4.1-4.3 (standby reconstruction)	Replication streams whatever WAL bytes exist (repl.c, unchanged) — one record replicates as one entry by construction	None — falls out of 1+3 once built; no separate replication-layer work
5.1-5.3 (identity preservation)	base_jobid + i arithmetic (job.c:238), idgen() (idgen.c:22) reserving a contiguous range	None — verification-only; new expansion loop reuses this arithmetic identically
6.1-6.5 (compact push)	sched_job_new_fun/apiserver_job_new_fun (sched.c:271, apiserver.c:127), Data union (controller.capnp:355) — already N-capable by call count, not by wire shape	Missing — no compact Data union variant; no scheduler-side (handler.c:112) or apiserver-side (Go backend/) local-expansion logic
7.1-7.2 (bounded record size)	idgen()'s MAX_ARRAY_IDX cap (config.h:68, idgen.c:31), enforced upstream in the Go apiserver (clib.go:165) before a request reaches the controller	None — pre-existing cap plus the record's O(1) payload satisfies this by construction; no chunking policy needed
8.1-8.4 (non-regression)	Existing single-job path (job.c), existing /xce-test-ha harness	Constraint — array_size == 1 must stay byte-for-byte identical; verified achievable by branching before any shared state touches
9.1-9.2 (pre-commit validation)	inflight_range_try_add/jobid_range_exists (job.c:90-212) — already range-shaped from bug-1419	None — consumed unchanged, called once per array submission exactly as today
9.3-9.5 (deterministic apply / live fence)	persist_recover_loud_fence_if_unrecoverable/recovery_unrecoverable (persist.c:4259) — boot-replay-only fence shape	Missing — persist_apply_pump_run() (persist.c:3244) discards apply_committed_entry()'s return value (a real, load-bearing bug, not array-specific); no live-path fence exists
9.6 (fault injection)	This codebase's #ifdef XCE_TESTING-guarded test-only-symbol convention (structure.md)	Missing — no injection seam exists yet in any job-construction path
10.1, 10.4 (jobj.c NULL-check, leak)	job_init()/job_destroy() (jobj.c:532, 565)	Missing — both are concrete, located bugs with no fix yet
10.2 (jobsub_table ordering)	jobsub_table_init()/job_obj_init() (jobj.c:80)	Unknown — root cause suspected (init-ordering) but not confirmed; flagged as a task-level investigation in design.md, not pre-decided
10.3, 10.5 (MOVE_PTR safety, ckptinfo coverage)	job_init()'s MOVE_PTR fields, decode_Job/free_Job (generated code)	None — both verified as non-issues during design (freshly-calloc'd payloads are always NULL on these fields; ckptinfo has no wire field to leak)

Implementation Approach Options

Option A: Extend Existing Components Only

Every change lands inside the file that already owns that domain — job.c (submit), persist.c (apply/commit/pump), sched.c/apiserver.c (push), ha.c (promotion), jobj.c (construction) — with no new non-test files.

• ✅ Matches structure.md's "one domain per file" convention exactly; no domain boundary is invented.
• ✅ Reuses jobsubmit_async_ctx_t, the in-flight range registry, and the commit engine wiring completely untouched.
• ❌ persist.c is already one of the largest files in the component; this adds more surface to it.
• ❌ ha.c gets touched for a narrow addition (2 conditions, 2 call sites) but still requires the full TLA+/protocol-verification checklist review (src/controller/CLAUDE.md) since it's HA-critical.

Option B: New Standalone Component (e.g. a job_array.c)

Isolate the array-specific logic (restore_job_array, array-submit helpers) into a new file.

• ✅ Cleaner surface for unit-testing restore_job_array() in isolation.
• ❌ restore_job_array() must still be called from persist.c's two switch statements and share persist_t/cluster_t internals plus locking discipline (apply_pump_lock, c->jobmgr's RTTI lock) — a separate file buys little real isolation given this tight coupling, and risks violating the "No Hidden Shared Ownership" design principle by either exposing currently-internal persist.c state or duplicating its lock semantics.
• ❌ Array submission isn't a new domain in this codebase's terms — it's a new WAL record shape within the existing job domain — so a new file would be an organizational mismatch, not a genuine boundary.

Option C: Hybrid (extend in place, new functions not new domain files)

Extend the six existing files (Option A's list) plus controller.capnp, adding clearly-named new functions (restore_job_array, persist_submit_job_array_async, xcesched_push_jobarray, apiserver_push_jobarray, persist_apply_resource_fenced/_mark/_clear) rather than modifying existing function bodies destructively. New files are created only where isolation is genuinely warranted: test files (test_job_array_apply.c, test_job_array_fence.c, test_jobj_hardening.c).

• ✅ Every new function has one clear, single-file home matching its existing domain — no fragmentation, no fake isolation.
• ✅ Genuine isolation exactly where it earns its cost (tests).
• ❌ Touches 8 existing files across 2 languages (C controller/scheduler, Go apiserver) plus 1 schema file — coordination surface is wide even though each individual touch is small.
• ❌ The "two live dispatchers" fact (apply_committed_entry vs. apply_wal_entry) must be correctly carried into task generation and implementation by every task that touches either — a real risk if that fact isn't threaded through explicitly (mitigated: design.md's Revalidation Triggers and File Structure Plan both name both dispatchers explicitly).

Effort & Risk

• Effort: L (1-2 weeks) — significant functionality (new WAL type, apply-seam expansion, HA promotion-gate change, two new push consumers across two languages, five jobj.c fixes) spanning multiple files and one cross-language boundary, but no single piece requires inventing a new architectural primitive — every piece composes an already-proven existing mechanism (commit engine, hook system, RTTI object system, Data union).
• Risk: Medium — no unknown technology and no foundational primitive being invented from scratch, but two of the touch points are genuinely HA-sensitive (ha.c's promotion gate, persist_apply_pump_run()'s return-value handling) where a subtle mistake has correctness consequences beyond this feature. Mitigated by the design's scratch-build atomicity discipline and by full /xce-test-ha regression being a stated, non-optional gate (Requirement 8.2). Not High: nothing here requires new consensus/replication mechanics — those are consumed unchanged.

Recommendation

Option C (Hybrid) is the right fit and is what design.md already selected — confirmed by this analysis, not contradicted by it. Option A alone would have been viable too (the "new components" in Option C are functions, not files, in every case except tests), so in practice Option C and a disciplined Option A converge here; the distinction matters mainly for the test files, where isolation is unambiguously worth it.

Research items carried forward (already flagged in design.md's own Risk notes, repeated here for gap-analysis completeness):

• Requirement 10.2's jobsub_table initialization-ordering root cause — confirmed unknown at design time, scoped to a bounded task-level investigation with an explicit boundary note (if the fix turns out to be outside jobj.c, split into its own task rather than silently widening scope).