Rongsong Shen
b469b174a2
brief/requirements updated to reflect the reopened-and-now-closed post-GO gap-closure phase (tasks 7.1-7.3); adds research.md capturing the investigation trail; roadmap.md reflects current spec status.
74 lines
5 KiB
Markdown
74 lines
5 KiB
Markdown
# Brief: job-array-single-wal-record
|
|
|
|
## Problem
|
|
|
|
A job-array submission fans out into **one Raft log record per array element**. Confirmed in the controller:
|
|
|
|
- `job_submit()` loops `for (i = 0; i < submit->array_size; i++)` calling `create_job()` once per element (`job.c:74`).
|
|
- `persist_job_hook_fun` then loops over the resulting job list calling `persist_wal_sync_append(p, WAL_JOB_SUBMIT, ...)` once per element (`persist.c:2556`), and **each** call runs its own `repl_wait_majority_ack` commit barrier (`persist.c:2248`).
|
|
- The scheduler receives one push per committed element (`sched.c`).
|
|
|
|
So a 10k-element array becomes 10k WAL appends + 10k sequential majority-commit barriers + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, scheduler push volume, and failover replay time **linearly in array size**, while holding the submit thread across every per-element barrier.
|
|
|
|
## Current State
|
|
|
|
- No batched/atomic array record exists; the array is a loop over the single-job path.
|
|
- The commit engine (`ha-commit-engine`) and barrier rollout (`ha-commit-barrier-rollout`) provide per-LSN majority commit and idempotency guards — the primitives needed to commit one larger record atomically.
|
|
- Per-element identity (jobid = base_id + i) and per-element addressing (status / cancel / completion) are assumed throughout the controller and scheduler.
|
|
|
|
## Desired Outcome
|
|
|
|
A job-array submission persists, replicates, and pushes as a **single** Raft log record:
|
|
|
|
- The controller writes **one** WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through one commit barrier.
|
|
- Replication carries the single record; standbys reconstruct the whole array from one committed LSN.
|
|
- The scheduler receives the array as one push and expands locally.
|
|
- Per-element identity and state addressing are preserved after expansion (status, cancel, completion still operate per task id).
|
|
|
|
## Approach
|
|
|
|
Single atomic array record on top of the post-decoupling apply seam:
|
|
|
|
- Define a `WAL_JOB_ARRAY_SUBMIT` record encoding base job spec + index range/count (no per-index overrides — array elements are uniform clones today, distinguished only by jobid; the client submit API has no per-index override field).
|
|
- Command handler appends + commits this one record through a single barrier; on commit, the shared `apply_committed_entry()` seam expands it into N in-memory `job_t` entries (reusing `create_job` per element) under one apply step.
|
|
- Restart replay decodes the single record and expands identically (same apply seam → no online/replay divergence).
|
|
- Scheduler push delivers one array descriptor; scheduler expands locally.
|
|
- Preserve idempotency: re-applying a committed array record (replay or retry) must not double-create elements.
|
|
|
|
## Scope
|
|
|
|
- **In**: single WAL record format for arrays; atomic single-barrier commit; standby reconstruction from one LSN; single scheduler push + local expansion; preserved per-element identity/addressing; replay expansion; idempotency on re-apply; requirement + regression tests (large-array commit count, failover replay time, per-element status/cancel still correct).
|
|
- **Out**: the apply-ordering/hook-decoupling refactor itself (sibling spec `raft-apply-decoupling`); non-array single-job path changes; scheduler policy changes.
|
|
|
|
## Boundary Candidates
|
|
|
|
- The `WAL_JOB_ARRAY_SUBMIT` record format + encode/decode.
|
|
- The single-barrier commit of the array record.
|
|
- The expand-on-apply step (online + replay, shared).
|
|
- The single scheduler push + scheduler-side expansion.
|
|
|
|
## Out of Boundary
|
|
|
|
- Per-element override semantics beyond what single-job submit already supports.
|
|
- The commit engine / barrier internals (consumed unchanged).
|
|
|
|
## Upstream / Downstream
|
|
|
|
- **Upstream**: `raft-apply-decoupling` (HARD — builds the array expand step on the clean `apply_committed_entry()` seam rather than the old optimistic-before-commit path); `ha-commit-engine` + `ha-commit-barrier-rollout` (single atomic commit + idempotency); `ha-consistency-invariants` (replicated scheduling state).
|
|
- **Downstream**: failover replay time and replication bandwidth improve for array-heavy workloads; lightweight rejoin re-attach optimization benefits.
|
|
|
|
## Existing Spec Touchpoints
|
|
|
|
- **Extends**: none (new record type).
|
|
- **Adjacent**: `raft-apply-decoupling` (shared apply seam — must land first); `ha-commit-barrier-rollout` (idempotency guards reused for re-apply safety); `scheduler-controller-msg-gap` (scheduler push path).
|
|
|
|
## Constraints
|
|
|
|
- Must preserve per-element identity/state addressing after expansion (status, cancel, completion per task id).
|
|
- Atomic commit: either the whole array commits or none of it (no partial array on a standby).
|
|
- Idempotent re-apply (replay/retry must not duplicate elements).
|
|
- Bounded record size / encoding for very large arrays (avoid a single pathological WAL entry; define a max or chunking policy in design).
|
|
|
|
## Provenance
|
|
|
|
Promoted 2026-06-24 from the roadmap backlog item `job-array-single-wal-record` (recorded 2026-06-16) via `/kiro-discovery`.
|