docs: promote raft-apply-decoupling spec family from /kiro-discovery
Adds the Raft Apply-Ordering & Hook Decoupling roadmap phase: canonical pre-hooks -> raft commit -> apply -> notify ordering, with the raft protocol step lifted out of the notification hook fan-out (root of the bug-795/828/ 851/1060 divergence family). Decision: apply-after-commit via a single shared apply_committed_entry() seam reused by online apply and restart replay. - raft-drive-async: full spec docs (brief/requirements/research/design) committed alongside its already-tracked tasks.md; async-drive prerequisite for the apply-decoupling work, sequenced to start first (threatens failover liveness today). - raft-apply-decoupling: new spec (brief + facts-in-raft.md) covering the apply seam and the facts (WAL_MODE_RETAIN) apply-timing question (open decision D1). - job-array-single-wal-record: promoted from backlog to a tracked spec, sequenced after raft-apply-decoupling. - ha-consistency-invariants: add criterion 2.9 (Facts-Apply invariant), documenting the current on-receipt fact-reaction deviation this family will reconcile. - raft-commit-safety / raft-log-replication: reconcile spec.json phase to "implemented" (both were mislabeled tasks-generated despite full 18/18 and 24/24 task completion + raft_sim GO). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
This commit is contained in:
parent
9318f9977e
commit
5f307e297b
14 changed files with 1075 additions and 16 deletions
74
.kiro/specs/job-array-single-wal-record/brief.md
Normal file
74
.kiro/specs/job-array-single-wal-record/brief.md
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
# Brief: job-array-single-wal-record
|
||||
|
||||
## Problem
|
||||
|
||||
A job-array submission fans out into **one Raft log record per array element**. Confirmed in the controller:
|
||||
|
||||
- `job_submit()` loops `for (i = 0; i < submit->array_size; i++)` calling `create_job()` once per element (`job.c:74`).
|
||||
- `persist_job_hook_fun` then loops over the resulting job list calling `persist_wal_sync_append(p, WAL_JOB_SUBMIT, ...)` once per element (`persist.c:2556`), and **each** call runs its own `repl_wait_majority_ack` commit barrier (`persist.c:2248`).
|
||||
- The scheduler receives one push per committed element (`sched.c`).
|
||||
|
||||
So a 10k-element array becomes 10k WAL appends + 10k sequential majority-commit barriers + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, scheduler push volume, and failover replay time **linearly in array size**, while holding the submit thread across every per-element barrier.
|
||||
|
||||
## Current State
|
||||
|
||||
- No batched/atomic array record exists; the array is a loop over the single-job path.
|
||||
- The commit engine (`ha-commit-engine`) and barrier rollout (`ha-commit-barrier-rollout`) provide per-LSN majority commit and idempotency guards — the primitives needed to commit one larger record atomically.
|
||||
- Per-element identity (jobid = base_id + i) and per-element addressing (status / cancel / completion) are assumed throughout the controller and scheduler.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
A job-array submission persists, replicates, and pushes as a **single** Raft log record:
|
||||
|
||||
- The controller writes **one** WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through one commit barrier.
|
||||
- Replication carries the single record; standbys reconstruct the whole array from one committed LSN.
|
||||
- The scheduler receives the array as one push and expands locally.
|
||||
- Per-element identity and state addressing are preserved after expansion (status, cancel, completion still operate per task id).
|
||||
|
||||
## Approach
|
||||
|
||||
Single atomic array record on top of the post-decoupling apply seam:
|
||||
|
||||
- Define a `WAL_JOB_ARRAY_SUBMIT` record encoding base job spec + index range/count + sparse per-index overrides.
|
||||
- Command handler appends + commits this one record through a single barrier; on commit, the shared `apply_committed_entry()` seam expands it into N in-memory `job_t` entries (reusing `create_job` per element) under one apply step.
|
||||
- Restart replay decodes the single record and expands identically (same apply seam → no online/replay divergence).
|
||||
- Scheduler push delivers one array descriptor; scheduler expands locally.
|
||||
- Preserve idempotency: re-applying a committed array record (replay or retry) must not double-create elements.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: single WAL record format for arrays; atomic single-barrier commit; standby reconstruction from one LSN; single scheduler push + local expansion; preserved per-element identity/addressing; replay expansion; idempotency on re-apply; requirement + regression tests (large-array commit count, failover replay time, per-element status/cancel still correct).
|
||||
- **Out**: the apply-ordering/hook-decoupling refactor itself (sibling spec `raft-apply-decoupling`); non-array single-job path changes; scheduler policy changes.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
- The `WAL_JOB_ARRAY_SUBMIT` record format + encode/decode.
|
||||
- The single-barrier commit of the array record.
|
||||
- The expand-on-apply step (online + replay, shared).
|
||||
- The single scheduler push + scheduler-side expansion.
|
||||
|
||||
## Out of Boundary
|
||||
|
||||
- Per-element override semantics beyond what single-job submit already supports.
|
||||
- The commit engine / barrier internals (consumed unchanged).
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `raft-apply-decoupling` (HARD — builds the array expand step on the clean `apply_committed_entry()` seam rather than the old optimistic-before-commit path); `ha-commit-engine` + `ha-commit-barrier-rollout` (single atomic commit + idempotency); `ha-consistency-invariants` (replicated scheduling state).
|
||||
- **Downstream**: failover replay time and replication bandwidth improve for array-heavy workloads; lightweight rejoin re-attach optimization benefits.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: none (new record type).
|
||||
- **Adjacent**: `raft-apply-decoupling` (shared apply seam — must land first); `ha-commit-barrier-rollout` (idempotency guards reused for re-apply safety); `scheduler-controller-msg-gap` (scheduler push path).
|
||||
|
||||
## Constraints
|
||||
|
||||
- Must preserve per-element identity/state addressing after expansion (status, cancel, completion per task id).
|
||||
- Atomic commit: either the whole array commits or none of it (no partial array on a standby).
|
||||
- Idempotent re-apply (replay/retry must not duplicate elements).
|
||||
- Bounded record size / encoding for very large arrays (avoid a single pathological WAL entry; define a max or chunking policy in design).
|
||||
|
||||
## Provenance
|
||||
|
||||
Promoted 2026-06-24 from the roadmap backlog item `job-array-single-wal-record` (recorded 2026-06-16) via `/kiro-discovery`.
|
||||
Loading…
Add table
Add a link
Reference in a new issue