docs: promote raft-apply-decoupling spec family from /kiro-discovery

Adds the Raft Apply-Ordering & Hook Decoupling roadmap phase: canonical
pre-hooks -> raft commit -> apply -> notify ordering, with the raft protocol
step lifted out of the notification hook fan-out (root of the bug-795/828/
851/1060 divergence family). Decision: apply-after-commit via a single
shared apply_committed_entry() seam reused by online apply and restart
replay.

- raft-drive-async: full spec docs (brief/requirements/research/design)
  committed alongside its already-tracked tasks.md; async-drive prerequisite
  for the apply-decoupling work, sequenced to start first (threatens
  failover liveness today).
- raft-apply-decoupling: new spec (brief + facts-in-raft.md) covering the
  apply seam and the facts (WAL_MODE_RETAIN) apply-timing question (open
  decision D1).
- job-array-single-wal-record: promoted from backlog to a tracked spec,
  sequenced after raft-apply-decoupling.
- ha-consistency-invariants: add criterion 2.9 (Facts-Apply invariant),
  documenting the current on-receipt fact-reaction deviation this family
  will reconcile.
- raft-commit-safety / raft-log-replication: reconcile spec.json phase to
  "implemented" (both were mislabeled tasks-generated despite full 18/18
  and 24/24 task completion + raft_sim GO).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
This commit is contained in:
Rongsong Shen 2026-07-03 02:41:11 +00:00
parent 9318f9977e
commit 5f307e297b
14 changed files with 1075 additions and 16 deletions

View file

@ -0,0 +1,74 @@
# Brief: job-array-single-wal-record
## Problem
A job-array submission fans out into **one Raft log record per array element**. Confirmed in the controller:
- `job_submit()` loops `for (i = 0; i < submit->array_size; i++)` calling `create_job()` once per element (`job.c:74`).
- `persist_job_hook_fun` then loops over the resulting job list calling `persist_wal_sync_append(p, WAL_JOB_SUBMIT, ...)` once per element (`persist.c:2556`), and **each** call runs its own `repl_wait_majority_ack` commit barrier (`persist.c:2248`).
- The scheduler receives one push per committed element (`sched.c`).
So a 10k-element array becomes 10k WAL appends + 10k sequential majority-commit barriers + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, scheduler push volume, and failover replay time **linearly in array size**, while holding the submit thread across every per-element barrier.
## Current State
- No batched/atomic array record exists; the array is a loop over the single-job path.
- The commit engine (`ha-commit-engine`) and barrier rollout (`ha-commit-barrier-rollout`) provide per-LSN majority commit and idempotency guards — the primitives needed to commit one larger record atomically.
- Per-element identity (jobid = base_id + i) and per-element addressing (status / cancel / completion) are assumed throughout the controller and scheduler.
## Desired Outcome
A job-array submission persists, replicates, and pushes as a **single** Raft log record:
- The controller writes **one** WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through one commit barrier.
- Replication carries the single record; standbys reconstruct the whole array from one committed LSN.
- The scheduler receives the array as one push and expands locally.
- Per-element identity and state addressing are preserved after expansion (status, cancel, completion still operate per task id).
## Approach
Single atomic array record on top of the post-decoupling apply seam:
- Define a `WAL_JOB_ARRAY_SUBMIT` record encoding base job spec + index range/count + sparse per-index overrides.
- Command handler appends + commits this one record through a single barrier; on commit, the shared `apply_committed_entry()` seam expands it into N in-memory `job_t` entries (reusing `create_job` per element) under one apply step.
- Restart replay decodes the single record and expands identically (same apply seam → no online/replay divergence).
- Scheduler push delivers one array descriptor; scheduler expands locally.
- Preserve idempotency: re-applying a committed array record (replay or retry) must not double-create elements.
## Scope
- **In**: single WAL record format for arrays; atomic single-barrier commit; standby reconstruction from one LSN; single scheduler push + local expansion; preserved per-element identity/addressing; replay expansion; idempotency on re-apply; requirement + regression tests (large-array commit count, failover replay time, per-element status/cancel still correct).
- **Out**: the apply-ordering/hook-decoupling refactor itself (sibling spec `raft-apply-decoupling`); non-array single-job path changes; scheduler policy changes.
## Boundary Candidates
- The `WAL_JOB_ARRAY_SUBMIT` record format + encode/decode.
- The single-barrier commit of the array record.
- The expand-on-apply step (online + replay, shared).
- The single scheduler push + scheduler-side expansion.
## Out of Boundary
- Per-element override semantics beyond what single-job submit already supports.
- The commit engine / barrier internals (consumed unchanged).
## Upstream / Downstream
- **Upstream**: `raft-apply-decoupling` (HARD — builds the array expand step on the clean `apply_committed_entry()` seam rather than the old optimistic-before-commit path); `ha-commit-engine` + `ha-commit-barrier-rollout` (single atomic commit + idempotency); `ha-consistency-invariants` (replicated scheduling state).
- **Downstream**: failover replay time and replication bandwidth improve for array-heavy workloads; lightweight rejoin re-attach optimization benefits.
## Existing Spec Touchpoints
- **Extends**: none (new record type).
- **Adjacent**: `raft-apply-decoupling` (shared apply seam — must land first); `ha-commit-barrier-rollout` (idempotency guards reused for re-apply safety); `scheduler-controller-msg-gap` (scheduler push path).
## Constraints
- Must preserve per-element identity/state addressing after expansion (status, cancel, completion per task id).
- Atomic commit: either the whole array commits or none of it (no partial array on a standby).
- Idempotent re-apply (replay/retry must not duplicate elements).
- Bounded record size / encoding for very large arrays (avoid a single pathological WAL entry; define a max or chunking policy in design).
## Provenance
Promoted 2026-06-24 from the roadmap backlog item `job-array-single-wal-record` (recorded 2026-06-16) via `/kiro-discovery`.