docs: mark scheduler-dispatch-commit tasks complete; add parked commit-batch-window spec
scheduler-dispatch-commit's 5 parent tasks were already fully implemented (all subtasks checked); this rolls up the parent checkboxes to match. commit-batch-window is a new spec, requirements-only, explicitly parked (not actively pursued) pending a dedicated future session. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
parent
f2ff45f18e
commit
8a2e22f48c
2 changed files with 45 additions and 0 deletions
21
.kiro/specs/commit-batch-window/requirements.md
Normal file
21
.kiro/specs/commit-batch-window/requirements.md
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
# Requirements Document
|
||||||
|
|
||||||
|
## Project Description (Input)
|
||||||
|
A general, time-windowed batch-commit facility: gather multiple WAL-worthy facts arriving within a short window and commit them together in a single majority-ack round (group commit), rather than each individual fact paying its own full commit-barrier round-trip. Any operation that needs an externally-visible side effect gated on durability (e.g. "don't tell the worker to start a job until that job's dispatch fact is majority-committed") should be able to enqueue its fact into the shared batch and be notified once the BATCH — not just its own individual entry — achieves majority commit.
|
||||||
|
|
||||||
|
This builds on infrastructure that already partially exists: the "group-commit flusher" already advances `durable_lsn` asynchronously (bug-1013, bug-760), and job-array-single-wal-record already batches multiple array elements' submission into a single WAL record. This spec generalizes that pattern into a reusable facility any fact type can use, rather than a one-off per call site.
|
||||||
|
|
||||||
|
**Motivating first use case (not the only one), and the strongest candidate specifically: batch/array job dispatch.** Batching only pays off when multiple facts genuinely cluster together in time — a single, isolated job dispatch gains essentially nothing from this facility (there is nothing else to amortize its round-trip cost against). A job ARRAY's dispatch decisions, by contrast, naturally arrive in a tight burst (the scheduler decides many elements together), which is exactly where group commit earns its cost — and exactly the workload pattern both bug-1698 and bug-1741 were reproduced against live this session (a 50-element array dispatching multiple elements to one worker within well under a second). Any design here should prioritize the array/burst case as the primary target, not attempt to make every standalone dispatch faster. dispatch_a_job_() (src/controller/job.c) currently sends xce_startjob_req to the worker BEFORE persist_submit_job_dispatch_fact() is even called — confirmed as a real, non-hypothetical gap via bug-1698's own live WAL evidence (job [12]'s dispatch was never even locally WAL-written before the primary froze). Making dispatch commit-gated would close that gap at the source (Raft's own leader-completeness guarantee then protects it), but only if batching keeps the added latency (measured ~167ms in this codebase's own bug-1487 live soak, dominated by standby fdatasync per bug-561) acceptable for high-throughput dispatch patterns like large job arrays.
|
||||||
|
|
||||||
|
**Additional scope dimension: hook/notification timing.** dispatch_a_job_() currently mutates job state and fires HOOK_DETAIL_JOB_START/notifies the scheduler+apiserver synchronously and inline at dispatch time (persist_notify_job_updated_already_persisted, guarded by the t_apply_seam_notifying exactly-once flag) specifically BECAUSE the ordinary apply-time hook path (apply_committed_entry) would otherwise see a no-op transition and silently skip notifying (bug-1679's root cause). A general batch-commit facility moves "this fact is true" to a deferred, asynchronous, batch-boundary-aligned moment for EVERY fact type that uses it, not just dispatch — so hook/notification firing timing needs to be redesigned around that general batch-commit-confirmed moment, without reintroducing bug-1679's original gap (notify never fires) or a double-notify race (the exact class scheduler-dispatch-commit's own design already had to fix once for dispatch specifically). This is the same hook/event-notification machinery this session's bug-1731/1737/1738/1739 fixes were untangling elsewhere in persist.c — any batch-commit design here should be reviewed against that same family of fixes, not designed in isolation.
|
||||||
|
|
||||||
|
Sibling specs: scheduler-dispatch-commit (implementation-complete, moved the RUN-state transition timing and added dispatch-failure retry classification in dispatch_a_job_() — a different, already-solved concern) and job-array-single-wal-record (the existing batching precedent, scoped to job submission only). worker-rejoin-resync (in progress, requirements phase) remains valuable as a reconciliation backstop regardless of whether this spec is ever pursued, since it also covers other classes of state divergence, not just the one dispatch-ordering gap this spec's motivating use case addresses.
|
||||||
|
|
||||||
|
**Scope exclusion: user-synchronous-wait operations are not batch-eligible.** Not every WAL-worthy fact should participate in this facility — only facts where nothing is synchronously, human-observably waiting on the result (e.g. job dispatch, decided internally by the scheduler with no client blocked on it; a worker's job-finish report, arriving async off the critical path of any interactive command). Operations where a user is actively blocked at a terminal or client waiting for the reply — job submission (jobsub) foremost, and by the same reasoning any other admin/control command that already commits synchronously today (config push, job kill/control) — must be explicitly excluded from batching and continue to commit as soon as possible, not deliberately held to wait for a batch window to fill. Any design for this facility needs an explicit per-fact-type eligibility decision, not a blanket "everything batches" default.
|
||||||
|
|
||||||
|
**Candidate mechanism: an explicit classifier, mirroring existing precedent.** This codebase already has exactly this shape of decision elsewhere: xce_classify_dispatch_fail() (dispatch_a_job_()'s own failure classifier, scheduler-dispatch-commit) maps an exhaustive site-identity enum to a decision (TERMINAL vs RETRYABLE) rather than scattering ad hoc logic per call site. A parallel classifier — mapping each fact type/call site to BATCHABLE vs IMMEDIATE commit eligibility — would be the idiomatically consistent way to implement the user-synchronous-wait exclusion above, and should be the starting design candidate rather than an ad hoc per-call-site flag.
|
||||||
|
|
||||||
|
**Status note (2026-08-31): parked, not actively pursued.** Assessed as a substantial cross-cutting change touching the core commit pipeline and hook/notification timing broadly, not a small project — revisit in a dedicated future session.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
<!-- Will be generated in /kiro-spec-requirements phase -->
|
||||||
24
.kiro/specs/commit-batch-window/spec.json
Normal file
24
.kiro/specs/commit-batch-window/spec.json
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
{
|
||||||
|
"feature_name": "commit-batch-window",
|
||||||
|
"created_at": "2026-08-31T14:07:43Z",
|
||||||
|
"updated_at": "2026-08-31T14:20:00Z",
|
||||||
|
"language": "en",
|
||||||
|
"phase": "initialized",
|
||||||
|
"approvals": {
|
||||||
|
"requirements": {
|
||||||
|
"generated": false,
|
||||||
|
"approved": false
|
||||||
|
},
|
||||||
|
"design": {
|
||||||
|
"generated": false,
|
||||||
|
"approved": false
|
||||||
|
},
|
||||||
|
"tasks": {
|
||||||
|
"generated": false,
|
||||||
|
"approved": false
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"ready_for_implementation": false,
|
||||||
|
"sibling_specs": ["scheduler-dispatch-commit", "job-array-single-wal-record", "worker-rejoin-resync"],
|
||||||
|
"note": "Parked/tracked, not actively being pursued. Originally scoped narrowly as job-dispatch-commit-batching; broadened (2026-08-31, same session) to a general time-windowed batch-commit facility for ANY WAL-worthy fact needing a durability-gated external side effect, not just job dispatch -- job dispatch remains the motivating first use case. User assessed this as a substantial cross-cutting change (core commit pipeline, latency/throughput tradeoffs, hook/notification timing, soak-level correctness verification), not a small project. Revisit in a dedicated future session."
|
||||||
|
}
|
||||||
Loading…
Add table
Add a link
Reference in a new issue