Extends the HA soak harness with --array-chaos-retry: submits an
array, kills the primary essentially immediately (before commit is
known), retries the identical submission against the new primary, and
hard-asserts exactly one committed instance results -- no silent loss,
no double-submit, no partial duplication across the array's full
index range.
Confirms (no production code changes needed) that the existing
single-job cross-primary dedup mechanism (bug-1419: client-side jobid
minted once and never re-sent on retry; server-side
jobid_range_exists()/inflight_range_try_add(), both genuinely
range-shaped over [base,base+array_size); xce_failover_classify()'s
idempotent-replay rule) generalizes correctly to arrays, treating the
whole range as one dedup unit rather than per-element.
No mutation-test RED phase against production code was performed
(correctly out of this task's boundary); reviewer compensated by
extracting the harness's own assertion logic into a standalone
mutation-harness with stubbed collaborators, confirming real
fail()-gated teeth on all four outcomes (clean/silent-loss/
double-submit/partial-duplication).
Live-verified across 4 independent fresh cluster deployments (2 by
implementer, 2 by reviewer), array sizes 6/10/15, 17 total chaotic
rounds, zero failures, genuine election/term diversity each time.
Task 6.2 of job-array-single-wal-record (Requirements 8.2, 8.3).
Extends the HA soak harness with a new --array-soak mode: small (N=2)
and large (N=10000) array submission sub-rounds, each with one clean
kill/failover, asserting one WAL record per submission regardless of
array size, per-element addressability (status/cancel/completion)
spread across low/mid/high indices post-failover, and a real
millisecond-resolution replay-time bound comparing large-array vs
small-array kill-to-addressable latency.
waldump.c gains a WAL_JOB_ARRAY_SUBMIT case in wal_type_name() so the
scenario can grep an exact type name for its live WAL-record-count
delta check.
Went through one remediation round: the replay-time measurement
initially used whole-second resolution (collapsed to 0s/n-a on a real
run) and never asserted on the comparison, giving zero actual evidence
for the O(1)-replay-time requirement. Fixed with $EPOCHREALTIME
(millisecond resolution -- this sandbox's BusyBox date silently
mangles the more common `date +%s%3N` idiom) and a real bound
assertion that hard-fails the run on a genuine O(N) regression,
independently verified to have real teeth.
Live-verified 3 times with genuinely discriminating numbers each time;
final run: small=210ms, large=163ms -- large-array replay was not
slower, live evidence of the feature's O(1) claim.
Task 6.1 of job-array-single-wal-record (Requirements 4.3, 5.2, 5.3,
8.2, 8.3). Depends on and reuses task 6.0a's own soak verification.
replay_wal() (persist.c ~3902) tallied *new_job_submits_out only for
hdr.type == WAL_JOB_SUBMIT. WAL_JOB_ARRAY_SUBMIT entries were
dispatched and correctly materialized N real jobmgr entries via
apply_committed_entry()'s array case, but never counted --
persist_verify_apply_correctness() then saw real jobmgr growth exceed
the undercounted expectation and declared APPLY_DIVERGENT, engaging a
PERMANENT, non-self-healing fence (distinct from task 4.2's
self-healing Live Apply Resource Fence).
Live-reproduced during task 6.1's soak attempt on 2 of 3 nodes in a
single small-array (N=2) sub-round via two different trigger paths.
Deterministic, not size-dependent -- the first array submission to a
live cluster would fence any node that restarts or hits an
election-path recovery fallback before its next checkpoint, shrinking
the promotable pool. A pre-existing comment at persist.c ~3863-3874
(predating this spec) had already flagged this exact code region as
needing re-verification before tasks 4.1/6.1+ went live; this is that
re-verification, and it failed.
Fix adds a WAL_JOB_ARRAY_SUBMIT branch tallying apply_rc (the genuine
per-record element count apply_committed_entry()'s array case already
returns) when apply_rc > 0, mirroring the existing dedup-safe pattern
for WAL_JOB_SUBMIT. Traced the full return-value space of
restore_job_array()/apply_committed_entry()'s array case to confirm
every tallied value is provably equal to real jobmgr growth -- the fix
cannot overcount and so cannot mask a genuine divergence.
Verified via a from-scratch live 3-node HA soak re-run (--array-soak),
independently by both implementer and reviewer: zero APPLY_DIVERGENT
across all three original trigger paths, one-WAL-record/one-commit-
barrier confirmed for N=2 and N=10000, per-element post-failover
status/cancel/completion confirmed.
Task 6.0a of job-array-single-wal-record (Requirements 3.1, 3.2, 4.1,
9.4). Inserted mid-run after live soak testing surfaced the bug; user
confirmed sequencing (fix now, blocking task 6.1+).
Cross-daemon integration test, test-only, spanning the scheduler (C)
and API-server (Go/CGO) receive paths built in tasks 5.1-5.4/5.3b.
Drives real xce_encode()/xce_decode() wire bytes through the actual
production dispatch chain on both daemons -- extract_data() ->
scheduler_req_handler() -> sched_handle_dataset() ->
sched_on_job_array() and go_push_handler() -> handleDataset() ->
cJobArrayToModels()/dispatchJobArray() -- skipping only vtask_entry()'s
~10 lines of static glue, mirrored verbatim in test code (the
if (callnext) treats-any-nonzero-as-proceed behavior that caused
5.3b's bug is reproduced exactly).
Redelivery is tested via genuinely independent fresh decodes of
separately-encoded bytes, not a decode-once-call-twice shortcut.
Bug-catching capability proven by temporarily reverting task 5.3b's
fix and confirming both the C and Go tests fail meaningfully, then
restoring to green -- confirmed independently by both the implementer
and reviewer.
Task 5.5 of job-array-single-wal-record (Requirements 6.1-6.5, 8.4).
This completes Phase 5 (Compact Scheduler/API-Server Push).
Adds the Go/CGO receive path for Data.jobArray, mirroring task 5.3's
scheduler-side C implementation: rawJobArrayToModels() (pure-Go
base_jobid + i expansion), cJobArrayToModels() (CGO adapter sibling to
cJobToModel), dispatchJobArray() (dedup via the same store.Jobs.Has()
the ordinary Data_data_jobs path already writes through, single
confirmation log only when at least one element was newly
dispatched), and a new case C.Data_data_jobArray in handleDataset.
Elements route through OnJobUpdated, matching the ordinary single-job
push's own routing. The CGO boundary needed no changes -- both
C.xce_job_submit_t and C.Data_data_jobArray were already exposed from
prior tasks, including task 5.3b's extract_data() fix which this
receive path depends on identically to the scheduler's.
store.ArrayJobs (the array-level rollup aggregate) is deliberately
left unpopulated -- confirmed out of this spec's scope, deferred to a
future spec per requirements.md's Boundary Context.
Task 5.4 of job-array-single-wal-record (Requirements 6.2, 6.4, 6.5, 8.4).
extract_data() (src/lib/request.c), the shared reassembly step every
inbound push passes through inside vtask_entry() before any receiver's
handler runs, had no case for Data_data_jobArray -- it hit
default: rc=-EINVAL. Because vtask_entry()'s callnext check treats any
nonzero return (including a negative errno) as "reassembly complete,
proceed," this never surfaced as an error: the real payload was freed
and the receiver silently handed an empty, zeroed dataset_t, with no
log anywhere in the chain. Tasks 5.1-5.3 were each individually
correct and tested in isolation, but the feature was dead on the real
wire end-to-end.
Adds case Data_data_jobArray: passing the single, unchunked
descriptor straight through (no pagination needed, unlike the
existing LIST-shaped cases) and nulls the source pointer after
transfer -- verified this is not cosmetic, since free_Request()'s
pushmsg case would otherwise double-free it via free_Data()'s live
Data_data_jobArray branch.
Proven with a real xce_encode()/xce_decode() round-trip through the
fixed extract_data() -> sched_handle_dataset() -> sched_on_job_array()
(task 5.3, unmodified) chain, not just a unit-level check on
extract_data() alone. Also confirms task 5.4's future API-server
receive path shares this exact fix, since both go through the same
xcecli_ctx_init() -> vtask_entry() -> extract_data() pipeline.
One self-flagged, non-blocking residual gap (pre-existing since task
5.3, out of this task's request.c-only boundary): dataset->rows for
the jobArray case is never deep-freed downstream, leaking one bounded
xce_job_submit_t per successfully-delivered array push commit.
Task 5.3b of job-array-single-wal-record (Requirements 6.1, 6.2, 6.4, 8.4).
Inserted mid-run after independent review confirmed the gap; user
confirmed sequencing (fix now, before task 5.4).
Adds sched_on_job_array() (handler.c), a sibling case to
Data_data_schedjobs decoding the new Data.jobArray variant and
expanding it locally into N ordinary schedulable tasks via
base_jobid + i, routed through the existing unmodified per-job
pipeline (sched_on_jobs()). Each element gets its own deep clone
(clone_job_submit_data/clone_job_startinfo) of the shared submit
spec, since per-element teardown (free_Job_ptr) frees it
independently and a shared pointer would double-free.
Dedup reuses sched_get_job() against the same kvtab sched_add_job()
guards, rather than a new mechanism. A confirmation log line fires
once per successful expansion (skipped on a fully-duplicate
redelivery).
Adds a new task (5.3b) discovered during this task's review: the
shared wire dispatcher extract_data() (src/lib/request.c) has no case
for Data_data_jobArray and silently drops every real array push
before any receiver's handler runs -- outside every current task's
boundary, confirmed to affect the future API-server receive path
(task 5.4) identically. design.md's File Structure Plan corrected to
record this. User confirmed sequencing: fix it now, before 5.4.
Task 5.3 of job-array-single-wal-record (Requirements 6.3, 6.4, 6.5, 8.4).
Adds apiserver_push_jobarray() (apiserver.c), mirroring task 5.1's
xcesched_push_jobarray() exactly: same (base jobid, array_size, one
spec) Data.jobArray derivation, same direct xce_request()/OPCode_push
transport. apiserver_job_new_fun() branches on l->count > 1; the
existing single-job path (apiserver_push_joblist()) is byte-for-byte
unchanged.
Additionally, and unlike the scheduler (single target), replicates
apiserver_push_joblist()'s existing per-connected-server fan-out loop
(N_APISERVER_MAX iteration, same lock discipline) -- verified against
2 simulated connected servers, each receiving exactly one push.
New standalone apiserver_push_unit_test binary (same link-collision
rationale as task 5.1's sched_push_unit_test), correctly omitting
egress.c since apiserver_push_joblist() has no WAL-commit filter.
Task 5.2 of job-array-single-wal-record (Requirements 6.2, 6.4).
Adds xcesched_push_jobarray() (sched.c), encoding one compact
(base jobid, array_size, one spec) Data.jobArray descriptor -- reusing
the JobSubmit capnp struct already added to the Data union in task 1.2
-- instead of N individually-encoded job structs. sched_job_new_fun()
branches on l->count > 1 to call it; the existing single-job path
(l->count == 1 -> xcesched_push_joblist()) is byte-for-byte unchanged.
Sends via direct xce_request()/OPCode_push/Request_body_pushmsg rather
than the existing xce_push_data_req() helper, which hard-rejects any
payload outside jobs/workers/resources/resvals and cannot carry a
single non-list descriptor. design.md corrected to record this
transport deviation for task 5.2 to follow.
New standalone sched_push_unit_test binary (registered in
meson.build): sched.c/dataex.c/egress.c were never linked into
controller_unit_test, and two existing test files there ship local
fakes of on_scheduler_join()/notify()/collect_confvars() that would
collide with the real definitions at link time.
Task 5.1 of job-array-single-wal-record (Requirements 6.1, 6.3, 6.4).
New test_job_array_fence.c exercises the full Requirement 9.4-9.6
engage-refuse-clear-recover sequence through real production seams:
forced array-expansion failure -> fence engages -> promotion refused
at a real ha.c gate site -> zero partial elements visible -> a second,
independent node's progress is unaffected -> clearing the failure and
retrying self-heals the fence and re-opens promotion eligibility.
Deliberately drives this through task 3.1's construction-phase
(scratch-build) seam rather than the bulk-insert-phase seam: the
bulk-insert seam has a known, separately-tested residual gap (an
already-inserted element prefix can survive a mid-pass key_add()
failure with no key_del to undo it), so asserting "no partial set"
against it would misrepresent an already-documented open gap as
resolved. That gap remains open and unowned (see tasks.md
Implementation Notes) -- Phase 4 delivers the generic halt/fence/gate
machinery for it, not the partial-visibility guarantee itself.
Task 4.4 of job-array-single-wal-record (Requirements 9.4, 9.5, 9.6).
This completes Phase 4 (Live Apply Resource Fence).
Threads !persist_apply_resource_fenced(&c->persist) as a required
conjunct into both existing promotion-eligibility call sites in ha.c:
the periodic fence-release check (promotion_fence_timeout_fn) and the
election-time check (raft_cb_become_leader). A node with the fence
engaged is refused promotion at both points; once it self-clears
(task 4.2), the node is eligible again with no other action.
Task 4.3 of job-array-single-wal-record (Requirements 9.4, 9.5).
Adds a new atomic LSN field on persist_t (distinct from the existing
boot-only unrecoverable-fence flag) plus three primitives:
persist_apply_resource_fenced(), persist_mark_apply_resource_fenced(),
persist_clear_apply_resource_fence(). Wired into
persist_apply_pump_run(): engages on task 4.1's halt condition, clears
unconditionally on the walk's next successful advance — provably
equivalent to "clears only when the same stuck LSN's retry succeeds"
since the walk is strictly sequential and never advances past a
failed position.
Known residual gap, out of this task's boundary (documented in
tasks.md Implementation Notes): replay_wal() during boot/restart
replay does not yet share this protection, so a node crashing while
fenced can silently re-pass the same failure on restart. Flagged for
a follow-up task, not fixed here.
Task 4.2 of job-array-single-wal-record (Requirements 9.4, 9.5).
persist_apply_pump_run() no longer discards apply_committed_entry()'s
return value. On a negative rc for any WAL type (not just the new
array type), the walk halts at that LSN — no last_applied_lsn advance,
no pending-apply hand-off — logs a loud error, and retries from the
same position on the next kick, mirroring the existing
apply_pump_read_entry failure branch's shape.
Investigated whether this newly turns the pre-existing job-fact
-ENOENT path into a permanent-wedge hazard: it does not, since
c->jobmgr never deletes entries and facts are only ever applied after
their causally-prior submit already succeeded, so a live -ENOENT there
can only mean genuine divergence — halting is the correct behavior.
Task 4.1 of job-array-single-wal-record (Requirement 9.4).
Adds 2 targeted tests after confirming tasks 3.1/3.2/3.3 already
cover most of this task's observable criteria:
- N-way expansion swept across sizes {2,3,100,1000} with exact-
endpoint non-leakage checks (existing coverage only tested size 5)
- Requirement 5.2 (individually addressable exactly as a per-element
job) exercised through the real production command dispatch
(job_ctrl(), not the raw key_find() every existing test used) --
confirms an array-expanded element can be cancelled by identity,
siblings stay untouched, and an out-of-range jobid is rejected
Idempotent no-op, forced-failure-then-retry, and partial-reapply
were already fully covered by 3.1/3.2 -- no new tests needed.
Requirement 9.4's node-fencing behavior is confirmed out of this
task's boundary (Live Apply Resource Fence, task 4.x); restore_job_
array()'s own atomicity contract that the fence depends on is
already proven. No production code changed -- test-only.
Task 3.4 of job-array-single-wal-record.
Adds case WAL_JOB_ARRAY_SUBMIT to apply_wal_entry() (standby
catch-up dispatch), calling restore_job_array() hook-free, matching
the existing WAL_JOB_SUBMIT case's shape exactly.
Review found apply_wal_entry() has zero production call sites -- a
2026-07-10 refactor (ha_standby_advance_committed_and_apply) moved
standby entity materialization onto the same apply_committed_entry()
pump the primary uses; the code comment claiming otherwise predated
that refactor and was never updated. Requirement 3.1/3.2/4.1 are
fully delivered by task 3.2 alone. This task's wiring is kept as
defensive parity with WAL_JOB_SUBMIT's existing, equally-dead case
in the same switch, not as an independent delivery path -- design.md
and tasks.md corrected accordingly.
Task 3.3 of job-array-single-wal-record.
Adds case WAL_JOB_ARRAY_SUBMIT to apply_committed_entry() (the live
apply pump / boot replay dispatch seam): calls restore_job_array()
(task 3.1) instead of restore_job(), and on success fires
HOOK_DETAIL_JOB_NEW exactly once carrying every newly created
element in restore_job_array()'s own out_new_jobs list -- never once
per element. No hook fires on an idempotent no-op (-EEXIST) or any
genuine construction failure, honestly propagating task 3.1's
documented bulk-insert-phase residual gap rather than papering over
it. apply_wal_entry() (standby catch-up) is untouched -- task 3.3.
Task 3.2 of job-array-single-wal-record.
Adds the shared N-way array expansion primitive: decodes a committed
WAL_JOB_ARRAY_SUBMIT payload, scratch-builds every element via a
throwaway kv-table (never touching c->jobmgr during construction),
then makes the whole set visible in one bulk pass under a single
write-lock acquisition -- no window where a concurrent reader can
observe a partially-constructed array. Idempotent per index (skips
already-existing jobids; whole-record no-op if all exist). Includes
an XCE_TESTING-guarded fault-injection seam. Not yet wired into any
live dispatch path (tasks 3.2/3.3).
Two review rounds: round 1 found the bulk-insert pass silently
claimed success even when key_add() itself could fail on allocation
exhaustion (c->jobmgr grows via realloc, no reservation primitive
exists anywhere in kv.c/kvmgr.c/zarray.c to pre-check this). Fixed
by checking key_add()'s return and reporting failure honestly
instead of a false success. The underlying table has no key_del, so
an already-inserted prefix on this specific failure mode can remain
genuinely visible while the call reports failure -- documented as a
residual gap in design.md and tasks.md for tasks 3.2/4.x to account
for (that prefix is never pushed to scheduler/API server, even on a
later successful retry, since it's never included in out_new_jobs).
Task 3.1 of job-array-single-wal-record.
Adds 3 targeted tests after confirming tasks 2.1/2.2 already cover
most of this task's observable criteria:
- one-entry/one-continuation swept across sizes {2,3,100,1000}
(existing coverage only tested size 5)
- payload-size boundedness at the real idgen()-enforced ceiling
(MAX_ARRAY_IDX+1 = 32768), plus an absolute byte-cap assertion
(existing coverage only compared 2 vs 10000)
- array_size==1 baseline: explicit WAL-byte readback proving the
entry type is WAL_JOB_SUBMIT (not WAL_JOB_ARRAY_SUBMIT) and the
payload decodes via struct Job, not struct JobSubmit (existing
coverage only proved "old tests still pass" implicitly)
No production code changed -- test-only.
Task 2.3 of job-array-single-wal-record.
Branches handle_jobsubmit() on array_size: ==1 continues through
today's exact unmodified single-job path; >1 routes through a new
handle_jobsubmit_array() helper that reuses jobsubmit_async_ctx_t
with pending=1 and calls task 2.1's persist_submit_job_array_async()
instead of looping N single-element appends. Range validation
(jobid_range_exists/inflight_range_try_add) unchanged, same order,
strictly before any WAL append, for both branches.
Retires two pre-existing tests whose premise (N independent
per-element WAL entries with mixed outcomes) is now structurally
impossible with one entry per array; restores array-scoped
job-non-existence coverage for the failed-commit case that an
earlier draft of this change had dropped (Req 2.2/9.2).
Task 2.2 of job-array-single-wal-record.
Adds persist_submit_job_array_async(): encodes one JobSubmit-shaped
payload (base jobid, array size, one shared spec), appends exactly
one WAL_JOB_ARRAY_SUBMIT entry, and registers exactly one commit
continuation regardless of array size -- reuses the existing
job_commit_adapter_ctx_t/job_commit_resolve_adapter machinery, no
per-element loop. Not yet wired into handle_jobsubmit (task 2.2).
Also fixes a test-helper bug (WAL offset back-computed from the
post-append cursor, only valid for zero-payload entries) that
corrupted state across tests when exercising a nonzero-payload
array-submit entry.
Task 2.1 of job-array-single-wal-record.
- job_init() now checks makeJobsub()/makeString(ckptinfo) return
values and propagates failure instead of storing NULL with a
success return (Req 10.1).
- Root-caused the previously observed makeJobsub() NULL-with-EINVAL
failure to a test-binary-only hazard: multiple controller test
files each called job_obj_init() from their own setUp() with no
cross-file coordination, re-running jobsub_table_init() against an
already-live lock/table. Fixed via atomic_flag-guarded idempotent
job_obj_init()/job_obj_fini(), plus a distinct -ENODEV return from
makeJobsub()/makeJobstartinfo() when the table isn't ready
(previously fell through to undefined behavior) (Req 10.2).
- Verified (test) MOVE_PTR fields are always NULL on a freshly-built
submission payload -- no code change needed (Req 10.3).
- Fixed job_destroy()'s accounts leak via the existing
free_job_account() helper, replacing the stale FIXME (Req 10.4).
- Verified (inspection + test) struct Job's capnp schema has no
ckptinfo field, so decode_Job/free_Job have nothing to leak there
-- no code change needed (Req 10.5).
Independently re-verified: reviewer reproduced a pre-existing,
unrelated rtti-library leak-on-init-failure against a stashed
pre-task copy of jobj.c to confirm it predates this change.
Task 1.3 of job-array-single-wal-record.
Adds Data union variant jobArray@6, reusing JobSubmit's existing
(jobid, arraySize, submit) shape verbatim -- no new struct, no
existing variant renumbered. Not yet wired into any push path
(tasks 5.1/5.2 own that).
Task 1.2 of job-array-single-wal-record.
Adds the array-shaped WAL entry type (distinct from WAL_JOB_SUBMIT,
value 11 in the unused WAL_JOB_* gap) and classifies it WAIT-mode via
wal_classify, matching the single-job type's commit-barrier semantics.
No dispatch/apply wiring yet -- foundation for task 2.x/3.x.
Task 1.1 of job-array-single-wal-record.
The umbrella's implementation_status note still described the Facts-Apply
invariant as an open documented deviation owned by raft-apply-decoupling's
D1. raft-apply-decoupling has since closed it: task 4 ("Fact apply-after-
commit split") moved WAL_MODE_RETAIN fact state mutation + reactions to
the post-commit apply_committed_entry() seam (persist.c:2721) via
apply_job_update_transition(); the old apply-on-receipt path
(apply_wal_entry()) has zero production call sites. D1 (block-but-durable
under quorum loss) is implemented and soak-tested (task 7.2: kill primary
mid-fact-stream). Update the note so it doesn't read as outstanding work.
Redesign the plugin picker hierarchy to project→queue→job+preempt
(was 3-level). Adds LSF-style cross-queue preemption (SIGSTP) and a
unified v1/v2 dispatch loop. Requirements grow 6→9 to cover the new
preemption and cross-queue levels.
Replace the deliberate election-carrier leak with an atomic refcount:
a bias reference held by ha->elect_ctx plus one reference per captured
vote endpoint, released via a new endpoint_t.on_fini teardown-completion
hook (fired once, before ep->data is freed). Both real-election and
PreVote capture sites (raft_cb_send_vote_request,
raft_cb_send_pre_vote_request) increment on capture; both publish sites
(ha_start_election_async, ha_start_prevote_async) now atomic_exchange
the outgoing carrier and release it instead of leaking or freeing
outright, so a still-outstanding vote endpoint keeps the superseded
carrier alive until its own teardown completes.
Verified: 758+ controller tests green, isolated LeakSanitizer run shows
zero ha_elect_ctx_t leaks across many election/PreVote rounds, no
premature-free/UAF under supersession, and a live 10-round HA soak
(36 jobs, zero loss, all 3 nodes served as primary).
Election carrier (ha_elect_ctx_t) is never freed today -- each election/PreVote
round deliberately leaks the previous one because per-peer vote endpoints hold a
bare pointer into it and tear down asynchronously, so a naive free risks UAF.
Requirements scope this to a memory-lifetime-only fix: free once no vote
endpoint can reference it, never free while one might, uniformly across the
real-election and PreVote paths. No election/failover semantic change.
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).
bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.
Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
Closes the two non-blocking follow-ups from the 2026-07-07 validate-impl
audit:
- design.md had 3 sections describing the original plan rather than what
landed, each already self-disclosed in tasks.md's Implementation Notes
but never reconciled back: repl.c's planned repl_submit_majority_commit()
was never built (task 3.2 calls commit_submit_async() directly instead);
the apply pump is wired via commit_resolve_pending() in commit.c, not
ha.c's on_commit_advanced; fact hooks fire one generic
HOOK_DETAIL_JOB_UPDATE, not per-type hooks (verified safe -- sched.c's
subscribers already consume the generic form).
- task 5.3 touched ha.c (two promotion/fence call sites) without a formal
_Boundary:_ exception the way task 1.4 got one for its own out-of-file
touch -- added retroactively, no behavior change.
controller-status, ha-commit-engine, ha-heartbeat-committed-propagation,
ha-prevote, ha-single-leader-role, ha-wal-replication, libxce-failover, and
raft-ha-core-safety each had every numbered sub-task marked [x] but their
group-header lines (e.g. "1. Foundation: ...") left unchecked -- a cosmetic
gap, confirmed by roadmap.md's own IMPLEMENTED/VERIFIED notes and zero real
unchecked sub-tasks in any of the eight files. No task content changed.
raft-apply-decoupling: 32/32 tasks complete, validate-impl GO (79/79 test
groups, live smoke boot verified) -- update roadmap's In Flight entry with
outcome, residuals, and follow-ups.
raft-drive-async: all 17 sub-tasks were already [x] but the 4 phase-header
checkboxes were never ticked, leaving tasks.md looking incomplete. Confirmed
via source (g_drive_async_configpush_offload defaults to 1) and prior
session notes that all 4 phases genuinely landed and flipped ON in
production; checkboxes now match reality.
Measured sustained worker-fact throughput against the pre-feature baseline.
Commit throughput is unregressed (~2.5% delta, within noise) and no per-fact
synchronous commit barrier exists, confirming Req 11.4's core claim. But
apply/observable throughput -- the new hop this spec introduces -- regressed
56% with clearly superlinear scaling (2x volume -> 5.2x elapsed).
Root cause: apply_pump_read_entry has no memory across calls -- every LSN
lookup re-lists WAL segments and rescans its segment from byte 0 -- while its
only caller applies committed entries one at a time in strict ascending
order, turning a sequential access pattern into O(n^2) total work.
Fixed with a minimal segment-read cache on persist_t (fd + offset + expected
next LSN, guarded by the existing apply_pump_lock): a fast path reads the
next expected LSN directly at a remembered offset, falling back unchanged to
full rediscovery on any miss. Correctness never depends on the cache, only
performance does. Invalidated on init, fini, recovery, and WAL truncation --
the four points a stale cursor could otherwise misread from.
Two review rounds: round 1 caught a missing lock guard in the recovery
invalidation path and an inaccurate risk claim about the cache/trim
interaction (a real, bounded behavior change deferred to the existing
APPLY_DIVERGENT fence, not "no worse than before") -- both corrected and
re-verified in round 2. Post-fix apply-throughput scales sublinearly (1.46x
elapsed for 2x volume) and exceeds the pre-feature baseline. Zero new test
failures across the pump, correctness-verification, recovery, and truncation
groups.
An intermittent, pre-existing client hang in the synchronous jobsub path and
a trim-floor/apply-lag architectural gap were surfaced but are explicitly
out of scope for this fix -- flagged for separate follow-up.
Full 79-group unit/integration suite re-run found exactly 2 stale tests in
test_commit_barrier_reply.c asserting task 3.2's documented interim
contract ("no observable effect until task 6 completes it"), which the
already-committed ordered apply pump (tasks 6.1/6.2) closed by construction.
Updated both to assert the real, deterministic apply-after-commit settled
state instead of the no-longer-true no-effect-yet state, matching the idiom
already used elsewhere in the same file. No production code changed.
Full suite: 0 failures across 79 groups. HA soak (--sequential, 5 rounds):
zero committed-job loss, zero divergence, all 3 nodes served as primary.
Both independently re-verified by adversarial review.
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.
Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.
Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
Extend raft_sim.c's deterministic TLA+-conformance harness with two
fault-injectable checks: no reaction before commit, and idempotent
re-apply of an already-committed entry. Wired as dedicated tests only,
not into raft_sim_check_all's fixed four-invariant contract (owned by
a separate spec). RED/GREEN independently verified by adversarial
review; zero regression to existing scenario functions.
Extend test_apply_pump_integration.c with 2 tests, no production
changes. Test 3 interleaves commands and facts through a real
quorum-loss outage and proves strict original-log-order catch-up via
a sequence spy. Test 4 wires a real 4-vcpu vtaskmgr_t to drive 4
commands + 2 facts through one covering ACK, creating genuine N-way
concurrent races across task 6.2's self-healing fix rather than one
isolated forced pair -- stress-testing bug-1415's fix under a
realistic catch-up burst.
Closes out task-group 6 (ordered apply pump integration): the pump
is now live, hands off command replies correctly, and is proven
correct for isolated and mixed/bursty command+fact scenarios under
both normal and quorum-loss conditions.
Add test_apply_pump_integration.c (2 tests), no production changes.
Exercises the full live pump (real commit engine, real apply
dispatch, real hook subscribers) for both a job submission and a
fact: proves reply-after-visibility ordering for commands, and zero
fact-reaction side effects during a delayed-commit/quorum-loss
window (polled repeatedly across real time, not checked once).
Confirms Req 4.3 by construction, not assertion: the apply walk only
ever invokes apply_committed_entry() for LSNs already <= committed_lsn,
so a consumer acting on the post-apply notification never needs to
separately re-check a commit watermark.
Independently verifies apply_committed_entry()'s generic
HOOK_DETAIL_JOB_UPDATE routing (unchanged by task 6.1's _pad fix)
makes the type-specific sched.c/apiserver.c hook handlers
structurally unreachable for live-applied facts -- confirmed their
va_arg-typed signatures would type-confuse-crash if ever called with
the job_t* apply_committed_entry() actually passes, making the
single generic reaction spy a sound design, not a coverage gap.
Repo-wide audit confirms exactly 2 live call sites ever fire these
hook details (the guarded apply seam, and job_ctrl's legitimate
SIGCONT/SIGSTOP path) -- no orphaned third site.
Wire persist_apply_pump_run() to call persist_take_pending_apply()
immediately after each entry's apply_committed_entry() call,
invoking the registered continuation strictly after the mutation.
Adds jobsubmit_reply_success_once() (job.c) so a successful job
submission finally sends a client reply once every array element
applies -- closes the standing gap from task 3.2 where committed
submissions never got a response.
Fixes bug-1415: commit_resolve_pending() dispatches the commit
continuation and the apply-pump kick as two independent
VTASK_LIVE_BLOCK vtasks with no completion-order guarantee. On any
real multi-vcpu pool (the production default, n_vcpus=8), if the
apply pump won the race and advanced last_applied_lsn past an LSN
before its continuation registered, the registration was permanently
orphaned -- the job would be constructed and visible, but the client
would hang forever. Untested by every existing fixture in this spec
(all use pool=NULL, which trivially serializes and can't reproduce
the race).
Fixed with a self-healing check in persist_register_pending_apply():
if the LSN is already <= last_applied_lsn, invoke the callback
immediately instead of registering for later pickup. Verified by the
spec's first genuine multi-vcpu regression test (real vtaskmgr_t,
n_vcpus=4), independently RED->GREEN reproduced by two review rounds.
Build persist_apply_pump_run()/persist_apply_pump_kick() -- the
ordered walk from last_applied_lsn+1 through committed_lsn, applying
every entry via apply_committed_entry() regardless of commit mode.
This is the first live (non-replay) consumer of that dispatch
function. Kick is wired into commit_resolve_pending() (the single
choke point all 6 internal committed_lsn-advancing paths funnel
through, discovered in task 1.2), dispatched via
vtask_enq_classified(VTASK_LIVE_BLOCK) with a CAS-coalesced
fire-and-forget trigger.
Fixes the two hazards flagged since task 4.1 as required before
going live:
1. Double-append: a thread-local call-provenance guard
(t_apply_seam_notifying) prevents persist_job_hook_fun/
persist_queue_hook_fun/persist_worker_hook_fun from re-appending
entries the apply pump itself just applied, while preserving
job_ctrl's legitimate live command-path appends (a naive
detail-based filter would have broken SIGCONT/SIGSTOP). Also
closes an undocumented instance of the same hazard in the
queue/worker hook functions.
2. Merge-staleness: record_job_fail_fact/postdone's fact recording
previously baked a receipt-time-computed final job state into the
WAL entry, which goes stale under deferred apply -- a FINISH
followed shortly by an ordinary post-exec FAIL could silently
clobber a correct DONE completion with a stale RUN|EXIT|PERR, no
guard catching it. Fixed by repurposing wal_jobstate_t's unused
_pad field to carry just enough data (FAIL's stage) to recompute
against the job's actual current state at apply time via the real
apply_job_fail_transition()/new apply_job_postdone_transition()
guard cores, instead of overwriting with a stale absolute value.
Backward compatible: _pad==0 (all prior data) routes to the same
legacy overwrite behavior.
Also fixes bug-1409 (unbounded busy-loop in apply_pump_drain when
committed_lsn advances without a corresponding on-disk entry) and a
genuine test-flakiness hazard (a second, independent apply-pump
kicker from the fsync_worker thread that existing tests didn't
account for).
Two pre-existing tests in test_commit_barrier_reply.c now fail as
expected -- their own assertions state "no job may be constructed
until task 6's apply pump exists," now correctly invalidated by this
task landing; updating them is task 7.3's explicit job.
Add explicit test coverage for the DIVERGENT fence's stay-alive and
restart-only-clear guarantees (task 5.3), which surfaced a critical,
pre-existing safety gap: the sticky recovery_unrecoverable flag is
trigger-agnostic, and two pre-existing live call sites in ha.c
(promotion_fence_timeout_fn's periodic timer, and
raft_cb_become_leader's promotion-eligibility gate) clear it based
purely on structural conditions, never checking the apply verdict.
A node whose local apply dispatch diverges from the committed log,
but whose WAL bytes are otherwise structurally healthy, would have
its fence silently released by the live timer, then be eligible for
promotion to primary on a later election -- serving genuinely
divergent state.
Fix: gate both clear-call-sites on persist_apply_verdict(&c->persist)
!= APPLY_DIVERGENT in addition to their existing structural checks.
An apply-divergence-triggered fence can now only clear via a fresh
process restart's clean persist_recover() replay.
New tests: test_promotion_fence_apply_divergent_survives_structural_catchup
(test_commit_durability.c) covers the timer site; and
test_promotion_refused_when_apply_divergent_even_if_eligible
(test_convergence_verification.c) covers the promotion-gate site,
structurally isolated so it cannot pass for the wrong reason.
Closes out task-group 5 (restart apply-correctness verification).
Add bool apply_divergent to the existing recover_policy_inputs_t;
persist_recover_decision() unconditionally short-circuits to the
existing RECOVER_DECISION_FENCE_UNRECOVERABLE when true, reusing the
same sticky recovery_unrecoverable flag and
persist_recover_loud_fence_if_unrecoverable() -- no new fence
mechanism, no ha.c changes needed (the existing promotion gate
already reads the flag independently), no new self-termination path.
Fixes a real bug found during review: the fence's operator-facing
ERROR() log unconditionally asserted a "no healthy peer reachable...
durable_floor_lsn < hwm" clause that was only truthful for the
pre-existing structural-fence trigger. For the new apply_divergent
trigger this produced literally false claims in test output
("100 < 100", "2 < 0"). Fixed by branching the log call itself on
apply_divergent so it states only the verified reason.
Add apply_verdict_t (APPLY_CLEAN/APPLY_DIVERGENT) computed alongside
the existing structural (CRC) recover_verdict_t. Dual-check design:
an entity-count check (WAL_JOB_SUBMIT entries scanned vs live jobmgr
growth, genuinely independent of apply_committed_entry()'s return
code) and a position check against a new last_applied_lsn field
(forward-looking infrastructure for task 6's live pump).
Fixes a real bug found during review: the position check's
expectation was computed from the physical WAL-scan tip
(current_lsn - 1), which advances past checkpoint markers and
unknown entry types that never reach apply_committed_entry(). A
crash between persist_checkpoint()'s marker-fsync step and its
later snapshot-persist step leaves a trailing checkpoint marker on
next boot, which falsely reported APPLY_DIVERGENT on a perfectly
clean recovery -- reachable today, not gated behind a future task.
Fixed by reading last_applied_lsn directly, which only advances for
entries actually dispatched through the apply seam.
Known accepted gap: entity-count check covers jobs only, not
workers (which can genuinely be created via WAL_WORKER_CTRL) --
documented for future extension.
Add test_fact_state_unchanged_throughout_quorum_loss_outage to
test_commit_barrier_reply.c (30->31), no production changes. Drives
a real repl_remove_standby quorum-loss window through all 4
fact-receipt handlers, polls live state and committed_lsn at 5
time-spread points (not once), asserts a real hook-spy never sees
AFTER_EVENT fire, and confirms durable WAL presence via direct
on-disk read.
Documents an architectural asymmetry for task 6 planning: facts have
no commit-timeout concept (COMMIT_MODE_RETAIN registers no
continuation, "never honors the timeout") unlike commands'
commit_submit_async -- the apply pump must discover newly-committed
retained facts purely via committed_lsn advancement, not any
per-fact signal.
Closes out task-group 4 (fact apply-after-commit split).
Convert on_job_start/on_job_finish/on_job_fail/on_job_postdone to
validate and durably append only -- BEFORE_EVENT validation hook
unchanged, AFTER_EVENT (post-mutation) hook no longer fires at
receipt, live state mutation removed. Adds persist_submit_job_fact()
(persist.c) since the durable append previously lived inside the
very AFTER_EVENT hook this task silences; wire format is unchanged
and identical to what apply_committed_entry() already routes.
mark_job_start/mark_job_finish (single caller each) renamed in place
to record_job_start_fact/record_job_finish_fact. mark_job_fail/
mark_job_update (shared with job_ctrl's client-command path, out of
boundary) left untouched; new durable-append-only siblings
record_job_fail_fact/record_job_update_fact added instead.
Accepts a narrow, permanent regression: on_job_postdone's finish_time
anomaly backstop has no wire-format equivalent and is dropped
(affects only a rare no-prior-exit-status path).
Two critical hazards discovered by review, documented in tasks.md,
MUST be resolved before task 6 wires apply_committed_entry() into a
live path: (1) persist_job_hook_fun still fires on HOOK_DETAIL_JOB_UPDATE
and will double-append every live-applied fact; (2) record_job_fail_fact/
record_job_update_fact precompute a final state that can be silently
clobbered by a concurrent intervening fact once apply is deferred by
the commit round-trip.
Add 4 tests to test_commit_barrier_reply.c (25->29), no production
changes: explicit-FAILED via truncation (distinct from timeout),
repeated failure loops (5 cycles) for both timeout and truncation,
and an array_size>1 mixed-outcome test proving a committed sibling's
apply hand-off isn't blocked by failing siblings while producing
zero phantom jobs for the failures and exactly one batch-level
failure reply.
Closes out task-group 3 (command apply-after-commit ordering).
Replace handle_jobsubmit()'s blocking commit call (task 3.1) with a
real async registration against commit_submit_async(). On
FAILED/timeout, replies failure immediately with no job constructed.
On COMMITTED, hands off to a new pending-apply registration
(persist_register_pending_apply) for the ordered apply pump (task
6.1, not yet built) to consume -- does not apply or reply directly,
per design.
Deviates from design.md's stated integration site: appends via the
same ha_primary_log_append/persist_wal_async primitive the RETAIN-fact
path already uses and calls commit_submit_async directly on
p->repl.ce, bypassing repl_wait_majority_ack/repl_submit_majority_commit
in repl.c. Verified safe -- majority-ack detection is driven entirely
by (ce, lsn) events inside the commit engine, agnostic to which
wrapper registered the continuation -- but narrows the reusable
primitive design.md intended; documented in tasks.md.
Deletes (not preserves-dead) task 3.1's bug-1408 notification fix --
apply_committed_entry()'s WAL_JOB_SUBMIT case (task 2.2) already
fires the notification per-entry, immune to the batch-suppression
class bug-1408 fixed.
IMPORTANT: no live path applies a successfully-committed job
submission until task 6 lands (only restart replay does) -- every
successful submission commits durably but gets zero reply until
then. This commit range must not be soak-tested or deployed against
live traffic before task 6 completes.
Reorder handle_jobsubmit() so job objects are constructed via
create_job() only after the WAL commit succeeds, not before. Adds
jobid_range_exists() for a fast-fail pre-check with no WAL work, and
persist_submit_job_payloads_and_commit() (persist.c) operating on raw
xce_job_t* payloads instead of pre-existing job_t*. Commit remains
synchronous -- converting to the continuation-based commit API is
task 3.2.
Fixes bug-1408 (found during review): the post-commit array-
construction loop suppressed notification for an entire array batch
if any one sibling element failed to construct, stranding other
durably-committed and successfully-constructed jobs invisible until
the next restart's replay. Notification now fires unconditionally
for every successfully-constructed element, independent of sibling
failures; the client reply still separately reports the batch's
construction status.
