/xce-audit-invariants controller found two live violations of the async
I/O thread-role contract: ha_demotion_convergence_fn (ha_convergence.c)
ran WAL truncation + tlog_exit()+execv() directly on the delaymgr thread,
and ha_catchup_continuation_fn (config_repl.c) ran unbounded WAL-streaming
to a standby directly on delaymgr -- the same single thread that drives
heartbeats and election timeouts cluster-wide, so either blocks liveness
for the whole cluster and risks a spurious election.
A prior vtask-pool offload attempt (2026-07-10, bug-1429) deadlocked
controller_unit_test because c->ha.pool is never initialized by the
lightweight unit-test fixtures these functions run under, and was fully
reverted. This fix instead uses the codebase's established slow-lane
pattern (the same mechanism already used for snapshot-send, config-push,
and peer connect) -- a 4th and 5th instance, not a new mechanism.
New g_drive_async_demotion_offload / g_drive_async_catchup_offload flags
ship default OFF, matching the original rollout discipline of the three
reference offloads (shipped OFF, flipped ON later after a dedicated
compatibility audit). Flipping ON was tried during validation and
reproduced two distinct hazards against pre-existing fixtures that never
call slowlane_init: the already-known hang, and a new ASan SEGV in
commit_advance_locked (logged as bug-1452, not root-caused, deferred to
the future flip-to-ON audit). Both flags verified OFF and the full
976-test suite clean, then the committed state re-verified clean against
a live 10-round chaotic and 10-round sequential HA soak (zero job loss,
zero divergence, both PASS).
Raft-drive-async's reconnect-offload burst (Task 2.1/2.2) exposed a
latent race in io_worker's epoll dispatch loop: ha.c's
on_standby_connected() evicts a stale duplicate peer channel via
iomgr_unmonitor()+endpoint_unref(), but epoll_ctl(DEL) only stops
FUTURE epoll_wait() calls -- it does not invalidate an event for that
same endpoint already sitting in the CURRENT batch's events[] array.
A later event in the same batch then dereferences the freed
endpoint_t/SSL*, producing the ASAN SEGV observed deep inside
OpenSSL's ssl3_read_bytes during raft-drive-async's round-9 soak.
The original "concurrent SSL_do_handshake from two io_worker threads"
hypothesis was wrong and is retracted -- iomgr_start() creates exactly
one io_worker thread per iomgr_t.
Fix: extract io_worker_dispatch_batch(), which takes endpoint_ref() on
every endpoint in the batch before dispatching any of them and
endpoint_unref()s each only after the whole batch completes -- the
same lock-free ref/unref bracket already used for bug-1298 and
INVOKE_MSG_HANDLER, just never applied to io_worker's own loop. Pure
atomics, no locks, no blocking added to the hot path.
Adds test_netio_batch_dispatch_uaf.c, which drives the real dispatch
loop with a synthetic epoll_event[] batch (TDD RED confirmed via ASAN
heap-use-after-free before the fix, GREEN after). Verified further
with two independent clean 10-round live sequential HA soaks (33 and
36 jobs, zero loss) and full ASAN suite runs (xcelib_test 294/0,
controller_unit_test 972/0). raft-drive-async's kiro-validate-impl
re-run flips NO-GO to GO.
Only 21 of 86 test_*.c group functions called their own nested
UNITY_BEGIN()/UNITY_END(). Each nested call reset Unity's global
NumberOfTests/TestFailures counters, so controller_test.c's final
`return UNITY_END();` -- the value meson test actually gates on --
only ever reflected whichever small window of tests ran since the
LAST nested reset, silently discarding pass/fail evidence for
everything before it (measured: 16 of 972 total tests in one run).
The same nested-BEGIN calls also caused PASS/FAIL banners to show
the wrong source file for tests in files that don't reset it
themselves (Unity.TestFile just carries over from whichever group
last called UNITY_BEGIN()).
Fix: replace every nested UNITY_BEGIN() with UnitySetTestFile(__FILE__)
(sets file attribution only, no counter reset) and drop the matching
UNITY_END(); add the same UnitySetTestFile(__FILE__) as the first
statement to the other group functions that had neither. The 2
genuinely standalone binaries (test_job_array_sched_push.c,
test_job_array_apiserver_push.c -- each has its own int main(), not
registered via controller_test.c's TEST() macro) are untouched.
Verified: single consolidated summary now reads
"972 Tests 0 Failures 0 Ignored OK", exactly matching the
ground-truth RUN_TEST call-site count across all 83 registered
group files (976 total minus 4 belonging to the 2 standalone
binaries). Every PASS/FAIL banner now names its own true source
file. Full xce-build.sh all regression pass clean (Ok: 9, Fail: 0).
No production code touched -- test-harness plumbing only.
raft_cb_become_leader()'s eligibility gate read
persist_apply_resource_fenced() before this process's own first
cluster_init_phase2()/persist_recover()/replay_wal() call ever ran --
so a node whose first-ever replay hit a resource-exhaustion apply
failure could still become CTRL_PRIMARY, because the fence that
replay would engage didn't exist yet when the gate checked it. Every
LATER promotion evaluation was already correctly protected; this was
a narrow gap specific to a process's single very-first election.
Fix: re-check persist_apply_resource_fenced() immediately after
cluster_init_phase2() completes on first boot, refusing promotion
(same WARN + ha_arm_promotion_fence_timer() + return pattern as the
existing refusal branch) if that first replay just engaged the fence.
Confirmed ha_elect_evaluate_and_signal (the other election path) is
dead in production for the promotion outcome -- every live call site
of handle_request_vote_reply passes vote_granted=0 since bug-828
redirected real grant recording to the record-only, non-signaling
ha_record_vote_reply -- so raft_cb_become_leader is the sole live
promotion path and this fix's scope is complete.
TDD: new test test_first_boot_promotion_evaluates_own_first_replay_fence
wires ha_phase2_hook to actually call persist_recover(), reproducing
production ordering exactly. RED-confirmed against the unfixed code
at the precise predicted assertion; GREEN-confirmed after the fix.
Full controller_unit_test regression (bundles every HA-adjacent
testsuite/test_*.c) green, 0 new failures.
Originally documented (not fixed) in job-array-single-wal-record
task 7.2's Implementation Notes; logged as bug-1445 and fixed here,
outside that spec's own task list per its persist.c-only boundary.
librtti master 7212ba1 -> 53cf11e (revCount 20->21): makeInstance() was
not freeing its calloc'd allocation when a type's .init() returned
negative, causing a 240-byte LeakSanitizer leak (SIGABRT under
abort_on_error=1) on any job_init() failure path -- most visibly
libxce test's test_job_init_propagates_makeJobsub_failure.
Verified fixed: fresh rebuild resolves the new rev, full libxce test
suite green, and test_jobj_hardening passes clean under
ASAN_OPTIONS=detect_leaks=1:abort_on_error=1 with no leak stack.
sched_handle_dataset()'s Data_data_jobArray case never freed
dataset->rows (the decoded xce_job_submit_t*) after sched_on_job_array()
consumed it -- free_Dataset() is an empty no-op for every table type, so
nothing else owned this call. One bounded leak per delivered array push.
Free it via the existing free_JobSubmit_ptr() teardown on both the
success and error path, mirroring the sibling list-shaped cases'
ownership-transfer convention.
Rewrote test_job_array_expand.c's init_job_array() fixture to
heap-allocate + strdup its fields (matching the real wire-decode shape)
since freeing the old stack/string-literal fixture crashed. Redelivery
test now builds a fresh descriptor per delivery, matching how a real
redelivery is always a freshly-decoded wire message.
Closes task 7.3 (job-array-single-wal-record), the last of the three
post-GO residual gaps reopened 2026-07-16.
replay_wal() now halts logical apply and engages persist_mark_apply_resource_fenced()
on a negative apply_committed_entry() return during boot/restart replay, mirroring
persist_apply_pump_run()'s existing live-path halt-and-fence behavior (task 4.1).
Physical scan bookkeeping (current_lsn, CRC chain) stays unconditional per entry,
consistent with persist_recover()'s own established current_lsn/last_applied_lsn
divergence precedent.
Went through one remediation round: round 1 exercised the wrong ha.c gate (periodic
re-alert timer); round 2 corrected the test to drive the real election-time gate
(raft_cb_become_leader_with_cluster_for_test()), with teeth independently confirmed
via a temporary fence-clear RED/GREEN cycle. Round-2 reviewer APPROVED.
Documents a real, NOT-fixed residual gap found during this task: a node's very
first-ever promotion evaluation in a fresh process runs before that node's own
first persist_recover()/replay_wal() call, so this fence is invisible to that one
promotion check (ha.c:1303-1408 gate runs before ha.c:1427-1439's first-boot
recovery). Every promotion evaluation after a process's first one is protected.
Out of this task's persist.c-only boundary; flagged for a dedicated ha.c follow-up.
restore_job_array()'s bulk-insert pass could fail partway through
(key_add() on realloc growth exhaustion), leaving an already-inserted
prefix permanently visible in c->jobmgr but excluded from
out_new_jobs -- so those elements existed and were individually
addressable but never reached HOOK_DETAIL_JOB_NEW, never scheduled or
pushed, not on this attempt or any later successful retry. A literal
violation of Requirements 2.3 and 9.4, tracked as bug-1443.
Adds kvmgr_reserve()/xzarray_reserve(), new purely-additive primitives
in shared kvmgr.c/zarray.c (following the existing kvmgr_capture_keys()
template -- bare extern function, no RTTI vtable slot, caller-holds-
write-lock contract), that pre-grow c->jobmgr's backing store before
the bulk-insert pass begins, closing the dominant N-scaling failure
trigger structurally.
Found a genuine off-by-one in the existing, unmodified
xzarray_ensure_items(): its strict less-than sufficiency check means
reserving exactly array_size and then performing array_size real
stores still triggers one more growth cycle on the last store.
Reserving array_size + 1 is the exact minimal correct compensation,
independently re-derived and confirmed by review with concrete
arithmetic.
For the now-much-narrower residual case (kvtab_add()'s own small
per-entry allocation, or a content collision -- both non-N-scaling,
honestly documented as still theoretically possible): restore_job_array()
now populates out_new_jobs with whatever prefix was genuinely inserted
before a failure, and apply_committed_entry() fires HOOK_DETAIL_JOB_NEW
for that non-empty partial prefix even when the overall return is
negative, while still propagating the negative rc so the LSN correctly
halts and the Live Apply Resource Fence still engages for the
remainder. A genuinely-inserted element is never left un-pushed again.
Task 7.1 of job-array-single-wal-record (Requirements 2.3, 9.4, 9.6).
Extends the HA soak harness with a new --array-soak mode: small (N=2)
and large (N=10000) array submission sub-rounds, each with one clean
kill/failover, asserting one WAL record per submission regardless of
array size, per-element addressability (status/cancel/completion)
spread across low/mid/high indices post-failover, and a real
millisecond-resolution replay-time bound comparing large-array vs
small-array kill-to-addressable latency.
waldump.c gains a WAL_JOB_ARRAY_SUBMIT case in wal_type_name() so the
scenario can grep an exact type name for its live WAL-record-count
delta check.
Went through one remediation round: the replay-time measurement
initially used whole-second resolution (collapsed to 0s/n-a on a real
run) and never asserted on the comparison, giving zero actual evidence
for the O(1)-replay-time requirement. Fixed with $EPOCHREALTIME
(millisecond resolution -- this sandbox's BusyBox date silently
mangles the more common `date +%s%3N` idiom) and a real bound
assertion that hard-fails the run on a genuine O(N) regression,
independently verified to have real teeth.
Live-verified 3 times with genuinely discriminating numbers each time;
final run: small=210ms, large=163ms -- large-array replay was not
slower, live evidence of the feature's O(1) claim.
Task 6.1 of job-array-single-wal-record (Requirements 4.3, 5.2, 5.3,
8.2, 8.3). Depends on and reuses task 6.0a's own soak verification.
replay_wal() (persist.c ~3902) tallied *new_job_submits_out only for
hdr.type == WAL_JOB_SUBMIT. WAL_JOB_ARRAY_SUBMIT entries were
dispatched and correctly materialized N real jobmgr entries via
apply_committed_entry()'s array case, but never counted --
persist_verify_apply_correctness() then saw real jobmgr growth exceed
the undercounted expectation and declared APPLY_DIVERGENT, engaging a
PERMANENT, non-self-healing fence (distinct from task 4.2's
self-healing Live Apply Resource Fence).
Live-reproduced during task 6.1's soak attempt on 2 of 3 nodes in a
single small-array (N=2) sub-round via two different trigger paths.
Deterministic, not size-dependent -- the first array submission to a
live cluster would fence any node that restarts or hits an
election-path recovery fallback before its next checkpoint, shrinking
the promotable pool. A pre-existing comment at persist.c ~3863-3874
(predating this spec) had already flagged this exact code region as
needing re-verification before tasks 4.1/6.1+ went live; this is that
re-verification, and it failed.
Fix adds a WAL_JOB_ARRAY_SUBMIT branch tallying apply_rc (the genuine
per-record element count apply_committed_entry()'s array case already
returns) when apply_rc > 0, mirroring the existing dedup-safe pattern
for WAL_JOB_SUBMIT. Traced the full return-value space of
restore_job_array()/apply_committed_entry()'s array case to confirm
every tallied value is provably equal to real jobmgr growth -- the fix
cannot overcount and so cannot mask a genuine divergence.
Verified via a from-scratch live 3-node HA soak re-run (--array-soak),
independently by both implementer and reviewer: zero APPLY_DIVERGENT
across all three original trigger paths, one-WAL-record/one-commit-
barrier confirmed for N=2 and N=10000, per-element post-failover
status/cancel/completion confirmed.
Task 6.0a of job-array-single-wal-record (Requirements 3.1, 3.2, 4.1,
9.4). Inserted mid-run after live soak testing surfaced the bug; user
confirmed sequencing (fix now, blocking task 6.1+).
Cross-daemon integration test, test-only, spanning the scheduler (C)
and API-server (Go/CGO) receive paths built in tasks 5.1-5.4/5.3b.
Drives real xce_encode()/xce_decode() wire bytes through the actual
production dispatch chain on both daemons -- extract_data() ->
scheduler_req_handler() -> sched_handle_dataset() ->
sched_on_job_array() and go_push_handler() -> handleDataset() ->
cJobArrayToModels()/dispatchJobArray() -- skipping only vtask_entry()'s
~10 lines of static glue, mirrored verbatim in test code (the
if (callnext) treats-any-nonzero-as-proceed behavior that caused
5.3b's bug is reproduced exactly).
Redelivery is tested via genuinely independent fresh decodes of
separately-encoded bytes, not a decode-once-call-twice shortcut.
Bug-catching capability proven by temporarily reverting task 5.3b's
fix and confirming both the C and Go tests fail meaningfully, then
restoring to green -- confirmed independently by both the implementer
and reviewer.
Task 5.5 of job-array-single-wal-record (Requirements 6.1-6.5, 8.4).
This completes Phase 5 (Compact Scheduler/API-Server Push).
Adds the Go/CGO receive path for Data.jobArray, mirroring task 5.3's
scheduler-side C implementation: rawJobArrayToModels() (pure-Go
base_jobid + i expansion), cJobArrayToModels() (CGO adapter sibling to
cJobToModel), dispatchJobArray() (dedup via the same store.Jobs.Has()
the ordinary Data_data_jobs path already writes through, single
confirmation log only when at least one element was newly
dispatched), and a new case C.Data_data_jobArray in handleDataset.
Elements route through OnJobUpdated, matching the ordinary single-job
push's own routing. The CGO boundary needed no changes -- both
C.xce_job_submit_t and C.Data_data_jobArray were already exposed from
prior tasks, including task 5.3b's extract_data() fix which this
receive path depends on identically to the scheduler's.
store.ArrayJobs (the array-level rollup aggregate) is deliberately
left unpopulated -- confirmed out of this spec's scope, deferred to a
future spec per requirements.md's Boundary Context.
Task 5.4 of job-array-single-wal-record (Requirements 6.2, 6.4, 6.5, 8.4).
extract_data() (src/lib/request.c), the shared reassembly step every
inbound push passes through inside vtask_entry() before any receiver's
handler runs, had no case for Data_data_jobArray -- it hit
default: rc=-EINVAL. Because vtask_entry()'s callnext check treats any
nonzero return (including a negative errno) as "reassembly complete,
proceed," this never surfaced as an error: the real payload was freed
and the receiver silently handed an empty, zeroed dataset_t, with no
log anywhere in the chain. Tasks 5.1-5.3 were each individually
correct and tested in isolation, but the feature was dead on the real
wire end-to-end.
Adds case Data_data_jobArray: passing the single, unchunked
descriptor straight through (no pagination needed, unlike the
existing LIST-shaped cases) and nulls the source pointer after
transfer -- verified this is not cosmetic, since free_Request()'s
pushmsg case would otherwise double-free it via free_Data()'s live
Data_data_jobArray branch.
Proven with a real xce_encode()/xce_decode() round-trip through the
fixed extract_data() -> sched_handle_dataset() -> sched_on_job_array()
(task 5.3, unmodified) chain, not just a unit-level check on
extract_data() alone. Also confirms task 5.4's future API-server
receive path shares this exact fix, since both go through the same
xcecli_ctx_init() -> vtask_entry() -> extract_data() pipeline.
One self-flagged, non-blocking residual gap (pre-existing since task
5.3, out of this task's request.c-only boundary): dataset->rows for
the jobArray case is never deep-freed downstream, leaking one bounded
xce_job_submit_t per successfully-delivered array push commit.
Task 5.3b of job-array-single-wal-record (Requirements 6.1, 6.2, 6.4, 8.4).
Inserted mid-run after independent review confirmed the gap; user
confirmed sequencing (fix now, before task 5.4).
Adds sched_on_job_array() (handler.c), a sibling case to
Data_data_schedjobs decoding the new Data.jobArray variant and
expanding it locally into N ordinary schedulable tasks via
base_jobid + i, routed through the existing unmodified per-job
pipeline (sched_on_jobs()). Each element gets its own deep clone
(clone_job_submit_data/clone_job_startinfo) of the shared submit
spec, since per-element teardown (free_Job_ptr) frees it
independently and a shared pointer would double-free.
Dedup reuses sched_get_job() against the same kvtab sched_add_job()
guards, rather than a new mechanism. A confirmation log line fires
once per successful expansion (skipped on a fully-duplicate
redelivery).
Adds a new task (5.3b) discovered during this task's review: the
shared wire dispatcher extract_data() (src/lib/request.c) has no case
for Data_data_jobArray and silently drops every real array push
before any receiver's handler runs -- outside every current task's
boundary, confirmed to affect the future API-server receive path
(task 5.4) identically. design.md's File Structure Plan corrected to
record this. User confirmed sequencing: fix it now, before 5.4.
Task 5.3 of job-array-single-wal-record (Requirements 6.3, 6.4, 6.5, 8.4).
Adds apiserver_push_jobarray() (apiserver.c), mirroring task 5.1's
xcesched_push_jobarray() exactly: same (base jobid, array_size, one
spec) Data.jobArray derivation, same direct xce_request()/OPCode_push
transport. apiserver_job_new_fun() branches on l->count > 1; the
existing single-job path (apiserver_push_joblist()) is byte-for-byte
unchanged.
Additionally, and unlike the scheduler (single target), replicates
apiserver_push_joblist()'s existing per-connected-server fan-out loop
(N_APISERVER_MAX iteration, same lock discipline) -- verified against
2 simulated connected servers, each receiving exactly one push.
New standalone apiserver_push_unit_test binary (same link-collision
rationale as task 5.1's sched_push_unit_test), correctly omitting
egress.c since apiserver_push_joblist() has no WAL-commit filter.
Task 5.2 of job-array-single-wal-record (Requirements 6.2, 6.4).
Adds xcesched_push_jobarray() (sched.c), encoding one compact
(base jobid, array_size, one spec) Data.jobArray descriptor -- reusing
the JobSubmit capnp struct already added to the Data union in task 1.2
-- instead of N individually-encoded job structs. sched_job_new_fun()
branches on l->count > 1 to call it; the existing single-job path
(l->count == 1 -> xcesched_push_joblist()) is byte-for-byte unchanged.
Sends via direct xce_request()/OPCode_push/Request_body_pushmsg rather
than the existing xce_push_data_req() helper, which hard-rejects any
payload outside jobs/workers/resources/resvals and cannot carry a
single non-list descriptor. design.md corrected to record this
transport deviation for task 5.2 to follow.
New standalone sched_push_unit_test binary (registered in
meson.build): sched.c/dataex.c/egress.c were never linked into
controller_unit_test, and two existing test files there ship local
fakes of on_scheduler_join()/notify()/collect_confvars() that would
collide with the real definitions at link time.
Task 5.1 of job-array-single-wal-record (Requirements 6.1, 6.3, 6.4).
New test_job_array_fence.c exercises the full Requirement 9.4-9.6
engage-refuse-clear-recover sequence through real production seams:
forced array-expansion failure -> fence engages -> promotion refused
at a real ha.c gate site -> zero partial elements visible -> a second,
independent node's progress is unaffected -> clearing the failure and
retrying self-heals the fence and re-opens promotion eligibility.
Deliberately drives this through task 3.1's construction-phase
(scratch-build) seam rather than the bulk-insert-phase seam: the
bulk-insert seam has a known, separately-tested residual gap (an
already-inserted element prefix can survive a mid-pass key_add()
failure with no key_del to undo it), so asserting "no partial set"
against it would misrepresent an already-documented open gap as
resolved. That gap remains open and unowned (see tasks.md
Implementation Notes) -- Phase 4 delivers the generic halt/fence/gate
machinery for it, not the partial-visibility guarantee itself.
Task 4.4 of job-array-single-wal-record (Requirements 9.4, 9.5, 9.6).
This completes Phase 4 (Live Apply Resource Fence).
Threads !persist_apply_resource_fenced(&c->persist) as a required
conjunct into both existing promotion-eligibility call sites in ha.c:
the periodic fence-release check (promotion_fence_timeout_fn) and the
election-time check (raft_cb_become_leader). A node with the fence
engaged is refused promotion at both points; once it self-clears
(task 4.2), the node is eligible again with no other action.
Task 4.3 of job-array-single-wal-record (Requirements 9.4, 9.5).
Adds a new atomic LSN field on persist_t (distinct from the existing
boot-only unrecoverable-fence flag) plus three primitives:
persist_apply_resource_fenced(), persist_mark_apply_resource_fenced(),
persist_clear_apply_resource_fence(). Wired into
persist_apply_pump_run(): engages on task 4.1's halt condition, clears
unconditionally on the walk's next successful advance — provably
equivalent to "clears only when the same stuck LSN's retry succeeds"
since the walk is strictly sequential and never advances past a
failed position.
Known residual gap, out of this task's boundary (documented in
tasks.md Implementation Notes): replay_wal() during boot/restart
replay does not yet share this protection, so a node crashing while
fenced can silently re-pass the same failure on restart. Flagged for
a follow-up task, not fixed here.
Task 4.2 of job-array-single-wal-record (Requirements 9.4, 9.5).
persist_apply_pump_run() no longer discards apply_committed_entry()'s
return value. On a negative rc for any WAL type (not just the new
array type), the walk halts at that LSN — no last_applied_lsn advance,
no pending-apply hand-off — logs a loud error, and retries from the
same position on the next kick, mirroring the existing
apply_pump_read_entry failure branch's shape.
Investigated whether this newly turns the pre-existing job-fact
-ENOENT path into a permanent-wedge hazard: it does not, since
c->jobmgr never deletes entries and facts are only ever applied after
their causally-prior submit already succeeded, so a live -ENOENT there
can only mean genuine divergence — halting is the correct behavior.
Task 4.1 of job-array-single-wal-record (Requirement 9.4).
Adds 2 targeted tests after confirming tasks 3.1/3.2/3.3 already
cover most of this task's observable criteria:
- N-way expansion swept across sizes {2,3,100,1000} with exact-
endpoint non-leakage checks (existing coverage only tested size 5)
- Requirement 5.2 (individually addressable exactly as a per-element
job) exercised through the real production command dispatch
(job_ctrl(), not the raw key_find() every existing test used) --
confirms an array-expanded element can be cancelled by identity,
siblings stay untouched, and an out-of-range jobid is rejected
Idempotent no-op, forced-failure-then-retry, and partial-reapply
were already fully covered by 3.1/3.2 -- no new tests needed.
Requirement 9.4's node-fencing behavior is confirmed out of this
task's boundary (Live Apply Resource Fence, task 4.x); restore_job_
array()'s own atomicity contract that the fence depends on is
already proven. No production code changed -- test-only.
Task 3.4 of job-array-single-wal-record.
Adds case WAL_JOB_ARRAY_SUBMIT to apply_wal_entry() (standby
catch-up dispatch), calling restore_job_array() hook-free, matching
the existing WAL_JOB_SUBMIT case's shape exactly.
Review found apply_wal_entry() has zero production call sites -- a
2026-07-10 refactor (ha_standby_advance_committed_and_apply) moved
standby entity materialization onto the same apply_committed_entry()
pump the primary uses; the code comment claiming otherwise predated
that refactor and was never updated. Requirement 3.1/3.2/4.1 are
fully delivered by task 3.2 alone. This task's wiring is kept as
defensive parity with WAL_JOB_SUBMIT's existing, equally-dead case
in the same switch, not as an independent delivery path -- design.md
and tasks.md corrected accordingly.
Task 3.3 of job-array-single-wal-record.
Adds case WAL_JOB_ARRAY_SUBMIT to apply_committed_entry() (the live
apply pump / boot replay dispatch seam): calls restore_job_array()
(task 3.1) instead of restore_job(), and on success fires
HOOK_DETAIL_JOB_NEW exactly once carrying every newly created
element in restore_job_array()'s own out_new_jobs list -- never once
per element. No hook fires on an idempotent no-op (-EEXIST) or any
genuine construction failure, honestly propagating task 3.1's
documented bulk-insert-phase residual gap rather than papering over
it. apply_wal_entry() (standby catch-up) is untouched -- task 3.3.
Task 3.2 of job-array-single-wal-record.
Adds the shared N-way array expansion primitive: decodes a committed
WAL_JOB_ARRAY_SUBMIT payload, scratch-builds every element via a
throwaway kv-table (never touching c->jobmgr during construction),
then makes the whole set visible in one bulk pass under a single
write-lock acquisition -- no window where a concurrent reader can
observe a partially-constructed array. Idempotent per index (skips
already-existing jobids; whole-record no-op if all exist). Includes
an XCE_TESTING-guarded fault-injection seam. Not yet wired into any
live dispatch path (tasks 3.2/3.3).
Two review rounds: round 1 found the bulk-insert pass silently
claimed success even when key_add() itself could fail on allocation
exhaustion (c->jobmgr grows via realloc, no reservation primitive
exists anywhere in kv.c/kvmgr.c/zarray.c to pre-check this). Fixed
by checking key_add()'s return and reporting failure honestly
instead of a false success. The underlying table has no key_del, so
an already-inserted prefix on this specific failure mode can remain
genuinely visible while the call reports failure -- documented as a
residual gap in design.md and tasks.md for tasks 3.2/4.x to account
for (that prefix is never pushed to scheduler/API server, even on a
later successful retry, since it's never included in out_new_jobs).
Task 3.1 of job-array-single-wal-record.
Adds 3 targeted tests after confirming tasks 2.1/2.2 already cover
most of this task's observable criteria:
- one-entry/one-continuation swept across sizes {2,3,100,1000}
(existing coverage only tested size 5)
- payload-size boundedness at the real idgen()-enforced ceiling
(MAX_ARRAY_IDX+1 = 32768), plus an absolute byte-cap assertion
(existing coverage only compared 2 vs 10000)
- array_size==1 baseline: explicit WAL-byte readback proving the
entry type is WAL_JOB_SUBMIT (not WAL_JOB_ARRAY_SUBMIT) and the
payload decodes via struct Job, not struct JobSubmit (existing
coverage only proved "old tests still pass" implicitly)
No production code changed -- test-only.
Task 2.3 of job-array-single-wal-record.
Branches handle_jobsubmit() on array_size: ==1 continues through
today's exact unmodified single-job path; >1 routes through a new
handle_jobsubmit_array() helper that reuses jobsubmit_async_ctx_t
with pending=1 and calls task 2.1's persist_submit_job_array_async()
instead of looping N single-element appends. Range validation
(jobid_range_exists/inflight_range_try_add) unchanged, same order,
strictly before any WAL append, for both branches.
Retires two pre-existing tests whose premise (N independent
per-element WAL entries with mixed outcomes) is now structurally
impossible with one entry per array; restores array-scoped
job-non-existence coverage for the failed-commit case that an
earlier draft of this change had dropped (Req 2.2/9.2).
Task 2.2 of job-array-single-wal-record.
Adds persist_submit_job_array_async(): encodes one JobSubmit-shaped
payload (base jobid, array size, one shared spec), appends exactly
one WAL_JOB_ARRAY_SUBMIT entry, and registers exactly one commit
continuation regardless of array size -- reuses the existing
job_commit_adapter_ctx_t/job_commit_resolve_adapter machinery, no
per-element loop. Not yet wired into handle_jobsubmit (task 2.2).
Also fixes a test-helper bug (WAL offset back-computed from the
post-append cursor, only valid for zero-payload entries) that
corrupted state across tests when exercising a nonzero-payload
array-submit entry.
Task 2.1 of job-array-single-wal-record.
- job_init() now checks makeJobsub()/makeString(ckptinfo) return
values and propagates failure instead of storing NULL with a
success return (Req 10.1).
- Root-caused the previously observed makeJobsub() NULL-with-EINVAL
failure to a test-binary-only hazard: multiple controller test
files each called job_obj_init() from their own setUp() with no
cross-file coordination, re-running jobsub_table_init() against an
already-live lock/table. Fixed via atomic_flag-guarded idempotent
job_obj_init()/job_obj_fini(), plus a distinct -ENODEV return from
makeJobsub()/makeJobstartinfo() when the table isn't ready
(previously fell through to undefined behavior) (Req 10.2).
- Verified (test) MOVE_PTR fields are always NULL on a freshly-built
submission payload -- no code change needed (Req 10.3).
- Fixed job_destroy()'s accounts leak via the existing
free_job_account() helper, replacing the stale FIXME (Req 10.4).
- Verified (inspection + test) struct Job's capnp schema has no
ckptinfo field, so decode_Job/free_Job have nothing to leak there
-- no code change needed (Req 10.5).
Independently re-verified: reviewer reproduced a pre-existing,
unrelated rtti-library leak-on-init-failure against a stashed
pre-task copy of jobj.c to confirm it predates this change.
Task 1.3 of job-array-single-wal-record.
Adds Data union variant jobArray@6, reusing JobSubmit's existing
(jobid, arraySize, submit) shape verbatim -- no new struct, no
existing variant renumbered. Not yet wired into any push path
(tasks 5.1/5.2 own that).
Task 1.2 of job-array-single-wal-record.
Adds the array-shaped WAL entry type (distinct from WAL_JOB_SUBMIT,
value 11 in the unused WAL_JOB_* gap) and classifies it WAIT-mode via
wal_classify, matching the single-job type's commit-barrier semantics.
No dispatch/apply wiring yet -- foundation for task 2.x/3.x.
Task 1.1 of job-array-single-wal-record.
xcecli_joblist/reslist/workerlist/resval/jobinfo return a non-negative ROW
COUNT on success (rc = getreply.nrows), not a 0/status code. They were
wrapped in XCECLI_FAILOVER_CALL, which unconditionally classifies the raw
return value against fixed constants (E_XCE_* and POSIX errno magnitudes
accepted in either sign) — a row count that happens to equal one of those
constants (e.g. exactly 111 jobs == ECONNREFUSED) was silently
misclassified as a transport failure, forcing a spurious
xce_failover_rotate() on every call with that exact count. In
/xce-test-ha's arithmetic progression (3 + 3*round) this hit at round 36,
burning the retry budget on self-inflicted churn until it coincided with
a genuine transient and surfaced a real ECONNRESET to the caller.
Add XCECLI_FAILOVER_CALL_COUNT: only enters the retry/classify loop when
EXPR's raw result is negative; any non-negative result (0 or a genuine
row count of any magnitude) is unconditionally success. Switch the five
xce_get-family wrappers to it. Verified via 40-round sequential and
40-round chaotic HA soak past round 36, zero committed-job loss.
raft-drive-async's default-ON pool dispatch lets messages from different
peer endpoints (two vote replies, or a vote reply racing a vote request)
run concurrently on different vtask pool workers. raft_state_t's fields
(votes_granted, replies_received, election_decided, term, role) are plain
non-atomic — the raft library assumes a single-threaded caller. Two
concurrent raft_on_vote_reply() calls could race: one reaches quorum and
promotes to PRIMARY while the other, reading pre-increment vote count,
independently satisfies "all-replied, no quorum" and fires
on_become_follower for the same term, silently reverting the promotion —
every request afterward correctly (but wrongly) rejected as STANDBY.
Add ha->raft_lock (PTHREAD_MUTEX_RECURSIVE — a WON/quorum outcome can
legitimately re-enter the state machine synchronously on the same thread,
e.g. PreVote WON -> real election) around every raft_state_t entry point
on the election/vote path. Verified via 40-round sequential and 40-round
chaotic HA soak, zero committed-job loss, all 3 nodes led.
Replace the deliberate election-carrier leak with an atomic refcount:
a bias reference held by ha->elect_ctx plus one reference per captured
vote endpoint, released via a new endpoint_t.on_fini teardown-completion
hook (fired once, before ep->data is freed). Both real-election and
PreVote capture sites (raft_cb_send_vote_request,
raft_cb_send_pre_vote_request) increment on capture; both publish sites
(ha_start_election_async, ha_start_prevote_async) now atomic_exchange
the outgoing carrier and release it instead of leaking or freeing
outright, so a still-outstanding vote endpoint keeps the superseded
carrier alive until its own teardown completes.
Verified: 758+ controller tests green, isolated LeakSanitizer run shows
zero ha_elect_ctx_t leaks across many election/PreVote rounds, no
premature-free/UAF under supersession, and a live 10-round HA soak
(36 jobs, zero loss, all 3 nodes served as primary).
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).
bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.
Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
- xce-build.sh: `controller` component now always builds `controller/xce_test`
alongside `xce_controller`, in every profile — xce-test-ha needs both
binaries and previously had to build xce_test as a separate step.
- xce-build/SKILL.md: removed `context: fork` — reproducibly broken for this
skill (confirmed twice): whenever ARGUMENTS is non-empty, the forked
subagent never receives a usable task and silently does nothing. Documented
that this skill should instead always be dispatched via the plain Agent
tool with a fully-resolved, self-contained prompt (not `context: fork`,
which failed) — proven to work end-to-end.
- xce-test-ha/SKILL.md, xce-deploy-ha/SKILL.md: replaced the blanket "never
invoke via subagent" warning with the precise finding — `context: fork`
(Skill-tool isolation) is the confirmed-broken mechanism (its own teardown
SIGHUPs the whole controller process group mid-soak/mid-deploy), but
Agent-tool subagent dispatch is a different mechanism and is safe
(empirically verified: a `setsid nohup ... & disown` process launched from
inside an Agent subagent survives that subagent returning, reparented to
init). Documented both the foreground-blocking and fire-and-forget
subagent patterns for each skill; xce-deploy-ha additionally flagged that
its `start.sh &` should gain `setsid` to match the soak driver's stronger
detach pattern before relying on unattended subagent dispatch.
A primary never learns of a follower's higher term via AppendEntries-reject
responses: ha_raft_on_repl_ack only routes AE replies through
raft_on_append_entries_reply for success=1 ACKs, and that function itself
drops any reply whose term != s->term (raft.c stale-term drop) before it
could adopt a higher term anyway. Net effect: once a follower outpaces a
stale primary's term, every AppendEntries the primary sends is rejected
forever and it can never regain write-quorum — a permanent election livelock
(surfaced as "no primary elected at term>8" in HA soak testing).
Adds two fallback term-discovery vectors that fire regardless of AE-reject
outcome:
- StateReport heartbeat-reply now carries the standby's current term
(controller.capnp, typemap.h) so the primary learns a higher term on every
convergence cycle.
- New raft_on_higher_term_observed() mirrors the UpdateTerm step (strictly
greater-than only; touches term/voted_for/role; persists before notifying
on_become_follower) and is wired into both the heartbeat state-report path
and the WAL-NACK path in repl_ack_decode_and_dispatch.
Validated: 265 controller + 287 lib unit tests pass; 10-round sequential and
chaotic HA soaks both PASS with zero job loss and full leadership coverage.
Found via extended chaotic /xce-test-ha soaking on top of the bug-1419 fix
(same session). Both bugs independently caused the 10-round chaotic soak to
fail; sequential and chaotic now pass cleanly and repeatably.
1. Term-desync liveness deadlock (new): a standby that loses a near-
simultaneous election race to a peer can legitimately end up at a HIGHER
term than the peer that actually won and became primary. The primary's
now-stale-term AppendEntries/heartbeats are correctly rejected by the raft
library, but every one of those rejected messages was ALSO resetting the
standby's own liveness deadline (ha_on_heartbeat_received), so its
liveness timeout never fired and it could never start a fresh,
higher-term election to let the cluster self-heal — a permanent stall.
Fix: ha_msg_term_is_stale() gates the liveness reset (and
ha_discover_primary step-down) on the incoming message's own term stamp,
skipping it for provably-stale messages, in cluster.c's wal_catchup_entry
and nop/heartbeat handlers.
2. bug-847 (root-caused 2026-06-30, never fixed): a per-sub-entry `payload`
Data blob in a multi-entry WAL catch-up batch loses its capnp pointer once
the message crosses a segment boundary — the receiver decodes
raw_payload_len=0 for entries at/after the boundary and silently persists
empty WAL entries (structurally intact: correct LSN/term/CRC chain, empty
content). Only surfaces when a node rejoins and catches up a real batch,
which is why it needed a live chaotic soak (not the 2-entry unit test) to
reproduce. Fix: carry every sub-entry's payload concatenated in ONE
top-level batchPayloads blob instead of one Data blob per entry — a
single top-level blob does not exhibit the cross-segment loss. Sliced
back apart on decode using each sub-entry's own wirePayloadLen, a plain
scalar that (like lsn/entryType/term) decodes via the generic generated
decode path and was never subject to the bug. Confirmed via targeted
diagnostic logging that the corrupted entry arrives already-empty over
the wire (not a receive-side race), and via a faithful reproduction at
live JOB_SUBMIT payload size (5×3560B) after the original 2-entry/
14×872B tests turned out not to cross a segment boundary in this
environment.
Incidental fixes found and fixed along the way:
- A real self-deadlock (persist_entry_term() re-acquiring p->lock from
inside a caller that already held it) introduced by an initial, ultimately
unnecessary WAL-read locking fix attempted before bug-847 was pinned down;
fixed via a new persist_entry_term_locked() lock-free counterpart. Left in
place as a genuine (if incidental) hardening of ha_stream_wal_to_standby's
read consistency.
- .claude/skills/xce-test-ha/SKILL.md's `context: fork` frontmatter caused
the skill's own forked sub-session teardown to SIGHUP the whole
xce_controller process group mid-soak, killing all 3 nodes silently with
no error logged, whenever the soak was launched via the Skill tool itself
(setsid/nohup inside run-ha-soak.sh only protects against a controlling
terminal's own SIGHUP, not a parent forked-session's teardown). Removed;
added explicit guidance to launch long soaks via direct Bash from the main
session instead.
Verified: full 78-group unit regression sweep clean (0 failures) after each
change; two consecutive clean 10-round chaotic soaks and one clean 10-round
sequential soak on the final binary (zero committed divergence, zero job
loss, all 3 nodes led in both modes).
Root cause (confirmed via targeted DIAG1419 logging on a live chaotic soak):
a client's commit-timeout retry during a genuine quorum-loss window appends
a second WAL_JOB_SUBMIT for the same jobid; the timed-out entry is never
retracted, so if it later achieves majority commit anyway both entries end
up committed. This is a legitimate, harmless duplicate — restore_job()'s own
-EEXIST correctly no-ops the second one — but replay_wal() tallied every raw
WAL_JOB_SUBMIT occurrence instead of genuine creations, so the duplicate
made new_job_submits disagree with jobmgr's real growth, firing a false
APPLY_DIVERGENT and deadlocking promotion.
Reverts the layer-3 claim/release machinery (job_wal_commit_claim/release,
claim_committed_job_submits, and their wiring): three attempts at preventing
the duplicate at submit/dispatch/commit time all failed to close the gap,
because the duplicate isn't a bug to prevent — a WAL entry can't be
retracted once appended, so any submit-time guard is structurally the wrong
layer.
Real fix: apply_committed_entry()'s WAL_JOB_SUBMIT case now returns 1 for a
genuine creation vs 0 for an idempotent no-op (restore_job() -EEXIST), and
replay_wal() only tallies new_job_submits_out on rc==1. Updates the affected
apply-seam/reapply-idempotency tests for the new return contract, and
reverses two apply-correctness tests whose premise (duplicate-jobid replay
must diverge) is now understood to be the actual bug rather than the
expected behavior — they now force divergence directly via the same
production fencing seam instead of relying on that trigger.
Verified: two consecutive 10-round chaotic soaks, zero divergence, zero job
loss, no election stall (previously reproduced within ~5 rounds).
jobid_range_exists() (c->jobmgr) and the in-flight registry (job.c,
per-process, only populated by this node's own handle_jobsubmit calls)
both miss a real cross-node case: a client's XCECLI_FAILOVER_CALL retry
resends the same client-generated jobid after a connection drop, and the
resend lands on a newly elected primary that never processed the original
request. If the original was already committed before the old primary
died, the new primary's own local WAL already holds it (Raft Leader
Completeness) but jobmgr hasn't applied it yet — both existing guards see
nothing, so the resend appends a second, genuinely committed WAL_JOB_SUBMIT
entry for the same jobid. A live chaotic soak reproduced this exactly
(waldump: two committed entries, same term, matching apiserverId/timestamp/
sequence, one second apart) — apply's own idempotent dedup keeps jobmgr
correct, but replay's raw WAL_JOB_SUBMIT count no longer matches jobmgr's
growth, firing a false APPLY_DIVERGENT fence and deadlocking promotion.
Fix: apply_committed_entry()'s WAL_JOB_SUBMIT case now claims the jobid in
job.c's in-flight registry (new job_wal_commit_claim/release, wrapping the
existing inflight_range_try_add/remove) for the duration of dispatch,
regardless of whether this node originated the submission or is applying a
replicated/replayed entry. A resend landing mid-dispatch on any node now
sees the same -EEXIST rejection self-originated in-flight submits already
get. Release only fires when the claim call added the entry, so a
self-originated submission's own claim/release lifecycle (unchanged) is
never double-released or stolen.
Also fixes 4 pre-existing tests that broke under the earlier checkpoint_lsn
fix (0eb1054) and were missed in that commit's regression sweep: cluster-
less fixtures where last_applied_lsn never advances now correctly get
checkpoint_lsn=0 unless the test simulates apply catch-up, matching the
new (correct) semantics. One assertion in
test_checkpoint_tail_with_no_snapshot_is_apply_clean compared Phase 1's
checkpoint_lsn against Phase 2's independently-reconstructed physical WAL
tip — that equality only held under the pre-fix current_lsn-tracking
behavior and is removed, not patched around.
Also hardens run-ha-soak.sh's start_node() with setsid+nohup: a forked/
background test-harness invocation whose session tears down mid-run was
silently killing every controller process via SIGHUP forwarding (observed
twice — deploy completes, ~1s of activity, then all 3 nodes die together
with no error logged).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
xce_test's cmd_jsub hardcoded idgen(&idctx, 0, 0, ...) (clusterId=apiserverId=0),
and run-ha-soak.sh's submit_jobs() spawns a fresh xce_test process per retry
attempt, resetting idgen's static sequence counter each time. Two unrelated
job submissions from separate process launches landing in the same
wall-clock second collided on an identical jobid, surfacing as a spurious
APPLY_DIVERGENT fence in chaotic soak that was initially misattributed to
bug-1419's controller-side dedup logic.
Confirmed not a production risk: apiserver's GenerateJobID holds one
process-lifetime idgen_ctx_t via sync.Once, and each deployed instance has
its own distinct apiserverId, so config.h's jobid bit layout already
partitions the production namespace. Fixed by folding getpid() into the
harness's apiserverId so cross-process collisions become structurally
impossible there too.
Also adds test_jobsubmit_resubmit_after_full_resolution_rejected_no_second_job
to test_commit_barrier_reply.c, proving the post-resolution dedup window
(retry arriving after commit+apply+reply+release) is airtight, complementing
the existing before-resolution test -- both known bug-1419 dedup windows are
now provably race-free, confirming those fixes were never the cause of this
soak failure.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
persist_checkpoint() stamped checkpoint_lsn from p->current_lsn (physical
WAL append tip) instead of p->last_applied_lsn (the true "jobmgr reflects
up through here" cursor), uncoordinated with the async apply pump
introduced by raft-apply-decoupling. write_snapshot() serializes live
jobmgr state, so a checkpoint taken while apply lags the WAL tip mislabels
the snapshot: on restart last_applied_lsn is re-seeded from checkpoint_lsn
and replay resumes strictly above it, silently skipping committed-but-not-
yet-applied entries jobmgr never received. This is the actual mechanism
behind kill-mid-apply producing a permanent APPLY_DIVERGENT fence —
apply_committed_entry() is deterministic/idempotent and write_snapshot()
is crash-atomic, so the bug was a mislabeled watermark, not a structural
divergence requiring a safety/liveness tradeoff.
Fix stamps checkpoint_lsn from last_applied_lsn and holds apply_pump_lock
across the label capture + write_snapshot() so the label and the jobmgr
state it describes are captured atomically w.r.t. the live apply pump.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
jobid_range_exists() only checks c->jobmgr, but task 3.1 deliberately
defers job construction until after commit -- so a submission that is
durably WAL-appended and awaiting its commit outcome is invisible to
that check. A client-side retry (XCECLI_FAILOVER_CALL resending the
same jobid after a transport error or failover mid-flight) sails past
the guard and appends a second, non-identical WAL_JOB_SUBMIT entry for
the same jobid -- confirmed via waldump on a live chaotic-soak node
(duplicate submits, same jobid, differing subtime/CRC) -- which can
trip persist_recover()'s post-replay apply-correctness check into a
genuine APPLY_DIVERGENT verdict.
Adds a process-lifetime in-flight-jobid-range registry in job.c that
handle_jobsubmit atomically check-and-claims immediately after the
existing jobid_range_exists check, releasing from
jobsubmit_async_ctx_release (the single choke point already shared by
the commit-continuation, apply-complete, and OOM early-out paths). A
resubmit for a still-in-flight jobid now gets -EEXIST with no second
WAL append.
Adds a matching XCE_TESTING reset seam (declared in cluster.h) so a
test that deliberately leaves a submission's commit continuation
unresolved through teardown doesn't leak its jobid range into a later
test reusing the same range -- commit_engine_fini frees pending
continuations without invoking their callback.
Regression test added to test_commit_barrier_reply.c (RED->GREEN
confirmed). Full 78-group controller_unit_test sweep: 782 tests, 0
failures.
This fixes only the WAL duplicate-submit mechanism (layer 1 of
bug-1419). The deeper PreVote-freshness/apply-divergence liveness
deadlock (layer 2 -- no recovery path when the only fresh-enough node
is apply-divergent-fenced) remains open and needs its own design
discussion; buglog.json updated accordingly.
vote_reply_dispatch passed the compressed, self-excluded local votes_idx
(vctx->peer_idx, set by raft_cb_send_vote_request's absolute->local remap)
straight through to raft_on_vote_reply, which indexes vote_replies[] in
ABSOLUTE cluster space -- the same space raft_start_election's self-mark
and per-peer send loop use.
For a 3-node cluster's middle node (self_idx=1), the peer at absolute
index 2 compresses to votes_idx=1, colliding with vote_replies[self_idx=1]
-- the slot raft_start_election already marked for the self-vote. The
library's per-peer dedup guard then silently drops the genuine reply as a
duplicate, so votes_granted never reaches quorum even though the peer
granted. Peer 0 is unaffected (compressed index equals absolute), which is
why sequential elections and prior soak runs never surfaced this -- only a
middle-priority node losing its higher-priority peer and depending on its
lower peer's vote hits the collision.
Fixed by routing through ha_repl_ack_abs_peer_idx (already existed,
already unit-tested, already used for the analogous heartbeat-ACK path)
before calling raft_on_vote_reply. ha_record_vote_reply immediately above
is unchanged -- it correctly wants the compressed index for ctx->votes[].
Confirmed pre_vote_reply_dispatch needs no equivalent fix: its send path
already stores the absolute index.
New regression test drives a real 3-node election end-to-end through the
actual decode/dispatch path (not the test-hook bypass every other
real-vote test in this file uses): Expected 2 Was 1 pre-fix, GREEN
post-fix. Full 79-group suite: zero new failures. 10-round --chaotic HA
soak: zero committed divergence, zero committed job loss.
Measured sustained worker-fact throughput against the pre-feature baseline.
Commit throughput is unregressed (~2.5% delta, within noise) and no per-fact
synchronous commit barrier exists, confirming Req 11.4's core claim. But
apply/observable throughput -- the new hop this spec introduces -- regressed
56% with clearly superlinear scaling (2x volume -> 5.2x elapsed).
Root cause: apply_pump_read_entry has no memory across calls -- every LSN
lookup re-lists WAL segments and rescans its segment from byte 0 -- while its
only caller applies committed entries one at a time in strict ascending
order, turning a sequential access pattern into O(n^2) total work.
Fixed with a minimal segment-read cache on persist_t (fd + offset + expected
next LSN, guarded by the existing apply_pump_lock): a fast path reads the
next expected LSN directly at a remembered offset, falling back unchanged to
full rediscovery on any miss. Correctness never depends on the cache, only
performance does. Invalidated on init, fini, recovery, and WAL truncation --
the four points a stale cursor could otherwise misread from.
Two review rounds: round 1 caught a missing lock guard in the recovery
invalidation path and an inaccurate risk claim about the cache/trim
interaction (a real, bounded behavior change deferred to the existing
APPLY_DIVERGENT fence, not "no worse than before") -- both corrected and
re-verified in round 2. Post-fix apply-throughput scales sublinearly (1.46x
elapsed for 2x volume) and exceeds the pre-feature baseline. Zero new test
failures across the pump, correctness-verification, recovery, and truncation
groups.
An intermittent, pre-existing client hang in the synchronous jobsub path and
a trim-floor/apply-lag architectural gap were surfaced but are explicitly
out of scope for this fix -- flagged for separate follow-up.
Full 79-group unit/integration suite re-run found exactly 2 stale tests in
test_commit_barrier_reply.c asserting task 3.2's documented interim
contract ("no observable effect until task 6 completes it"), which the
already-committed ordered apply pump (tasks 6.1/6.2) closed by construction.
Updated both to assert the real, deterministic apply-after-commit settled
state instead of the no-longer-true no-effect-yet state, matching the idiom
already used elsewhere in the same file. No production code changed.
Full suite: 0 failures across 79 groups. HA soak (--sequential, 5 rounds):
zero committed-job loss, zero divergence, all 3 nodes served as primary.
Both independently re-verified by adversarial review.
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.
Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.
Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
Extend raft_sim.c's deterministic TLA+-conformance harness with two
fault-injectable checks: no reaction before commit, and idempotent
re-apply of an already-committed entry. Wired as dedicated tests only,
not into raft_sim_check_all's fixed four-invariant contract (owned by
a separate spec). RED/GREEN independently verified by adversarial
review; zero regression to existing scenario functions.
Extend test_apply_pump_integration.c with 2 tests, no production
changes. Test 3 interleaves commands and facts through a real
quorum-loss outage and proves strict original-log-order catch-up via
a sequence spy. Test 4 wires a real 4-vcpu vtaskmgr_t to drive 4
commands + 2 facts through one covering ACK, creating genuine N-way
concurrent races across task 6.2's self-healing fix rather than one
isolated forced pair -- stress-testing bug-1415's fix under a
realistic catch-up burst.
Closes out task-group 6 (ordered apply pump integration): the pump
is now live, hands off command replies correctly, and is proven
correct for isolated and mixed/bursty command+fact scenarios under
both normal and quorum-loss conditions.
Add test_apply_pump_integration.c (2 tests), no production changes.
Exercises the full live pump (real commit engine, real apply
dispatch, real hook subscribers) for both a job submission and a
fact: proves reply-after-visibility ordering for commands, and zero
fact-reaction side effects during a delayed-commit/quorum-loss
window (polled repeatedly across real time, not checked once).
Confirms Req 4.3 by construction, not assertion: the apply walk only
ever invokes apply_committed_entry() for LSNs already <= committed_lsn,
so a consumer acting on the post-apply notification never needs to
separately re-check a commit watermark.
Independently verifies apply_committed_entry()'s generic
HOOK_DETAIL_JOB_UPDATE routing (unchanged by task 6.1's _pad fix)
makes the type-specific sched.c/apiserver.c hook handlers
structurally unreachable for live-applied facts -- confirmed their
va_arg-typed signatures would type-confuse-crash if ever called with
the job_t* apply_committed_entry() actually passes, making the
single generic reaction spy a sound design, not a coverage gap.
Repo-wide audit confirms exactly 2 live call sites ever fire these
hook details (the guarded apply seam, and job_ctrl's legitimate
SIGCONT/SIGSTOP path) -- no orphaned third site.
Wire persist_apply_pump_run() to call persist_take_pending_apply()
immediately after each entry's apply_committed_entry() call,
invoking the registered continuation strictly after the mutation.
Adds jobsubmit_reply_success_once() (job.c) so a successful job
submission finally sends a client reply once every array element
applies -- closes the standing gap from task 3.2 where committed
submissions never got a response.
Fixes bug-1415: commit_resolve_pending() dispatches the commit
continuation and the apply-pump kick as two independent
VTASK_LIVE_BLOCK vtasks with no completion-order guarantee. On any
real multi-vcpu pool (the production default, n_vcpus=8), if the
apply pump won the race and advanced last_applied_lsn past an LSN
before its continuation registered, the registration was permanently
orphaned -- the job would be constructed and visible, but the client
would hang forever. Untested by every existing fixture in this spec
(all use pool=NULL, which trivially serializes and can't reproduce
the race).
Fixed with a self-healing check in persist_register_pending_apply():
if the LSN is already <= last_applied_lsn, invoke the callback
immediately instead of registering for later pickup. Verified by the
spec's first genuine multi-vcpu regression test (real vtaskmgr_t,
n_vcpus=4), independently RED->GREEN reproduced by two review rounds.
