Cause: handle_jobsubmit() never looked at the target queue, so a
submission that bypassed the apiserver's check (libxce, harness
xce_test, or a REST submit without a queue name) was accepted into a
closed queue, an unknown queue, or the reserved lost_and_found queue.
Fix: jobsubmit_check_queue() reads the queue's in-memory state under the
qmgr/queue read locks. handle_jobsubmit() calls it after the field
checks and before the in-flight jobid registry and any WAL append, for
single jobs and job arrays. It replies -E_XCE_QUEUE_CLOSED (new code,
"queue is closed") for a closed queue and -E_XCE_BAD_QUEUE for an
unknown or reserved one. Apply/replay (restore_job) is unchanged, so
committed submit facts for a now-closed queue still apply.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
update_job_cpu_stats() (misc.c), run after each usage report:
- each host's cpu_rate = CPU-seconds gained / wall seconds since its
previous report; the job's cores = the sum over hosts
- peak_cpu = highest job cores seen (peak_cpu_time = it rounded, the
wire field is an integer); peak_cpu_duration = the interval that set it
- avg_cpu_efficiency = cpu_time / elapsed / slots, peak_cpu_efficiency =
peak_cpu / slots, mem_efficiency = highest used_mem / rusage[mem]
request (job-wide total, per task by default), all in percent
aggregate_job_runusage() no longer treats cumulative cpu_time as a peak.
New fields host sample_time/cpu_rate and job peak_cpu are appended to
the capnp structs (old WAL entries still decode) and copied by
merge_job_runusage(), so a promoted standby keeps the rate baselines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
handle_jobsubmit now registers the submission first and checks the job
table second; a hit is answered -EEXIST and released through the atomic
reply-and-release step. For array submissions, array_identity_exists()
does one read-locked scan of the 32768 tagged element keys of the
array's block (element 0 included for pre-upgrade arrays), covering
both the element range and the array identity; unmarked submissions
keep jobid_range_exists. An existing X|3 now blocks a new X|5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
handle_jobsubmit classifies (first_jobid, array_size) before the field
checks, the before-event hook and any registry access: a tagged array
starting at index 0 is answered with the new E_XCE_ARRAY_START0 and an
ERROR log asking for the client to be upgraded (7.1, wins per 7.9);
size 0, tags 2/3, untagged multi-element submissions and arrays whose
last element exceeds 32767 (checked without addition) get -EINVAL.
Nothing is stored for a rejected submission. This also closes a crash:
a size-0 submission used to reach payload building and overflow calloc.
Existing array tests move to tagged block|1 first IDs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A worker reported job start, finish and failure with a fire-and-forget
OPCode_notify the controller never answers, so a report lost while the
worker stayed connected (the live incident: 6 of 7 finish reports in a
burst) left the job RUN forever; the rejoin resync (165fc158) only
recovers reports lost around a reconnect.
- controller.capnp: OPCode_notifyAcked -- the same NotifyData body,
answered. A separate opcode because replies are matched through the
client's single wbuf, not by sequence number: answering plain notifies
would feed stale statuses to the next synchronous request.
- controller (cluster.c): do_notify_acked() applies the fact like
do_notify, then waits on the commit barrier and replies 0 only once it
is majority-committed. A barrier failure (the fact is still appended and
retained) replies -E_XCE_NOT_READY, so the worker retries in place
instead of rotating away from a healthy primary (cf. bug-1850). Admitted
by the no-quorum fact-bearing gate like OPCode_notify.
- libxce: xce_notify_acked_req/xce_notify_acked (send, wait for the
status) and xcecli_notify_acked (XCECLI_FAILOVER_CALL retry/failover).
xcecli_notify_jobstart/jobfinish/jobfail use it; on -E_XCE_BAD_OPCODE
(a controller predating the opcode) they fall back to the plain
fire-and-forget notify, so workers and controllers can be upgraded in
either order.
- Resends are safe: START/FINISH are duplicate-guarded and a repeated FAIL
yields the same state.
- worker: logs a WARN when a report is still unacknowledged after the
retry budget (the rejoin resync remains the backstop). These calls run
on the worker daemon's IPC callback, so while the controller lacks
quorum other jobs' IPC messages wait (buffered, not lost) up to the
retry budget.
Tests: test_notify_acked_reply_only_after_commit and
test_notify_acked_withheld_replies_not_ready (no reply before the covering
ACK; -E_XCE_NOT_READY with the fact retained when ACKs are withheld).
Written with the opcode, so no separate RED run. Full /xce-build all:
111 suites pass; controller suite 1245/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
"ERROR incorrect worker idx -1" on every node after a worker-join commit
timeout (during a PreVote/election) was the visible end of a real
dangling pointer. init_worker/rejoin_worker bind the endpoint to the
worker (apply_join_state: w->ep = ep, UNAVAIL cleared) before the commit
barrier; on barrier failure handle_worker_join returned without setting
priv->u.idx, so when that connection closed handle_endpoint_error saw
idx -1, logged the ERROR and never ran handle_worker_leave -- the worker
kept w->ep pointing at the freed endpoint, available for dispatch, until
it rejoined.
- handle_worker_join: on barrier failure, clear w->ep (if still this
endpoint) and mark the worker unavailable, as if the join never
happened; the worker retries its join.
- handle_endpoint_error: an idx -1 worker endpoint is now the legitimate
"closed before its join completed" case -- WARN, nothing to clean up.
test_worker_join_withheld_acks_replies_error also asserts the worker is
unbound and unavailable after the -ETIMEDOUT reply (RED before).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two independent gaps in misc.c/job.c, both silently zeroing every real
job's resource-usage data:
- update_host_runusage() matched worker rusage report resnames
"cputime"/"mem", but the real worker's collector
(collect_job_rusages(), xcecd_cgroups.c) sends "cpu_time_sec"/
"memory_rss" -- the two never matched, so cpu_time/mem_usage stayed
at their calloc()'d zero for every non-simulated job. Aligned both
sides, including xce_test's own simulated-worker fixture.
- update_job_rusages() had a standing FIXME: xce_job_runusage_t's
job-level max_mem/avg_mem/used_mem/cpu_time were never derived from
the per-host host_runusage[] entries. Added aggregate_job_runusage()
(sum/max/avg across hosts) and wired it in right after the per-host
update.
Added test_host_runusage.c (standalone unit test, mirrors
resval_typeresolve_unit_test's isolation pattern) plus the
controller_unit_test stub aggregate_job_runusage() needs since that
target excludes the real misc.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Same class as bug-1678: apply_job_start_transition() sets the real
job->job.pidinfo (worker-reported PIDs/PGIDs/exdata) and start_time, but
the WAL fact for WAL_JOB_START only carried the shared wal_jobstate_t
{jobid, state} header -- both apply seams (apply_committed_entry() for
primary boot replay, apply_wal_entry() for standby catch-up) restored
state only. pidinfo stayed permanently NULL after a restart; start_time
was fabricated with time(NULL) at replay instead of being carried,
drifting from the true value.
Mirrors bug-1678's exact fix shape: capnp_encode_pidinfo()/decode_pidinfo()
(persist.c, using the pre-existing JobPIDInfo capnp struct -- no schema
change needed) serialize job_pidinfo_t into the WAL payload, trailing a
new uint64_t start_time field appended after wal_jobstate_t.
persist_submit_job_start_fact() replaces the old state-only
persist_submit_job_fact() call for this type. Both apply seams updated;
a payload of exactly sizeof(wal_jobstate_t) (any entry written before
this fix) decodes with pidinfo left NULL and start_time fabricated via
time(NULL), exactly matching pre-fix behavior -- full backward
compatibility, verified by every pre-existing WAL_JOB_START test using
the old bare-struct shape.
Also fixed a genuine, unrelated deadlock this change's larger payload
exposed in test_commit_barrier_reply.c: cbrp_count_jobstate_entries_ex()
matched WAL entries by exact payload size (==), so once WAL_JOB_START
legitimately grew it silently undercounted to 0 and a downstream test
spun forever polling an uncounted target. Fixed the match to >= (jobid/
state always live at a fixed leading offset) -- bisected via /proc/*/wchan
futex inspection (no ptrace in this sandbox), confirmed as the actual
cause, not the teardown-ordering red herring first suspected.
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension, added after bug-1678). 6 new tests in
test_wal_job_start_fact.c; RED/GREEN independently verified (4 tests
genuinely failed pre-fix with exact expected-vs-fabricated value
mismatches, 2 backward-compat tests correctly passed both before and
after). Full controller suite: 1146 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Fixes test_commit_barrier_reply.c's test_jobdispatch_receipt_ack_no_wal_append
(owned by ha-commit-barrier-rollout), the last remaining assertion this
feature's earlier tasks left stale:
- WAL-count delta corrected from "must be zero" to lsn_before + 1 -- the
batch's successful dispatch now appends one WAL_JOB_DISPATCH fact
(task 3.1); mark_job_fail() was traced and confirmed to append
nothing via any path.
- The batch's terminal-failure job was replaced with a genuinely
retryable one (WORKER_NULL site, unregistered host, new job id)
rather than degrading the notify-count assertion to zero, which
would also have silently killed the test's own reply-before-notify
ordering pin (nothing left to order against). This keeps the test
meaningfully exercising notify_dispatch()'s wiring through
handle_jobdispatch().
Full regression, independently re-verified by review:
- Scheduler suite at documented baseline (51/52, 1 pre-existing
unrelated failure). Investigated and logged bug-1632, a new sibling
of the already-known bug-1598 stack-lifetime pattern in a different
test file -- confirmed pre-existing, untouched by this feature, and
non-deterministic.
- Controller suite clean: 1133 tests, only the 2 pre-existing bug-1561
failures.
- Sequential HA soak: PASS (10 rounds, 36 jobs, zero loss, all 3 nodes
led).
- Chaotic HA soak: PASS (10 rounds, committed_divergence=0
committed_job_loss=0).
This completes the scheduler-dispatch-commit spec: bug-1624
(resource-in-use round-boundary loss) and bug-1625 (scheduler discards
dispatch-failure notify) are both fixed and regression-tested.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
The four retryable-classified failure sites in dispatch_a_job_() (worker
lookup, worker null, worker not ready, RPC failure) now resolve instead
of just returning an error code: lock(j, RTTI_LOCK_WRITE) -> genuinely
classify via xce_classify_dispatch_fail() -> increment dispatch_fail_count
-> within XCE_DISPATCH_FAIL_RETRY_BOUND (3) resolves retryable (nothing
further under the lock), past it resolves terminal via mark_job_fail()
exactly like task 3.2's sites -> unlock(j) -> dereference(j) (closing
bug-1627's 4-site reference leak) -> outside the lock, notify_dispatch()
fires only on the retryable outcome. Each site's natural error code is
captured into a stable local before any intervening call, since errno
isn't guaranteed stable once lock()/ERROR()/free_launchinfo() sit between
the original failure and the eventual return.
handle_jobdispatch() is intentionally untouched -- it still unconditionally
calls notify_dispatch() itself for every outcome, so the retryable path
now temporarily double-notifies until task 4.1 removes that caller-side
call. Expected, not a defect.
New test_dispatch_fail_retry.c (4 tests, including a bounded-escalation
test driving 4 consecutive RPC failures on the same job: 3rd retries,
4th resolves EXIT) plus a fixed stale assertion in test_dispatch_commit.c
and additive, read-only notify-capture accessors in
test_commit_barrier_reply.c (the binary's sole notify() stub host).
Independently reviewed and approved, including explicit scrutiny of the
cross-file test accessor addition and the stale-assertion fix -- both
confirmed legitimate, not scope creep or regression-hiding.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
Bundles the fixes validated this session across ha.c/persist.c/worker.c:
- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
closed with a recovery-complete gate that closes at process-lifecycle
start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
check mirroring the existing job-count check (count_jobmgr_entries and
count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
applies mark_worker_unavail()'s mutations via a shared
apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
fixed for promoted vote endpoints via a shared votes_idx_for_peer()
index-scale converter.
Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds test_batch_payload_persist_failure_propagates_to_whole_batch,
proving persist_submit_job_payloads_async resolves every payload in a
3-entry batch with the same first-failure rc when the first payload's
append fails, rather than letting later payloads independently
succeed (Req 5.3).
Fault injection: chmod the WAL leaf directory to 0500 to force
open_new_segment's EACCES (surfaced as -EIO), restored the instant the
first failure is observed -- a transient fault, not a permanent one,
so the test genuinely distinguishes propagation from independent
per-payload failure. No existing fault hook was reachable from this
code path; this is a real, previously-uncovered gap.
1066 -> 1067 tests, 0 failures. Task 5 (regression-lock tests for
already-shipped/already-conformant behavior) now complete -- Core
phase of raft-adapter-conformance finished (tasks 1-5).
Only 21 of 86 test_*.c group functions called their own nested
UNITY_BEGIN()/UNITY_END(). Each nested call reset Unity's global
NumberOfTests/TestFailures counters, so controller_test.c's final
`return UNITY_END();` -- the value meson test actually gates on --
only ever reflected whichever small window of tests ran since the
LAST nested reset, silently discarding pass/fail evidence for
everything before it (measured: 16 of 972 total tests in one run).
The same nested-BEGIN calls also caused PASS/FAIL banners to show
the wrong source file for tests in files that don't reset it
themselves (Unity.TestFile just carries over from whichever group
last called UNITY_BEGIN()).
Fix: replace every nested UNITY_BEGIN() with UnitySetTestFile(__FILE__)
(sets file attribution only, no counter reset) and drop the matching
UNITY_END(); add the same UnitySetTestFile(__FILE__) as the first
statement to the other group functions that had neither. The 2
genuinely standalone binaries (test_job_array_sched_push.c,
test_job_array_apiserver_push.c -- each has its own int main(), not
registered via controller_test.c's TEST() macro) are untouched.
Verified: single consolidated summary now reads
"972 Tests 0 Failures 0 Ignored OK", exactly matching the
ground-truth RUN_TEST call-site count across all 83 registered
group files (976 total minus 4 belonging to the 2 standalone
binaries). Every PASS/FAIL banner now names its own true source
file. Full xce-build.sh all regression pass clean (Ok: 9, Fail: 0).
No production code touched -- test-harness plumbing only.
Branches handle_jobsubmit() on array_size: ==1 continues through
today's exact unmodified single-job path; >1 routes through a new
handle_jobsubmit_array() helper that reuses jobsubmit_async_ctx_t
with pending=1 and calls task 2.1's persist_submit_job_array_async()
instead of looping N single-element appends. Range validation
(jobid_range_exists/inflight_range_try_add) unchanged, same order,
strictly before any WAL append, for both branches.
Retires two pre-existing tests whose premise (N independent
per-element WAL entries with mixed outcomes) is now structurally
impossible with one entry per array; restores array-scoped
job-non-existence coverage for the failed-commit case that an
earlier draft of this change had dropped (Req 2.2/9.2).
Task 2.2 of job-array-single-wal-record.
Root cause (confirmed via targeted DIAG1419 logging on a live chaotic soak):
a client's commit-timeout retry during a genuine quorum-loss window appends
a second WAL_JOB_SUBMIT for the same jobid; the timed-out entry is never
retracted, so if it later achieves majority commit anyway both entries end
up committed. This is a legitimate, harmless duplicate — restore_job()'s own
-EEXIST correctly no-ops the second one — but replay_wal() tallied every raw
WAL_JOB_SUBMIT occurrence instead of genuine creations, so the duplicate
made new_job_submits disagree with jobmgr's real growth, firing a false
APPLY_DIVERGENT and deadlocking promotion.
Reverts the layer-3 claim/release machinery (job_wal_commit_claim/release,
claim_committed_job_submits, and their wiring): three attempts at preventing
the duplicate at submit/dispatch/commit time all failed to close the gap,
because the duplicate isn't a bug to prevent — a WAL entry can't be
retracted once appended, so any submit-time guard is structurally the wrong
layer.
Real fix: apply_committed_entry()'s WAL_JOB_SUBMIT case now returns 1 for a
genuine creation vs 0 for an idempotent no-op (restore_job() -EEXIST), and
replay_wal() only tallies new_job_submits_out on rc==1. Updates the affected
apply-seam/reapply-idempotency tests for the new return contract, and
reverses two apply-correctness tests whose premise (duplicate-jobid replay
must diverge) is now understood to be the actual bug rather than the
expected behavior — they now force divergence directly via the same
production fencing seam instead of relying on that trigger.
Verified: two consecutive 10-round chaotic soaks, zero divergence, zero job
loss, no election stall (previously reproduced within ~5 rounds).
jobid_range_exists() (c->jobmgr) and the in-flight registry (job.c,
per-process, only populated by this node's own handle_jobsubmit calls)
both miss a real cross-node case: a client's XCECLI_FAILOVER_CALL retry
resends the same client-generated jobid after a connection drop, and the
resend lands on a newly elected primary that never processed the original
request. If the original was already committed before the old primary
died, the new primary's own local WAL already holds it (Raft Leader
Completeness) but jobmgr hasn't applied it yet — both existing guards see
nothing, so the resend appends a second, genuinely committed WAL_JOB_SUBMIT
entry for the same jobid. A live chaotic soak reproduced this exactly
(waldump: two committed entries, same term, matching apiserverId/timestamp/
sequence, one second apart) — apply's own idempotent dedup keeps jobmgr
correct, but replay's raw WAL_JOB_SUBMIT count no longer matches jobmgr's
growth, firing a false APPLY_DIVERGENT fence and deadlocking promotion.
Fix: apply_committed_entry()'s WAL_JOB_SUBMIT case now claims the jobid in
job.c's in-flight registry (new job_wal_commit_claim/release, wrapping the
existing inflight_range_try_add/remove) for the duration of dispatch,
regardless of whether this node originated the submission or is applying a
replicated/replayed entry. A resend landing mid-dispatch on any node now
sees the same -EEXIST rejection self-originated in-flight submits already
get. Release only fires when the claim call added the entry, so a
self-originated submission's own claim/release lifecycle (unchanged) is
never double-released or stolen.
Also fixes 4 pre-existing tests that broke under the earlier checkpoint_lsn
fix (0eb1054) and were missed in that commit's regression sweep: cluster-
less fixtures where last_applied_lsn never advances now correctly get
checkpoint_lsn=0 unless the test simulates apply catch-up, matching the
new (correct) semantics. One assertion in
test_checkpoint_tail_with_no_snapshot_is_apply_clean compared Phase 1's
checkpoint_lsn against Phase 2's independently-reconstructed physical WAL
tip — that equality only held under the pre-fix current_lsn-tracking
behavior and is removed, not patched around.
Also hardens run-ha-soak.sh's start_node() with setsid+nohup: a forked/
background test-harness invocation whose session tears down mid-run was
silently killing every controller process via SIGHUP forwarding (observed
twice — deploy completes, ~1s of activity, then all 3 nodes die together
with no error logged).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
xce_test's cmd_jsub hardcoded idgen(&idctx, 0, 0, ...) (clusterId=apiserverId=0),
and run-ha-soak.sh's submit_jobs() spawns a fresh xce_test process per retry
attempt, resetting idgen's static sequence counter each time. Two unrelated
job submissions from separate process launches landing in the same
wall-clock second collided on an identical jobid, surfacing as a spurious
APPLY_DIVERGENT fence in chaotic soak that was initially misattributed to
bug-1419's controller-side dedup logic.
Confirmed not a production risk: apiserver's GenerateJobID holds one
process-lifetime idgen_ctx_t via sync.Once, and each deployed instance has
its own distinct apiserverId, so config.h's jobid bit layout already
partitions the production namespace. Fixed by folding getpid() into the
harness's apiserverId so cross-process collisions become structurally
impossible there too.
Also adds test_jobsubmit_resubmit_after_full_resolution_rejected_no_second_job
to test_commit_barrier_reply.c, proving the post-resolution dedup window
(retry arriving after commit+apply+reply+release) is airtight, complementing
the existing before-resolution test -- both known bug-1419 dedup windows are
now provably race-free, confirming those fixes were never the cause of this
soak failure.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
jobid_range_exists() only checks c->jobmgr, but task 3.1 deliberately
defers job construction until after commit -- so a submission that is
durably WAL-appended and awaiting its commit outcome is invisible to
that check. A client-side retry (XCECLI_FAILOVER_CALL resending the
same jobid after a transport error or failover mid-flight) sails past
the guard and appends a second, non-identical WAL_JOB_SUBMIT entry for
the same jobid -- confirmed via waldump on a live chaotic-soak node
(duplicate submits, same jobid, differing subtime/CRC) -- which can
trip persist_recover()'s post-replay apply-correctness check into a
genuine APPLY_DIVERGENT verdict.
Adds a process-lifetime in-flight-jobid-range registry in job.c that
handle_jobsubmit atomically check-and-claims immediately after the
existing jobid_range_exists check, releasing from
jobsubmit_async_ctx_release (the single choke point already shared by
the commit-continuation, apply-complete, and OOM early-out paths). A
resubmit for a still-in-flight jobid now gets -EEXIST with no second
WAL append.
Adds a matching XCE_TESTING reset seam (declared in cluster.h) so a
test that deliberately leaves a submission's commit continuation
unresolved through teardown doesn't leak its jobid range into a later
test reusing the same range -- commit_engine_fini frees pending
continuations without invoking their callback.
Regression test added to test_commit_barrier_reply.c (RED->GREEN
confirmed). Full 78-group controller_unit_test sweep: 782 tests, 0
failures.
This fixes only the WAL duplicate-submit mechanism (layer 1 of
bug-1419). The deeper PreVote-freshness/apply-divergence liveness
deadlock (layer 2 -- no recovery path when the only fresh-enough node
is apply-divergent-fenced) remains open and needs its own design
discussion; buglog.json updated accordingly.
Full 79-group unit/integration suite re-run found exactly 2 stale tests in
test_commit_barrier_reply.c asserting task 3.2's documented interim
contract ("no observable effect until task 6 completes it"), which the
already-committed ordered apply pump (tasks 6.1/6.2) closed by construction.
Updated both to assert the real, deterministic apply-after-commit settled
state instead of the no-longer-true no-effect-yet state, matching the idiom
already used elsewhere in the same file. No production code changed.
Full suite: 0 failures across 79 groups. HA soak (--sequential, 5 rounds):
zero committed-job loss, zero divergence, all 3 nodes served as primary.
Both independently re-verified by adversarial review.
Wire persist_apply_pump_run() to call persist_take_pending_apply()
immediately after each entry's apply_committed_entry() call,
invoking the registered continuation strictly after the mutation.
Adds jobsubmit_reply_success_once() (job.c) so a successful job
submission finally sends a client reply once every array element
applies -- closes the standing gap from task 3.2 where committed
submissions never got a response.
Fixes bug-1415: commit_resolve_pending() dispatches the commit
continuation and the apply-pump kick as two independent
VTASK_LIVE_BLOCK vtasks with no completion-order guarantee. On any
real multi-vcpu pool (the production default, n_vcpus=8), if the
apply pump won the race and advanced last_applied_lsn past an LSN
before its continuation registered, the registration was permanently
orphaned -- the job would be constructed and visible, but the client
would hang forever. Untested by every existing fixture in this spec
(all use pool=NULL, which trivially serializes and can't reproduce
the race).
Fixed with a self-healing check in persist_register_pending_apply():
if the LSN is already <= last_applied_lsn, invoke the callback
immediately instead of registering for later pickup. Verified by the
spec's first genuine multi-vcpu regression test (real vtaskmgr_t,
n_vcpus=4), independently RED->GREEN reproduced by two review rounds.
Build persist_apply_pump_run()/persist_apply_pump_kick() -- the
ordered walk from last_applied_lsn+1 through committed_lsn, applying
every entry via apply_committed_entry() regardless of commit mode.
This is the first live (non-replay) consumer of that dispatch
function. Kick is wired into commit_resolve_pending() (the single
choke point all 6 internal committed_lsn-advancing paths funnel
through, discovered in task 1.2), dispatched via
vtask_enq_classified(VTASK_LIVE_BLOCK) with a CAS-coalesced
fire-and-forget trigger.
Fixes the two hazards flagged since task 4.1 as required before
going live:
1. Double-append: a thread-local call-provenance guard
(t_apply_seam_notifying) prevents persist_job_hook_fun/
persist_queue_hook_fun/persist_worker_hook_fun from re-appending
entries the apply pump itself just applied, while preserving
job_ctrl's legitimate live command-path appends (a naive
detail-based filter would have broken SIGCONT/SIGSTOP). Also
closes an undocumented instance of the same hazard in the
queue/worker hook functions.
2. Merge-staleness: record_job_fail_fact/postdone's fact recording
previously baked a receipt-time-computed final job state into the
WAL entry, which goes stale under deferred apply -- a FINISH
followed shortly by an ordinary post-exec FAIL could silently
clobber a correct DONE completion with a stale RUN|EXIT|PERR, no
guard catching it. Fixed by repurposing wal_jobstate_t's unused
_pad field to carry just enough data (FAIL's stage) to recompute
against the job's actual current state at apply time via the real
apply_job_fail_transition()/new apply_job_postdone_transition()
guard cores, instead of overwriting with a stale absolute value.
Backward compatible: _pad==0 (all prior data) routes to the same
legacy overwrite behavior.
Also fixes bug-1409 (unbounded busy-loop in apply_pump_drain when
committed_lsn advances without a corresponding on-disk entry) and a
genuine test-flakiness hazard (a second, independent apply-pump
kicker from the fsync_worker thread that existing tests didn't
account for).
Two pre-existing tests in test_commit_barrier_reply.c now fail as
expected -- their own assertions state "no job may be constructed
until task 6's apply pump exists," now correctly invalidated by this
task landing; updating them is task 7.3's explicit job.
Add test_fact_state_unchanged_throughout_quorum_loss_outage to
test_commit_barrier_reply.c (30->31), no production changes. Drives
a real repl_remove_standby quorum-loss window through all 4
fact-receipt handlers, polls live state and committed_lsn at 5
time-spread points (not once), asserts a real hook-spy never sees
AFTER_EVENT fire, and confirms durable WAL presence via direct
on-disk read.
Documents an architectural asymmetry for task 6 planning: facts have
no commit-timeout concept (COMMIT_MODE_RETAIN registers no
continuation, "never honors the timeout") unlike commands'
commit_submit_async -- the apply pump must discover newly-committed
retained facts purely via committed_lsn advancement, not any
per-fact signal.
Closes out task-group 4 (fact apply-after-commit split).
Convert on_job_start/on_job_finish/on_job_fail/on_job_postdone to
validate and durably append only -- BEFORE_EVENT validation hook
unchanged, AFTER_EVENT (post-mutation) hook no longer fires at
receipt, live state mutation removed. Adds persist_submit_job_fact()
(persist.c) since the durable append previously lived inside the
very AFTER_EVENT hook this task silences; wire format is unchanged
and identical to what apply_committed_entry() already routes.
mark_job_start/mark_job_finish (single caller each) renamed in place
to record_job_start_fact/record_job_finish_fact. mark_job_fail/
mark_job_update (shared with job_ctrl's client-command path, out of
boundary) left untouched; new durable-append-only siblings
record_job_fail_fact/record_job_update_fact added instead.
Accepts a narrow, permanent regression: on_job_postdone's finish_time
anomaly backstop has no wire-format equivalent and is dropped
(affects only a rare no-prior-exit-status path).
Two critical hazards discovered by review, documented in tasks.md,
MUST be resolved before task 6 wires apply_committed_entry() into a
live path: (1) persist_job_hook_fun still fires on HOOK_DETAIL_JOB_UPDATE
and will double-append every live-applied fact; (2) record_job_fail_fact/
record_job_update_fact precompute a final state that can be silently
clobbered by a concurrent intervening fact once apply is deferred by
the commit round-trip.
Add 4 tests to test_commit_barrier_reply.c (25->29), no production
changes: explicit-FAILED via truncation (distinct from timeout),
repeated failure loops (5 cycles) for both timeout and truncation,
and an array_size>1 mixed-outcome test proving a committed sibling's
apply hand-off isn't blocked by failing siblings while producing
zero phantom jobs for the failures and exactly one batch-level
failure reply.
Closes out task-group 3 (command apply-after-commit ordering).
Replace handle_jobsubmit()'s blocking commit call (task 3.1) with a
real async registration against commit_submit_async(). On
FAILED/timeout, replies failure immediately with no job constructed.
On COMMITTED, hands off to a new pending-apply registration
(persist_register_pending_apply) for the ordered apply pump (task
6.1, not yet built) to consume -- does not apply or reply directly,
per design.
Deviates from design.md's stated integration site: appends via the
same ha_primary_log_append/persist_wal_async primitive the RETAIN-fact
path already uses and calls commit_submit_async directly on
p->repl.ce, bypassing repl_wait_majority_ack/repl_submit_majority_commit
in repl.c. Verified safe -- majority-ack detection is driven entirely
by (ce, lsn) events inside the commit engine, agnostic to which
wrapper registered the continuation -- but narrows the reusable
primitive design.md intended; documented in tasks.md.
Deletes (not preserves-dead) task 3.1's bug-1408 notification fix --
apply_committed_entry()'s WAL_JOB_SUBMIT case (task 2.2) already
fires the notification per-entry, immune to the batch-suppression
class bug-1408 fixed.
IMPORTANT: no live path applies a successfully-committed job
submission until task 6 lands (only restart replay does) -- every
successful submission commits durably but gets zero reply until
then. This commit range must not be soak-tested or deployed against
live traffic before task 6 completes.
Reorder handle_jobsubmit() so job objects are constructed via
create_job() only after the WAL commit succeeds, not before. Adds
jobid_range_exists() for a fast-fail pre-check with no WAL work, and
persist_submit_job_payloads_and_commit() (persist.c) operating on raw
xce_job_t* payloads instead of pre-existing job_t*. Commit remains
synchronous -- converting to the continuation-based commit API is
task 3.2.
Fixes bug-1408 (found during review): the post-commit array-
construction loop suppressed notification for an entire array batch
if any one sibling element failed to construct, stranding other
durably-committed and successfully-constructed jobs invisible until
the next restart's replay. Notification now fires unconditionally
for every successfully-constructed element, independent of sibling
failures; the client reply still separately reports the batch's
construction status.
Root cause (found via live 3-node E2E + a verification test): lsn was 0-based
(hdr.lsn = current_lsn++ from current_lsn=0), and 0 was overloaded as a
sentinel, causing three cold-start defects on a freshly-promoted leader:
- the first WAL entry (own-term promotion NOOP at lsn 0) was structurally
uncommittable: commit_advance_locked only stores when cand>cur and
committed_lsn starts at 0, so lsn 0 can never advance it;
- commit_set_writable_lsn(noop_lsn=0) set writable_lsn=0, which wait_done
reads as 'gate off' — the own-term safety gate was a no-op on the first leader;
- bug-543 (WAL hole): ha_stream_wal_to_standby filters hdr.lsn>standby_lsn;
a fresh empty standby reports standby_lsn=0 (ambiguous 'empty' vs 'has lsn 0'),
so lsn 0 failed 0>0 and was dropped -> non-contiguous standby WAL.
Fix: a fresh persist store initializes current_lsn=1, reserving lsn 0 as the
none/empty sentinel. The first real entry is lsn 1 -> committable (0->1),
writable_lsn=1 -> gate effective, and standby_lsn=0 unambiguously means 'empty'
so catch-up streams from the first entry with no hole. persist_recover is
data-driven (current_lsn=max(hdr.lsn+1)), so existing on-disk WALs and non-fresh
recovery are unaffected; the change only sets the base for brand-new stores.
No wire/capnp change.
Verification: test_coldstart_lsn0 drives the real commit engine + real streamer
and proves H1 (first entry commits 0->1), H3 (gate effective + blocks when
committed<writable), H4 (empty-standby catch-up is contiguous [1,2], control
standby_lsn=1 gets only [2]). Full controller suite green under ASAN. Tests that
relied on the lsn-0 base were shifted (several strengthened); commit.c /
config_repl.c / ha.c / wire format untouched.
Owning specs: ha-commit-engine / ha-wal-replication / persist. Follow-up:
sweep stale lsn-0 comments (repl.c:429-436, persist.c:1055-1075/1240-1243/1296).
3 RED carriers (duplicate FINISH on DONE appends a 2nd WAL entry +
overwrites state; START on terminal clobbers it; replay non-equivalent)
+ 2 preservation pins (forward postdone PDONE-merge, fail-after-finish
PERR-merge) that 6.2 must not break. Reviewer-confirmed: the
UPDATE-clears-terminal arm is production-unreachable (pinned, not
driven). 22 tests / 3 RED; census 508 distinct triples, only those 3
fail.
Green integration proof: 4 facts driven during total quorum loss append
RETAIN with rc 0 in <1000ms (vs 5000ms timeout — wrongly-applied WAIT
tripwire), pending with retained-depth=4 logged, present exactly once
on disk (real WAL header scan); reconnect + one covering ACK commits
the suffix with current_lsn unchanged (no redelivery). Completes the
bug-396 fix evidence (umbrella Req 2.8). Census 495/495.
4 green pins: handle_jobdispatch replies receipt-only with zero WAL
appends through a real 2-alloc dispatch cycle (notify outcomes intact,
reply-precedes-processing); set/clr_preason and do_update_op rusage
stay receipt-style with RETAIN-pending fact appends ACK-committed.
Umbrella Reply-Ordering Contract cited per pin. DISCOVERY bug-451
(reviewer-verified): RUSAGE hook va_list type confusion — 4-byte OOB
read + garbage state in WAL_JOB_RUSAGE payloads; fix deferred to 7.1.
Census 488; zero failures.
