jobid_range_exists() only checks c->jobmgr, but task 3.1 deliberately
defers job construction until after commit -- so a submission that is
durably WAL-appended and awaiting its commit outcome is invisible to
that check. A client-side retry (XCECLI_FAILOVER_CALL resending the
same jobid after a transport error or failover mid-flight) sails past
the guard and appends a second, non-identical WAL_JOB_SUBMIT entry for
the same jobid -- confirmed via waldump on a live chaotic-soak node
(duplicate submits, same jobid, differing subtime/CRC) -- which can
trip persist_recover()'s post-replay apply-correctness check into a
genuine APPLY_DIVERGENT verdict.
Adds a process-lifetime in-flight-jobid-range registry in job.c that
handle_jobsubmit atomically check-and-claims immediately after the
existing jobid_range_exists check, releasing from
jobsubmit_async_ctx_release (the single choke point already shared by
the commit-continuation, apply-complete, and OOM early-out paths). A
resubmit for a still-in-flight jobid now gets -EEXIST with no second
WAL append.
Adds a matching XCE_TESTING reset seam (declared in cluster.h) so a
test that deliberately leaves a submission's commit continuation
unresolved through teardown doesn't leak its jobid range into a later
test reusing the same range -- commit_engine_fini frees pending
continuations without invoking their callback.
Regression test added to test_commit_barrier_reply.c (RED->GREEN
confirmed). Full 78-group controller_unit_test sweep: 782 tests, 0
failures.
This fixes only the WAL duplicate-submit mechanism (layer 1 of
bug-1419). The deeper PreVote-freshness/apply-divergence liveness
deadlock (layer 2 -- no recovery path when the only fresh-enough node
is apply-divergent-fenced) remains open and needs its own design
discussion; buglog.json updated accordingly.
Wire persist_apply_pump_run() to call persist_take_pending_apply()
immediately after each entry's apply_committed_entry() call,
invoking the registered continuation strictly after the mutation.
Adds jobsubmit_reply_success_once() (job.c) so a successful job
submission finally sends a client reply once every array element
applies -- closes the standing gap from task 3.2 where committed
submissions never got a response.
Fixes bug-1415: commit_resolve_pending() dispatches the commit
continuation and the apply-pump kick as two independent
VTASK_LIVE_BLOCK vtasks with no completion-order guarantee. On any
real multi-vcpu pool (the production default, n_vcpus=8), if the
apply pump won the race and advanced last_applied_lsn past an LSN
before its continuation registered, the registration was permanently
orphaned -- the job would be constructed and visible, but the client
would hang forever. Untested by every existing fixture in this spec
(all use pool=NULL, which trivially serializes and can't reproduce
the race).
Fixed with a self-healing check in persist_register_pending_apply():
if the LSN is already <= last_applied_lsn, invoke the callback
immediately instead of registering for later pickup. Verified by the
spec's first genuine multi-vcpu regression test (real vtaskmgr_t,
n_vcpus=4), independently RED->GREEN reproduced by two review rounds.
Build persist_apply_pump_run()/persist_apply_pump_kick() -- the
ordered walk from last_applied_lsn+1 through committed_lsn, applying
every entry via apply_committed_entry() regardless of commit mode.
This is the first live (non-replay) consumer of that dispatch
function. Kick is wired into commit_resolve_pending() (the single
choke point all 6 internal committed_lsn-advancing paths funnel
through, discovered in task 1.2), dispatched via
vtask_enq_classified(VTASK_LIVE_BLOCK) with a CAS-coalesced
fire-and-forget trigger.
Fixes the two hazards flagged since task 4.1 as required before
going live:
1. Double-append: a thread-local call-provenance guard
(t_apply_seam_notifying) prevents persist_job_hook_fun/
persist_queue_hook_fun/persist_worker_hook_fun from re-appending
entries the apply pump itself just applied, while preserving
job_ctrl's legitimate live command-path appends (a naive
detail-based filter would have broken SIGCONT/SIGSTOP). Also
closes an undocumented instance of the same hazard in the
queue/worker hook functions.
2. Merge-staleness: record_job_fail_fact/postdone's fact recording
previously baked a receipt-time-computed final job state into the
WAL entry, which goes stale under deferred apply -- a FINISH
followed shortly by an ordinary post-exec FAIL could silently
clobber a correct DONE completion with a stale RUN|EXIT|PERR, no
guard catching it. Fixed by repurposing wal_jobstate_t's unused
_pad field to carry just enough data (FAIL's stage) to recompute
against the job's actual current state at apply time via the real
apply_job_fail_transition()/new apply_job_postdone_transition()
guard cores, instead of overwriting with a stale absolute value.
Backward compatible: _pad==0 (all prior data) routes to the same
legacy overwrite behavior.
Also fixes bug-1409 (unbounded busy-loop in apply_pump_drain when
committed_lsn advances without a corresponding on-disk entry) and a
genuine test-flakiness hazard (a second, independent apply-pump
kicker from the fsync_worker thread that existing tests didn't
account for).
Two pre-existing tests in test_commit_barrier_reply.c now fail as
expected -- their own assertions state "no job may be constructed
until task 6's apply pump exists," now correctly invalidated by this
task landing; updating them is task 7.3's explicit job.
Convert on_job_start/on_job_finish/on_job_fail/on_job_postdone to
validate and durably append only -- BEFORE_EVENT validation hook
unchanged, AFTER_EVENT (post-mutation) hook no longer fires at
receipt, live state mutation removed. Adds persist_submit_job_fact()
(persist.c) since the durable append previously lived inside the
very AFTER_EVENT hook this task silences; wire format is unchanged
and identical to what apply_committed_entry() already routes.
mark_job_start/mark_job_finish (single caller each) renamed in place
to record_job_start_fact/record_job_finish_fact. mark_job_fail/
mark_job_update (shared with job_ctrl's client-command path, out of
boundary) left untouched; new durable-append-only siblings
record_job_fail_fact/record_job_update_fact added instead.
Accepts a narrow, permanent regression: on_job_postdone's finish_time
anomaly backstop has no wire-format equivalent and is dropped
(affects only a rare no-prior-exit-status path).
Two critical hazards discovered by review, documented in tasks.md,
MUST be resolved before task 6 wires apply_committed_entry() into a
live path: (1) persist_job_hook_fun still fires on HOOK_DETAIL_JOB_UPDATE
and will double-append every live-applied fact; (2) record_job_fail_fact/
record_job_update_fact precompute a final state that can be silently
clobbered by a concurrent intervening fact once apply is deferred by
the commit round-trip.
Replace handle_jobsubmit()'s blocking commit call (task 3.1) with a
real async registration against commit_submit_async(). On
FAILED/timeout, replies failure immediately with no job constructed.
On COMMITTED, hands off to a new pending-apply registration
(persist_register_pending_apply) for the ordered apply pump (task
6.1, not yet built) to consume -- does not apply or reply directly,
per design.
Deviates from design.md's stated integration site: appends via the
same ha_primary_log_append/persist_wal_async primitive the RETAIN-fact
path already uses and calls commit_submit_async directly on
p->repl.ce, bypassing repl_wait_majority_ack/repl_submit_majority_commit
in repl.c. Verified safe -- majority-ack detection is driven entirely
by (ce, lsn) events inside the commit engine, agnostic to which
wrapper registered the continuation -- but narrows the reusable
primitive design.md intended; documented in tasks.md.
Deletes (not preserves-dead) task 3.1's bug-1408 notification fix --
apply_committed_entry()'s WAL_JOB_SUBMIT case (task 2.2) already
fires the notification per-entry, immune to the batch-suppression
class bug-1408 fixed.
IMPORTANT: no live path applies a successfully-committed job
submission until task 6 lands (only restart replay does) -- every
successful submission commits durably but gets zero reply until
then. This commit range must not be soak-tested or deployed against
live traffic before task 6 completes.
Reorder handle_jobsubmit() so job objects are constructed via
create_job() only after the WAL commit succeeds, not before. Adds
jobid_range_exists() for a fast-fail pre-check with no WAL work, and
persist_submit_job_payloads_and_commit() (persist.c) operating on raw
xce_job_t* payloads instead of pre-existing job_t*. Commit remains
synchronous -- converting to the continuation-based commit API is
task 3.2.
Fixes bug-1408 (found during review): the post-commit array-
construction loop suppressed notification for an entire array batch
if any one sibling element failed to construct, stranding other
durably-committed and successfully-constructed jobs invisible until
the next restart's replay. Notification now fires unconditionally
for every successfully-constructed element, independent of sibling
failures; the client reply still separately reports the batch's
construction status.
Split mark_job_start/finish/fail/update in job.c into four hook-free
core functions (apply_job_*_transition) containing only the
monotonic-rank guard and mutation. Wrappers now peek the same guard
to decide hook-firing, then delegate to the core, preserving exact
pre-extraction hook-firing timing. Prerequisite for task 2.2's
apply_committed_entry() dispatch and the ordered apply pump (6.1).
Replaces the naive task-5.1 stopgap with the real restore contract:
detects legacy vs entity-bearing checkpoints by remaining file size;
restores every job then every queue status via restore_job/
restore_queue_state, in file order, with bounds-checked length-prefixed
reads; only after every entity restores successfully does
current_lsn/checkpoint_lsn commit to the checkpoint's position. Every
failure path (bounds check, decode, restore error, unrecognized
format_version) returns before that single commit point -- position is
provably untouched on failure, never partially advanced.
Also fixes the pre-existing restore_job decode leak (job.c): job_init
deep-copies submit (never nulled) and MOVE_PTRs allocs/pidinfo/
pend_reasons (nulled on the one path that runs) -- free_Job(&j) is safe
unconditionally on every reachable return path (EEXIST early-return,
create_job success, create_job failure), verified against the exact
generated free_*_ptr null-check guards.
Two pre-existing test_config_repl.c wire-protocol tests needed their
synthetic garbage snapshot bodies updated to a well-formed empty-entity
tail, since load_snapshot now genuinely parses past the legacy header
(previously ignored); their actual assertions (chunk-reassembly
correctness) are unchanged.
This is the exact function/contract that caused task 2.2's rejection
(position advancing before restore validated). Independent review
traced the memory-safety fix through the generated capnp code field-by-
field, confirmed the test_config_repl.c change doesn't weaken any
assertion, and confirmed restore_queue_state's ENOENT-aborts-the-whole-
checkpoint behavior is faithful to design.md's unconditional contract
(flagged as a watch-item for task 7.1's soak, not a defect).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
raft-apply-decoupling item #1 (down payment). xce_run_hooks returns only the
LAST-registered hook's rc, so a later JOB AFTER_EVENT subscriber (sched/
apiserver) returning 0 clobbered persist_job_hook_fun's commit -ETIMEDOUT.
handle_jobsubmit then replied 'jobsubmit ok' for an entry that never reached
majority, and a new-term leader legally discarded it (committed-job-loss,
Commit Durability violation).
- persist.c/.h: new persist_submit_wal_and_commit(p, jlist) — the explicit raft
step (WAL append + majority-commit barrier) returning the commit verdict
directly. persist_job_hook_fun's WAL_MODE_WAIT branch is now a no-op.
- job.c handle_jobsubmit: call the explicit step, read commit_rc DIRECTLY,
reject on <0 before firing the now notification-only AFTER hook
(canonical commit -> notify -> reply).
- tests: +test_job_hook_new_is_notify_only regression pin; 4 hook tests
relocated to drive the explicit step. persist_hooks 19/0, e2e_hooks 3/0,
commit_barrier_reply 22/0, commit_durability 38/0.
Proven: 10-round chaotic soak committed_job_loss 5 -> 0 (same election code).
Residual soak FAILs (divergence catch-up lag + a round-4 election-escalation
tail) are NOT job loss — tracked as task 5.10.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Guards at the top of mark_job_start/finish/update under the caller's
job lock, before BEFORE_EVENT: duplicate (FINISH on terminal) → INFO
no-op skipping the AFTER_EVENT hook (no WAL append, lsn frozen);
terminal→non-terminal backward (START/UPDATE) → WARN reject preserving
state. Shared job_state_is_backward predicate provably admits every
production forward merge (PDONE/PERR/suspend bit-OR) while rejecting
overwrites. mark_job_fail intentionally unguarded (its merges are
always terminal-forward; job_ctrl pre-gates terminal). 5/5 guard tests
green; full job-lifecycle suite green; census 0 FAIL.
WAL payloads:
- Job state transitions (START/FINISH/FAIL/STOP/RESUME/UPDATE etc):
wal_jobstate_t {uint64_t jobid, uint32_t state} — restores exact state flags
- Queue state (QUEUE_CTRL): wal_queuehdr_t {uint32_t status} + name string —
re-applies admin open/close/active/inactive after config queues are built
- Worker CTRL: existing [uint32_t detail][capnp xce_worker_t] unchanged
New recovery functions:
- restore_job_state(c, jobid, state): looks up job, overwrites state flags
- restore_queue_state(c, name, status): looks up queue, overwrites status
apply_wal_entry: dispatches all WAL types including job state and queue
test_repl_apply.c: weak stubs for restore_job_state/restore_queue_state
so unit test binary links without job.c/queue.c
WAL hooks now write real Cap'n Proto payloads instead of stub uint32_t:
- WAL_JOB_SUBMIT: full xce_job_t serialized via encode_Job/write_Job
- WAL_WORKER_CTRL: [uint32_t detail][xce_worker_t capnp] via encode_Worker
- Other job events (START/FINISH/FAIL/STOP etc): uint64_t jobid only
restore_job(): deserializes Cap'n Proto xce_job_t, guards against duplicate
replay with key_find check, then calls create_job()
worker_from_capnp(): skips 4-byte detail prefix, deserializes xce_worker_t,
guards against duplicate replay, then calls restore_worker()
join.c: pass worker_t* (looked up by idx via vget) to WORKER_JOIN hook
so persist hook can serialize worker data without touching workermgr RTTI
Add terminal-state constants and GC retention predicates for the
checkpoint xzarray_thaw() GC pass:
- types.h: add JOB_STAT_TRANSIENT (0x400), XCE_WORKER_HIDDEN (0x00200000),
QUEUE_STAT_HIDDEN (0x0100)
- persist.c: implement job_is_terminal(), job_should_retain(),
worker_should_retain() — placed here so the unit test binary
(which links only persist.c) can access them without RTTI deps
- persist.h: declare the three GC predicates; include xce/types.h
- worker.c: set dead_since = time(NULL) in mark_worker_unavail()
when dead_since is not already set
- cluster.c: add TODO P0-8b marker on zarray_del endpoint cleanup
- meson.build: register test_gc_retention.c; raise test timeout 60→120s
- testsuite/test_gc_retention.c: 19 tests covering job_is_terminal,
job_should_retain (TRANSIENT TTL, terminal TTL, non-terminal always
retained), and worker_should_retain (HIDDEN/UNAVAIL/healthy/zero
dead_since); all pass (63 total, 0 failures)
- cluster_def.h: add struct tag `cluster_s` to cluster_t to resolve
conflicting typedef with persist.h forward declaration
- cluster_def.h: add HOOK_DETAIL_JOB_STOP and HOOK_DETAIL_JOB_RESUME enum
values after HOOK_DETAIL_JOB_RUSAGE_UPDATE
- job.c on_job_stop(): replace mark_job_update() call with direct
xce_run_hooks(BEFORE/AFTER, HOOK_DETAIL_JOB_STOP) so WAL writer
records stop events with the correct detail value
- job.c on_job_resume(): same pattern with HOOK_DETAIL_JOB_RESUME
- job.c do_clr_preason(): fix typo — second hook call was HOOK_BEFORE_EVENT
instead of HOOK_AFTER_EVENT (double-before, zero-after bug)
- join.c handle_worker_join(): move xce_reply() to after HOOK_AFTER_EVENT
so client ACK is not sent before WAL write completes (durability fix)
- testsuite/test_hooks.c: new TDD tests for JOB_STOP/RESUME enum values
- testsuite/controller_test.c + meson.build: register test_hooks suite
Reply ordering audit results:
job.c handle_jobsubmit: reply before AFTER_EVENT — intentional (jobid
returned to client before scheduler notification; WAL hook belongs in
BEFORE_EVENT for this path, handled in Task 13)
job.c do_set_preason: reply before set_preason() loop — intentional
fire-and-forget (no AFTER_EVENT hook at top level)
job.c do_clr_preason: reply before loop — same fire-and-forget pattern
job.c do_jobcontrol_op: no AFTER_EVENT hook, reply only — OK
join.c handle_worker_join: FIXED (reply moved after AFTER_EVENT)
worker.c on_worker_update/dead: reply after all hooks — OK
cluster.c: AFTER_EVENT hooks are internal (no reply) — OK
