jobid_range_exists() only checks c->jobmgr, but task 3.1 deliberately
defers job construction until after commit -- so a submission that is
durably WAL-appended and awaiting its commit outcome is invisible to
that check. A client-side retry (XCECLI_FAILOVER_CALL resending the
same jobid after a transport error or failover mid-flight) sails past
the guard and appends a second, non-identical WAL_JOB_SUBMIT entry for
the same jobid -- confirmed via waldump on a live chaotic-soak node
(duplicate submits, same jobid, differing subtime/CRC) -- which can
trip persist_recover()'s post-replay apply-correctness check into a
genuine APPLY_DIVERGENT verdict.
Adds a process-lifetime in-flight-jobid-range registry in job.c that
handle_jobsubmit atomically check-and-claims immediately after the
existing jobid_range_exists check, releasing from
jobsubmit_async_ctx_release (the single choke point already shared by
the commit-continuation, apply-complete, and OOM early-out paths). A
resubmit for a still-in-flight jobid now gets -EEXIST with no second
WAL append.
Adds a matching XCE_TESTING reset seam (declared in cluster.h) so a
test that deliberately leaves a submission's commit continuation
unresolved through teardown doesn't leak its jobid range into a later
test reusing the same range -- commit_engine_fini frees pending
continuations without invoking their callback.
Regression test added to test_commit_barrier_reply.c (RED->GREEN
confirmed). Full 78-group controller_unit_test sweep: 782 tests, 0
failures.
This fixes only the WAL duplicate-submit mechanism (layer 1 of
bug-1419). The deeper PreVote-freshness/apply-divergence liveness
deadlock (layer 2 -- no recovery path when the only fresh-enough node
is apply-divergent-fenced) remains open and needs its own design
discussion; buglog.json updated accordingly.
Build persist_apply_pump_run()/persist_apply_pump_kick() -- the
ordered walk from last_applied_lsn+1 through committed_lsn, applying
every entry via apply_committed_entry() regardless of commit mode.
This is the first live (non-replay) consumer of that dispatch
function. Kick is wired into commit_resolve_pending() (the single
choke point all 6 internal committed_lsn-advancing paths funnel
through, discovered in task 1.2), dispatched via
vtask_enq_classified(VTASK_LIVE_BLOCK) with a CAS-coalesced
fire-and-forget trigger.
Fixes the two hazards flagged since task 4.1 as required before
going live:
1. Double-append: a thread-local call-provenance guard
(t_apply_seam_notifying) prevents persist_job_hook_fun/
persist_queue_hook_fun/persist_worker_hook_fun from re-appending
entries the apply pump itself just applied, while preserving
job_ctrl's legitimate live command-path appends (a naive
detail-based filter would have broken SIGCONT/SIGSTOP). Also
closes an undocumented instance of the same hazard in the
queue/worker hook functions.
2. Merge-staleness: record_job_fail_fact/postdone's fact recording
previously baked a receipt-time-computed final job state into the
WAL entry, which goes stale under deferred apply -- a FINISH
followed shortly by an ordinary post-exec FAIL could silently
clobber a correct DONE completion with a stale RUN|EXIT|PERR, no
guard catching it. Fixed by repurposing wal_jobstate_t's unused
_pad field to carry just enough data (FAIL's stage) to recompute
against the job's actual current state at apply time via the real
apply_job_fail_transition()/new apply_job_postdone_transition()
guard cores, instead of overwriting with a stale absolute value.
Backward compatible: _pad==0 (all prior data) routes to the same
legacy overwrite behavior.
Also fixes bug-1409 (unbounded busy-loop in apply_pump_drain when
committed_lsn advances without a corresponding on-disk entry) and a
genuine test-flakiness hazard (a second, independent apply-pump
kicker from the fsync_worker thread that existing tests didn't
account for).
Two pre-existing tests in test_commit_barrier_reply.c now fail as
expected -- their own assertions state "no job may be constructed
until task 6's apply pump exists," now correctly invalidated by this
task landing; updating them is task 7.3's explicit job.
Split mark_job_start/finish/fail/update in job.c into four hook-free
core functions (apply_job_*_transition) containing only the
monotonic-rank guard and mutation. Wrappers now peek the same guard
to decide hook-firing, then delegate to the core, preserving exact
pre-extraction hook-firing timing. Prerequisite for task 2.2's
apply_committed_entry() dispatch and the ordered apply pump (6.1).
commit_resolve_pending_above(ce, lsn_floor, rc) fails every registered
continuation for lsn > lsn_floor, reusing commit_resolve()'s existing
exactly-once/idempotent guarantee (no parallel bookkeeping path).
Truncation: a new on_truncate/truncate_cb_ctx callback on persist_t
(same inversion-of-control shape as the existing on_durable_advance
hook) fires from persist_truncate_wal_above after releasing p->lock,
only when entries were actually removed. This one wrapper covers all
three real truncation call sites: raft_cb_become_leader's
post-election truncate, the library's on_truncate_from
conflict-reconcile, and ha_demotion_convergence_fn's post-demotion
truncate.
Demotion: raft_cb_become_follower calls
commit_resolve_pending_above(&c->commit, 0, -XCE_ERR_STANDBY) --
lsn_floor=0 fails everything, since a demoted node can never
authoritatively confirm anything still pending (CP posture).
Without this, a demoted or truncated node would leave any pending
async command permanently unresolved -- the requester would simply
never get a reply, worse than today's bounded-timeout behavior.
Also closes bug-1407 (pre-existing test_config_repl flake, unrelated
to this feature) -- confirmed fixed as a side effect of prior tasks
in this series, 70/70 passing across 3 independent runs.
Converting COMMIT_MODE_WAIT away from a blocking wait removed the only
mechanism that ever detected a commit timing out (the caller's own
pthread_cond_timedwait deadline). Adds a dedicated per-commit-engine
sweep thread that arms an equivalent timeout for async-registered
continuations, resolving FAILED/-ETIMEDOUT within the same
DEFAULT_SYNC_TIMEOUT_MS window the old blocking path used, with no
thread parked to detect it.
Chose a private sweep thread over the shared delaymgr: wiring
delaymgr would touch cluster.c/ha.c (out of boundary) and its 200ms
poll granularity vs the 500ms default timeout would add jitter,
breaking the "identical failure timing" no-regression bar.
Round 1 review rejected a lost-wakeup race: the sweep loop computed
its next-deadline snapshot under lock, unlocked to dispatch, then
slept on the stale snapshot -- a registration landing mid-dispatch
had its broadcast lost (no thread parked on the condvar), so the
sweep thread could sleep on an absent/late deadline indefinitely.
Fixed by holding the lock continuously across scan-then-wait
decisions and always re-scanning immediately before ever waiting
(recheck-then-wait idiom). Round 2 review independently reproduced
the round-1 bug shape against the fix to confirm it actually closes
the race, not just narrows it.
commit_resolve() delivers a registered continuation's outcome exactly
once (idempotent re-resolve), dispatched off-thread via
vtask_enq_classified(VTASK_LIVE_BLOCK) so it never runs inline on the
replication-ACK or iomgr path. Wired into all 6 commit.c-internal
paths that can advance committed_lsn/writable_lsn.
Review caught a real task-graph gap, not a defect: converting
COMMIT_MODE_WAIT away from a blocking wait also removes the only
mechanism that ever detected a commit timing out (the caller's own
cond_timedwait deadline). Nothing yet arms an equivalent timeout for
async-registered continuations, and nothing wires demotion/log-truncation
detection to commit_resolve(FAILED, ...) -- left as-is, a pending async
command during quorum loss would hang forever instead of failing,
which is worse than today. Split into new tasks 1.3 (timeout arming)
and 1.4 (demotion/truncation wiring), renumbered the old concurrency
test to 1.5, and made 3.2 depend on both since its failure/timeout
observable is meaningless without them. design.md's File Structure
Plan updated to reflect that ha.c/persist.c need a narrow touch for
1.4's detection hooks.
commit_submit_async() lets a WAIT-mode caller register a continuation
against an appended LSN instead of blocking on pthread_cond_timedwait,
reusing the existing admission-cap discipline (wait_admit_locked/
wait_release_locked) so the bounded-capacity guarantee carries over
unchanged. Purely additive: no existing call site (repl_wait_majority_ack)
is touched yet -- that switch lands in task 3.2 once commit_resolve()
(task 1.2) exists to drive it.
vtaskmgr gains vtask_class_t (LIVENESS/LIVE_BLOCK/REQ_BLOCK) + a pool-level
blocking reservation: REQ_BLOCK fast-rejects -XCE_ERR_NOT_READY past
n_vcpus-R; LIVENESS never gated (deadlock-freedom). commit_submit folds its
inflight_waits onto the pool via commit_set_pool; production wired in
cluster.c phase2 (vtaskmgr_set_liveness_reserved(R) + commit_set_pool).
Lock order p->lock->ce->lock->blk_lock. Dormant until 3.1. Commit suites
38/30/22/38 green; lib 282/0, controller 196/0, ASAN clean. Phase 1 done.
Dedicated-pthread pool isolated from the vtask pool; non-blocking slowlane_submit
(0 enqueued / -EEXIST deduped / -EAGAIN cap-full-deferred), admission cap on
running tasks, per-key in-flight dedup, run-exactly-once, drain+join fini. Wired
into controller lifecycle (dormant; consumers land in 2.x/3.x). 7 unit tests,
mutation-verified cap+dedup; full suite 194/0 green, 15x ASAN.
Two coupled fixes for a live commit-stall (committed_lsn pinned at the own-term
NOOP while a 3-node cluster was durable to LSN 5 — jobs acked but never committed):
bug-991: the WAL catch-up applied_lsn was advanced to lsn+1 (next-to-apply) in
the three live/catch-up apply paths, but recovery/checkpoint/init and the leader's
catch-up streamer treat it as last-held. The over-advertise made the streamer skip
an LSN -> library logOk wedge -> no WalAck. Fixed to last-held (= lsn).
Option A (design reconsideration of REVIEW ISSUE 3): removed the
committed = min(durable-majority, raft_commit_index) ceiling. Under single-writer
the durable-majority watermark is already (term,lsn)-current-term-safe (term-carrying
replication + (term,lsn) reconcile/truncate-before-durable + own-term NOOP gate +
post-election truncate-to-safe), so the library commit_index was a redundant,
starvation-prone gate that pinned committed below the safe durable majority whenever
the library lagged. The durable-majority barrier + own-term writable gate are now
authoritative; raft_commit_index is inert-advisory.
Safety: reviewer-verified the durable-majority watermark is (term,lsn)-safe and
Figure-8 holds via the own-term gate (new RED->GREEN test); bug-851 cannot recur.
Live: commit trace 4->5 + walcompare committed-prefix identical across 3 nodes.
177 unit tests pass.
vote_request_thread connects to peers via SSL. iomgr was started before
election but c->ep was only registered with epoll inside cluster_start(),
which runs after election. The TCP connect succeeded (kernel accepts it)
but SSL_accept was never called server-side, so SSL_connect failed
immediately — both vote threads went to deny in <1ms and ctrl-a could
never win a quorum.
Fix: add cluster_register_listener() that wires c->ep into iomgr's
epoll watch. Call it from myapp_run() right after iomgr_start/vtaskmgr_start,
before election. cluster_start() still calls iomgr_monitor again
(idempotent via EPOLL_CTL_MOD).
Add the full call chain for host control operations from the Go API
server through the C library to the controller:
- schema/controller.capnp: HostControl struct + hostctrlmsg @21
in Request.body union
- include/typemap.h: xce_hostctrl_t* hostctrlmsg in body union
(placed after @20 clrpreasonmsg, matching capnp ordinal)
- include/client.h, xceapi.h, request.h: declarations for
xcecli_hostctrl, xce_hostctrl, xce_hostctrl_req
- lib/client.c: xcecli_hostctrl (validate + delegate to xce_hostctrl)
- lib/xceapi.c: xce_hostctrl (validate ctx state + send + wait reply)
- lib/request.c: xce_hostctrl_req (validate hostname/empty/nil,
accept only HOST_OPEN/HOST_CLOSE, reject HOST_TAG/UNTAG)
- lib/helper.c: Request_body_hostctrlmsg cases in
xce_request_body() and xce_request_msg_name()
- controller/cluster.c: OPCode_hostcontrol dispatch
- controller/include/cluster.h: do_hostcontrol_op declaration
- controller/worker.c: do_hostcontrol_op + fix pre-existing
goto-clean bug in worker_ctrl that leaked write lock and
reference on HOST_OPEN/HOST_CLOSE no-op paths
TDD: 8 xce_hostctrl_req validation tests (null ep, null ctrl,
null/empty hostname, HOST_DO_NONE no-op, HOST_TAG/UNTAG reject,
negative action reject, HOST_OPEN/CLOSE accept via xce_request).
116/116 xcelib tests pass. CGO_ENABLED=0 Go backend tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
