lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(raft-apply-decoupling): arm timeout for async commit continuations (task 1.3)

Browse source 
Converting COMMIT_MODE_WAIT away from a blocking wait removed the only
mechanism that ever detected a commit timing out (the caller's own
pthread_cond_timedwait deadline). Adds a dedicated per-commit-engine
sweep thread that arms an equivalent timeout for async-registered
continuations, resolving FAILED/-ETIMEDOUT within the same
DEFAULT_SYNC_TIMEOUT_MS window the old blocking path used, with no
thread parked to detect it.

Chose a private sweep thread over the shared delaymgr: wiring
delaymgr would touch cluster.c/ha.c (out of boundary) and its 200ms
poll granularity vs the 500ms default timeout would add jitter,
breaking the "identical failure timing" no-regression bar.

Round 1 review rejected a lost-wakeup race: the sweep loop computed
its next-deadline snapshot under lock, unlocked to dispatch, then
slept on the stale snapshot -- a registration landing mid-dispatch
had its broadcast lost (no thread parked on the condvar), so the
sweep thread could sleep on an absent/late deadline indefinitely.
Fixed by holding the lock continuously across scan-then-wait
decisions and always re-scanning immediately before ever waiting
(recheck-then-wait idiom). Round 2 review independently reproduced
the round-1 bug shape against the fix to confirm it actually closes
the race, not just narrows it.
This commit is contained in:
Rongsong Shen 2026-07-03 10:35:37 +00:00
parent fd7a0746eb
commit 530a02bda3
4 changed files with 461 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

36
src/controller/include/commit.h
View file

@ -10,6 +10,7 @@
*/
#include <pthread.h>
#include <stdint.h>
#include <time.h> /* struct timespec — per-continuation timeout deadline (task 1.3) */
#include "persist.h" /* persist_t (owns committed_lsn atomic + replicator_t registry) */
#include "vtaskmgr.h" /* vtaskmgr_t — LivenessAdmission blocking-admission seam */
@ -47,6 +48,15 @@ typedef struct commit_continuation {
uint64_t lsn;
commit_continuation_fn fn;
void* ctx;
/* raft-apply-decoupling task 1.3: absolute CLOCK_REALTIME deadline this
* registration must resolve by if majority is never reached -- computed
* at registration time via the SAME wait_deadline() helper and the SAME
* DEFAULT_SYNC_TIMEOUT_MS constant commit_submit's blocking WAIT path
* uses, so an async registration observes identical failure timing to
* today's blocking path (Req 11.1). Only meaningful while lsn != 0;
* overwritten on every fresh registration of a reused slot. Swept by the
* commit engine's dedicated timeout-sweep thread (see commit.c). */
struct timespec deadline;
} commit_continuation_t;
/*
@ -90,6 +100,17 @@ typedef struct commit_engine {
* inert -- nothing reads it back. */
commit_continuation_t* continuations;
int n_continuations; /* capacity, == n_vcpus */
/* raft-apply-decoupling task 1.3: dedicated timeout-sweep thread, private
* to this commit_engine_t (NOT the shared controller delaymgr/iomgr --
* see commit.c file header for why). Sleeps on `cond` to the earliest
* pending continuation deadline (or indefinitely when none are pending)
* and resolves any still-registered continuation FAILED/-ETIMEDOUT once
* its deadline passes. Started in commit_engine_init, stopped+joined in
* commit_engine_fini before `lock`/`cond`/`continuations` are torn down.
* sweep_stop is read/written only under `lock`, alongside the broadcast
* that wakes the thread to observe it. */
pthread_t sweep_thread;
int sweep_stop;
} commit_engine_t;
/* Lifecycle. init validates n_peers >= 0, n_vcpus >= 1, non-NULL persist_t;
@ -121,9 +142,18 @@ int commit_submit(commit_engine_t* ce, uint64_t lsn, commit_mode_t mode,
* -XCE_ERR_NOT_READY when the admission cap is exhausted (the same shared
* discipline commit_submit's WAIT mode uses -- see wait_admit_locked).
* Never blocks: the calling thread returns immediately, before lsn's
* outcome is known. Registering here does not yet cause `fn` to be
* invoked -- delivery is commit_resolve's job, added by a follow-up task
* that wires it onto the majority-commit and failure paths. */
* outcome is known.
*
* raft-apply-decoupling task 1.3: registration also arms an implicit
* DEFAULT_SYNC_TIMEOUT_MS deadline (identical constant and clock the
* blocking WAIT path uses) -- if lsn has not resolved by then, the commit
* engine's own timeout-sweep thread resolves it FAILED/-ETIMEDOUT with no
* caller action required. A genuine majority ack that lands at or after the
* deadline still wins: the sweep thread always checks the majority
* predicate before the deadline in the same lock hold, mirroring
* commit_submit's own last-look-before-timeout tie-break. There is no
* per-call timeout parameter (unlike commit_submit) -- see design.md
* §Commit Engine Resolve for the fixed signature. */
int commit_submit_async(commit_engine_t* ce, uint64_t lsn,
commit_continuation_fn fn, void* ctx);