lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(lib): keep per-endpoint message order in the vtask dispatcher (bug-1864)

Browse source 
The dispatcher checked each queued task's endpoint busy flag one item at a
time, while the vcpu that finished the endpoint's previous message cleared
the flag without the taskq lock (a lock there deadlocked, bug-1282). Within
one scan, task N was skipped (endpoint still busy), the endpoint freed up,
and task N+1 of the SAME endpoint was dequeued first. Standby controllers
then handled WAL catch-up entry 258 before 257 and NACKed (log mismatch):
26 reorders -> ~576 NACKs in a 45-min production-like run, both standbys
at once at 21:39:24, primary lost its write quorum and answered
jobdispatch/join with E_XCE_NOT_READY, the scheduler client failed over and
rejoined, and the rejoin JOBS_PUSH carried lagging committed state
("refusing resync state ... existing job is RUN" WARNs).

dataq gains dataq_deq_ctx() (a per-scan matcher context). The scan is now
vtask_deq_eligible(taskq, is_blocked): it remembers every endpoint found
busy and keeps all later tasks of that endpoint ineligible for the rest of
the scan (past 64 distinct busy endpoints, every later IO task waits for
the next scan) -- per-endpoint FIFO without any new lock.

Test lib/testsuite/test_vtask_order.c injects is_blocked to free the
endpoint mid-scan: task N+1 was dequeued before the fix, none after; other
endpoints and FIFO/generic tasks unaffected. Suites: lib 1/1, controller
5/5, worker 10/10, scheduler 87/87, harness 5/5. Chaotic HA soak 2/2 PASS
with 0 WAL reorders (2 and 5 per run before), no ASAN.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-25 14:44:14 +00:00
parent 6247bf052b
commit 0d3bb965e5
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
7 changed files with 293 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

6
src/include/dataq.h
View file

@ -9,6 +9,8 @@ typedef struct _data_item {
} data_item_t;
typedef int (*data_item_match_fn_t)(data_item_t*);
/* like data_item_match_fn_t, plus a caller-owned per-scan context */
typedef int (*data_item_match_ctx_fn_t)(data_item_t*, void*);
typedef void (*data_item_killer_t)(data_item_t*);
typedef struct {
@ -22,6 +24,10 @@ typedef struct {
extern int dataq_init(dataq_t*, data_item_killer_t);
extern data_item_t* dataq_deq(dataq_t*, data_item_match_fn_t);
/* bug-1864: first item matching f, with ctx passed to every f call of this
* one scan (under the queue lock) -- lets a matcher remember what it saw
* earlier in the same scan */
extern data_item_t* dataq_deq_ctx(dataq_t*, data_item_match_ctx_fn_t, void*);
extern int dataq_enq(dataq_t*, data_item_t*);
extern void dataq_fini(dataq_t*);

7
src/include/vtaskmgr.h
View file

@ -99,6 +99,13 @@ extern int vtask_init(vtask_t*, vtask_func_t, void*);
extern void vtask_fini(vtask_t*);
extern int vtask_enq(vtaskmgr_t*, vtask_t*);
/* bug-1864: dequeue the first dispatchable task from a vtaskmgr task queue.
* is_blocked reports whether an endpoint already has a task running
* (production: endpoint_is_task_blocked); it is a parameter so a test can
* make an endpoint unblock part-way through one scan. */
extern vtask_t* vtask_deq_eligible(dataq_t* taskq,
int (*is_blocked)(endpoint_t*));
extern int vtask_handle_msg(vtaskmgr_t*, vtask_func_t, void* arg);
/* LivenessAdmission (raft-drive-async task 1.6) — pool-level reservation.