fix(lib): keep per-endpoint message order in the vtask dispatcher (bug-1864)

The dispatcher checked each queued task's endpoint busy flag one item at a
time, while the vcpu that finished the endpoint's previous message cleared
the flag without the taskq lock (a lock there deadlocked, bug-1282). Within
one scan, task N was skipped (endpoint still busy), the endpoint freed up,
and task N+1 of the SAME endpoint was dequeued first. Standby controllers
then handled WAL catch-up entry 258 before 257 and NACKed (log mismatch):
26 reorders -> ~576 NACKs in a 45-min production-like run, both standbys
at once at 21:39:24, primary lost its write quorum and answered
jobdispatch/join with E_XCE_NOT_READY, the scheduler client failed over and
rejoined, and the rejoin JOBS_PUSH carried lagging committed state
("refusing resync state ... existing job is RUN" WARNs).

dataq gains dataq_deq_ctx() (a per-scan matcher context). The scan is now
vtask_deq_eligible(taskq, is_blocked): it remembers every endpoint found
busy and keeps all later tasks of that endpoint ineligible for the rest of
the scan (past 64 distinct busy endpoints, every later IO task waits for
the next scan) -- per-endpoint FIFO without any new lock.

Test lib/testsuite/test_vtask_order.c injects is_blocked to free the
endpoint mid-scan: task N+1 was dequeued before the fix, none after; other
endpoints and FIFO/generic tasks unaffected. Suites: lib 1/1, controller
5/5, worker 10/10, scheduler 87/87, harness 5/5. Chaotic HA soak 2/2 PASS
with 0 WAL reorders (2 and 5 per run before), no ASAN.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-25 14:44:14 +00:00
parent 6247bf052b
commit 0d3bb965e5
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
7 changed files with 293 additions and 22 deletions

View file

@ -55,6 +55,7 @@ extern void test_wal_wire(void);
extern void test_ctrlstatus(void);
extern void test_vtask_admission(void);
extern void test_vtask_enq_race(void);
extern void test_vtask_order(void);
extern void test_vtaskmgr_teardown(void);
extern void test_xcealloc(void);
@ -181,6 +182,7 @@ int main(int argc, char* argv[])
TEST(test_ctrlstatus);
TEST(test_vtask_admission);
TEST(test_vtask_enq_race);
TEST(test_vtask_order);
TEST(test_vtaskmgr_teardown);
TEST(test_xcealloc);