lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(raft-drive-async): peer-control vtask dispatch shim, dormant (task 3.1)

Browse source 
Lands the PeerCtrlDispatch machinery behind a DEFAULT-OFF flag
(g_drive_async_peerctrl_dispatch_enabled); production behavior is
UNCHANGED. peer_ctrl_msg_handler becomes a thin shim that takes
ownership of ep->fsm_in.pkt and either runs the routing body inline
(flag OFF, byte-for-byte the prior path) or dispatches it to the vtask
pool via vtask_enq_classified (flag ON). The 3 routers now take pkt as
an explicit param (no cross-thread ep->fsm_in.pkt re-read). Per-endpoint
arrival order is enforced by bind_vtask's endpoint_block_tasks.

Packet ownership: shim clears ep->fsm_in.pkt at dispatch (sole-owner
transfer); peer_ctrl_vtask_entry NULLs iod->pkt before the body frees
it once. New vtask_t.owns_io_pkt (default 0) lets taskq_item_killer free
the carried packet for an un-executed queued task at vtaskmgr_fini WITHOUT
double-freeing the libxce endpoint-owned path (bug-1199).

Adversarial review found two FLAG-ON hazards, documented with loud
comments (flag is OFF so production is unaffected), to be resolved before
task 3.4 flips the flag ON:
  - bug-1197 (CRITICAL): raft_cb_become_follower endpoint_fini()s peer
    vote-eps from a pool worker, racing the io_worker's read of ep->data;
    needs iomgr_unmonitor before endpoint_fini in every pool-reachable
    teardown.
  - bug-1198 (HIGH): peer_ctrl_classify tags all work VTASK_LIVENESS, but
    the CTRL_PRIMARY repl-ack path can run a blocking snapshot checkpoint
    inline; resolved by task 3.2 (offload snapshot to SlowLane).

Per spec, 3.1 is not 'done' until the task 3.4 deadlock gate is green.
Tests: 3 new peerctrl tests; controller 196/196, lib 282/0; 0 net leaks.
This commit is contained in:
Rongsong Shen 2026-06-26 11:12:04 +00:00
parent 21e26a17b6
commit e17a6a613c
7 changed files with 1678 additions and 31 deletions
Download patch file Download diff file Expand all files Collapse all files

32
src/controller/ha.h
View file

@ -173,6 +173,12 @@ typedef struct {
_Atomic uint64_t committed_config_tag; /* Req 8.1: set on quorum ACK */
/* ── New fields added by election-iomgr-refactor spec (Task 2.1) ── */
void *client_ssl_ctx; /* SSL_CTX* (TLS_client_method); created in myapp_run */
/* ── New field added by raft-drive-async spec (Task 3.1) ──────── */
void *pool; /* vtaskmgr_t* (cast; avoids pulling vtaskmgr.h
* into every ha.h consumer). Set once at init
* (cluster_init_phase2 → &c->vtaskmgr). Used ONLY
* by peer_ctrl_msg_handler when peer-control
* dispatch is enabled; NULL otherwise (gate OFF). */
int phase2_done; /* 1 after cluster_init_phase2 runs */
int n_peers; /* total cluster size (including self), matches raft convention */
/* ── New fields added by raft-library spec (Task 6.1) ──────── */
@ -1221,6 +1227,32 @@ void ha_start_primary_heartbeat(cluster_t *c);
void ha_set_reconnect_offload_for_test(int on);
int ha_reconnect_offload_enabled(void);
/*
* ha_set_peerctrl_dispatch_for_test / ha_peerctrl_dispatch_enabled:
* Feature-flag accessors (raft-drive-async Task 3.1). The dispatch flag gates
* whether the registered peer-control handler (peer_ctrl_msg_handler) runs its
* routing body INLINE on the iomgr io_worker thread (OFF — the production
* default for this commit; behavior is byte-for-byte identical to before the
* task) or hands the decoded packet to the vtask pool via vtask_enq_classified
* and returns promptly (ON — the target behavior, validated by the task 3.4
* deadlock gate before it becomes the default). These accessors are compiled
* in ALL builds; the setter is used by tests to flip RED<->GREEN.
*
* DEFAULT: OFF (0). Production behavior is UNCHANGED by this commit.
*/
void ha_set_peerctrl_dispatch_for_test(int on);
int ha_peerctrl_dispatch_enabled(void);
/*
* ha_peerctrl_body_probe: test-only observation seam (raft-drive-async Task
* 3.1). When non-NULL it is invoked at the TOP of the peer-control routing
* body (peer_ctrl_msg_handler_inline) with the endpoint, BEFORE any role
* routing. Tests use it to (a) count body invocations, (b) capture the thread
* the body ran on (proving inline-vs-pool execution), and (c) optionally park
* the body to prove per-endpoint arrival-order serialization. NULL in
* production (zero cost) — a plain global like ha_nop_req_hook. */
extern void (*ha_peerctrl_body_probe)(endpoint_t *ep);
/* ── Task 2.4: Non-blocking async election initiator ────────────────────── */
/*