lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race

Browse source 
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).

bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.

Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
This commit is contained in:
Rongsong Shen 2026-07-11 12:16:28 +00:00
parent ea0ac81c92
commit 40f510fb14
25 changed files with 1617 additions and 226 deletions
Download patch file Download diff file Expand all files Collapse all files

45
src/controller/join.c
View file

@ -5,7 +5,6 @@
#include <unistd.h>
#include "cluster.h"
#include "config_repl.h"
#include "ha.h"
#include "persist.h"
#include "xce/cluster_def.h"
@ -172,40 +171,36 @@ static int handle_scheduler_join(cluster_t* c, endpoint_t* ep,
return rc;
}
/*
* handle_controller_join: reject a peer-controller join request.
*
* Controller-to-controller connection establishment is NOT this path — per
* ha-async-io.md's Connection Ownership, it is always primary-initiated
* outbound (primary_heartbeat_fn -> on_standby_connected -> ctrlhello ->
* repl_add_standby), with the resulting endpoint owned by
* vote_ep_err_handler / standby_ep_err_handler_primary. This arm used to
* accept an INBOUND role=HostRole_controller join and store it directly
* into standby_endpoints, bypassing both documented owners and their
* err-handler-driven eviction — a live ownership-model violation, even
* though no current sender in the tree ever builds such a request
* (xce_join_req structurally rejects role==HostRole_controller). Reject
* outright rather than leave the bypass reachable from the wire.
*/
static int handle_controller_join(cluster_t* c, endpoint_t* ep,
xce_request_t* request)
{
xce_reply_t reply;
init_reply_by_req(&reply, request);
reply.status = 0;
reply.status = -E_XCE_BAD_ROLE;
reply.data_which = RequestReply_data_nodata;
xce_reply(ep, c->myhost, &reply);
/* Register this endpoint for OP_upgradeConfig fan-out to standbys */
if (c->ha.standby_endpoints) {
ep_private_t *priv = (ep_private_t *)ep->data;
if (priv)
priv->role = HostRole_controller;
WARN("%s rejecting controller-role join from %s — controller peering is "
"primary-initiated only, never via join",
__func__, request->from ? request->from->name : "?");
pthread_mutex_lock(&c->ha.lock);
zarray_store(c->ha.standby_endpoints, ep);
pthread_mutex_unlock(&c->ha.lock);
INFO("controller %s joined — %d standby(s) tracked",
request->from ? request->from->name : "?",
c->ha.standby_endpoints->count);
/* Immediately push current config to the newly-joined standby so it
* doesn't have to wait for the next scheduled push. Transport-only
* (no barrier wait — task 5.2 remediation r1): this is a best-effort
* catch-up stream; convergence is owned by the OP_replConfig
* anti-entropy pull and the heartbeat decision loop. */
if (ha_is_primary(&c->ha))
push_config_transport_only(c, NULL, 0, NULL, 0, NULL);
}
return 0;
return -E_XCE_BAD_ROLE;
}
static int handle_apiserver_join(cluster_t* c, endpoint_t* ep,