lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): never close() the primary channel's raw fd (bug-1865)

Browse source 
handle_ctrlhello stored the netio endpoint's fd (primary_ctrl_fd = ep->fd)
and ha_on_primary_ctrl_fd_rdhup close()d it in three branches. netio owns
that socket and closes it on endpoint teardown, so by the time the
liveness poll fired the RDHUP the number could be reused -- in a chaotic
soak by the standby's WAL segment: HA's close() shut the WAL, fdatasync
failed 2305 times (EBADF, then EINVAL once a socket took the number),
durable_lsn stuck, and one round's jobsub failed EIO.

handle_ctrlhello now also holds a reference on the endpoint
(primary_ctrl_ep), which keeps its socket open and its fd number ours.
ha_drop_primary_ctrl_conn() replaces the raw closes: shutdown() the
still-owned socket so netio sees the hang-up and tears it down itself,
then release the reference. A bare fd with no endpoint is only forgotten.

Test test_rdhup_never_closes_an_fd_netio_already_released: real
ctrlhello -> netio teardown -> file opened on the freed number -> RDHUP;
the file was closed before, survives now. test_rdhup_closes_and_nulls_fd
updated to the new contract (peer reads EOF, fields cleared). Controller
suite 5/5.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-25 14:55:04 +00:00
parent 98c52b7d96
commit 910b43cd37
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
5 changed files with 127 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

37
src/controller/ha.c
View file

@ -2889,6 +2889,7 @@ int ha_init(ha_state_t *ha, const ha_config_t *cfg, int my_priority)
ha->jitter_seed = 0;
ha->standby_endpoints = NULL;
ha->primary_ctrl_fd = -1;
ha->primary_ctrl_ep = NULL;
/* election-iomgr-refactor Task 2.1: explicit zero-init for documentation clarity
* (memset above already zeroes these, but being explicit is better documentation) */
@ -8645,6 +8646,27 @@ void ha_begin_failover(ha_state_t *ha, cluster_t *c)
ha_start_prevote_async(ha, c);
}
/* bug-1865: drop the standby's channel to the primary. primary_ctrl_fd is a
* netio endpoint's socket: netio owns it and closes it on endpoint teardown,
* after which the fd number can be reused (it was, by the WAL segment) --
* so it is never close()d here. The endpoint is held with a reference
* (handle_ctrlhello), which keeps the socket open and its number ours until
* the reference is released: shutdown() it so netio sees the hang-up and
* tears the connection down itself, then release the reference. */
void ha_drop_primary_ctrl_conn(ha_state_t *ha)
{
endpoint_t *ep = (endpoint_t *)ha->primary_ctrl_ep;
if (ep != NULL) {
if (ep->fd >= 0) {
(void)shutdown(ep->fd, SHUT_RDWR);
}
ha->primary_ctrl_ep = NULL;
endpoint_unref(ep);
}
ha->primary_ctrl_fd = -1;
}
void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c)
{
if (!ha) return;
@ -8665,10 +8687,7 @@ void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c)
ctrl_role_t cur = (ctrl_role_t)atomic_load_explicit(&ha->role,
memory_order_acquire);
if (cur == CTRL_ELECTING || ha->raft.pre_vote_in_progress) {
if (ha->primary_ctrl_fd >= 0) {
close(ha->primary_ctrl_fd);
ha->primary_ctrl_fd = -1;
}
ha_drop_primary_ctrl_conn(ha);
INFO("HA: primary_ctrl_fd RDHUP but an election is already in progress "
"(role=%d, prevote=%d) — dropping the RDHUP, not restarting "
"failover (task 5.6: don't preempt the in-progress election / "
@ -8696,10 +8715,7 @@ void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c)
memory_order_acquire);
uint64_t cooldown_ms = 2u * (uint64_t)ha->config.election_timeout_ms;
if (last != 0 && now_ms >= last && (now_ms - last) < cooldown_ms) {
if (ha->primary_ctrl_fd >= 0) {
close(ha->primary_ctrl_fd);
ha->primary_ctrl_fd = -1;
}
ha_drop_primary_ctrl_conn(ha);
INFO("HA: primary_ctrl_fd RDHUP within failover cooldown (%lums since "
"last trigger, cooldown=%lums) — debouncing, not restarting "
"failover (5.10b: a sustained RDHUP cadence must not ladder the "
@ -8728,10 +8744,7 @@ void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c)
/* Close and null the primary control fd. Connection-teardown bookkeeping
* stays at the trigger regardless of the PreVote outcome (ha-prevote task
* 4.2). */
if (ha->primary_ctrl_fd >= 0) {
close(ha->primary_ctrl_fd);
ha->primary_ctrl_fd = -1;
}
ha_drop_primary_ctrl_conn(ha);
/* raft-ha-core-safety task 5.2: SPURIOUS-FAILOVER SUPPRESSION.
*