lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): never close() the primary channel's raw fd (bug-1865)

Browse source 
handle_ctrlhello stored the netio endpoint's fd (primary_ctrl_fd = ep->fd)
and ha_on_primary_ctrl_fd_rdhup close()d it in three branches. netio owns
that socket and closes it on endpoint teardown, so by the time the
liveness poll fired the RDHUP the number could be reused -- in a chaotic
soak by the standby's WAL segment: HA's close() shut the WAL, fdatasync
failed 2305 times (EBADF, then EINVAL once a socket took the number),
durable_lsn stuck, and one round's jobsub failed EIO.

handle_ctrlhello now also holds a reference on the endpoint
(primary_ctrl_ep), which keeps its socket open and its fd number ours.
ha_drop_primary_ctrl_conn() replaces the raw closes: shutdown() the
still-owned socket so netio sees the hang-up and tears it down itself,
then release the reference. A bare fd with no endpoint is only forgotten.

Test test_rdhup_never_closes_an_fd_netio_already_released: real
ctrlhello -> netio teardown -> file opened on the freed number -> RDHUP;
the file was closed before, survives now. test_rdhup_closes_and_nulls_fd
updated to the new contract (peer reads EOF, fields cleared). Controller
suite 5/5.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-25 14:55:04 +00:00
parent 98c52b7d96
commit 910b43cd37
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
5 changed files with 127 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

15
src/controller/ha.h
View file

@ -178,6 +178,12 @@ typedef struct {
uint64_t voted_for_term; /* Req 9: term of last granted vote */
char voted_for_name[XCE_MAXNAME_LEN]; /* Req 9: candidate of last granted vote */
int primary_ctrl_fd; /* Req 6: persistent channel fd (-1 = none) */
/* bug-1865: the netio endpoint primary_ctrl_fd belongs to, held with a
* reference (handle_ctrlhello) so its socket -- which netio owns and
* closes -- cannot be closed and its fd number reused while HA still
* treats it as the primary channel. Dropped via ha_drop_primary_ctrl_conn,
* never by close(primary_ctrl_fd). NULL = none. */
struct endpoint_t* primary_ctrl_ep;
bool defer_config_fetch; /* Req 3: tag=0 node defers fetch to post-election */
void * _Atomic elect_ctx; /* Req 2: ha_elect_ctx_t* during election; NULL otherwise */
_Atomic uint64_t last_known_committed_lsn; /* Req 5: from primary heartbeats */
@ -1605,12 +1611,17 @@ ctrl_role_t ha_wait_for_heartbeat_or_timeout(ha_state_t *ha);
/* ── Task 9.4: EPOLLRDHUP Failover and Heartbeat Timeout (Req 6.2–6.4, 6.8) ─
* ha_on_primary_ctrl_fd_rdhup: EPOLLRDHUP event handler.
* Closes primary_ctrl_fd, transitions to CTRL_ELECTING, truncates WAL to
* last_known_committed_lsn, fires ha_start_election_async (via cluster when non-NULL).
* Drops the primary channel (ha_drop_primary_ctrl_conn), transitions to
* CTRL_ELECTING, truncates WAL to last_known_committed_lsn, fires
* ha_start_election_async (via cluster when non-NULL).
* Pass c == NULL to skip persist/election (unit test mode).
*/
void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c);
/* bug-1865: shut down + release the standby's primary channel. Never
* close()s primary_ctrl_fd -- netio owns that socket. */
void ha_drop_primary_ctrl_conn(ha_state_t *ha);
/*
* ha_arm_heartbeat_timeout_vtask: start/re-arm the heartbeat watchdog thread.
* Fires ha_on_primary_ctrl_fd_rdhup after election_timeout_ms + rand_jitter of