lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(ha): passive standby model — primary connects outbound to all peers

Browse source 
Remove fetch_config_from_primary, ha_find_and_connect_to_primary, and the
tier-1/2/3 discovery path. Standbys never initiate outbound connections to
the primary; the primary owns all inter-controller channels.

Key changes:
- ha.c: add primary heartbeat loop (ha_start_primary_heartbeat) that sends
  OPCode_nop to connected standbys every heartbeat_interval_ms; loop
  self-cancels on demotion via role check before re-arm.
- ha.c: add outbound reconnect in primary_heartbeat_fn — for peers absent
  from standby_endpoints, open fresh outbound TLS connection (on_standby_connected
  sends ctrlhello; standby_ep_ctx_t mirrors vote_ep_ctx_t layout).
- ha.c: remove ha_find_and_connect_to_primary + helpers (ha_raw_tcp_connect,
  ha_tcp_connect_hook); remove election_timeout_fn (superseded by
  raft_election_timeout_fn); remove fetch_config_from_primary call from
  ha_elect_evaluate_and_signal STANDBY branch.
- cluster.c: OPCode_requestVote denial no longer reuses inbound connection;
  late joiners are bootstrapped via primary heartbeat outbound connect.
- config_repl.c: remove connect_ssl_raw, do_one_config_fetch,
  fetch_config_from_primary (~250 LOC).
- myapp.c: remove my_tag > 0 fetch branch; all standbys wait passively.
- ha.h: remove ha_find_and_connect_to_primary declaration + macros +
  ha_tcp_connect_hook seam; add ha_start_primary_heartbeat declaration.
- meson.build: remove test_ha_connect.c (tests deleted function).
- controller_test.c: remove test_ha_connect registration.
- CLAUDE.md, src/controller/CLAUDE.md: document passive standby model,
  inbound-vs-outbound connection ownership, async I/O constraint,
  heartbeat model.
- request.h: add xce_request declaration (removes scattered local externs).
This commit is contained in:
Rongsong Shen 2026-06-01 04:17:05 +00:00
parent 842b9a838e
commit 7b39650229
11 changed files with 716 additions and 919 deletions
Download patch file Download diff file Expand all files Collapse all files

36
src/controller/CLAUDE.md
View file

@ -25,6 +25,42 @@ Links against `libxce` from `src/lib/`.
- Component-specific headers in `include/`
- Shared types defined in `src/include/xce/` (libxce)
## HA Design Constraints
### Inbound vs. Outbound Connection Ownership (CRITICAL)
**Strict rule: inbound connections are for voting only. All persistent control channels are outbound from the primary.**
| Connection type | Direction | Purpose | Who closes |
|---|---|---|---|
| Vote RPC | Standby → Primary (outbound from standby) | `OPCode_requestVote` + reply only | Remote closes after reply |
| Standby control channel | Primary → Standby (outbound from primary) | ctrlhello, config push, WAL replication, heartbeat | `vote_ep_err_handler` / `standby_ep_err_handler_primary` on disconnect |
Rules:
- **Inbound vote connections** (accepted on the listener) are used **exclusively** for the vote exchange. After the reply is sent the primary does NOT reuse the connection for config/replication. The inbound ep is left open until the remote closes it.
- **Primary never uses an inbound connection to push state.** All state push (ctrlhello, config, WAL, heartbeat nop) goes over **outbound connections** stored in `ha->standby_endpoints`.
- During election: candidate opens outgoing vote connections (one per peer). After winning, `raft_cb_become_leader` promotes surviving vote connections to `standby_endpoints`.
- Late joiners: the primary's heartbeat task (`primary_heartbeat_fn`) scans the peers list and **opens a fresh outbound connection** to any peer absent from `standby_endpoints`. On connect, `on_standby_connected` sends ctrlhello and registers the ep.
- `CTRL_STANDBY` must not call `ha_find_and_connect_to_primary` or schedule `fetch_config_from_primary` after an election result — the primary will connect to it.
### Asynchronous, Non-Blocking I/O (CRITICAL)
All inter-controller communication uses iomgr (epoll-based) and vtaskmgr:
- `iomgr` dispatches inbound packets to vtask handlers — never block inside a vtask.
- `vtaskmgr` runs handlers on a thread pool; each handler must return quickly.
- Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — non-blocking.
- Blocking I/O (e.g., `fetch_config_from_primary`, `ha_find_and_connect_to_primary`) is only permitted from `delaymgr` worker threads, **never** from iomgr/vtask context.
### Heartbeat Model
XCE uses an asymmetric heartbeat model — different from standard Raft:
- The **standby** arms a watchdog timer (`ha_arm_heartbeat_timeout_vtask`) when it enters `CTRL_STANDBY`. If no message arrives from the primary within `heartbeat_timeout_ms`, it triggers re-election.
- The **primary does not send periodic heartbeats** to standbys. Instead, WAL replication entries, config pushes, and ctrlhello messages serve as liveness signals.
- `raft.c` (the election library) has no heartbeat loop — it is election-only. Do not add heartbeat callbacks to `raft_cbs_t`; that belongs in the HA layer.
### Connection Ownership
- Promoted vote connections (`ha->standby_endpoints`): owned by `vote_ep_err_handler`; freed when the standby disconnects.
- Late-joiner inbound connections: owned by the cluster endpoint array (`c->endpoints`); the `ep_private_t` role identifies `HostRole_controller` for cleanup in `handle_endpoint_error`.
- `ha->primary_ctrl_fd`: set by `handle_ctrlhello` on the standby when the first ctrlhello arrives; used by `ha_on_primary_ctrl_fd_rdhup` to trigger re-election on disconnect.
## Key Files
- `src/controller/` — Controller source files
- `include/` — Controller-specific headers