wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity
myxce/.kiro/specs/ha-peer-connection-reuse/requirements.md
Rongsong Shen e31673dd88 chore: checkpoint all prior work for clean state 
Captures all uncommitted work from ha-config-bootstrap,
ha-heartbeat-liveness, and associated spec/tooling updates:

- src: ha.c, ha.h, cluster.c, config_repl.c, myapp.c, raft.c —
  heartbeat liveness, RDHUP integration, committing phase fixes
- tests: test_ha_role.c, test_config_repl.c, test_ha_rdhup_integration.c,
  test_p1_integration.c, test_p3_hardening.c, scheduler test
- specs: election-iomgr-refactor, ha-heartbeat-liveness,
  ha-peer-connection-reuse, raft-library (new); ha-config-bootstrap
  and election-iomgr-refactor task/research updates
- tooling: .wolf/ state, ha/data cluster fixtures, raft/raft.tla,
  src/build symlink, xce-build skill update
2026-06-04 13:04:16 +00:00

6.3 KiB
Raw Blame History

Requirements Document

Introduction

The XCE controller election currently creates outgoing vote connections that are discarded after the vote exchange, forcing standbys to create new outbound connections back to the primary for the persistent control channel. This double-connection pattern causes a 10-second heartbeat timeout delay before standbys detect the primary, retry warnings from premature config fetch attempts, and a complex tier-1/2/3 primary discovery path. This feature reuses the candidate's outgoing vote connections as persistent standby control channels after the election: the winner promotes its connections to replication channels; the loser closes its outbound connections and waits passively for the primary's ctrlhello to arrive on the listener.

Boundary Context

  • In scope: Lifecycle of outbound vote connections from creation through promotion or close; standby passive connection model; primary control channel setup via promoted connection; heartbeat watchdog against the promoted connection.
  • Out of scope: SSL/TLS certificate management; vote grant/deny protocol logic; WAL entry format and replication content; config file format; cluster membership changes.
  • Adjacent expectations: iomgr must support reassigning a msg_handler on an existing endpoint without closing and reopening the connection. The existing endpoint_t lifetime model is not changed. The existing heartbeat timeout and RDHUP detection semantics are preserved on the promoted connection.

Requirements

1. Candidate Initiates All Peer Connections

Objective: As a cluster operator, I want the controller that initiates an election to create and own all peer connections, so that standbys never need to discover or dial out to the primary.

Acceptance Criteria

  1. When a controller enters CTRL_ELECTING state and sends vote requests, the controller shall establish one outbound TLS connection per peer for the vote RPC.
  2. The controller shall reuse each surviving outbound vote connection as the persistent peer control channel for the duration of that node's role after the election result is known, without closing and re-establishing the connection.
  3. When a controller wins the election, the controller shall promote each surviving outbound vote connection to a standby replication channel without creating new outbound connections to standbys.
  4. When a controller loses the election, the controller shall close all outbound vote connections it created and remain fully passive on its listener.

2. Standby Fully Passive Connection Model

Objective: As a cluster operator, I want standby controllers to never initiate outbound connections to the primary, so that connection management is uniform and the standby cannot mis-identify the primary.

Acceptance Criteria

  1. While in CTRL_STANDBY state, the controller shall not initiate any outbound TCP connections to peer controllers.
  2. When the primary sends a ctrlhello, config push, or WAL replication message on the inbound connection accepted from the winning candidate, the standby controller shall accept and process it on that existing connection.
  3. If a controller transitions to CTRL_STANDBY and receives no inbound communication from the primary within heartbeat_timeout_ms, the controller shall trigger a new election.
  4. When a controller becomes a standby, the controller shall not invoke the tier-1/2/3 primary discovery path; the inbound connection from the winning candidate is the only primary control channel for that election round.

3. Primary Control Channel via Promoted Connection

Objective: As a cluster operator, I want the standby's primary control channel to be the inbound connection established by the winning candidate during the vote, so that heartbeat detection is immediate and no reconnection delay occurs.

Acceptance Criteria

  1. When the primary sends the first ctrlhello on a promoted vote connection, the standby controller shall designate that connection as its primary control channel and arm the heartbeat timeout watchdog against it.
  2. When the primary control channel drops (RDHUP), the controller shall detect the failure within one iomgr epoll cycle and transition to CTRL_ELECTING.
  3. The controller shall arm the heartbeat timeout watchdog against the promoted inbound connection with the same timeout semantics currently applied to primary_ctrl_fd.

4. Explicit Close of Outbound Connections on Election Loss

Objective: As a cluster operator, I want the losing controller to cleanly close its outbound vote connections, so that the primary's promoted connections are the only active inter-controller channels after election.

Acceptance Criteria

  1. When a controller determines the election is lost (via quorum denial or election timeout), the controller shall explicitly close all outbound vote endpoints it created for that election round.
  2. If an outbound vote connection fails before a reply is received, the controller shall count that peer as a denied vote and not attempt to reconnect to that peer for replication purposes in that election round.
  3. When a controller starts a new election after a failover event, the controller shall create fresh outbound connections to all peers for the new round.

5. No Regression in Existing Behaviors

Objective: As a cluster operator, I want all current HA behaviors preserved after connection reuse, so that failover, WAL replication, and config push continue to work correctly.

Acceptance Criteria

  1. When the primary wins election, the controller shall push cluster configuration to standbys via the promoted connection with the same correctness guarantees as the current config push path.
  2. When the primary enters CTRL_PRIMARY_COMMITTING, the controller shall replicate WAL entries to standbys via the promoted connections using the existing catch-up replication protocol.
  3. When a single-node cluster starts, the controller shall self-elect to primary without creating any peer connections.
  4. If the majority ack threshold is not reached within committing_timeout_ms, the controller shall promote to CTRL_PRIMARY in degraded mode (Req 8.4 behavior preserved).
  5. When all controller tests pass (unit and integration), the controller shall produce no new ASAN errors compared to the baseline before this feature.