Rongsong Shen
e85100d761
Task 7.1 (unit tests) was already complete with 8 tests in test_ha_connection_reuse.c. Added two new tests: Task 7.2 — test_follower_passive_no_outbound_connect (Req 2.1, 2.4): Verifies raft_cb_become_follower sets role=CTRL_STANDBY and nulls all votes[i].ep. The old fetch_config_from_primary path is gone; standbys never initiate outbound connects after election loss. Task 7.3 — test_promoted_ep_repl_ack_promotes_to_primary (Req 5.2, 5.4): Injects two OPCode_replCatchUpDone packets via peer_ctrl_msg_handler in CTRL_PRIMARY_COMMITTING with n_peers=4 (5-node cluster, quorum=2 ACKs). Verifies repl_ack_count reaches 2 and role transitions to CTRL_PRIMARY. Added vote_ep_ctx_set_cluster_for_test() seam so vctx->c is non-NULL when ha_repl_ack_received fires ha_promote_to_primary. All 62 controller tests pass; ASAN clean. spec: .kiro/specs/ha-peer-connection-reuse/tasks.md all tasks [x]
92 lines
7.9 KiB
Markdown
92 lines
7.9 KiB
Markdown
# Implementation Plan: ha-peer-connection-reuse
|
|
|
|
- [x] 1. Foundation: track outgoing vote endpoint per peer
|
|
|
|
- [x] 1.1 Add `ep` field to `ha_vote_entry_t` and populate it during vote endpoint creation
|
|
- Add `struct endpoint_t *ep` field to `ha_vote_entry_t` in `ha.h` (forward-declare `endpoint_t` if not already present in that header)
|
|
- In `raft_cb_send_vote_request` (ha.c): after `iomgr_monitor(ep)` succeeds, store `ep` in `ectx->votes[votes_idx].ep`; leave `votes[votes_idx].ep = NULL` on connect or monitor failure
|
|
- In the test helper `vote_ep_ctx_build_for_test`: also accept an `endpoint_t *ep` parameter and store it in `votes[peer_idx].ep` so tests can verify the field
|
|
- Observable: ASAN-clean build; `ectx->votes[i].ep` is non-NULL for successfully connected peers and NULL for failed peers in existing vote-send tests
|
|
- _Requirements: 1.1, 1.2, 1.3, 1.4, 4.1_
|
|
|
|
- [x] 2. Core: implement state-aware message dispatcher
|
|
|
|
- [x] 2.1 (P) Implement `peer_ctrl_msg_handler` as a role-dispatching iomgr handler
|
|
- Add `static int peer_ctrl_msg_handler(void *ep_raw)` in ha.c
|
|
- On `CTRL_ELECTING`: call `vote_reply_decode_and_dispatch(ep, vctx)` — factor the body of the existing `vote_reply_msg_handler` into this static helper (preserving XCE_TESTING term-fencing and `xce_vote_reply_hook` seam)
|
|
- On `CTRL_PRIMARY_COMMITTING` or `CTRL_PRIMARY`: call `repl_ack_decode_and_dispatch(ep, vctx)` — new static helper that decodes an inbound capnp packet, checks opcode == `OPCode_replCatchUpDone`, then calls `ha_repl_ack_received(ha, c)`; logs WARN and drops on unexpected opcode
|
|
- On any other role (`CTRL_STANDBY`, `CTRL_INIT`): free packet, return 0, log WARN
|
|
- Observable: unit test verifies that injecting a vote-reply packet with role=ELECTING calls `xce_vote_reply_hook`; injecting a replCatchUpDone packet with role=COMMITTING calls the repl-ack hook; injecting with role=STANDBY results in packet dropped and no hook called
|
|
- _Requirements: 1.2_
|
|
- _Boundary: peer_ctrl_msg_handler_
|
|
|
|
- [x] 3. Core: promote winner's vote endpoints to standby control channels
|
|
|
|
- [x] 3.1 (P) Extend `raft_cb_become_leader` to populate `ha->standby_endpoints` from surviving vote endpoints
|
|
- Read `ha->elect_ctx`; if NULL (single-node path), skip the loop
|
|
- Iterate `ctx->votes[i].ep` for `i` in `[0, ctx->n_peers)` (note: votes[0] is self in the raft library; peer eps start at index 1 — verify against `raft_cb_send_vote_request` peer_idx mapping)
|
|
- For each non-NULL `votes[i].ep`: acquire `ha->lock`, append to `ha->standby_endpoints`, release lock
|
|
- Log `INFO "HA: promoted vote ep[%d] → standby channel"` for each promoted endpoint
|
|
- Observable: after `raft_cb_become_leader` fires in a unit test with 2 populated `votes[i].ep`, `ha->standby_endpoints->count == 2`; subsequent call to `ha_send_ctrlhello_to_all_standbys` iterates the promoted endpoints
|
|
- _Requirements: 1.3, 5.1, 5.2_
|
|
- _Boundary: raft_cb_become_leader_
|
|
|
|
- [x] 4. Core: passive standby path — close vote endpoints and arm watchdog
|
|
|
|
- [x] 4.1 (P) Extend `raft_cb_become_follower` to close vote endpoints and arm heartbeat watchdog; remove `fetch_config_from_primary` call
|
|
- Read `ha->elect_ctx`; if NULL, skip close loop
|
|
- Iterate `ctx->votes[i].ep` for all i: if non-NULL, call `endpoint_fini(votes[i].ep)`, set `votes[i].ep = NULL`
|
|
- Log `INFO "HA: closing vote ep[%d] (loser path)"` for each closed endpoint
|
|
- Remove the `#ifndef XCE_TESTING` block that calls `fetch_config_from_primary` via `delay_task_ms` — delete it entirely
|
|
- After `ha->role = CTRL_STANDBY` is stored, call `ha_arm_heartbeat_timeout_vtask(ha, c)` (no-op when `c == NULL` in tests)
|
|
- Observable: unit test with mocked `ha_arm_heartbeat_timeout_vtask` hook verifies it is called exactly once; `votes[i].ep == NULL` for all i after the call; no `delay_task_ms` for `fetch_config_from_primary` is dispatched
|
|
- _Requirements: 1.4, 2.1, 2.3, 2.4, 3.3, 4.1_
|
|
- _Boundary: raft_cb_become_follower_
|
|
|
|
- [x] 5. Core: designate primary control channel from ctrlhello
|
|
|
|
- [x] 5.1 (P) Extend `handle_ctrlhello` to set `primary_ctrl_fd` on first ctrlhello when role is CTRL_STANDBY
|
|
- In the ctrlhello handler (config_repl.c): if `ha->role == CTRL_STANDBY` and `ha->primary_ctrl_fd == -1`, set `ha->primary_ctrl_fd = ep->fd`
|
|
- Call `ha_on_heartbeat_received(ha)` to reset the watchdog deadline (marks the first live message from primary)
|
|
- Guard with `primary_ctrl_fd == -1` for idempotency — repeated ctrlhellos on the same connection do not re-register
|
|
- Observable: unit test calls `handle_ctrlhello` with `ha->role = CTRL_STANDBY`, `ha->primary_ctrl_fd = -1`; afterward `ha->primary_ctrl_fd == ep->fd`; calling again with the same ep leaves `primary_ctrl_fd` unchanged
|
|
- _Requirements: 3.1, 3.2, 3.3_
|
|
- _Boundary: handle_ctrlhello_
|
|
|
|
- [x] 6. Integration: wire dispatcher and promoted-connection lifecycle
|
|
|
|
- [x] 6.1 Replace `vote_reply_msg_handler` assignment in `on_vote_connected` with `peer_ctrl_msg_handler`
|
|
- In `on_vote_connected` (ha.c): change `ep->msg_handler = vote_reply_msg_handler` to `ep->msg_handler = peer_ctrl_msg_handler`
|
|
- `vote_reply_msg_handler` retained for `vote_reply_msg_handler_for_test` under XCE_TESTING; body factored into `vote_reply_decode_and_dispatch`
|
|
- Observable: existing vote-send tests still pass with no behavior change; the `xce_vote_reply_hook` seam path is reached via `peer_ctrl_msg_handler` → `vote_reply_decode_and_dispatch`
|
|
- _Depends: 2.1_
|
|
- _Requirements: 1.2_
|
|
|
|
- [x] 6.2 Extend `vote_ep_err_handler` to remove promoted endpoint from `standby_endpoints` on disconnect
|
|
- When `vote_ep_err_handler` fires and `ha->role` is `CTRL_PRIMARY_COMMITTING` or `CTRL_PRIMARY`: acquire `ha->lock`, scan `ha->standby_endpoints` for `ep`, remove it, release lock
|
|
- Existing deny-vote path (fired during CTRL_ELECTING) is preserved unchanged — role gate ensures no double-action
|
|
- Observable: unit test populates `standby_endpoints` with a mock endpoint, sets role=COMMITTING, calls `vote_ep_err_handler`; verifies endpoint removed from array and `standby_endpoints->count` decremented
|
|
- _Requirements: 4.2, 5.5_
|
|
|
|
- [x] 7. Validation: tests and regression coverage
|
|
|
|
- [x] 7.1 Unit tests for dispatcher, promotion, and passive path
|
|
- `peer_ctrl_msg_handler` role-routing: three scenarios (ELECTING → vote hook; COMMITTING → repl-ack hook; STANDBY → drop), each confirmed by mock hooks
|
|
- `raft_cb_become_leader` promotion: `standby_endpoints` populated from non-NULL `votes[i].ep` entries; NULL entries skipped
|
|
- `raft_cb_become_follower` close path: all `votes[i].ep` set to NULL; `ha_arm_heartbeat_timeout_vtask` called; no `fetch_config_from_primary` dispatched
|
|
- `handle_ctrlhello` idempotency: first call sets `primary_ctrl_fd`; second call is a no-op
|
|
- `vote_ep_err_handler` standby removal: endpoint removed from `standby_endpoints` when role=COMMITTING
|
|
- Observable: `make test` for controller testsuite passes with no new failures; ASAN reports no leaks or UAF
|
|
- _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 4.1, 4.2, 5.5_
|
|
|
|
- [x] 7.2 Integration test: standby receives ctrlhello without active outbound primary connect
|
|
- Using existing two-node test harness in `test_ha_bootstrap.c` or `test_p3_integration.c`: boot two nodes, verify the standby never invokes `ha_find_and_connect_to_primary` after election
|
|
- Verify ctrlhello arrives on the standby within `election_timeout_ms + 200ms` via the promoted inbound connection
|
|
- Verify `ha->primary_ctrl_fd` is set on the standby after ctrlhello
|
|
- Observable: test passes end-to-end; ASAN reports no new errors compared to baseline
|
|
- _Requirements: 2.2, 2.4, 3.1, 5.3, 5.5_
|
|
|
|
- [x] 7.3 Integration test: promoted connection carries WAL replication ACK
|
|
- Verify `OPCode_replCatchUpDone` decoded on the promoted ep increments `ha->repl_ack_count` and triggers `ha_promote_to_primary` when majority threshold reached
|
|
- Observable: `repl_ack_count` reaches quorum - 1 and role transitions to CTRL_PRIMARY; ASAN clean
|
|
- _Requirements: 5.2, 5.4_
|