lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/ha-peer-connection-reuse/research.md

15 KiB
Raw Blame History

Research & Design Decisions: ha-peer-connection-reuse

Discovery Scope

Type: Extension — modifies the existing election endpoint lifecycle in ha.c and standby connection model in cluster.c.

Discovery method: Light discovery — codebase analysis of existing patterns.

Key Investigations

1. Existing msg_handler Reassignment Pattern

Finding: on_vote_connected (ha.c:1869) already performs ep->msg_handler = vote_reply_msg_handler without a lock. This is safe because iomgr is single-threaded and msg_handler is only written from the iomgr thread.

Implication: A state-aware dispatcher (peer_ctrl_msg_handler) that reads ha->role atomically can replace this explicit reassignment without any iomgr changes. The role is already atomic (_Atomic int32_t role).

2. vote_ep_ctx_t Lifetime

Finding: vctx (vote_ep_ctx_t) lives in ep->data. It is currently freed by ha_election_cleanup. The vote_ep_err_handler intentionally does NOT free it because vote_reply_msg_handler also references ep->data.

Implication: With connection reuse, vctx outlives the election for promoted connections. Lifetime is extended: freed only when the promoted endpoint closes (via err_handler) or when election cleanup runs for the loser path.

3. ha->standby_endpoints Population

Finding: Currently populated in cluster.c when standbys create inbound connections to the primary and register with HostRole_controller. With connection reuse, the primary populates it directly from promoted outgoing vote endpoints in raft_cb_become_leader.

Implication: The existing ha_send_ctrlhello_to_all_standbys and push_config_to_standbys iterate ha->standby_endpoints. Populating it from promoted connections means no changes are needed to these functions — they work on whatever is in the array.

4. Standby Side: fetch_config_from_primary

Finding: raft_cb_become_follower (ha.c:457-462) schedules fetch_config_from_primary via delay_task_ms with 100ms delay. This is the tier-1/2/3 discovery path the spec wants to eliminate.

Implication: Remove this call. The primary pushes config via promoted connections. The standby receives it passively on the inbound connection (handled by existing vtaskmgr path).

5. Heartbeat Watchdog Arming

Finding: ha_arm_heartbeat_timeout_vtask is called after ha_find_and_connect_to_primary succeeds (production path). With connection reuse, ha_find_and_connect_to_primary is not called on the standby path.

Implication: Arm the watchdog in raft_cb_become_follower instead, immediately when transitioning to CTRL_STANDBY. The ha_on_heartbeat_received call from ctrlhello/heartbeat handlers resets the deadline.

6. Primary Control Channel (primary_ctrl_fd) on Standby

Finding: ha->primary_ctrl_fd is set by ha_find_and_connect_to_primary. The RDHUP and heartbeat watchdog use it as the anchor for failover. With connection reuse, the primary's inbound connection (accepted from winner's outgoing vote connection) is the primary control channel.

Implication: Set ha->primary_ctrl_fd = ep->fd in handle_ctrlhello when ctrlhello arrives and role == CTRL_STANDBY. The existing controller leave handler in cluster.c checks ep->fd == ha->primary_ctrl_fd to trigger re-election.

7. Promoted Connection Leave Handling

Finding: The cluster.c controller leave handler removes endpoints from ha->standby_endpoints. This fires for inbound connections by checking priv->role == HostRole_controller. Promoted outgoing vote endpoints have ep->data = vctx (vote_ep_ctx_t), not a role struct, so the existing leave handler cannot identify them.

Decision: Handle promoted endpoint disconnect in vote_ep_err_handler instead of the cluster.c leave handler. When role is COMMITTING/PRIMARY, the err_handler removes the endpoint from ha->standby_endpoints.

Architecture Decisions

Decision 1: State-Aware Dispatcher vs. Explicit Reassignment

Chosen: State-aware dispatcher (peer_ctrl_msg_handler) that reads ha->role atomically.

Why not explicit reassignment: Explicit reassignment in raft_cb_become_leader (which runs in the raft callback chain, still in iomgr thread) would be safe, but requires coordinating "which endpoints to reassign" and creates a narrow window where a message could arrive between role update and handler swap. The state-aware dispatcher eliminates this concern entirely.

Boundary: peer_ctrl_msg_handler is set once at endpoint creation and never changed.

Decision 2: No Changes to iomgr

Chosen: Zero changes to src/lib/netio.c or src/include/netio.h.

Why: endpoint_t already supports msg_handler reassignment via endpoint_set_msg_handler. The state-aware dispatcher approach doesn't reassign at all — it reads state at dispatch time. No iomgr API changes are needed.

Decision 3: vote_ep_ctx_t Reuse for Promoted Connections

Chosen: Keep vote_ep_ctx_t as ep->data for promoted connections. No separate replication context struct.

Why: vote_ep_ctx_t already carries ha, c, and peer_idx — exactly what replication ACK handling needs. Adding a promoted flag (or reading role) is sufficient to differentiate phases. No new struct reduces code churn.

Decision 4: ha_vote_entry_t.ep for Endpoint Tracking

Chosen: Add endpoint_t *ep to ha_vote_entry_t to track the outgoing vote endpoint per peer.

Why: raft_cb_become_leader and raft_cb_become_follower receive only ha_state_t*. They read ha->elect_ctx to get ha_elect_ctx_t* and iterate votes[] to find endpoints. This is the cleanest way to pass endpoints from raft_cb_send_vote_request to the callbacks without additional global state.

Synthesis Outcomes

Generalization: The "peer control channel" concept generalizes both the vote phase and the replication phase. One endpoint, two roles, one dispatcher. The role is the state machine.

Build vs. Adopt: All changes are purely additive/modificative within existing ha.c/cluster.c patterns. No new libraries.

Simplification: Removing fetch_config_from_primary from the standby path eliminates the tier-1/2/3 discovery complexity. The standby path becomes purely passive.

Post-Implementation Gap Analysis (2026-05-31)

Implementation Status

Tasks 1–6 (all core implementation) are complete. The following was verified in the live codebase:

Component Status Evidence
ha_vote_entry_t.ep field in ha.h ✅ Done ha.h:82
peer_ctrl_msg_handler (state-aware dispatcher) ✅ Done ha.c:2082
on_vote_connected uses peer_ctrl_msg_handler ✅ Done ha.c:1905
raft_cb_become_leader promotes vote eps to standby_endpoints ✅ Done ha.c:353–430
raft_cb_become_follower closes vote eps + arms watchdog ✅ Done ha.c:440–496
handle_ctrlhello sets primary_ctrl_fd (config_repl.c) ✅ Done config_repl.c:1596–1603
vote_ep_err_handler removes ep from standby_endpoints ✅ Done ha.c:2437–2442
Test file test_ha_connection_reuse.c (8 unit tests) ✅ Done testsuite/
Test file wired into meson.build ✅ Done meson.build:120

Gap 1 (CRITICAL): fetch_config_from_primary Still Present in ha_elect_evaluate_and_signal

Location: src/controller/ha.c, lines 1737–1743

/* ha_elect_evaluate_and_signal — STANDBY result path (old election path) */
#ifndef XCE_TESTING
if (c != NULL) {
    extern int fetch_config_from_primary(cluster_t *c);
    delay_task_ms(&c->delaymgr, 100, 1,
        (delay_task_entry_fn_t)fetch_config_from_primary, c);
}
#endif

Analysis: There are two code paths that can transition a node to CTRL_STANDBY:

  1. raft_cb_become_follower (raft library callback, ha.c:440–496) — the new path introduced by the election-iomgr-refactor spec. This path correctly does NOT call fetch_config_from_primary. ✅
  2. ha_elect_evaluate_and_signal (ha.c:~1702–1744) — the older custom election path (invoked directly by handle_request_vote_reply / vote-thread path). This path still calls fetch_config_from_primary. ❌

Requirement violated: Req 2.1 ("CTRL_STANDBY does not initiate outbound TCP connections") and Req 2.4 ("CTRL_STANDBY does not invoke tier-1/2/3 discovery").

Risk: Whether this matters depends on which path is actually active at runtime. If the system now exclusively uses the raft library callback path (and ha_elect_evaluate_and_signal is never called with a STANDBY result), the bug is dormant. But the old path still exists in the code and can be triggered by direct vote-reply handling, so it is a latent correctness issue and contradicts the spec intent.

Fix: Remove the #ifndef XCE_TESTING block at ha.c:1737–1743 (the fetch_config_from_primary call) from ha_elect_evaluate_and_signal. Matches task 4.1 intent, which was scoped only to raft_cb_become_follower.

Gap 2 (LOW): ha_arm_heartbeat_timeout_vtask Not Called from ha_elect_evaluate_and_signal

Analysis: Similarly to Gap 1, ha_elect_evaluate_and_signal does not call ha_arm_heartbeat_timeout_vtask in its STANDBY result path, while raft_cb_become_follower does (ha.c:495). If the old path is still reachable, the heartbeat watchdog is not armed from that path.

Requirement potentially violated: Req 2.3 and Req 3.3.

Fix: Mirror the watchdog arm from raft_cb_become_follower into ha_elect_evaluate_and_signal's STANDBY path, or confirm the old path is dead code and remove it.

Gap 3 (MEDIUM): Unit Tests Written But Not Verified Passing

Location: src/controller/testsuite/test_ha_connection_reuse.c

Analysis: The test file exists with 8 unit tests (task 7.1) and is wired into meson.build. However, task 7.1 is marked open in tasks.md. The tests reference several extern symbols that must be defined under XCE_TESTING in ha.c:

  • peer_ctrl_msg_handler_for_test — present (ha.c:2325)
  • vote_ep_err_handler_for_test — present (ha.c:2316)
  • vote_reply_build_pkt_for_test — present (ha.c:2123)
  • repl_catchup_done_pkt_for_test — present (ha.c:2282)
  • vote_ep_ctx_build_for_test — present (ha.c:2210)
  • raft_cb_become_leader_for_test — present (ha.c:2336)
  • raft_cb_become_follower_for_test — present (ha.c:2343)
  • ha_push_config_hook, ha_send_ctrlhello_hook — present (ha.c:1523, 1530)

All seams appear to be in place. The tests should pass, but a build run (/xce-build controller) is needed to confirm.

Gap 4 (LOW): Integration Tests Not Written (Tasks 7.2, 7.3)

Tasks open:

  • 7.2: Two-node integration test verifying standby never calls ha_find_and_connect_to_primary
  • 7.3: Integration test confirming WAL replication ACK on promoted ep increments repl_ack_count

Risk: Without integration tests, the full end-to-end behaviour (standby passive model, WAL replication via promoted connection) is unverified at the system level. The unit tests cover the individual components but not their interaction with the running cluster state machine.

Requirements Feasibility Assessment

Req Summary Status Gap
1.1 Candidate establishes outbound TLS per peer ✅ existing path unchanged None
1.2 Vote connection reused as peer channel ✅ peer_ctrl_msg_handler implemented None
1.3 Winner promotes to standby_endpoints ✅ raft_cb_become_leader extended None
1.4 Loser closes vote endpoints ✅ raft_cb_become_follower closes None
2.1 CTRL_STANDBY no outbound TCP ⚠️ partially Old path still calls fetch_config_from_primary
2.2 Standby accepts ctrlhello/config/WAL passively ✅ existing path unchanged None
2.3 No heartbeat → new election ✅ watchdog armed in new path Old path unarmed
2.4 No tier-1/2/3 discovery ⚠️ partially Old path still schedules it
3.1 ctrlhello → designate primary control channel ✅ handle_ctrlhello updated None
3.2 RDHUP detected in one epoll cycle ✅ existing mechanism None
3.3 Watchdog arm semantics preserved ✅ new path Old path unarmed
4.1 Loser closes all outbound vote eps ✅ raft_cb_become_follower None
4.2 Failed connect = denied vote, no reconnect ✅ vote_ep_err_handler unchanged None
4.3 New election = fresh connections ✅ ha_start_election_async unchanged None
5.1–5.4 No regression in HA behaviors ✅ promoted path uses existing iterators Integration tests needed
5.5 No new ASAN errors Unknown Needs build + test run

Implementation Approach Summary

Chosen approach: Option A (extend existing components) — completed for tasks 1–6.

Remaining work for full compliance:

  1. (Critical — Req 2.1, 2.4) Remove fetch_config_from_primary call from ha_elect_evaluate_and_signal STANDBY path (ha.c:1737–1743), or confirm the function is never reached on the STANDBY branch in production. Mirror ha_arm_heartbeat_timeout_vtask call if the path is still live.
  2. (Verification — task 7.1) Run /xce-build controller to confirm all unit tests in test_ha_connection_reuse.c pass and ASAN is clean.
  3. (Integration — task 7.2) Add two-node test verifying standby does not call ha_find_and_connect_to_primary.
  4. (Integration — task 7.3) Add promoted-connection WAL ACK test.

Effort and Risk

Item Effort Risk
Remove fetch_config_from_primary from old path S (1 hour) Low — isolated removal; old path analysis needed first
Verify unit tests pass S (build + inspect) Low — seams all present
Task 7.2 integration test M (1–2 days) Medium — requires two-node test harness plumbing
Task 7.3 integration test M (1–2 days) Medium — requires promoted ep + WAL harness

Overall: The core feature is implemented. The remaining work is test validation + one latent correctness issue in the old election path.

Cross-Spec Gap Audit (2026-06-08)

Batch /kiro-validate-gap across 9 specs on persistent_ha branch.

Analysis Summary

  • ha-peer-connection-reuse is ~98% complete. All 7 task groups (1–7) are [x] in tasks.md.
  • Prior CRITICAL gap RESOLVED: The fetch_config_from_primary call in ha_elect_evaluate_and_signal's STANDBY path was removed (confirmed: only comments remain in ha.c). The standby is fully passive — no outbound TCP connections from CTRL_STANDBY.
  • Integration tests (7.2, 7.3): Both marked [x]. The two-node integration test and WAL ACK test are done.
  • Remaining: Build verification (/xce-build controller) to confirm all unit tests in test_ha_connection_reuse.c pass under ASAN.
  • Cross-spec relationships: ha-repl-hardening commit 0762577 (task 5.1: "reduce handle_ctrlhello to connection setup only") directly depends on this spec's peer_ctrl_msg_handler + promoted connection model. The two specs are consistent — ha-repl-hardening moves all replication decisions to the heartbeat decision loop, which works over the promoted connections established by this spec.