myxce/.kiro/specs/ha-heartbeat-liveness/tasks.md

Implementation Plan

• 1. Foundation: state fields and startup validation

• 1.1 Add liveness state fields to ha_state_t and standby_ep_ctx_t

  • Add time_t last_activity to ha_state_t in ha.h; zero-initialize in ha_init()
  • Add time_t last_push_time to standby_ep_ctx_t struct (ha.c internal); initialize to 0 in all allocation sites
  • Remove hb_vtask_t typedef declaration from ha.h (struct moved to ha.c or fully removed)
  • Observable: project builds; ha_init() test verifies last_activity == 0 after init
  • Requirements: 1.1, 1.3, 3.1

• 1.2 Add heartbeat timing validation at startup

  • Implement validate_heartbeat_config(heartbeat_timeout_ms, heartbeat_interval_ms, err_buf, err_len) in startup_validation.c
  • Returns -EINVAL when heartbeat_timeout_ms < 2 * heartbeat_interval_ms; writes human-readable error into err_buf
  • Wire call into main.c after ha_config_t is populated, before ha_init()
  • Observable: starting controller with heartbeat_timeout_ms = heartbeat_interval_ms prints error and exits non-zero
  • Requirements: 2.4
  • Boundary: startup_validation.c, main.c

• 2. Core: standby liveness poll (replaces watchdog thread)

• 2.1 (P) Implement liveness_poll_fn and replace hb_timeout_thread

  • Define liveness_poll_ctx_t struct in ha.c: { ha_state_t *ha; cluster_t *c; _Atomic int cancelled; }
  • Implement liveness_poll_fn(void *arg): check cancelled first; check atomic_load(&ha->role) == CTRL_STANDBY to guard against duplicate fires with EPOLLRDHUP path; read ha->last_activity, compute elapsed; if elapsed >= heartbeat_timeout_ms call ha_on_primary_ctrl_fd_rdhup(ha, c); else re-arm via delay_task_ms at heartbeat_interval_ms / 2
  • Replace ha_arm_heartbeat_timeout_vtask: allocate liveness_poll_ctx_t, write clock_gettime to ha->last_activity, call delay_task_ms to schedule first poll; log INFO "HA: standby liveness poll armed"
  • Replace ha_disarm_heartbeat_timeout_vtask: set ctx->cancelled = 1, free ctx, set static ptr to NULL; no pthread_join needed
  • Add ha_disarm_heartbeat_timeout_vtask call at the top of ha_on_primary_ctrl_fd_rdhup so that EPOLLRDHUP cancels any pending poll task before transitioning to CTRL_ELECTING (Req 5.4)
  • Update ha_on_heartbeat_received: write clock_gettime(CLOCK_MONOTONIC) seconds to ha->last_activity; log DEBUG3 on each reset (not INFO to avoid log spam)
  • Remove hb_vtask_t struct, hb_timeout_thread function, s_hb_vtask static variable from ha.c
  • Observable: grep -r hb_timeout_thread src/controller/*.c returns zero matches; build passes with no new warnings; SIGKILL primary → standbys show exactly one ha_on_primary_ctrl_fd_rdhup log each (no duplicate fires)
  • Requirements: 1.2, 1.4, 1.5, 1.6, 2.1, 2.2, 2.3, 4.2, 4.3, 5.2, 5.3, 5.4, 5.6
  • Boundary: ha.c (standby liveness path)

• 2.2 (P) Extend primary_heartbeat_fn with per-standby idle tracking

  • In primary_heartbeat_fn: for each ep in ha->standby_endpoints, cast ep->data to standby_ep_ctx_t and read sctx->last_push_time
  • Compute elapsed = now - sctx->last_push_time; if elapsed >= heartbeat_interval_ms call xce_nop_req(ep) and write sctx->last_push_time = now; else skip NOP for this standby
  • In on_standby_connected: write sctx->last_push_time = clock_gettime(...) immediately after TLS handshake completes (before ctrlhello send); this counts the ctrlhello as the first push
  • Remove the workaround xce_nop_req(ep) call added at the end of on_standby_connected (no longer needed)
  • Observable: with 2 standbys active and no WAL traffic, primary sends NOP to each standby at most once per heartbeat_interval_ms; verified via DEBUG3 log counting
  • Requirements: 3.1, 3.2, 3.3, 3.4, 3.5
  • Boundary: ha.c (primary heartbeat path, standby_ep_ctx_t)

• 3. Integration: wire liveness reset into opcode dispatch

• 3.1 Update cluster.c opcode handlers to reset standby liveness on any primary message

  • OPCode_nop handler: replace if (ep->fd == c->ha.primary_ctrl_fd) guard with if (atomic_load(&c->ha.role) == CTRL_STANDBY) — call ha_on_heartbeat_received(&c->ha)
  • OPCode_hello handler (ctrlhello path): after the existing handle_ctrlhello call, additionally call ha_on_heartbeat_received(&c->ha) when role is CTRL_STANDBY (ctrlhello counts as primary activity per Req 4.1)
  • OPCode_replEntry and OPCode_walCatchupEntry handlers: add ha_on_heartbeat_received(&c->ha) call when role is CTRL_STANDBY
  • Observable: in a running 3-node cluster, after primary sends ctrlhello, ha->last_activity timestamp is fresh and standby liveness poll does not fire within heartbeat_timeout_ms
  • Requirements: 1.3, 4.1
  • Depends: 2.1
  • Boundary: cluster.c (opcode dispatch)

• 3.2 Verify config_repl.c ctrlhello integration and EPOLLRDHUP cancellation path

  • Confirm ha_arm_heartbeat_timeout_vtask call in handle_ctrlhello (config_repl.c:1380) still correctly arms the delaymgr poll (not a thread)
  • Confirm ha_on_heartbeat_received call at config_repl.c:1379 writes last_activity (new impl)
  • Confirm ha_on_primary_ctrl_fd_rdhup now calls ha_disarm_heartbeat_timeout_vtask as its first step (Req 5.4)
  • Observable: build passes; handle_ctrlhello unit test (existing) still green; ha_arm_heartbeat_timeout_vtask no longer spawns a pthread; SIGKILL primary test shows exactly one RDHUP log per standby (no duplicate from poll)
  • Requirements: 1.3, 4.1, 4.2, 5.2, 5.3, 5.4
  • Depends: 2.1, 3.1
  • Boundary: config_repl.c, ha.c (ha_on_primary_ctrl_fd_rdhup)

• 4. Validation: unit and integration tests

• 4.1 (P) Unit tests for standby liveness poll

  • Test liveness_poll_fn fires ha_on_primary_ctrl_fd_rdhup when now - last_activity >= heartbeat_timeout_ms
  • Test liveness_poll_fn re-arms (no failover) when now - last_activity < heartbeat_timeout_ms
  • Test liveness_poll_fn with cancelled = 1 is a no-op (no failover, no re-arm)
  • Test ha_on_heartbeat_received updates ha->last_activity to a non-zero value
  • Test ha_arm_heartbeat_timeout_vtask does NOT spawn a pthread (thread count unchanged before/after call)
  • Test validate_heartbeat_config: timeout = 2 × interval passes; timeout = interval - 1 fails
  • Observable: meson test --suite controller exits 0 with all new test cases green; zero ASAN errors
  • Requirements: 1.2, 1.4, 1.5, 1.6, 2.1, 2.2, 2.3, 2.4
  • Boundary: ha.c unit tests, startup_validation.c unit tests
  • Depends: 2.1, 1.2

• 4.2 (P) Unit tests for primary idle NOP suppression

  • Test primary_heartbeat_fn: sends NOP to standby when last_push_time elapsed ≥ heartbeat_interval_ms
  • Test primary_heartbeat_fn: skips NOP when last_push_time elapsed < heartbeat_interval_ms
  • Test on_standby_connected writes non-zero last_push_time immediately
  • Observable: all 3 test cases pass; meson test --suite controller exits 0
  • Requirements: 3.1, 3.2, 3.3, 3.4
  • Boundary: ha.c unit tests
  • Depends: 2.2

• 4.3 Integration test: 3-node cold start — no false RDHUP loops and race elimination

  • Deploy 3-node cluster (ctrl-1 has config, ctrl-2/3 tag=0); verify ctrl-1 elects PRIMARY
  • Verify ctrl-2/3 receive ctrlhello, set last_activity, and do NOT trigger RDHUP before heartbeat_timeout_ms elapses from ctrlhello
  • Verify cluster is stable for 60s with no hb_timeout_thread log entries (thread removed) and no RDHUP false-positive entries
  • Observable: grep "hb_timeout_thread\|heartbeat timeout" /tmp/xce-deploy/logs/ctrl-{2,3}.log returns 0 lines during steady state; all 3 processes alive after 60s
  • Requirements: 1.3, 1.4, 4.1, 4.2, 4.3, 5.1, 5.3
  • Depends: 3.1, 3.2

• 4.4 Integration test: primary crash triggers failover within timeout window

  • Start 3-node cluster; verify ctrl-1 PRIMARY and cluster stable
  • SIGKILL ctrl-1; verify ctrl-2 or ctrl-3 detects silence within heartbeat_timeout_ms; promotes to PRIMARY via RDHUP path
  • Verify promoted primary serves traffic; verify no infinite RDHUP loop
  • Observable: promoted standby log shows ha_on_primary_ctrl_fd_rdhup → CTRL_ELECTING → CTRL_PRIMARY exactly once; cluster operational within heartbeat_timeout_ms + election_timeout_ms + committing_timeout_ms + 5s
  • Requirements: 1.4, 2.1, 2.2, 2.3, 5.2
  • Depends: 4.3

• * 4.5 Integration test: missed-heartbeat tolerance (SIGSTOP/SIGCONT)

  • Start 3-node cluster; SIGSTOP ctrl-1 for 1 × heartbeat_interval_ms; SIGCONT; verify no failover fires
  • SIGSTOP ctrl-1 for heartbeat_timeout_ms + 500ms; verify failover fires exactly once
  • Observable: first scenario — no RDHUP log on standbys; second scenario — exactly one ha_on_primary_ctrl_fd_rdhup log per standby
  • Requirements: 2.2, 2.3
  • Depends: 4.4