Rongsong Shen
06135954ff
Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x), heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup. New: ha-consistency-invariants umbrella (requirements/design/research) and its four implementation sub-specs (specs generated via batch, cross-spec review applied), steering roadmap + product.md Raft+CP decision.
103 lines
8.7 KiB
Markdown
103 lines
8.7 KiB
Markdown
# Implementation Plan
|
||
|
||
- [x] 1. Foundation: state fields and startup validation
|
||
- [x] 1.1 Add liveness state fields to ha_state_t and standby_ep_ctx_t
|
||
- Add `time_t last_activity` to `ha_state_t` in ha.h; zero-initialize in `ha_init()`
|
||
- Add `time_t last_push_time` to `standby_ep_ctx_t` struct (ha.c internal); initialize to 0 in all allocation sites
|
||
- Remove `hb_vtask_t` typedef declaration from ha.h (struct moved to ha.c or fully removed)
|
||
- Observable: project builds; `ha_init()` test verifies `last_activity == 0` after init
|
||
- _Requirements: 1.1, 1.3, 3.1_
|
||
|
||
- [x] 1.2 Add heartbeat timing validation at startup
|
||
- Implement `validate_heartbeat_config(heartbeat_timeout_ms, heartbeat_interval_ms, err_buf, err_len)` in `startup_validation.c`
|
||
- Returns `-EINVAL` when `heartbeat_timeout_ms < 2 * heartbeat_interval_ms`; writes human-readable error into `err_buf`
|
||
- Wire call into `main.c` after `ha_config_t` is populated, before `ha_init()`
|
||
- Observable: starting controller with `heartbeat_timeout_ms = heartbeat_interval_ms` prints error and exits non-zero
|
||
- _Requirements: 2.4_
|
||
- _Boundary: startup_validation.c, main.c_
|
||
|
||
- [x] 2. Core: standby liveness poll (replaces watchdog thread)
|
||
- [x] 2.1 (P) Implement liveness_poll_fn and replace hb_timeout_thread
|
||
- Define `liveness_poll_ctx_t` struct in ha.c: `{ ha_state_t *ha; cluster_t *c; _Atomic int cancelled; }`
|
||
- Implement `liveness_poll_fn(void *arg)`: check `cancelled` first; check `atomic_load(&ha->role) == CTRL_STANDBY` to guard against duplicate fires with EPOLLRDHUP path; read `ha->last_activity`, compute elapsed; if `elapsed >= heartbeat_timeout_ms` call `ha_on_primary_ctrl_fd_rdhup(ha, c)`; else re-arm via `delay_task_ms` at `heartbeat_interval_ms / 2`
|
||
- Replace `ha_arm_heartbeat_timeout_vtask`: allocate `liveness_poll_ctx_t`, write `clock_gettime` to `ha->last_activity`, call `delay_task_ms` to schedule first poll; log INFO "HA: standby liveness poll armed"
|
||
- Replace `ha_disarm_heartbeat_timeout_vtask`: set `ctx->cancelled = 1`, free ctx, set static ptr to NULL; no pthread_join needed
|
||
- Add `ha_disarm_heartbeat_timeout_vtask` call at the top of `ha_on_primary_ctrl_fd_rdhup` so that EPOLLRDHUP cancels any pending poll task before transitioning to CTRL_ELECTING (Req 5.4)
|
||
- Update `ha_on_heartbeat_received`: write `clock_gettime(CLOCK_MONOTONIC)` seconds to `ha->last_activity`; log DEBUG3 on each reset (not INFO to avoid log spam)
|
||
- Remove `hb_vtask_t` struct, `hb_timeout_thread` function, `s_hb_vtask` static variable from ha.c
|
||
- Observable: `grep -r hb_timeout_thread src/controller/*.c` returns zero matches; build passes with no new warnings; SIGKILL primary → standbys show exactly one `ha_on_primary_ctrl_fd_rdhup` log each (no duplicate fires)
|
||
- _Requirements: 1.2, 1.4, 1.5, 1.6, 2.1, 2.2, 2.3, 4.2, 4.3, 5.2, 5.3, 5.4, 5.6_
|
||
- _Boundary: ha.c (standby liveness path)_
|
||
|
||
- [x] 2.2 (P) Extend primary_heartbeat_fn with per-standby idle tracking
|
||
- In `primary_heartbeat_fn`: for each ep in `ha->standby_endpoints`, cast `ep->data` to `standby_ep_ctx_t` and read `sctx->last_push_time`
|
||
- Compute `elapsed = now - sctx->last_push_time`; if `elapsed >= heartbeat_interval_ms` call `xce_nop_req(ep)` and write `sctx->last_push_time = now`; else skip NOP for this standby
|
||
- In `on_standby_connected`: write `sctx->last_push_time = clock_gettime(...)` immediately after TLS handshake completes (before ctrlhello send); this counts the ctrlhello as the first push
|
||
- Remove the workaround `xce_nop_req(ep)` call added at the end of `on_standby_connected` (no longer needed)
|
||
- Observable: with 2 standbys active and no WAL traffic, primary sends NOP to each standby at most once per `heartbeat_interval_ms`; verified via DEBUG3 log counting
|
||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5_
|
||
- _Boundary: ha.c (primary heartbeat path, standby_ep_ctx_t)_
|
||
|
||
- [x] 3. Integration: wire liveness reset into opcode dispatch
|
||
- [x] 3.1 Update cluster.c opcode handlers to reset standby liveness on any primary message
|
||
- `OPCode_nop` handler: replace `if (ep->fd == c->ha.primary_ctrl_fd)` guard with `if (atomic_load(&c->ha.role) == CTRL_STANDBY)` — call `ha_on_heartbeat_received(&c->ha)`
|
||
- `OPCode_hello` handler (ctrlhello path): after the existing `handle_ctrlhello` call, additionally call `ha_on_heartbeat_received(&c->ha)` when role is CTRL_STANDBY (ctrlhello counts as primary activity per Req 4.1)
|
||
- `OPCode_replEntry` and `OPCode_walCatchupEntry` handlers: add `ha_on_heartbeat_received(&c->ha)` call when role is CTRL_STANDBY
|
||
- Observable: in a running 3-node cluster, after primary sends ctrlhello, `ha->last_activity` timestamp is fresh and standby liveness poll does not fire within `heartbeat_timeout_ms`
|
||
- _Requirements: 1.3, 4.1_
|
||
- _Depends: 2.1_
|
||
- _Boundary: cluster.c (opcode dispatch)_
|
||
|
||
- [x] 3.2 Verify config_repl.c ctrlhello integration and EPOLLRDHUP cancellation path
|
||
- Confirm `ha_arm_heartbeat_timeout_vtask` call in `handle_ctrlhello` (config_repl.c:1380) still correctly arms the delaymgr poll (not a thread)
|
||
- Confirm `ha_on_heartbeat_received` call at config_repl.c:1379 writes `last_activity` (new impl)
|
||
- Confirm `ha_on_primary_ctrl_fd_rdhup` now calls `ha_disarm_heartbeat_timeout_vtask` as its first step (Req 5.4)
|
||
- Observable: build passes; `handle_ctrlhello` unit test (existing) still green; `ha_arm_heartbeat_timeout_vtask` no longer spawns a pthread; SIGKILL primary test shows exactly one RDHUP log per standby (no duplicate from poll)
|
||
- _Requirements: 1.3, 4.1, 4.2, 5.2, 5.3, 5.4_
|
||
- _Depends: 2.1, 3.1_
|
||
- _Boundary: config_repl.c, ha.c (ha_on_primary_ctrl_fd_rdhup)_
|
||
|
||
- [x] 4. Validation: unit and integration tests
|
||
- [x] 4.1 (P) Unit tests for standby liveness poll
|
||
- Test `liveness_poll_fn` fires `ha_on_primary_ctrl_fd_rdhup` when `now - last_activity >= heartbeat_timeout_ms`
|
||
- Test `liveness_poll_fn` re-arms (no failover) when `now - last_activity < heartbeat_timeout_ms`
|
||
- Test `liveness_poll_fn` with `cancelled = 1` is a no-op (no failover, no re-arm)
|
||
- Test `ha_on_heartbeat_received` updates `ha->last_activity` to a non-zero value
|
||
- Test `ha_arm_heartbeat_timeout_vtask` does NOT spawn a pthread (thread count unchanged before/after call)
|
||
- Test `validate_heartbeat_config`: `timeout = 2 × interval` passes; `timeout = interval - 1` fails
|
||
- Observable: `meson test --suite controller` exits 0 with all new test cases green; zero ASAN errors
|
||
- _Requirements: 1.2, 1.4, 1.5, 1.6, 2.1, 2.2, 2.3, 2.4_
|
||
- _Boundary: ha.c unit tests, startup_validation.c unit tests_
|
||
- _Depends: 2.1, 1.2_
|
||
|
||
- [x] 4.2 (P) Unit tests for primary idle NOP suppression
|
||
- Test `primary_heartbeat_fn`: sends NOP to standby when `last_push_time` elapsed ≥ `heartbeat_interval_ms`
|
||
- Test `primary_heartbeat_fn`: skips NOP when `last_push_time` elapsed < `heartbeat_interval_ms`
|
||
- Test `on_standby_connected` writes non-zero `last_push_time` immediately
|
||
- Observable: all 3 test cases pass; `meson test --suite controller` exits 0
|
||
- _Requirements: 3.1, 3.2, 3.3, 3.4_
|
||
- _Boundary: ha.c unit tests_
|
||
- _Depends: 2.2_
|
||
|
||
- [x] 4.3 Integration test: 3-node cold start — no false RDHUP loops and race elimination
|
||
- Deploy 3-node cluster (ctrl-1 has config, ctrl-2/3 tag=0); verify ctrl-1 elects PRIMARY
|
||
- Verify ctrl-2/3 receive ctrlhello, set `last_activity`, and do NOT trigger RDHUP before `heartbeat_timeout_ms` elapses from ctrlhello
|
||
- Verify cluster is stable for 60s with no `hb_timeout_thread` log entries (thread removed) and no RDHUP false-positive entries
|
||
- Observable: `grep "hb_timeout_thread\|heartbeat timeout" /tmp/xce-deploy/logs/ctrl-{2,3}.log` returns 0 lines during steady state; all 3 processes alive after 60s
|
||
- _Requirements: 1.3, 1.4, 4.1, 4.2, 4.3, 5.1, 5.3_
|
||
- _Depends: 3.1, 3.2_
|
||
|
||
- [x] 4.4 Integration test: primary crash triggers failover within timeout window
|
||
- Start 3-node cluster; verify ctrl-1 PRIMARY and cluster stable
|
||
- SIGKILL ctrl-1; verify ctrl-2 or ctrl-3 detects silence within `heartbeat_timeout_ms`; promotes to PRIMARY via RDHUP path
|
||
- Verify promoted primary serves traffic; verify no infinite RDHUP loop
|
||
- Observable: promoted standby log shows `ha_on_primary_ctrl_fd_rdhup → CTRL_ELECTING → CTRL_PRIMARY` exactly once; cluster operational within `heartbeat_timeout_ms + election_timeout_ms + committing_timeout_ms + 5s`
|
||
- _Requirements: 1.4, 2.1, 2.2, 2.3, 5.2_
|
||
- _Depends: 4.3_
|
||
|
||
- [x]* 4.5 Integration test: missed-heartbeat tolerance (SIGSTOP/SIGCONT)
|
||
- Start 3-node cluster; SIGSTOP ctrl-1 for `1 × heartbeat_interval_ms`; SIGCONT; verify no failover fires
|
||
- SIGSTOP ctrl-1 for `heartbeat_timeout_ms + 500ms`; verify failover fires exactly once
|
||
- Observable: first scenario — no RDHUP log on standbys; second scenario — exactly one `ha_on_primary_ctrl_fd_rdhup` log per standby
|
||
- _Requirements: 2.2, 2.3_
|
||
- _Depends: 4.4_
|