docs(raft-adapter-conformance): confirm task 7.4 live soak (vote/PreVote post-connect failure)

Found a clean, reusable fault-injection technique: vote/PreVote
endpoints are TLS, and endpoint_connect() succeeds at the TCP level
independent of the handshake, so an accept-then-close impostor
listener on a peer's port deterministically causes an async
SSL_do_handshake failure post-connect -- landing in
vote_ep_err_handler/pre_vote_ep_err_handler (task 3.1's fix target),
never the already-fixed bug-1423 connect-refused path.

Independently reproduced by the reviewer from scratch: both a
RequestVote round and a PreVote round resolved via the fast
all-replied-no-quorum path in single-digit milliseconds, vs the
5000ms configured election timeout (confirmed firing later as a
no-op). Requirements 4.2, 4.3, 4.5 proven live, both vote kinds.

All 15 raft-adapter-conformance sub-tasks now complete. 1067 tests,
0 failures, 8 commits' worth of production fixes (config-push reply
integrity, vote/PreVote failure routing, dormant-sender
documentation), plus 6 regression-lock/confirmation/live-soak tasks.
Two residual gaps logged as new bugs during the sweep (bug-1483,
bug-1484), both pre-existing and out of this spec's scope.
This commit is contained in:
Rongsong Shen 2026-07-30 08:01:35 +00:00
parent a48ed5c018
commit 46bbdaa9a6

View file

@ -105,7 +105,7 @@
- _Requirements: 2.2, 2.3, 2.5, 7.4, 7.5_
- _Depends: 2.1, 2.2_
- [ ]* 7.4 (P) Live soak: vote/PreVote post-connect failure end-to-end (optional, defer if no fault-injection mechanism exists)
- [x]* 7.4 (P) Live soak: vote/PreVote post-connect failure end-to-end (optional, defer if no fault-injection mechanism exists) — NOT deferred, ran successfully
- Attempt a soak round that severs an already-established outbound vote connection before a reply arrives (e.g. a firewall-rule toggle or proxy in the soak harness); if no such mechanism exists cheaply in `xce-test-ha`, defer this task — the unit tests from 3.2 already give deterministic coverage of the fix itself
- Observable, if run: the soak log shows the affected round reaching a quorum decision within the fast-tally path rather than the full election timeout
- _Requirements: 4.2, 4.3, 4.5_
@ -123,3 +123,5 @@
- `pre_vote_reply_dispatch` has no `XCE_TESTING` mock-hook seam (unlike `vote_reply_dispatch`'s `xce_vote_reply_hook`) — it calls `raft_on_pre_vote_reply` directly. Tests verifying PreVote routing must assert on `raft_state_t`'s own tally fields (`pre_vote_replies[]`, `pre_votes_granted`, `pre_vote_in_progress`) instead, per `test_ha_prevote.c`'s established technique.
- Task 7.2 (bug-1484, residual gap, NOT fixed — outside this spec's task list): no test anywhere in `testsuite/` drives the real `repl_ack_decode_and_dispatch` path and asserts `raft_on_higher_term_observed` fires for either the heartbeat-reply (ha.c ~6690-6711) or WAL-NACK (ha.c ~6805-6814) term-discovery vector bug-1421 introduced. `test_intake_higher_term_heartbeat_adopts_term` looks like it covers this but actually drives a different AE-request code path. Requirement 7.5's "already verified" premise currently rests on unverified ground for these two vectors specifically (the third, config-push, vector added by this feature IS tested — task 2.2). A future task should add a dedicated test mirroring `test_config_push_reply_higher_term_adopts_and_steps_down`'s pattern for the other two.
- Task 7.3: the real elected-primary's own outbound config-push has no TOCTOU window to inject a sha256 mismatch without new production/relay code (primary hashes and pushes from the same in-memory buffer) — confirmed by reading `config_commit_change`/`push_config_to_standbys`. Live soaks of this specific reject path must act as a direct client/sender against a standby (using `xce_replconfigpush_req_raw`, the real production wire function) rather than corrupting a real primary's push; decode the `ReplConfigPushReply` envelope via `xce_reply()`'s `Request{opcode:reply, body.replymsg}` wrapper (not a bare `RequestReply`) to get real accepted/term content instead of only inferring correctness from the receiver's local log line.
- Task 7.4: found a clean, cheap, reusable live fault-injection technique for "post-connect, pre-reply" endpoint failures — vote/PreVote endpoints are TLS (`ep->ssl = SSL_new(...)`), and `endpoint_connect()`'s return value comes only from the TCP-level `socket_connect()`, never from the TLS handshake (`netio.c:875-903`). Standing up a throwaway TCP listener on a configured peer's port that accepts-then-closes without any TLS bytes makes `endpoint_connect()` succeed (so the candidate genuinely reaches "connected") while deterministically causing an async `SSL_do_handshake` failure milliseconds later — landing squarely in `vote_ep_err_handler`/`pre_vote_ep_err_handler` (task 3.1's fix target), never the already-fixed synchronous connect-refused path (bug-1423). `iptables`/`nft` aren't installed in this sandbox and `tc netem` needs `CAP_NET_ADMIN` (not available); ptrace/gdb mid-flight attach is blocked by `yama ptrace_scope=1`. This accept-then-close-impostor technique generalizes to any "TLS-tier endpoint fails after TCP connect" live-soak need.
- `xce-deploy-ha/SKILL.md` documents `VERBOSE=-vvv scripts/start.sh ctrl-N &`, but `start.sh` never reads `$VERBOSE` — only positional args. Use `-vvv` as a positional arg (`scripts/start.sh ctrl-N -vvv &`), matching `xce-test-ha/SKILL.md`'s documented form, not the env-var form.