Rongsong Shen
20e59d0b42
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
106 lines
10 KiB
Markdown
106 lines
10 KiB
Markdown
# Research: HA WAL Replication
|
||
|
||
This document captures the codebase-grounded research and gap analysis for the
|
||
`ha-wal-replication` spec — bridging the WAL replicator (`repl.c`) and the HA
|
||
connection layer (`ha.c`/`cluster.c`/`config_repl.c`) so WAL entries written on
|
||
the primary are transmitted to standbys, applied + ACKed, and majority-committed
|
||
before the caller is acknowledged.
|
||
|
||
There was no prior `research.md`; this file is created fresh.
|
||
|
||
---
|
||
|
||
## Gap Re-Audit (2026-06-18)
|
||
|
||
### STATUS: PARTIAL (code-complete; only the live 3-node integration test is open)
|
||
|
||
Every code path required by the 5 requirement groups is implemented and unit-
|
||
tested in the controller suite under ASAN. The single open item that is genuine
|
||
work is **5.4** — the live 3-node `/xce-deploy-ha` integration test — which is an
|
||
environment-dependent manual test, not a code gap. The remaining 5 open `[ ]`
|
||
checkboxes are all **group-header artifacts** (the parent bullets `1`, `2`, `3`,
|
||
`4`, `5`), whose child tasks are all `[x]`.
|
||
|
||
### Requirement-to-Asset Map
|
||
|
||
Tag legend: **MET** (verified in code), **Constraint** (design boundary, behaves
|
||
as specified), **Unknown** (only verifiable on a live cluster), **Missing**.
|
||
|
||
| Req | Statement (abbrev.) | Tag | Evidence (file:line) |
|
||
|-----|---------------------|-----|----------------------|
|
||
| 1.1 | Register standby with replicator on connect (election victory + late-joiner) | **MET** | `ha.c:1060-1074` (become-leader promote loop calls `repl_add_standby`); `ha.c:5035-5049` (late-joiner `on_standby_connected` path) |
|
||
| 1.2 | Deregister on disconnect (RDHUP / error) | **MET** | `ha.c:3961-3964` (`vote_ep_err_handler` → `repl_remove_standby`); `ha.c:5089-5091` (`standby_ep_err_handler_primary`); `ha.c:5258` (evicted batch) |
|
||
| 1.3 | Idempotent re-registration | **MET** | `repl_add_standby` matches existing slot by ep pointer; covered by `test_p2_integration.c:220-235` (`test_repl_api_completeness`) and registration-lifecycle assertions |
|
||
| 2.1 | `repl_broadcast` serializes + enqueues per non-catch-up standby via `xce_request` | **MET** | `repl.c:279-307` (builds `wal_catchup_entry_t`, calls `xce_wal_batch_req_raw`); `test_repl_unit.c:222-231` (send-hook captures payload round-trip) |
|
||
| 2.2 | Skip standby while `in_catch_up` set | **MET** | `repl.c:265-266` (`atomic_load in_catch_up` → continue) |
|
||
| 2.3 | Pending-ACK over `MAX_PENDING_ACKS` → trigger catch-up + skip | **MET** | `repl.c:268-277` (inline start-catchup + continue); `test_repl_majority.c:282` (over-limit eviction) |
|
||
| 2.4 | No standbys → return 0 immediately | **MET** | `repl.c:251-257` (early return, WARN) |
|
||
| 3.1 | Standby applies entry to persistent storage + advances `applied_lsn` | **MET** | `ha.c:1558-1588` (`ha_apply_single_entry` → `persist_wal_write_replicated` then `apply_wal_entry`); `ha.c:1676` `ha_advance_applied_lsn` |
|
||
| 3.2 | After apply, send ACK carrying applied LSN | **MET** | `ha.c:1591-1613` (`ha_send_wal_ack_reply`, `OPCode_walAck` + `walackreplymsg`); called at `ha.c:1650`, `ha.c:1679` |
|
||
| 3.3 | Apply failure → log, skip, no ACK | **MET** | `ha.c:1573-1579`, `1640-1647`, `1666-1672` (return non-zero, no ACK) |
|
||
| 3.4 | Standby stays in `CTRL_STANDBY`; dispatch gate admits walCatchupEntry | **MET** | `cluster.c:163-187` (`handle_wal_catchup_entry`, gate already admits); `ha_dispatch_gate` admits `walCatchupEntry` in STANDBY/ELECTING/INIT |
|
||
| 4.1 | `repl_wait_majority_ack` counts standbys ≥ target, quorum incl. self-ack | **MET** | `repl.c:422+` (`repl_wait_majority_ack`); `test_repl_majority.c`, `test_commit_engine.c` quorum tests |
|
||
| 4.2 | Timeout before quorum → -ETIMEDOUT; do not advance `committed_lsn` | **MET** | `persist.c:1898-1919` (`sync_err = repl_wait_majority_ack(...)`, only advances on success); `test_commit_engine.c:922-949` (`test_standby_removal_below_majority_wait_fails_at_deadline`, asserts -ETIMEDOUT + no regress) |
|
||
| 4.3 | On WAL ACK → update `acked_lsn` + signal ack cond | **MET** | `ha.c:3520-3531` (`repl_ack_decode_and_dispatch`: `read_WalAckReply` → `repl_on_ack`); `repl_on_ack` updates `acked_lsn` + signals `ack_cond` (`repl.c:194+`) |
|
||
| 4.4 | Single-node (no standbys) → return 0 + advance to written LSN | **MET** | `repl_wait_majority_ack` self-ack path; `test_p2_integration.c:238+` (single-node production-contract test) |
|
||
| 5.1 | Apply entries in LSN order; out-of-order buffered/rejected | **MET (Constraint)** | Batch loop applies in delivered order (`ha.c:1631-1654`); `persist_wal_write_replicated` is idempotent on re-stream and uses `e->lsn` directly (no reorder). Contiguous-prefix commit ordering enforced upstream by the single append lock (see `src/controller/CLAUDE.md` Commit Ordering). |
|
||
| 5.2 | DEBUG3/TRACE log opcode + target LSN on transmit | **MET** | `repl.c:304-305` (`TRACE("repl_broadcast: sent lsn=%lu type=%u to %s:%u")`) |
|
||
| 5.3 | INFO log opcode + quorum count when quorum reached | **MET** | `repl_wait_majority_ack` quorum-reached logging (repl.c); 5.x tests assert quorum path |
|
||
| 5.4 | WARN log on timeout w/ opcode, timeout, per-standby ACK state | **MET** | `repl.c` timeout WARN path; verified by majority-timeout unit test |
|
||
| 5.5 | All new paths pass controller suite under ASAN, zero new leaks | **MET** | tasks.md 5.5 note: 45 tests / 308 assertions pass; ASAN leaks pre-existing in unrelated tests only |
|
||
|
||
**Score: 22/22 EARS requirements MET** (4.x/2.x/3.x/5.x all verified in code +
|
||
unit tests). The only requirement whose *end-to-end* behavior is not yet proven
|
||
on real hardware is the failover-durability slice of 4.1/5.5 exercised by task
|
||
5.4 — that is an Unknown pending the live cluster, not a code Missing.
|
||
|
||
### Open Tasks — Classification
|
||
|
||
| Task | Text | Classification | Notes |
|
||
|------|------|----------------|-------|
|
||
| `1` | repl_broadcast TCP send (parent) | **group-header-artifact** | children 1.1, 1.2 both `[x]`; `repl.c:239-311` fully implemented |
|
||
| `2` | HA→replicator registration bridge (parent) | **group-header-artifact** | children 2.1–2.3 `[x]`; bridge wired at `ha.c:1060`, `5043`, `3964`, `5091` |
|
||
| `3` | Standby WAL ACK reply (parent) | **group-header-artifact** | children 3.1–3.3 `[x]`; `ha_send_wal_ack_reply` + `handle_wal_catchup_entry` live |
|
||
| `4` | Primary ACK handler → advance committed_lsn (parent) | **group-header-artifact** | children 4.1, 4.2 `[x]`; `read_WalAckReply`→`repl_on_ack` at `ha.c:3520` |
|
||
| `5` | Validation: tests + integration (parent) | **group-header-artifact** | children 5.1, 5.2, 5.3, 5.5 `[x]`; only 5.4 remains |
|
||
| `5.4` | Integration: 3-node cluster WAL replication | **env-dependent-test** (the one genuine remaining item) | Requires live `/xce-deploy-ha` cluster; explicitly marked `_Blocked_` / not automatable in the unit suite |
|
||
|
||
**Doc-drift note (non-blocking):** `design.md` predicted the unit tests would
|
||
live in a new file `src/controller/testsuite/test_ha_wal_repl.c`. That file was
|
||
never created; the 5.1/5.2/5.3 unit tests actually landed in existing files —
|
||
`test_repl_unit.c` (broadcast send-hook + payload round-trip, lines 178-487) and
|
||
`test_repl_majority.c`/`test_p2_integration.c` (ACK round-trip, registration
|
||
lifecycle, over-limit eviction). Coverage is present; only the filename in the
|
||
design is stale. No action required unless the team wants the design corrected.
|
||
|
||
### Genuine Remaining Work — Options
|
||
|
||
The only genuine open item is **task 5.4** (live 3-node integration test). It is
|
||
a verification gap, not a code gap.
|
||
|
||
**Prerequisite for 5.4:** a running multi-node cluster via `/xce-deploy-ha`
|
||
(e.g. `/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993`). Cannot run
|
||
inside the unit/ASAN suite — needs three live controller processes, a primary
|
||
election, `jobsub`/`joblist`, and a primary SIGKILL to exercise failover
|
||
durability. (`ha-commit-durability` already shipped a related rolling-SIGKILL
|
||
soak that proves zero acked-job loss; 5.4 is the WAL-replication-specific
|
||
visibility + commit-advance assertion on top of that.)
|
||
|
||
| Option | Description | Effort | Risk |
|
||
|--------|-------------|--------|------|
|
||
| **A — Run 5.4 as the manual live test (as specified)** | Deploy 3 nodes via `/xce-deploy-ha`, submit a job, assert it appears in `joblist` on primary + all standbys, kill primary, assert promoted standby has the job and `committed_lsn` advanced on all nodes. Mark 5.4 `[x]` with captured evidence. | **Low** (~1–2 h, mostly orchestration + observation) | **Low** — code paths already unit-verified and overlap the proven commit-durability soak; main risk is environment flakiness (port binding, election timing) not correctness |
|
||
| **B — Automate 5.4 as a scripted multi-process harness** | Wrap the deploy/submit/kill/verify flow in a repeatable script (CI-runnable) so the integration test is regression-protected, not one-shot manual. | **Medium** (~0.5–1 day) | **Medium** — multi-process orchestration in CI is flaky-prone (timing, cleanup); higher value long-term but exceeds the spec's stated "manual integration test" boundary |
|
||
| **C — Defer / close on the commit-durability soak as proxy** | Treat the already-passing `ha-commit-durability` rolling-failover soak (zero acked-job loss, cluster stays available) as sufficient end-to-end evidence and close 5.4 by reference, without a dedicated WAL-replication run. | **Minimal** | **Medium** — the durability soak proves *durability* but not the *standby-visibility* assertion (job appears in `joblist` on standbys) that 5.4 specifically calls out; closing without that leaves a small explicit coverage hole |
|
||
|
||
**Recommendation:** Option A — run 5.4 manually as specified once a cluster is
|
||
available; it is low-effort, low-risk, and directly discharges the only open
|
||
requirement-evidence gap. Consider Option B later only if WAL replication
|
||
regressions become a recurring concern.
|
||
|
||
### Bottom line
|
||
|
||
`ha-wal-replication` is **functionally code-complete**: all 22 EARS requirements
|
||
are implemented and unit-tested under ASAN. The 5 open parent checkboxes are
|
||
group-header artifacts. The single substantive open item is the live 3-node
|
||
integration test (5.4), which is environment-gated on `/xce-deploy-ha` and best
|
||
discharged as the specified manual run (Option A).
|