lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/ha-wal-replication/brief.md
Rongsong Shen 06135954ff chore: baseline snapshot — prior-session work + ha-consistency-invariants spec family 
Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x),
heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup.
New: ha-consistency-invariants umbrella (requirements/design/research) and
its four implementation sub-specs (specs generated via batch, cross-spec
review applied), steering roadmap + product.md Raft+CP decision.
2026-06-11 13:21:50 +00:00

3.2 KiB
Raw Blame History

Brief: ha-wal-replication

Problem

WAL entries are written to disk on the primary but never transmitted to standbys. repl_broadcast() is a no-send stub (casts payload to void, only increments counters). repl_wait_majority_ack() returns immediately because the replicator's standbys[] array is never populated from the HA layer's standby_endpoints. The result: job submits succeed locally but are invisible on standbys and not durable across failover.

Current State

  • repl.c: repl_add_standby(), repl_remove_standby(), repl_broadcast(), repl_wait_majority_ack() all exist with correct signatures. repl_broadcast is a stub (line 196-197: (void)payload; (void)payload_len).
  • persist.c: persist_wal_sync_append() calls repl_broadcast → repl_wait_majority_ack → advances committed_lsn. The call chain is wired; the send is what's missing.
  • ha.c / config_repl.c: ha->standby_endpoints manages per-standby connections (promoted vote eps, late-joiner connects). Nobody calls repl_add_standby() from here.
  • cluster.c: OPCode_walAck handler exists (ha_repl_ack_received) but is unreachable because standbys never receive WAL entries to ACK.

Desired Outcome

  1. WAL entries written on the primary are transmitted to all connected standbys.
  2. Each standby applies received WAL entries and sends an ACK.
  3. The primary advances committed_lsn only after a majority of standbys ACK.
  4. Job submits, state transitions, and queue changes survive primary failover.

Approach

Bridge the replicator and HA layer:

  1. Register: When the HA layer connects to a standby (promoted vote ep or late-joiner), call repl_add_standby() to register that endpoint with the replicator.
  2. Deregister: When a standby disconnects (RDHUP / err_handler), call repl_remove_standby().
  3. Send: Fill in repl_broadcast() to serialize the WAL header + payload into an OPCode_walAck or similar message and enqueue via xce_request().
  4. ACK: On the standby, when a WAL entry is applied, send a reply carrying the acked LSN. On the primary, handle_wal_ack calls repl_on_ack().

Scope

  • In: repl_broadcast TCP send, HA→replicator standby registration/deregistration, standby-side WAL ACK reply, integration with existing repl_wait_majority_ack.
  • Out: Changes to the heartbeat-driven decision loop (WAL streaming via NOP replies is already wired in ha-repl-hardening). Snapshot transfer path. Config replication path.

Boundary Candidates

  • repl_broadcast send implementation (repl.c)
  • Standby registration bridge (ha.c / config_repl.c → repl_add_standby)
  • Standby WAL ACK reply (cluster.c / ha.c)

Upstream / Downstream

  • Upstream: ha-config-bootstrap (repl_broadcast API, repl_wait_majority_ack), ha-repl-hardening (WAL catch-up payload, standby-side apply_wal_entry)
  • Downstream: None — this is the last missing link

Existing Spec Touchpoints

  • Reopens: ha-config-bootstrap task 7.6 (repl_broadcast TCP send deferred)
  • Adjacent: ha-repl-hardening (WAL payload already wired to standby apply path)

Constraints

  • Must use the existing repl_add_standby / repl_remove_standby API
  • Must reuse OPCode_walAck opcode for the ACK path
  • Standby WAL apply must be idempotent (replay-safe)