---
id: "bug-1386"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-02T08:15:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/config_repl.c"]
tags: ["raft-drive-async", "snapshot-transfer", "silent-failure", "force-snapshot-soak"]
related: ["bug-1197", "bug-1376", "bug-1381"]
legacy: true
---

# bug-1386: /xce-test-ha --force-snapshot round: after a wiped ctrl-2 rejoins empty and the primary sends a…

## Description
/xce-test-ha --force-snapshot round: after a wiped ctrl-2 rejoins empty and the primary sends a full-checkpoint snapshot, ctrl-2 sometimes never logs a completed install ("HA: snapshot loaded/legacy" line absent) though the cluster still reconverges to 3-of-3 -- OR the cluster fails to reconverge to 3-of-3 at all. Reproduced with g_drive_async_configpush_offload BOTH ON (0/3 pass) and OFF (1/2 pass) on the same freshly-built xce.build binary, so it is independent of that flag -- discovered while validating the bug-1381 configpush flip.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
ha_send_snapshot_to_standby (config_repl.c, ~line 2596) loops over up to 12 65536-byte chunks calling xce_snapshot_chunk_req_raw(ep, &chunk, buf, chunk_len) WITHOUT checking or propagating its return value. xce_snapshot_chunk_req_raw -> endpoint_enq can return -ECONNRESET (e.g. if the standby endpoint is mid-teardown or reconnecting during the empty-rejoin window) but the loop keeps incrementing offset/chunk_index regardless, and the function still logs "transfer complete" and returns rc=0 at the end. Confirmed via ctrl-2 log evidence: primary logs start+complete for a 735546-byte (12-chunk) snapshot within ~2ms, but ctrl-2 only ever receives a handful of leading chunks (2 in one run, 5 in another) before the stream silently stops -- no ERROR on either side.

## Fix
FIXED 2026-07-02 (config_repl.c): ha_send_snapshot_to_standby's chunk loop now captures the return value of xce_snapshot_chunk_req_raw (production path) and xce_snapshot_chunk_hook (XCE_TESTING path) into send_rc, breaks the loop on send_rc != 0, and propagates it as the function's rc instead of silently continuing. The 'transfer complete' INFO log is now strictly gated on rc==0; a failure path now logs a WARN with checkpoint_lsn/total_size/sent-offset/chunk_index/rc for observability. No receive-side timeout/re-request mechanism was added (out of scope, no existing seam) — the existing per-peer snapshot_inflight[] flag already clears unconditionally on send-thread exit (snapshot_lane_fn, ha.c) and the next heartbeat's SEND_SNAPSHOT decision naturally re-triggers a fresh transfer to a peer whose applied_lsn never advanced, so the existing retry cadence recovers once the endpoint is healthy again. New RED->GREEN unit test test_snapshot_transfer_stops_and_propagates_send_failure (test_config_repl.c) injects a mid-loop -ECONNRESET via the existing xce_snapshot_chunk_hook seam (new snaptest_fail_after_hook) and asserts rc == -ECONNRESET and exactly 2 chunks attempted (not the false-'complete' 3). RED confirmed via temporary revert ('Expected -104 Was 0'); GREEN restored, full controller_unit_test suite clean (0 failures across all groups).

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1197, bug-1376, bug-1381