feat(config-upgrade-snapshot): task 8.9 — live WAL-trim-under-chaos soak infra, blocked on bug-1474
Add run_config_upgrade_wal_trim_round() / --config-upgrade-wal-trim to run-ha-soak.sh: forces real WAL segment rotation via small --wal-segment-bytes, kills a non-primary node, drives a live config upgrade to fire the marker-triggered checkpoint's forced trim, then inspects survivors' on-disk WAL directory before the killed node rejoins to directly verify the mid-catch-up retention guard (repl_trim_lsn/trim_wal_segments, bug-1147) preserves what it still needs. Two independent live 3-node runs both reproduce a real, pre-existing, out-of-spec-boundary defect (bug-1474): a heartbeat reconnect-sweep race reuses a just-killed standby's freed replicator slot and clobbers its retention guard before trim ever runs. Root-caused via repl.c/ha.c and confirmed via git blame to predate this spec and be reachable under the old periodic-checkpoint trigger identically -- not caused by anything config-upgrade-snapshot changed. Task 8.9 stays blocked per tasks.md until bug-1474 is fixed upstream; infra is done and correctly proves the violation. Independently reviewed: round design verified faithful to the unit-level fixture, bug-1474 diagnosis independently re-derived from source and raw logs (not trusted from the implementer's report).
This commit is contained in:
parent
b78efd03b4
commit
8b3e2ab03c
3 changed files with 452 additions and 1 deletions
|
|
@ -343,6 +343,108 @@ inside `run_config_upgrade_round()`, so a violation aborts the run before
|
|||
this line is ever reached; there is no `verdict_emit`/bug-924 exclusion
|
||||
tally for this mode.
|
||||
|
||||
### `--config-upgrade-wal-trim` mode: marker-triggered WAL-segment trim under a genuinely mid-catch-up standby (config-upgrade-snapshot task 8.9)
|
||||
|
||||
Repeats the mid-catch-up WAL-trim assertion from the unit-level fixture
|
||||
(`test_marker_triggered_checkpoint_trim_preserves_standby_mid_catchup_segments`,
|
||||
`test_upgrade_snapshot_apply.c`) against a **real 3-node cluster** under
|
||||
chaotic timing, instead of hand-controlled LSNs — proving Requirement 7.4:
|
||||
the marker-triggered checkpoint's forced WAL trim
|
||||
(`persist_checkpoint_marker_triggered` → `trim_wal_segments` →
|
||||
`repl_trim_lsn`, bug-1147's pre-existing retention-floor guard, unmodified
|
||||
by this spec) never strands a genuinely mid-catch-up standby's still-needed
|
||||
WAL segment. See `run_config_upgrade_wal_trim_round()`'s own header comment
|
||||
(`run-ha-soak.sh`) for the full mechanism trace, including why the "chaotic"
|
||||
element is sequenced *before* the critical window rather than concurrent
|
||||
with it (a 3-node cluster has no majority left if 2 of 3 are ever
|
||||
simultaneously non-participants, and TARGET must stay down for the *whole*
|
||||
critical window to be genuinely mid-catch-up).
|
||||
|
||||
Sequence:
|
||||
|
||||
1. Deploys with a small `--wal-segment-bytes` (`4096`, same proven value
|
||||
`--force-snapshot` already uses) so segments genuinely rotate at soak job
|
||||
counts — the 64 MB production default never rotates within any bounded
|
||||
test's job volume. Deliberately does **not** lower `--max-wal-records`
|
||||
(unlike `--force-snapshot`): this scenario's whole point is the
|
||||
**marker**-triggered checkpoint specifically, not a periodic
|
||||
WAL-threshold one, so the periodic threshold is left at its default
|
||||
(`10000`, effectively never fires at this job volume) to keep the causal
|
||||
story unambiguous.
|
||||
2. One self-contained chaotic flap of the current primary (kill → restart
|
||||
*without* waiting for convergence, mirroring `--chaotic`'s own
|
||||
overlapping-kill style) runs first, then the cluster is allowed to settle
|
||||
to a clean, well-understood 3-of-3 baseline — genuine, unplanned
|
||||
term/timing churn *before* the critical window, since a literal
|
||||
concurrent second kill during TARGET's down window is structurally
|
||||
impossible (see above).
|
||||
3. Picks a non-primary `TARGET`. Primes ≥2 WAL segments on the primary if
|
||||
fewer than 2 exist yet (so there is real material below TARGET's eventual
|
||||
floor to trim — otherwise the round would prove nothing, per its own
|
||||
"not a single trivially-active segment" warning).
|
||||
4. Captures the primary's **currently-active segment** as `TARGET`'s floor
|
||||
(`seg_at_kill`) — valid because the cluster just fully reconverged, so
|
||||
TARGET cannot need anything the primary hasn't written yet, and being
|
||||
caught up it cannot be behind an earlier segment either. This is the live,
|
||||
on-disk analog of the unit fixture's `repl_on_ack(lsn2)` call, except here
|
||||
it is TARGET's real replication state, not a simulated ack.
|
||||
5. Kills `TARGET` — mid-catch-up from this instant: it has acked everything
|
||||
through `seg_at_kill`'s coverage and nothing past it, exactly "has not yet
|
||||
acknowledged the marker's position" (Req 7.4), since the marker has not
|
||||
even been built yet.
|
||||
6. While `TARGET` stays down, forces ≥2 **more** segment rotations past
|
||||
`seg_at_kill` on the surviving 2-of-3 majority — a genuine multi-segment
|
||||
below-floor/at-floor boundary, not a trivial single-segment case.
|
||||
7. Builds an upgrade-target `cluster.bin` (same scratch-copy + one-field
|
||||
"night queue" description edit as `--config-upgrade`, for the same
|
||||
bug-1469 reason — an update to an existing queue, never an added one) and
|
||||
submits it via `upgradeconfig` against the surviving majority. Its
|
||||
committed `WAL_CONFIG_MARKER` apply on the primary is what fires
|
||||
`persist_checkpoint_marker_triggered` → `trim_wal_segments`.
|
||||
8. **Before restarting `TARGET`**, inspects the primary's real on-disk
|
||||
`data/wal/` directory directly and hard-asserts: every below-floor
|
||||
segment captured at baseline is now **gone**; the floor segment
|
||||
(`seg_at_kill`) is **preserved**; the active segment is **preserved** —
|
||||
the live analog of the unit fixture's seg-1 (gone) / seg-2 (preserved) /
|
||||
seg-3 (preserved) assertions. Fails loud with the exact segment filenames
|
||||
if the boundary isn't genuinely observed (mirrors `--force-snapshot`'s
|
||||
"prove it, don't assume it" discipline — a false PASS here would prove
|
||||
nothing).
|
||||
9. Only *then* restarts `TARGET` and asserts full reconvergence + zero job
|
||||
loss — the behavioral capstone proving the preserved segment was
|
||||
genuinely *sufficient* for real incremental catch-up, not merely
|
||||
present-but-unused.
|
||||
|
||||
A wall-clock note baked into the round's own comments: bug-1147's
|
||||
disconnected-standby guard (`repl.c`'s `last_disconnected_lsn` /
|
||||
`last_disconnect_time`) expires after `snap_interval_s` (default 60s, not
|
||||
exposed via any CLI flag). The round logs kill-to-inspection elapsed time and
|
||||
warns if it is close to that budget — a failure under a slow environment may
|
||||
reflect a genuinely-expired guard (correct behavior) rather than a
|
||||
regression, so investigate environment slowness first if this round is
|
||||
flaky.
|
||||
|
||||
Single dedicated round, not a loop — like `--config-upgrade`/`--force-snapshot`.
|
||||
|
||||
```bash
|
||||
SKILL_DIR=.claude/skills/xce-test-ha
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --config-upgrade-wal-trim ~/tmp/xce-ha-waltrim
|
||||
# tune rotation volume / batch size / attempt budget
|
||||
XCE_HA_WALTRIM_ROTATIONS=3 XCE_HA_WALTRIM_BATCH=25 \
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --config-upgrade-wal-trim ~/tmp/xce-ha-waltrim
|
||||
```
|
||||
|
||||
Verdict line:
|
||||
|
||||
```
|
||||
[xce-test-ha] PASS (config-upgrade-wal-trim): 33 jobs, below-floor WAL segment(s) trimmed, floor + active segments preserved, zero job loss after real incremental catch-up
|
||||
```
|
||||
|
||||
Like `--config-upgrade`, `result` is unconditionally `PASS` here — every hard
|
||||
invariant is asserted with an immediate `fail()` inside
|
||||
`run_config_upgrade_wal_trim_round()`, so a violation aborts the run before
|
||||
this line is ever reached.
|
||||
|
||||
### Verdict self-test (no cluster)
|
||||
|
||||
The verdict parser has a deterministic, pure-bash self-test that feeds fixture
|
||||
|
|
@ -438,7 +540,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
| Env var | Meaning | Default |
|
||||
|---|---|---|
|
||||
| `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, `array-chaos-retry`, or `config-upgrade` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, `array-chaos-retry`, `config-upgrade`, or `config-upgrade-wal-trim` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
|
||||
| `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
|
||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–50) | `6` |
|
||||
|
|
@ -460,6 +562,9 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
| `XCE_HA_ARRAYRETRY_KILL_DELAY` | `--array-chaos-retry` seconds slept after backgrounding the submit before killing the primary | `0` |
|
||||
| `XCE_HA_CONFIGUPGRADE_KILLS` | `--config-upgrade` kill/restart rounds interleaved with the upgrade submission | `3` |
|
||||
| `XCE_HA_CONFIGUPGRADE_HOST` | `--config-upgrade` name of the synthetic worker host joined and admin-closed before the upgrade | `cfgupgrade-host` |
|
||||
| `XCE_HA_WALTRIM_ROTATIONS` | `--config-upgrade-wal-trim` number of new WAL segments to force past a reference point (priming, and while TARGET is down) | `2` |
|
||||
| `XCE_HA_WALTRIM_BATCH` | `--config-upgrade-wal-trim` jobs submitted per attempt while forcing segment rotation | `15` |
|
||||
| `XCE_HA_WALTRIM_MAX_ATTEMPTS` | `--config-upgrade-wal-trim` max batches attempted before failing loud that `--wal-segment-bytes` didn't produce the expected rotation | `20` |
|
||||
|
||||
## What the driver does (maps to the requested steps)
|
||||
|
||||
|
|
|
|||
|
|
@ -51,6 +51,40 @@
|
|||
# by rotating primary through all 3 nodes and querying
|
||||
# workerlist against each. Single dedicated round, not a
|
||||
# loop.
|
||||
# --config-upgrade-wal-trim config-upgrade-snapshot task 8.9 (Req 7.4):
|
||||
# repeats the mid-catch-up WAL-trim assertion from the unit
|
||||
# fixture (test_marker_triggered_checkpoint_trim_preserves_
|
||||
# standby_mid_catchup_segments, test_upgrade_snapshot_apply.c)
|
||||
# against a REAL 3-node cluster under chaotic timing, instead
|
||||
# of hand-controlled LSNs. Deploys with a small
|
||||
# --wal-segment-bytes (like --force-snapshot) so segments
|
||||
# genuinely rotate; kills a non-primary TARGET while it is
|
||||
# still caught up (its floor = the primary's currently-active
|
||||
# segment at that instant); forces >=2 MORE segment rotations
|
||||
# on the surviving majority while TARGET stays down; submits
|
||||
# a live configuration upgrade (reusing --config-upgrade's
|
||||
# upgradeconfig primitive) so its marker-triggered checkpoint
|
||||
# fires persist_checkpoint_marker_triggered -> trim_wal_
|
||||
# segments -> repl_trim_lsn (bug-1147's pre-existing
|
||||
# retention-floor guard) while TARGET is genuinely mid-catch-
|
||||
# up; then, BEFORE restarting TARGET, inspects the primary's
|
||||
# real on-disk wal/ directory directly and hard-asserts the
|
||||
# below-floor segment(s) are gone, the floor segment is
|
||||
# preserved, and the active segment is preserved — the live
|
||||
# analog of the unit fixture's seg-1/seg-2/seg-3 assertions.
|
||||
# Only THEN restarts TARGET and asserts full reconvergence +
|
||||
# zero job loss (proving the preserved segment was genuinely
|
||||
# sufficient for real incremental catch-up, not merely
|
||||
# present-but-unused). One self-contained chaotic flap of the
|
||||
# primary runs BEFORE the critical mid-catch-up window (a
|
||||
# literal concurrent second kill during TARGET's down window
|
||||
# is impossible in a 3-node cluster — 2 simultaneous non-
|
||||
# participants leaves no majority — see the round's own
|
||||
# header comment). Every empirical assumption (segment
|
||||
# rotation volume, guard-expiry timing) is confirmed by
|
||||
# direct evidence and fails loud, never assumed — mirroring
|
||||
# --force-snapshot's "prove it, don't assume it" discipline.
|
||||
# Single dedicated round, not a loop.
|
||||
# --factstream raft-apply-decoupling task 7.2 (Req 10.2/10.4): drives a
|
||||
# continuous WORKER-FACT stream (jstart/jfinish/jfail — real
|
||||
# wire-protocol worker reports, distinct from job-submission
|
||||
|
|
@ -181,6 +215,21 @@
|
|||
# XCE_HA_CONFIGUPGRADE_HOST name of the synthetic worker host joined and
|
||||
# admin-closed before the upgrade, used to prove
|
||||
# Req 6.4 (default "cfgupgrade-host")
|
||||
# XCE_HA_WALTRIM_ROTATIONS --config-upgrade-wal-trim: number of NEW WAL
|
||||
# segments to force past a reference point,
|
||||
# both while priming (before TARGET is picked)
|
||||
# and while TARGET is down (default 2 — enough
|
||||
# for a genuine multi-segment below-floor/at-
|
||||
# floor boundary, not just one trivially-active
|
||||
# segment)
|
||||
# XCE_HA_WALTRIM_BATCH --config-upgrade-wal-trim: jobs submitted per
|
||||
# attempt while forcing segment rotation
|
||||
# (default 15)
|
||||
# XCE_HA_WALTRIM_MAX_ATTEMPTS --config-upgrade-wal-trim: max batches
|
||||
# attempted before failing loud that
|
||||
# --wal-segment-bytes did not produce the
|
||||
# expected rotation at this job volume
|
||||
# (default 20)
|
||||
set -u
|
||||
|
||||
# ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─
|
||||
|
|
@ -202,6 +251,7 @@ while [ $# -gt 0 ]; do
|
|||
--array-soak) MODE="array-soak" ;;
|
||||
--array-chaos-retry) MODE="array-chaos-retry" ;;
|
||||
--config-upgrade) MODE="config-upgrade" ;;
|
||||
--config-upgrade-wal-trim) MODE="config-upgrade-wal-trim" ;;
|
||||
--selftest) exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;;
|
||||
*[!0-9]*|'') ARGS+=("$1") ;; # non-integer ⇒ deploy dir
|
||||
*) ROUNDS_CLI="$1" ;; # bare integer ⇒ rounds
|
||||
|
|
@ -251,9 +301,21 @@ WAITMS="${XCE_HA_WAITMS:-30000}"
|
|||
# reach, so it defaults --max-wal-records to persist's enforced floor (100)
|
||||
# unless the caller already set MAX_WAL_RECORDS explicitly. Every other mode
|
||||
# leaves it unset (empty ⇒ start_node passes no override, unchanged behavior).
|
||||
#
|
||||
# config-upgrade-snapshot task 8.9 (--config-upgrade-wal-trim): needs the SAME
|
||||
# small physical segment size as force-snapshot (the 64 MB production default
|
||||
# never rotates at soak job counts — see WAL_SEGMENT_BYTES doc above) so
|
||||
# segments genuinely rotate, but deliberately does NOT lower MAX_WAL_RECORDS:
|
||||
# this scenario's whole point is to exercise the MARKER-triggered checkpoint's
|
||||
# trim specifically (Req 7.4), not a periodic WAL-threshold one — leaving the
|
||||
# threshold at its default (10000) keeps the only forced checkpoint in this
|
||||
# round the marker-triggered one, so the causal story stays unambiguous.
|
||||
if [ "$MODE" = "force-snapshot" ]; then
|
||||
MAX_WAL_RECORDS="${MAX_WAL_RECORDS:-100}"
|
||||
WAL_SEGMENT_BYTES="${WAL_SEGMENT_BYTES:-4096}"
|
||||
elif [ "$MODE" = "config-upgrade-wal-trim" ]; then
|
||||
MAX_WAL_RECORDS="${MAX_WAL_RECORDS:-}"
|
||||
WAL_SEGMENT_BYTES="${WAL_SEGMENT_BYTES:-4096}"
|
||||
else
|
||||
MAX_WAL_RECORDS="${MAX_WAL_RECORDS:-}"
|
||||
WAL_SEGMENT_BYTES="${WAL_SEGMENT_BYTES:-}"
|
||||
|
|
@ -1314,6 +1376,267 @@ run_config_upgrade_round() {
|
|||
done
|
||||
}
|
||||
|
||||
# ── Config-upgrade WAL-trim primitives (config-upgrade-snapshot task 8.9) ───
|
||||
|
||||
# Echo a member's on-disk WAL directory (persist.c's p->wal_path convention:
|
||||
# $TOP/controller/<node>/data/wal — same root wipe_node_data already uses).
|
||||
wal_dir() { echo "$TOP/controller/$1/data/wal"; }
|
||||
|
||||
# Sorted (ascending by LSN) list of a member's WAL segment file BASENAMES
|
||||
# (seg-<lsn16hex>.wal — persist.c's WAL_SEG_FMT). Fixed-width hex filenames
|
||||
# sort lexicographically == numerically, so a plain `sort` is exact. Echoes
|
||||
# nothing (not an error) if the directory is empty/missing.
|
||||
list_wal_segments() {
|
||||
local node="$1"
|
||||
( cd "$(wal_dir "$node")" 2>/dev/null && ls -1 seg-*.wal 2>/dev/null | sort )
|
||||
}
|
||||
|
||||
count_wal_segments() { list_wal_segments "$1" | wc -l | tr -d ' '; }
|
||||
|
||||
# Submit job batches against the live primary (via $MASTERS — works
|
||||
# regardless of which member is currently down) until NODE's own on-disk WAL
|
||||
# segment count has grown by at least WANT_NEW beyond its count right now.
|
||||
# Bounded by MAX_ATTEMPTS batches of BATCH jobs each. Task 8.9's own
|
||||
# instruction is to confirm segment rotation EMPIRICALLY rather than assume
|
||||
# --wal-segment-bytes produces it at this job volume — a caller MUST treat a
|
||||
# non-zero return as a real round failure (fail loud), never proceed as if
|
||||
# rotation happened anyway.
|
||||
force_n_wal_rotations() {
|
||||
local node="$1" want_new="$2" batch="$3" max_attempts="$4"
|
||||
local baseline now attempt=0
|
||||
baseline="$(count_wal_segments "$node")"
|
||||
log "force_n_wal_rotations: $node baseline segment count=$baseline, target +$want_new new (batch=$batch jobs/attempt, max_attempts=$max_attempts)"
|
||||
while [ "$attempt" -lt "$max_attempts" ]; do
|
||||
submit_jobs "$batch" || log "force_n_wal_rotations: submit_jobs shortfall this batch (continuing — a partial batch still writes WAL bytes)"
|
||||
now="$(count_wal_segments "$node")"
|
||||
attempt=$((attempt + 1))
|
||||
log "force_n_wal_rotations: $node segment count=$now after $attempt batch(es) (+$((now - baseline)) vs baseline)"
|
||||
if [ "$((now - baseline))" -ge "$want_new" ]; then
|
||||
log "force_n_wal_rotations: $node reached target (+$((now - baseline)) >= +$want_new) after $attempt batches"
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
log "force_n_wal_rotations: $node only reached +$((now - baseline)) new segments (wanted +$want_new) after $max_attempts batches of $batch — WAL_SEGMENT_BYTES=$WAL_SEGMENT_BYTES may not be small enough for this job volume"
|
||||
return 1
|
||||
}
|
||||
|
||||
# config-upgrade-snapshot task 8.9 (Req 7.4): repeat the unit-level
|
||||
# test_marker_triggered_checkpoint_trim_preserves_standby_mid_catchup_segments
|
||||
# fixture (test_upgrade_snapshot_apply.c) against a REAL 3-node cluster under
|
||||
# chaotic timing — proving the marker-triggered checkpoint's WAL-segment trim
|
||||
# (persist_checkpoint_marker_triggered -> trim_wal_segments -> repl_trim_lsn,
|
||||
# bug-1147's pre-existing retention-floor guard, unmodified by this spec)
|
||||
# never strands a genuinely mid-catch-up standby's still-needed WAL
|
||||
# segment(s), under real kill/restart timing and real segment-rotation
|
||||
# volume instead of hand-controlled LSNs.
|
||||
#
|
||||
# Mirrors run_force_snapshot_round's "prove it, don't assume it" discipline:
|
||||
# every step that depends on empirical timing (segment rotation, guard
|
||||
# expiry) is confirmed by direct on-disk evidence and fails loud on a
|
||||
# violated precondition, rather than silently reporting a false PASS that
|
||||
# would prove nothing (task 8.9's own explicit instruction).
|
||||
#
|
||||
# bug-1147's guard has a WALL-CLOCK component the deterministic unit fixture
|
||||
# never has to reason about: a disconnected standby's floor guard
|
||||
# (repl.c's last_disconnected_lsn / last_disconnect_time) expires after
|
||||
# p->config.snap_interval_s (DEFAULT_SNAP_INTERVAL_S = 60s, persist.h/
|
||||
# config.h — not exposed by any CLI flag). If this round's kill-to-inspection
|
||||
# window ever exceeded that budget, the guard would correctly (not a bug)
|
||||
# stop protecting TARGET's floor segment, and the final assertion below would
|
||||
# then fail loud — which is CORRECT behavior for a round that ran that slow,
|
||||
# not a false positive, but it would mean the round proved nothing about the
|
||||
# guard's mid-catch-up behavior specifically. Elapsed kill-to-inspection time
|
||||
# is logged so a slow environment is diagnosable rather than silently masked.
|
||||
#
|
||||
# Chaos placement: Requirement 7.4 wants this exercised "under the chaotic
|
||||
# multi-kill conditions Requirement 6 exercises," but a literal CONCURRENT
|
||||
# second kill during TARGET's down window is structurally impossible here —
|
||||
# a 3-node cluster has no majority left if 2 of 3 are ever simultaneously
|
||||
# non-participants, and TARGET must stay down for the WHOLE critical window
|
||||
# to be genuinely "mid-catch-up" (mirroring the unit fixture's standby that
|
||||
# never advances past lsn2). Instead, one self-contained chaotic flap of the
|
||||
# CURRENT primary (kill -> restart WITHOUT waiting for convergence, mirroring
|
||||
# run_chaotic_rounds' own overlapping-kill style) runs BEFORE the critical
|
||||
# window begins, giving the round genuine, unplanned term/timing churn up
|
||||
# front; the cluster is then allowed to settle to a clean, well-understood
|
||||
# 3-of-3 baseline before TARGET is identified and killed. This is "the
|
||||
# sequence" Req 7.4 asks for, sequenced rather than concurrent, for a hard
|
||||
# quorum-math reason, not convenience.
|
||||
run_config_upgrade_wal_trim_round() {
|
||||
local rotations batch max_attempts xceadmin scratch_dir scratch_bin
|
||||
rotations="${XCE_HA_WALTRIM_ROTATIONS:-2}"
|
||||
batch="${XCE_HA_WALTRIM_BATCH:-15}"
|
||||
max_attempts="${XCE_HA_WALTRIM_MAX_ATTEMPTS:-20}"
|
||||
|
||||
xceadmin="$(find_xceadmin)" \
|
||||
|| fail "config-upgrade-wal-trim: xceadmin binary not found (run /xce-build all or /xce-build xceadmin)"
|
||||
|
||||
# 1. One early, self-contained chaotic flap of the CURRENT primary —
|
||||
# genuine unplanned term/timing churn, fully resolved BEFORE the
|
||||
# critical mid-catch-up window starts (see header comment: quorum
|
||||
# math rules out a literal concurrent second kill while TARGET is
|
||||
# down for the whole window).
|
||||
local flap_primary flap_prev_term flap_pid flap_new
|
||||
flap_primary="$(primary_from_log)"
|
||||
if [ -z "$flap_primary" ]; then
|
||||
wait_for_primary_log || log "config-upgrade-wal-trim: no primary elected yet before the early chaos flap"
|
||||
flap_primary="$(primary_from_log)"
|
||||
fi
|
||||
if [ -n "$flap_primary" ]; then
|
||||
flap_prev_term="$(latest_primary_term)"; flap_prev_term="${flap_prev_term:-0}"
|
||||
flap_pid="$(pgrep -f -- "$(node_pattern "$flap_primary")")"
|
||||
if [ -n "$flap_pid" ]; then
|
||||
log "config-upgrade-wal-trim: early chaos flap — killing primary $flap_primary (term=$flap_prev_term)"
|
||||
kill -9 $flap_pid 2>/dev/null
|
||||
sleep 1
|
||||
start_node "$flap_primary"
|
||||
flap_new="$(wait_for_new_primary "$flap_prev_term")"
|
||||
[ -n "$flap_new" ] && record_primary "$flap_new"
|
||||
log "config-upgrade-wal-trim: early chaos flap settled (new primary probe: ${flap_new:-none yet})"
|
||||
fi
|
||||
else
|
||||
log "config-upgrade-wal-trim: no primary found for the early chaos flap — skipping it (will still proceed once the cluster reconverges)"
|
||||
fi
|
||||
wait_for_full_convergence \
|
||||
|| fail "config-upgrade-wal-trim: cluster did not reconverge to 3-of-3 after the early chaos flap"
|
||||
log "config-upgrade-wal-trim: 3-of-3 converged after the early chaos flap — starting the critical mid-catch-up window"
|
||||
|
||||
# 2. Identify the primary and a non-primary TARGET to keep down through
|
||||
# the whole critical window (mirrors run_force_snapshot_round's
|
||||
# non-primary target-picking convention).
|
||||
local primary target n
|
||||
primary="$(discover_primary)" || fail "config-upgrade-wal-trim: no serving primary before picking TARGET"
|
||||
target=""
|
||||
for n in "${NODES[@]}"; do
|
||||
[ "$n" = "$primary" ] || { target="$n"; break; }
|
||||
done
|
||||
[ -n "$target" ] || fail "config-upgrade-wal-trim: could not pick a non-primary TARGET"
|
||||
log "config-upgrade-wal-trim: primary=$primary target=$target (target will be killed and kept down through the marker-triggered trim)"
|
||||
PRIMARY="$primary"
|
||||
|
||||
# 3. Prime enough segment volume that TARGET's eventual floor lands
|
||||
# strictly above segment 1 — otherwise there is nothing below the
|
||||
# floor to trim and the round would prove nothing (task 8.9's own
|
||||
# warning against a trivially-always-active single segment).
|
||||
# Confirmed empirically, not assumed.
|
||||
if [ "$(count_wal_segments "$primary")" -lt 2 ]; then
|
||||
force_n_wal_rotations "$primary" "$rotations" "$batch" "$max_attempts" \
|
||||
|| fail "config-upgrade-wal-trim: could not prime >=$rotations WAL segments on $primary before picking TARGET's floor"
|
||||
fi
|
||||
verify_no_loss || fail "config-upgrade-wal-trim: job loss during priming, before the critical window even started"
|
||||
|
||||
# 4. Baseline capture: TARGET is still up and the cluster just fully
|
||||
# reconverged, so the primary's CURRENTLY-ACTIVE segment at this exact
|
||||
# instant is where TARGET's own floor necessarily sits (it cannot need
|
||||
# anything the primary hasn't written yet, and being caught up it
|
||||
# cannot be behind an earlier segment either). This is the direct,
|
||||
# on-disk analog of the unit fixture's repl_on_ack(lsn2) call — except
|
||||
# here it is TARGET's real replication state, not a simulated ack.
|
||||
local baseline_segs seg_at_kill below_segs
|
||||
baseline_segs="$(list_wal_segments "$primary")"
|
||||
seg_at_kill="$(printf '%s\n' "$baseline_segs" | tail -1)"
|
||||
below_segs="$(printf '%s\n' "$baseline_segs" | sed '$d')" # every line except the last
|
||||
[ -n "$seg_at_kill" ] || fail "config-upgrade-wal-trim: $primary has no WAL segment at baseline capture — cannot identify TARGET's floor"
|
||||
[ -n "$below_segs" ] || fail "config-upgrade-wal-trim: no segment exists below the baseline active segment on $primary — priming did not produce trim material (should be unreachable after step 3)"
|
||||
log "config-upgrade-wal-trim: baseline on $primary — floor segment (must survive)=$seg_at_kill; below-floor segment(s) (must be trimmed)=$(printf '%s' "$below_segs" | tr '\n' ' ')"
|
||||
|
||||
# 5. Kill TARGET now, mid-catch-up: it has acked everything up through
|
||||
# seg_at_kill's coverage and NOTHING past it — exactly "has not yet
|
||||
# acknowledged the marker's position" (Req 7.4), since the marker has
|
||||
# not even been built yet at this point.
|
||||
kill_node "$target" || fail "config-upgrade-wal-trim: could not kill target $target"
|
||||
local kill_epoch; kill_epoch="$(date +%s)"
|
||||
log "config-upgrade-wal-trim: $target killed at epoch=$kill_epoch — kept down through the critical window (bug-1147 disconnect-guard budget: 60s default, DEFAULT_SNAP_INTERVAL_S)"
|
||||
|
||||
# 6. While TARGET is down, force >=$rotations MORE segment rotations past
|
||||
# seg_at_kill on the surviving 2-of-3 majority — a genuine boundary
|
||||
# case (multiple segments to trim, multiple to preserve), not a
|
||||
# single trivially-active segment.
|
||||
force_n_wal_rotations "$primary" "$rotations" "$batch" "$max_attempts" \
|
||||
|| fail "config-upgrade-wal-trim: could not force >=$rotations new WAL segment rotations past $target's floor while it was down"
|
||||
verify_no_loss || fail "config-upgrade-wal-trim: job loss while $target was down, before the upgrade was even submitted"
|
||||
|
||||
# 7. Build the upgrade-target cluster.bin (same scratch-copy + one-field
|
||||
# edit convention as run_config_upgrade_round — an UPDATE to an
|
||||
# existing queue's description, never an added queue; see that
|
||||
# function's own comment / bug-1469 for why an added queue would not
|
||||
# exercise a working live-upgrade path at all).
|
||||
scratch_dir="$TOP/.cfgupgrade-waltrim-src"
|
||||
scratch_bin="$TOP/.cfgupgrade-waltrim-src.bin"
|
||||
rm -rf "$scratch_dir" "$scratch_bin"
|
||||
cp -r "$REPO/src/configs" "$scratch_dir"
|
||||
sed -i 's/description: "night queue"/description: "night queue (config-upgrade-snapshot task 8.9 wal-trim soak marker)"/' \
|
||||
"$scratch_dir/queues.yaml"
|
||||
grep -q "task 8.9 wal-trim soak marker" "$scratch_dir/queues.yaml" \
|
||||
|| fail "config-upgrade-wal-trim: sed did not find the expected 'night queue' description line — src/configs/queues.yaml may have changed shape"
|
||||
"$xceadmin" config build "$scratch_dir" -o "$scratch_bin" \
|
||||
|| fail "config-upgrade-wal-trim: xceadmin config build failed for the upgrade-target config"
|
||||
log "config-upgrade-wal-trim: built upgrade-target cluster.bin: $scratch_bin"
|
||||
|
||||
# 8. Submit the upgrade against the surviving 2-of-3 majority (TARGET is
|
||||
# still down) — its committed WAL_CONFIG_MARKER apply on $primary is
|
||||
# what fires persist_checkpoint_marker_triggered -> trim_wal_segments.
|
||||
local upgrade_out
|
||||
upgrade_out="$(printf 'set masters %s\nset myname cfgupgrade-waltrim\nclient\nwait-primary %s\nupgradeconfig %s\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" "$scratch_bin" | xtest)"
|
||||
echo "$upgrade_out" | grep -q "Config upgrade submitted" \
|
||||
|| fail "config-upgrade-wal-trim: upgradeconfig did not report success: $upgrade_out"
|
||||
log "config-upgrade-wal-trim: upgrade submitted and committed on the surviving majority"
|
||||
|
||||
# 9. Give the primary's own apply pump a moment to apply the marker and
|
||||
# materialize+trim (Req 1.1/1.5 — synchronous from the applying
|
||||
# member's own perspective, but still async w.r.t. this shell).
|
||||
sleep 5
|
||||
|
||||
local kill_elapsed; kill_elapsed=$(( $(date +%s) - kill_epoch ))
|
||||
log "config-upgrade-wal-trim: kill-to-inspection elapsed=${kill_elapsed}s (bug-1147 disconnect-guard budget ~60s default)"
|
||||
if [ "$kill_elapsed" -ge 50 ]; then
|
||||
log "config-upgrade-wal-trim: WARNING — elapsed time is close to/over the disconnect-guard's default 60s budget; a failure below may reflect a genuinely-expired guard (correct behavior) rather than a real regression. If this round is flaky, investigate environment slowness before suspecting the guard."
|
||||
fi
|
||||
|
||||
# 10. THE assertion (Req 7.4's actual proof), BEFORE restarting TARGET:
|
||||
# inspect $primary's real on-disk WAL directory directly.
|
||||
local post_segs seg
|
||||
post_segs="$(list_wal_segments "$primary")"
|
||||
log "config-upgrade-wal-trim: post-trim segments on $primary: $(printf '%s' "$post_segs" | tr '\n' ' ')"
|
||||
|
||||
printf '%s\n' "$post_segs" | grep -qxF "$seg_at_kill" \
|
||||
|| fail "config-upgrade-wal-trim: TARGET's floor segment $seg_at_kill was TRIMMED — a mid-catch-up standby's still-needed WAL segment was stranded (bug-1147 guard violated under the marker-triggered path). post-trim segments: $post_segs"
|
||||
log "config-upgrade-wal-trim: floor segment $seg_at_kill correctly PRESERVED (still needed for $target's eventual catch-up)"
|
||||
|
||||
local stranded_ok=1
|
||||
while IFS= read -r seg; do
|
||||
[ -n "$seg" ] || continue
|
||||
if printf '%s\n' "$post_segs" | grep -qxF "$seg"; then
|
||||
log "config-upgrade-wal-trim: FAIL evidence — below-floor segment $seg is STILL PRESENT (should have been trimmed)"
|
||||
stranded_ok=0
|
||||
fi
|
||||
done <<<"$below_segs"
|
||||
[ "$stranded_ok" -eq 1 ] \
|
||||
|| fail "config-upgrade-wal-trim: at least one below-floor segment was NOT trimmed by the marker-triggered checkpoint — see FAIL evidence lines above. below-floor set: $below_segs"
|
||||
log "config-upgrade-wal-trim: all below-floor segment(s) correctly TRIMMED: $(printf '%s' "$below_segs" | tr '\n' ' ')"
|
||||
|
||||
local active_seg
|
||||
active_seg="$(printf '%s\n' "$post_segs" | tail -1)"
|
||||
[ -n "$active_seg" ] \
|
||||
|| fail "config-upgrade-wal-trim: $primary has no active WAL segment after the marker-triggered checkpoint"
|
||||
[ -f "$(wal_dir "$primary")/$active_seg" ] \
|
||||
|| fail "config-upgrade-wal-trim: active segment $active_seg reported by listing but missing on disk"
|
||||
log "config-upgrade-wal-trim: active segment $active_seg correctly PRESENT"
|
||||
|
||||
# 11. ONLY NOW restart TARGET and prove the preserved segment was
|
||||
# genuinely sufficient for real incremental catch-up, not merely
|
||||
# present-but-unused — the behavioral capstone.
|
||||
start_node "$target"
|
||||
wait_for_full_convergence \
|
||||
|| fail "config-upgrade-wal-trim: cluster did not reconverge to 3-of-3 after restarting $target — the preserved floor segment did not actually suffice for real incremental catch-up"
|
||||
log "config-upgrade-wal-trim: $target reconverged (3-of-3) using the preserved floor segment for incremental catch-up"
|
||||
|
||||
verify_no_loss \
|
||||
|| fail "config-upgrade-wal-trim: a committed job was LOST across the wal-trim-under-marker-trigger scenario"
|
||||
log "config-upgrade-wal-trim: zero job loss confirmed"
|
||||
}
|
||||
|
||||
# ── Fact-stream primitives (raft-apply-decoupling task 7.2, Req 10.2/10.4) ───
|
||||
#
|
||||
# Drives a continuous stream of WORKER-FACT messages (jstart/jfinish/jfail —
|
||||
|
|
@ -2245,6 +2568,26 @@ if [ "$MODE" = "config-upgrade" ]; then
|
|||
exit 0
|
||||
fi
|
||||
|
||||
# ── Config-upgrade WAL-trim mode: config-upgrade-snapshot task 8.9 ──────────
|
||||
# Opt-in (--config-upgrade-wal-trim / XCE_HA_MODE=config-upgrade-wal-trim).
|
||||
# See run_config_upgrade_wal_trim_round's own header comment for the full
|
||||
# mechanism trace and Req 7.4's mid-catch-up WAL-trim proof. Single dedicated
|
||||
# round, not a loop — like --config-upgrade/--force-snapshot; every hard
|
||||
# invariant (segment-rotation volume, floor-segment preservation, below-floor
|
||||
# trim, active-segment preservation, post-restart reconvergence, zero job
|
||||
# loss) is asserted with an immediate fail() inside
|
||||
# run_config_upgrade_wal_trim_round() itself, so reaching the PASS line below
|
||||
# already means every check passed. Does NOT run the sequential loop or
|
||||
# leadership-coverage maneuver.
|
||||
if [ "$MODE" = "config-upgrade-wal-trim" ]; then
|
||||
log "MODE=config-upgrade-wal-trim — exercising the marker-triggered checkpoint's WAL-segment trim against a genuinely mid-catch-up standby under chaotic timing"
|
||||
run_config_upgrade_wal_trim_round
|
||||
log "PASS (config-upgrade-wal-trim): ${#ALL_JOBIDS[@]} jobs, below-floor WAL segment(s) trimmed, floor + active segments preserved, zero job loss after real incremental catch-up"
|
||||
cleanup
|
||||
trap - EXIT INT TERM
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Chaotic mode: overlapping-kill rounds → isolated verdict ──────────────────
|
||||
# Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up
|
||||
# windows; outcomes are COUNTED (committed divergence + committed job loss) with
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue