feat(config-upgrade-snapshot): task 8.9 — live WAL-trim-under-chaos soak infra, blocked on bug-1474
Add run_config_upgrade_wal_trim_round() / --config-upgrade-wal-trim to run-ha-soak.sh: forces real WAL segment rotation via small --wal-segment-bytes, kills a non-primary node, drives a live config upgrade to fire the marker-triggered checkpoint's forced trim, then inspects survivors' on-disk WAL directory before the killed node rejoins to directly verify the mid-catch-up retention guard (repl_trim_lsn/trim_wal_segments, bug-1147) preserves what it still needs. Two independent live 3-node runs both reproduce a real, pre-existing, out-of-spec-boundary defect (bug-1474): a heartbeat reconnect-sweep race reuses a just-killed standby's freed replicator slot and clobbers its retention guard before trim ever runs. Root-caused via repl.c/ha.c and confirmed via git blame to predate this spec and be reachable under the old periodic-checkpoint trigger identically -- not caused by anything config-upgrade-snapshot changed. Task 8.9 stays blocked per tasks.md until bug-1474 is fixed upstream; infra is done and correctly proves the violation. Independently reviewed: round design verified faithful to the unit-level fixture, bug-1474 diagnosis independently re-derived from source and raw logs (not trusted from the implementer's report).
This commit is contained in:
parent
b78efd03b4
commit
8b3e2ab03c
3 changed files with 452 additions and 1 deletions
|
|
@ -343,6 +343,108 @@ inside `run_config_upgrade_round()`, so a violation aborts the run before
|
|||
this line is ever reached; there is no `verdict_emit`/bug-924 exclusion
|
||||
tally for this mode.
|
||||
|
||||
### `--config-upgrade-wal-trim` mode: marker-triggered WAL-segment trim under a genuinely mid-catch-up standby (config-upgrade-snapshot task 8.9)
|
||||
|
||||
Repeats the mid-catch-up WAL-trim assertion from the unit-level fixture
|
||||
(`test_marker_triggered_checkpoint_trim_preserves_standby_mid_catchup_segments`,
|
||||
`test_upgrade_snapshot_apply.c`) against a **real 3-node cluster** under
|
||||
chaotic timing, instead of hand-controlled LSNs — proving Requirement 7.4:
|
||||
the marker-triggered checkpoint's forced WAL trim
|
||||
(`persist_checkpoint_marker_triggered` → `trim_wal_segments` →
|
||||
`repl_trim_lsn`, bug-1147's pre-existing retention-floor guard, unmodified
|
||||
by this spec) never strands a genuinely mid-catch-up standby's still-needed
|
||||
WAL segment. See `run_config_upgrade_wal_trim_round()`'s own header comment
|
||||
(`run-ha-soak.sh`) for the full mechanism trace, including why the "chaotic"
|
||||
element is sequenced *before* the critical window rather than concurrent
|
||||
with it (a 3-node cluster has no majority left if 2 of 3 are ever
|
||||
simultaneously non-participants, and TARGET must stay down for the *whole*
|
||||
critical window to be genuinely mid-catch-up).
|
||||
|
||||
Sequence:
|
||||
|
||||
1. Deploys with a small `--wal-segment-bytes` (`4096`, same proven value
|
||||
`--force-snapshot` already uses) so segments genuinely rotate at soak job
|
||||
counts — the 64 MB production default never rotates within any bounded
|
||||
test's job volume. Deliberately does **not** lower `--max-wal-records`
|
||||
(unlike `--force-snapshot`): this scenario's whole point is the
|
||||
**marker**-triggered checkpoint specifically, not a periodic
|
||||
WAL-threshold one, so the periodic threshold is left at its default
|
||||
(`10000`, effectively never fires at this job volume) to keep the causal
|
||||
story unambiguous.
|
||||
2. One self-contained chaotic flap of the current primary (kill → restart
|
||||
*without* waiting for convergence, mirroring `--chaotic`'s own
|
||||
overlapping-kill style) runs first, then the cluster is allowed to settle
|
||||
to a clean, well-understood 3-of-3 baseline — genuine, unplanned
|
||||
term/timing churn *before* the critical window, since a literal
|
||||
concurrent second kill during TARGET's down window is structurally
|
||||
impossible (see above).
|
||||
3. Picks a non-primary `TARGET`. Primes ≥2 WAL segments on the primary if
|
||||
fewer than 2 exist yet (so there is real material below TARGET's eventual
|
||||
floor to trim — otherwise the round would prove nothing, per its own
|
||||
"not a single trivially-active segment" warning).
|
||||
4. Captures the primary's **currently-active segment** as `TARGET`'s floor
|
||||
(`seg_at_kill`) — valid because the cluster just fully reconverged, so
|
||||
TARGET cannot need anything the primary hasn't written yet, and being
|
||||
caught up it cannot be behind an earlier segment either. This is the live,
|
||||
on-disk analog of the unit fixture's `repl_on_ack(lsn2)` call, except here
|
||||
it is TARGET's real replication state, not a simulated ack.
|
||||
5. Kills `TARGET` — mid-catch-up from this instant: it has acked everything
|
||||
through `seg_at_kill`'s coverage and nothing past it, exactly "has not yet
|
||||
acknowledged the marker's position" (Req 7.4), since the marker has not
|
||||
even been built yet.
|
||||
6. While `TARGET` stays down, forces ≥2 **more** segment rotations past
|
||||
`seg_at_kill` on the surviving 2-of-3 majority — a genuine multi-segment
|
||||
below-floor/at-floor boundary, not a trivial single-segment case.
|
||||
7. Builds an upgrade-target `cluster.bin` (same scratch-copy + one-field
|
||||
"night queue" description edit as `--config-upgrade`, for the same
|
||||
bug-1469 reason — an update to an existing queue, never an added one) and
|
||||
submits it via `upgradeconfig` against the surviving majority. Its
|
||||
committed `WAL_CONFIG_MARKER` apply on the primary is what fires
|
||||
`persist_checkpoint_marker_triggered` → `trim_wal_segments`.
|
||||
8. **Before restarting `TARGET`**, inspects the primary's real on-disk
|
||||
`data/wal/` directory directly and hard-asserts: every below-floor
|
||||
segment captured at baseline is now **gone**; the floor segment
|
||||
(`seg_at_kill`) is **preserved**; the active segment is **preserved** —
|
||||
the live analog of the unit fixture's seg-1 (gone) / seg-2 (preserved) /
|
||||
seg-3 (preserved) assertions. Fails loud with the exact segment filenames
|
||||
if the boundary isn't genuinely observed (mirrors `--force-snapshot`'s
|
||||
"prove it, don't assume it" discipline — a false PASS here would prove
|
||||
nothing).
|
||||
9. Only *then* restarts `TARGET` and asserts full reconvergence + zero job
|
||||
loss — the behavioral capstone proving the preserved segment was
|
||||
genuinely *sufficient* for real incremental catch-up, not merely
|
||||
present-but-unused.
|
||||
|
||||
A wall-clock note baked into the round's own comments: bug-1147's
|
||||
disconnected-standby guard (`repl.c`'s `last_disconnected_lsn` /
|
||||
`last_disconnect_time`) expires after `snap_interval_s` (default 60s, not
|
||||
exposed via any CLI flag). The round logs kill-to-inspection elapsed time and
|
||||
warns if it is close to that budget — a failure under a slow environment may
|
||||
reflect a genuinely-expired guard (correct behavior) rather than a
|
||||
regression, so investigate environment slowness first if this round is
|
||||
flaky.
|
||||
|
||||
Single dedicated round, not a loop — like `--config-upgrade`/`--force-snapshot`.
|
||||
|
||||
```bash
|
||||
SKILL_DIR=.claude/skills/xce-test-ha
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --config-upgrade-wal-trim ~/tmp/xce-ha-waltrim
|
||||
# tune rotation volume / batch size / attempt budget
|
||||
XCE_HA_WALTRIM_ROTATIONS=3 XCE_HA_WALTRIM_BATCH=25 \
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --config-upgrade-wal-trim ~/tmp/xce-ha-waltrim
|
||||
```
|
||||
|
||||
Verdict line:
|
||||
|
||||
```
|
||||
[xce-test-ha] PASS (config-upgrade-wal-trim): 33 jobs, below-floor WAL segment(s) trimmed, floor + active segments preserved, zero job loss after real incremental catch-up
|
||||
```
|
||||
|
||||
Like `--config-upgrade`, `result` is unconditionally `PASS` here — every hard
|
||||
invariant is asserted with an immediate `fail()` inside
|
||||
`run_config_upgrade_wal_trim_round()`, so a violation aborts the run before
|
||||
this line is ever reached.
|
||||
|
||||
### Verdict self-test (no cluster)
|
||||
|
||||
The verdict parser has a deterministic, pure-bash self-test that feeds fixture
|
||||
|
|
@ -438,7 +540,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
| Env var | Meaning | Default |
|
||||
|---|---|---|
|
||||
| `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, `array-chaos-retry`, or `config-upgrade` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, `array-chaos-retry`, `config-upgrade`, or `config-upgrade-wal-trim` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
|
||||
| `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
|
||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–50) | `6` |
|
||||
|
|
@ -460,6 +562,9 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
| `XCE_HA_ARRAYRETRY_KILL_DELAY` | `--array-chaos-retry` seconds slept after backgrounding the submit before killing the primary | `0` |
|
||||
| `XCE_HA_CONFIGUPGRADE_KILLS` | `--config-upgrade` kill/restart rounds interleaved with the upgrade submission | `3` |
|
||||
| `XCE_HA_CONFIGUPGRADE_HOST` | `--config-upgrade` name of the synthetic worker host joined and admin-closed before the upgrade | `cfgupgrade-host` |
|
||||
| `XCE_HA_WALTRIM_ROTATIONS` | `--config-upgrade-wal-trim` number of new WAL segments to force past a reference point (priming, and while TARGET is down) | `2` |
|
||||
| `XCE_HA_WALTRIM_BATCH` | `--config-upgrade-wal-trim` jobs submitted per attempt while forcing segment rotation | `15` |
|
||||
| `XCE_HA_WALTRIM_MAX_ATTEMPTS` | `--config-upgrade-wal-trim` max batches attempted before failing loud that `--wal-segment-bytes` didn't produce the expected rotation | `20` |
|
||||
|
||||
## What the driver does (maps to the requested steps)
|
||||
|
||||
|
|
|
|||
|
|
@ -51,6 +51,40 @@
|
|||
# by rotating primary through all 3 nodes and querying
|
||||
# workerlist against each. Single dedicated round, not a
|
||||
# loop.
|
||||
# --config-upgrade-wal-trim config-upgrade-snapshot task 8.9 (Req 7.4):
|
||||
# repeats the mid-catch-up WAL-trim assertion from the unit
|
||||
# fixture (test_marker_triggered_checkpoint_trim_preserves_
|
||||
# standby_mid_catchup_segments, test_upgrade_snapshot_apply.c)
|
||||
# against a REAL 3-node cluster under chaotic timing, instead
|
||||
# of hand-controlled LSNs. Deploys with a small
|
||||
# --wal-segment-bytes (like --force-snapshot) so segments
|
||||
# genuinely rotate; kills a non-primary TARGET while it is
|
||||
# still caught up (its floor = the primary's currently-active
|
||||
# segment at that instant); forces >=2 MORE segment rotations
|
||||
# on the surviving majority while TARGET stays down; submits
|
||||
# a live configuration upgrade (reusing --config-upgrade's
|
||||
# upgradeconfig primitive) so its marker-triggered checkpoint
|
||||
# fires persist_checkpoint_marker_triggered -> trim_wal_
|
||||
# segments -> repl_trim_lsn (bug-1147's pre-existing
|
||||
# retention-floor guard) while TARGET is genuinely mid-catch-
|
||||
# up; then, BEFORE restarting TARGET, inspects the primary's
|
||||
# real on-disk wal/ directory directly and hard-asserts the
|
||||
# below-floor segment(s) are gone, the floor segment is
|
||||
# preserved, and the active segment is preserved — the live
|
||||
# analog of the unit fixture's seg-1/seg-2/seg-3 assertions.
|
||||
# Only THEN restarts TARGET and asserts full reconvergence +
|
||||
# zero job loss (proving the preserved segment was genuinely
|
||||
# sufficient for real incremental catch-up, not merely
|
||||
# present-but-unused). One self-contained chaotic flap of the
|
||||
# primary runs BEFORE the critical mid-catch-up window (a
|
||||
# literal concurrent second kill during TARGET's down window
|
||||
# is impossible in a 3-node cluster — 2 simultaneous non-
|
||||
# participants leaves no majority — see the round's own
|
||||
# header comment). Every empirical assumption (segment
|
||||
# rotation volume, guard-expiry timing) is confirmed by
|
||||
# direct evidence and fails loud, never assumed — mirroring
|
||||
# --force-snapshot's "prove it, don't assume it" discipline.
|
||||
# Single dedicated round, not a loop.
|
||||
# --factstream raft-apply-decoupling task 7.2 (Req 10.2/10.4): drives a
|
||||
# continuous WORKER-FACT stream (jstart/jfinish/jfail — real
|
||||
# wire-protocol worker reports, distinct from job-submission
|
||||
|
|
@ -181,6 +215,21 @@
|
|||
# XCE_HA_CONFIGUPGRADE_HOST name of the synthetic worker host joined and
|
||||
# admin-closed before the upgrade, used to prove
|
||||
# Req 6.4 (default "cfgupgrade-host")
|
||||
# XCE_HA_WALTRIM_ROTATIONS --config-upgrade-wal-trim: number of NEW WAL
|
||||
# segments to force past a reference point,
|
||||
# both while priming (before TARGET is picked)
|
||||
# and while TARGET is down (default 2 — enough
|
||||
# for a genuine multi-segment below-floor/at-
|
||||
# floor boundary, not just one trivially-active
|
||||
# segment)
|
||||
# XCE_HA_WALTRIM_BATCH --config-upgrade-wal-trim: jobs submitted per
|
||||
# attempt while forcing segment rotation
|
||||
# (default 15)
|
||||
# XCE_HA_WALTRIM_MAX_ATTEMPTS --config-upgrade-wal-trim: max batches
|
||||
# attempted before failing loud that
|
||||
# --wal-segment-bytes did not produce the
|
||||
# expected rotation at this job volume
|
||||
# (default 20)
|
||||
set -u
|
||||
|
||||
# ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─
|
||||
|
|
@ -202,6 +251,7 @@ while [ $# -gt 0 ]; do
|
|||
--array-soak) MODE="array-soak" ;;
|
||||
--array-chaos-retry) MODE="array-chaos-retry" ;;
|
||||
--config-upgrade) MODE="config-upgrade" ;;
|
||||
--config-upgrade-wal-trim) MODE="config-upgrade-wal-trim" ;;
|
||||
--selftest) exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;;
|
||||
*[!0-9]*|'') ARGS+=("$1") ;; # non-integer ⇒ deploy dir
|
||||
*) ROUNDS_CLI="$1" ;; # bare integer ⇒ rounds
|
||||
|
|
@ -251,9 +301,21 @@ WAITMS="${XCE_HA_WAITMS:-30000}"
|
|||
# reach, so it defaults --max-wal-records to persist's enforced floor (100)
|
||||
# unless the caller already set MAX_WAL_RECORDS explicitly. Every other mode
|
||||
# leaves it unset (empty ⇒ start_node passes no override, unchanged behavior).
|
||||
#
|
||||
# config-upgrade-snapshot task 8.9 (--config-upgrade-wal-trim): needs the SAME
|
||||
# small physical segment size as force-snapshot (the 64 MB production default
|
||||
# never rotates at soak job counts — see WAL_SEGMENT_BYTES doc above) so
|
||||
# segments genuinely rotate, but deliberately does NOT lower MAX_WAL_RECORDS:
|
||||
# this scenario's whole point is to exercise the MARKER-triggered checkpoint's
|
||||
# trim specifically (Req 7.4), not a periodic WAL-threshold one — leaving the
|
||||
# threshold at its default (10000) keeps the only forced checkpoint in this
|
||||
# round the marker-triggered one, so the causal story stays unambiguous.
|
||||
if [ "$MODE" = "force-snapshot" ]; then
|
||||
MAX_WAL_RECORDS="${MAX_WAL_RECORDS:-100}"
|
||||
WAL_SEGMENT_BYTES="${WAL_SEGMENT_BYTES:-4096}"
|
||||
elif [ "$MODE" = "config-upgrade-wal-trim" ]; then
|
||||
MAX_WAL_RECORDS="${MAX_WAL_RECORDS:-}"
|
||||
WAL_SEGMENT_BYTES="${WAL_SEGMENT_BYTES:-4096}"
|
||||
else
|
||||
MAX_WAL_RECORDS="${MAX_WAL_RECORDS:-}"
|
||||
WAL_SEGMENT_BYTES="${WAL_SEGMENT_BYTES:-}"
|
||||
|
|
@ -1314,6 +1376,267 @@ run_config_upgrade_round() {
|
|||
done
|
||||
}
|
||||
|
||||
# ── Config-upgrade WAL-trim primitives (config-upgrade-snapshot task 8.9) ───
|
||||
|
||||
# Echo a member's on-disk WAL directory (persist.c's p->wal_path convention:
|
||||
# $TOP/controller/<node>/data/wal — same root wipe_node_data already uses).
|
||||
wal_dir() { echo "$TOP/controller/$1/data/wal"; }
|
||||
|
||||
# Sorted (ascending by LSN) list of a member's WAL segment file BASENAMES
|
||||
# (seg-<lsn16hex>.wal — persist.c's WAL_SEG_FMT). Fixed-width hex filenames
|
||||
# sort lexicographically == numerically, so a plain `sort` is exact. Echoes
|
||||
# nothing (not an error) if the directory is empty/missing.
|
||||
list_wal_segments() {
|
||||
local node="$1"
|
||||
( cd "$(wal_dir "$node")" 2>/dev/null && ls -1 seg-*.wal 2>/dev/null | sort )
|
||||
}
|
||||
|
||||
count_wal_segments() { list_wal_segments "$1" | wc -l | tr -d ' '; }
|
||||
|
||||
# Submit job batches against the live primary (via $MASTERS — works
|
||||
# regardless of which member is currently down) until NODE's own on-disk WAL
|
||||
# segment count has grown by at least WANT_NEW beyond its count right now.
|
||||
# Bounded by MAX_ATTEMPTS batches of BATCH jobs each. Task 8.9's own
|
||||
# instruction is to confirm segment rotation EMPIRICALLY rather than assume
|
||||
# --wal-segment-bytes produces it at this job volume — a caller MUST treat a
|
||||
# non-zero return as a real round failure (fail loud), never proceed as if
|
||||
# rotation happened anyway.
|
||||
force_n_wal_rotations() {
|
||||
local node="$1" want_new="$2" batch="$3" max_attempts="$4"
|
||||
local baseline now attempt=0
|
||||
baseline="$(count_wal_segments "$node")"
|
||||
log "force_n_wal_rotations: $node baseline segment count=$baseline, target +$want_new new (batch=$batch jobs/attempt, max_attempts=$max_attempts)"
|
||||
while [ "$attempt" -lt "$max_attempts" ]; do
|
||||
submit_jobs "$batch" || log "force_n_wal_rotations: submit_jobs shortfall this batch (continuing — a partial batch still writes WAL bytes)"
|
||||
now="$(count_wal_segments "$node")"
|
||||
attempt=$((attempt + 1))
|
||||
log "force_n_wal_rotations: $node segment count=$now after $attempt batch(es) (+$((now - baseline)) vs baseline)"
|
||||
if [ "$((now - baseline))" -ge "$want_new" ]; then
|
||||
log "force_n_wal_rotations: $node reached target (+$((now - baseline)) >= +$want_new) after $attempt batches"
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
log "force_n_wal_rotations: $node only reached +$((now - baseline)) new segments (wanted +$want_new) after $max_attempts batches of $batch — WAL_SEGMENT_BYTES=$WAL_SEGMENT_BYTES may not be small enough for this job volume"
|
||||
return 1
|
||||
}
|
||||
|
||||
# config-upgrade-snapshot task 8.9 (Req 7.4): repeat the unit-level
|
||||
# test_marker_triggered_checkpoint_trim_preserves_standby_mid_catchup_segments
|
||||
# fixture (test_upgrade_snapshot_apply.c) against a REAL 3-node cluster under
|
||||
# chaotic timing — proving the marker-triggered checkpoint's WAL-segment trim
|
||||
# (persist_checkpoint_marker_triggered -> trim_wal_segments -> repl_trim_lsn,
|
||||
# bug-1147's pre-existing retention-floor guard, unmodified by this spec)
|
||||
# never strands a genuinely mid-catch-up standby's still-needed WAL
|
||||
# segment(s), under real kill/restart timing and real segment-rotation
|
||||
# volume instead of hand-controlled LSNs.
|
||||
#
|
||||
# Mirrors run_force_snapshot_round's "prove it, don't assume it" discipline:
|
||||
# every step that depends on empirical timing (segment rotation, guard
|
||||
# expiry) is confirmed by direct on-disk evidence and fails loud on a
|
||||
# violated precondition, rather than silently reporting a false PASS that
|
||||
# would prove nothing (task 8.9's own explicit instruction).
|
||||
#
|
||||
# bug-1147's guard has a WALL-CLOCK component the deterministic unit fixture
|
||||
# never has to reason about: a disconnected standby's floor guard
|
||||
# (repl.c's last_disconnected_lsn / last_disconnect_time) expires after
|
||||
# p->config.snap_interval_s (DEFAULT_SNAP_INTERVAL_S = 60s, persist.h/
|
||||
# config.h — not exposed by any CLI flag). If this round's kill-to-inspection
|
||||
# window ever exceeded that budget, the guard would correctly (not a bug)
|
||||
# stop protecting TARGET's floor segment, and the final assertion below would
|
||||
# then fail loud — which is CORRECT behavior for a round that ran that slow,
|
||||
# not a false positive, but it would mean the round proved nothing about the
|
||||
# guard's mid-catch-up behavior specifically. Elapsed kill-to-inspection time
|
||||
# is logged so a slow environment is diagnosable rather than silently masked.
|
||||
#
|
||||
# Chaos placement: Requirement 7.4 wants this exercised "under the chaotic
|
||||
# multi-kill conditions Requirement 6 exercises," but a literal CONCURRENT
|
||||
# second kill during TARGET's down window is structurally impossible here —
|
||||
# a 3-node cluster has no majority left if 2 of 3 are ever simultaneously
|
||||
# non-participants, and TARGET must stay down for the WHOLE critical window
|
||||
# to be genuinely "mid-catch-up" (mirroring the unit fixture's standby that
|
||||
# never advances past lsn2). Instead, one self-contained chaotic flap of the
|
||||
# CURRENT primary (kill -> restart WITHOUT waiting for convergence, mirroring
|
||||
# run_chaotic_rounds' own overlapping-kill style) runs BEFORE the critical
|
||||
# window begins, giving the round genuine, unplanned term/timing churn up
|
||||
# front; the cluster is then allowed to settle to a clean, well-understood
|
||||
# 3-of-3 baseline before TARGET is identified and killed. This is "the
|
||||
# sequence" Req 7.4 asks for, sequenced rather than concurrent, for a hard
|
||||
# quorum-math reason, not convenience.
|
||||
run_config_upgrade_wal_trim_round() {
|
||||
local rotations batch max_attempts xceadmin scratch_dir scratch_bin
|
||||
rotations="${XCE_HA_WALTRIM_ROTATIONS:-2}"
|
||||
batch="${XCE_HA_WALTRIM_BATCH:-15}"
|
||||
max_attempts="${XCE_HA_WALTRIM_MAX_ATTEMPTS:-20}"
|
||||
|
||||
xceadmin="$(find_xceadmin)" \
|
||||
|| fail "config-upgrade-wal-trim: xceadmin binary not found (run /xce-build all or /xce-build xceadmin)"
|
||||
|
||||
# 1. One early, self-contained chaotic flap of the CURRENT primary —
|
||||
# genuine unplanned term/timing churn, fully resolved BEFORE the
|
||||
# critical mid-catch-up window starts (see header comment: quorum
|
||||
# math rules out a literal concurrent second kill while TARGET is
|
||||
# down for the whole window).
|
||||
local flap_primary flap_prev_term flap_pid flap_new
|
||||
flap_primary="$(primary_from_log)"
|
||||
if [ -z "$flap_primary" ]; then
|
||||
wait_for_primary_log || log "config-upgrade-wal-trim: no primary elected yet before the early chaos flap"
|
||||
flap_primary="$(primary_from_log)"
|
||||
fi
|
||||
if [ -n "$flap_primary" ]; then
|
||||
flap_prev_term="$(latest_primary_term)"; flap_prev_term="${flap_prev_term:-0}"
|
||||
flap_pid="$(pgrep -f -- "$(node_pattern "$flap_primary")")"
|
||||
if [ -n "$flap_pid" ]; then
|
||||
log "config-upgrade-wal-trim: early chaos flap — killing primary $flap_primary (term=$flap_prev_term)"
|
||||
kill -9 $flap_pid 2>/dev/null
|
||||
sleep 1
|
||||
start_node "$flap_primary"
|
||||
flap_new="$(wait_for_new_primary "$flap_prev_term")"
|
||||
[ -n "$flap_new" ] && record_primary "$flap_new"
|
||||
log "config-upgrade-wal-trim: early chaos flap settled (new primary probe: ${flap_new:-none yet})"
|
||||
fi
|
||||
else
|
||||
log "config-upgrade-wal-trim: no primary found for the early chaos flap — skipping it (will still proceed once the cluster reconverges)"
|
||||
fi
|
||||
wait_for_full_convergence \
|
||||
|| fail "config-upgrade-wal-trim: cluster did not reconverge to 3-of-3 after the early chaos flap"
|
||||
log "config-upgrade-wal-trim: 3-of-3 converged after the early chaos flap — starting the critical mid-catch-up window"
|
||||
|
||||
# 2. Identify the primary and a non-primary TARGET to keep down through
|
||||
# the whole critical window (mirrors run_force_snapshot_round's
|
||||
# non-primary target-picking convention).
|
||||
local primary target n
|
||||
primary="$(discover_primary)" || fail "config-upgrade-wal-trim: no serving primary before picking TARGET"
|
||||
target=""
|
||||
for n in "${NODES[@]}"; do
|
||||
[ "$n" = "$primary" ] || { target="$n"; break; }
|
||||
done
|
||||
[ -n "$target" ] || fail "config-upgrade-wal-trim: could not pick a non-primary TARGET"
|
||||
log "config-upgrade-wal-trim: primary=$primary target=$target (target will be killed and kept down through the marker-triggered trim)"
|
||||
PRIMARY="$primary"
|
||||
|
||||
# 3. Prime enough segment volume that TARGET's eventual floor lands
|
||||
# strictly above segment 1 — otherwise there is nothing below the
|
||||
# floor to trim and the round would prove nothing (task 8.9's own
|
||||
# warning against a trivially-always-active single segment).
|
||||
# Confirmed empirically, not assumed.
|
||||
if [ "$(count_wal_segments "$primary")" -lt 2 ]; then
|
||||
force_n_wal_rotations "$primary" "$rotations" "$batch" "$max_attempts" \
|
||||
|| fail "config-upgrade-wal-trim: could not prime >=$rotations WAL segments on $primary before picking TARGET's floor"
|
||||
fi
|
||||
verify_no_loss || fail "config-upgrade-wal-trim: job loss during priming, before the critical window even started"
|
||||
|
||||
# 4. Baseline capture: TARGET is still up and the cluster just fully
|
||||
# reconverged, so the primary's CURRENTLY-ACTIVE segment at this exact
|
||||
# instant is where TARGET's own floor necessarily sits (it cannot need
|
||||
# anything the primary hasn't written yet, and being caught up it
|
||||
# cannot be behind an earlier segment either). This is the direct,
|
||||
# on-disk analog of the unit fixture's repl_on_ack(lsn2) call — except
|
||||
# here it is TARGET's real replication state, not a simulated ack.
|
||||
local baseline_segs seg_at_kill below_segs
|
||||
baseline_segs="$(list_wal_segments "$primary")"
|
||||
seg_at_kill="$(printf '%s\n' "$baseline_segs" | tail -1)"
|
||||
below_segs="$(printf '%s\n' "$baseline_segs" | sed '$d')" # every line except the last
|
||||
[ -n "$seg_at_kill" ] || fail "config-upgrade-wal-trim: $primary has no WAL segment at baseline capture — cannot identify TARGET's floor"
|
||||
[ -n "$below_segs" ] || fail "config-upgrade-wal-trim: no segment exists below the baseline active segment on $primary — priming did not produce trim material (should be unreachable after step 3)"
|
||||
log "config-upgrade-wal-trim: baseline on $primary — floor segment (must survive)=$seg_at_kill; below-floor segment(s) (must be trimmed)=$(printf '%s' "$below_segs" | tr '\n' ' ')"
|
||||
|
||||
# 5. Kill TARGET now, mid-catch-up: it has acked everything up through
|
||||
# seg_at_kill's coverage and NOTHING past it — exactly "has not yet
|
||||
# acknowledged the marker's position" (Req 7.4), since the marker has
|
||||
# not even been built yet at this point.
|
||||
kill_node "$target" || fail "config-upgrade-wal-trim: could not kill target $target"
|
||||
local kill_epoch; kill_epoch="$(date +%s)"
|
||||
log "config-upgrade-wal-trim: $target killed at epoch=$kill_epoch — kept down through the critical window (bug-1147 disconnect-guard budget: 60s default, DEFAULT_SNAP_INTERVAL_S)"
|
||||
|
||||
# 6. While TARGET is down, force >=$rotations MORE segment rotations past
|
||||
# seg_at_kill on the surviving 2-of-3 majority — a genuine boundary
|
||||
# case (multiple segments to trim, multiple to preserve), not a
|
||||
# single trivially-active segment.
|
||||
force_n_wal_rotations "$primary" "$rotations" "$batch" "$max_attempts" \
|
||||
|| fail "config-upgrade-wal-trim: could not force >=$rotations new WAL segment rotations past $target's floor while it was down"
|
||||
verify_no_loss || fail "config-upgrade-wal-trim: job loss while $target was down, before the upgrade was even submitted"
|
||||
|
||||
# 7. Build the upgrade-target cluster.bin (same scratch-copy + one-field
|
||||
# edit convention as run_config_upgrade_round — an UPDATE to an
|
||||
# existing queue's description, never an added queue; see that
|
||||
# function's own comment / bug-1469 for why an added queue would not
|
||||
# exercise a working live-upgrade path at all).
|
||||
scratch_dir="$TOP/.cfgupgrade-waltrim-src"
|
||||
scratch_bin="$TOP/.cfgupgrade-waltrim-src.bin"
|
||||
rm -rf "$scratch_dir" "$scratch_bin"
|
||||
cp -r "$REPO/src/configs" "$scratch_dir"
|
||||
sed -i 's/description: "night queue"/description: "night queue (config-upgrade-snapshot task 8.9 wal-trim soak marker)"/' \
|
||||
"$scratch_dir/queues.yaml"
|
||||
grep -q "task 8.9 wal-trim soak marker" "$scratch_dir/queues.yaml" \
|
||||
|| fail "config-upgrade-wal-trim: sed did not find the expected 'night queue' description line — src/configs/queues.yaml may have changed shape"
|
||||
"$xceadmin" config build "$scratch_dir" -o "$scratch_bin" \
|
||||
|| fail "config-upgrade-wal-trim: xceadmin config build failed for the upgrade-target config"
|
||||
log "config-upgrade-wal-trim: built upgrade-target cluster.bin: $scratch_bin"
|
||||
|
||||
# 8. Submit the upgrade against the surviving 2-of-3 majority (TARGET is
|
||||
# still down) — its committed WAL_CONFIG_MARKER apply on $primary is
|
||||
# what fires persist_checkpoint_marker_triggered -> trim_wal_segments.
|
||||
local upgrade_out
|
||||
upgrade_out="$(printf 'set masters %s\nset myname cfgupgrade-waltrim\nclient\nwait-primary %s\nupgradeconfig %s\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" "$scratch_bin" | xtest)"
|
||||
echo "$upgrade_out" | grep -q "Config upgrade submitted" \
|
||||
|| fail "config-upgrade-wal-trim: upgradeconfig did not report success: $upgrade_out"
|
||||
log "config-upgrade-wal-trim: upgrade submitted and committed on the surviving majority"
|
||||
|
||||
# 9. Give the primary's own apply pump a moment to apply the marker and
|
||||
# materialize+trim (Req 1.1/1.5 — synchronous from the applying
|
||||
# member's own perspective, but still async w.r.t. this shell).
|
||||
sleep 5
|
||||
|
||||
local kill_elapsed; kill_elapsed=$(( $(date +%s) - kill_epoch ))
|
||||
log "config-upgrade-wal-trim: kill-to-inspection elapsed=${kill_elapsed}s (bug-1147 disconnect-guard budget ~60s default)"
|
||||
if [ "$kill_elapsed" -ge 50 ]; then
|
||||
log "config-upgrade-wal-trim: WARNING — elapsed time is close to/over the disconnect-guard's default 60s budget; a failure below may reflect a genuinely-expired guard (correct behavior) rather than a real regression. If this round is flaky, investigate environment slowness before suspecting the guard."
|
||||
fi
|
||||
|
||||
# 10. THE assertion (Req 7.4's actual proof), BEFORE restarting TARGET:
|
||||
# inspect $primary's real on-disk WAL directory directly.
|
||||
local post_segs seg
|
||||
post_segs="$(list_wal_segments "$primary")"
|
||||
log "config-upgrade-wal-trim: post-trim segments on $primary: $(printf '%s' "$post_segs" | tr '\n' ' ')"
|
||||
|
||||
printf '%s\n' "$post_segs" | grep -qxF "$seg_at_kill" \
|
||||
|| fail "config-upgrade-wal-trim: TARGET's floor segment $seg_at_kill was TRIMMED — a mid-catch-up standby's still-needed WAL segment was stranded (bug-1147 guard violated under the marker-triggered path). post-trim segments: $post_segs"
|
||||
log "config-upgrade-wal-trim: floor segment $seg_at_kill correctly PRESERVED (still needed for $target's eventual catch-up)"
|
||||
|
||||
local stranded_ok=1
|
||||
while IFS= read -r seg; do
|
||||
[ -n "$seg" ] || continue
|
||||
if printf '%s\n' "$post_segs" | grep -qxF "$seg"; then
|
||||
log "config-upgrade-wal-trim: FAIL evidence — below-floor segment $seg is STILL PRESENT (should have been trimmed)"
|
||||
stranded_ok=0
|
||||
fi
|
||||
done <<<"$below_segs"
|
||||
[ "$stranded_ok" -eq 1 ] \
|
||||
|| fail "config-upgrade-wal-trim: at least one below-floor segment was NOT trimmed by the marker-triggered checkpoint — see FAIL evidence lines above. below-floor set: $below_segs"
|
||||
log "config-upgrade-wal-trim: all below-floor segment(s) correctly TRIMMED: $(printf '%s' "$below_segs" | tr '\n' ' ')"
|
||||
|
||||
local active_seg
|
||||
active_seg="$(printf '%s\n' "$post_segs" | tail -1)"
|
||||
[ -n "$active_seg" ] \
|
||||
|| fail "config-upgrade-wal-trim: $primary has no active WAL segment after the marker-triggered checkpoint"
|
||||
[ -f "$(wal_dir "$primary")/$active_seg" ] \
|
||||
|| fail "config-upgrade-wal-trim: active segment $active_seg reported by listing but missing on disk"
|
||||
log "config-upgrade-wal-trim: active segment $active_seg correctly PRESENT"
|
||||
|
||||
# 11. ONLY NOW restart TARGET and prove the preserved segment was
|
||||
# genuinely sufficient for real incremental catch-up, not merely
|
||||
# present-but-unused — the behavioral capstone.
|
||||
start_node "$target"
|
||||
wait_for_full_convergence \
|
||||
|| fail "config-upgrade-wal-trim: cluster did not reconverge to 3-of-3 after restarting $target — the preserved floor segment did not actually suffice for real incremental catch-up"
|
||||
log "config-upgrade-wal-trim: $target reconverged (3-of-3) using the preserved floor segment for incremental catch-up"
|
||||
|
||||
verify_no_loss \
|
||||
|| fail "config-upgrade-wal-trim: a committed job was LOST across the wal-trim-under-marker-trigger scenario"
|
||||
log "config-upgrade-wal-trim: zero job loss confirmed"
|
||||
}
|
||||
|
||||
# ── Fact-stream primitives (raft-apply-decoupling task 7.2, Req 10.2/10.4) ───
|
||||
#
|
||||
# Drives a continuous stream of WORKER-FACT messages (jstart/jfinish/jfail —
|
||||
|
|
@ -2245,6 +2568,26 @@ if [ "$MODE" = "config-upgrade" ]; then
|
|||
exit 0
|
||||
fi
|
||||
|
||||
# ── Config-upgrade WAL-trim mode: config-upgrade-snapshot task 8.9 ──────────
|
||||
# Opt-in (--config-upgrade-wal-trim / XCE_HA_MODE=config-upgrade-wal-trim).
|
||||
# See run_config_upgrade_wal_trim_round's own header comment for the full
|
||||
# mechanism trace and Req 7.4's mid-catch-up WAL-trim proof. Single dedicated
|
||||
# round, not a loop — like --config-upgrade/--force-snapshot; every hard
|
||||
# invariant (segment-rotation volume, floor-segment preservation, below-floor
|
||||
# trim, active-segment preservation, post-restart reconvergence, zero job
|
||||
# loss) is asserted with an immediate fail() inside
|
||||
# run_config_upgrade_wal_trim_round() itself, so reaching the PASS line below
|
||||
# already means every check passed. Does NOT run the sequential loop or
|
||||
# leadership-coverage maneuver.
|
||||
if [ "$MODE" = "config-upgrade-wal-trim" ]; then
|
||||
log "MODE=config-upgrade-wal-trim — exercising the marker-triggered checkpoint's WAL-segment trim against a genuinely mid-catch-up standby under chaotic timing"
|
||||
run_config_upgrade_wal_trim_round
|
||||
log "PASS (config-upgrade-wal-trim): ${#ALL_JOBIDS[@]} jobs, below-floor WAL segment(s) trimmed, floor + active segments preserved, zero job loss after real incremental catch-up"
|
||||
cleanup
|
||||
trap - EXIT INT TERM
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Chaotic mode: overlapping-kill rounds → isolated verdict ──────────────────
|
||||
# Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up
|
||||
# windows; outcomes are COUNTED (committed divergence + committed job loss) with
|
||||
|
|
|
|||
|
|
@ -147,9 +147,12 @@
|
|||
- Observable: across a full chaotic-soak run with configuration upgrades in flight, no standby ever loses a WAL segment it still needed for catch-up
|
||||
- _Requirements: 7.4_
|
||||
- _Depends: 7, 8.1, 8.2, 8.3, 8.4_
|
||||
- _Blocked: bug-1474 — pre-existing, out-of-spec-boundary HA connection-management defect (`repl_add_standby`'s first-empty-slot allocation, `repl.c:74-133`, unconditional `memset` at line 112; `primary_heartbeat_fn`'s reconnect sweep, `ha.c:9219-9254`; both blame to commits `7b396502`/`fa66d1ba9`/`8ef64a7e`, weeks before this spec). Sequence: killing a standby correctly frees its replicator slot and sets the bug-1147 `last_disconnected_lsn` retention guard; ~9s later the primary's routine heartbeat reconnect sweep spuriously treats an ALREADY-connected peer (connected via a promoted-vote-channel, not yet a heartbeat-task connection) as missing, spawns a redundant "late-joiner" reconnect, and `repl_add_standby` reallocates the just-freed slot — `memset`-clobbering the retention guard before any trim runs. Confirmed reproducible 2/2 against independent live 3-node clusters, and confirmed via `git blame` to be reachable identically under the pre-existing periodic-checkpoint trigger — nothing this spec added or changed. The live-soak round (`run_config_upgrade_wal_trim_round`, `run-ha-soak.sh`) is complete, committed, and correctly reproduces the violation; independently reviewed and confirmed sound (round design faithfully mirrors the unit fixture's mechanics; root-cause diagnosis independently re-derived from `repl.c`/`ha.c` plus raw preserved logs, not trusted from the implementer's report alone). No scoped adjustment to the round can dodge the race without suppressing the very live-cluster heartbeat behavior Req 7.4 requires testing under. Unblocks when bug-1474 is fixed by whoever owns HA connection/replicator-slot management — no further config-upgrade-snapshot work needed once that lands (re-run `run-ha-soak.sh --config-upgrade-wal-trim` to confirm clean PASS). See `.wolf/buglog.json` bug-1474 for the full forensic trace._
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- **Task 8.9 (live soak, genuinely requires chaos — unlike 8.5/8.7/8.8's unit-level redirections): infra complete, task BLOCKED on a new pre-existing bug (bug-1474).** Unlike every other 8.x subtask, this one's own bullet ("under the full chaotic multi-kill soak rather than an isolated fixture") makes the point of the task proving the mid-catch-up WAL-trim guard (`repl_trim_lsn`/`trim_wal_segments`, bug-1147) holds under REAL timing uncertainty, not just deterministic fixture control — so it could not be redirected to a cheaper unit-level proof the way 8.5/8.7/8.8 were. Built `run_config_upgrade_wal_trim_round()` + `--config-upgrade-wal-trim` in `run-ha-soak.sh` (deploys with small `--wal-segment-bytes`, kills a non-primary TARGET, forces ≥2 real segment rotations past its floor on survivors, submits a live config upgrade to fire the marker-triggered checkpoint's trim, then directly inspects survivors' on-disk `wal/` directory BEFORE restarting TARGET — mirrors the unit fixture's own seg-1-gone/seg-2-preserved/seg-3-preserved assertions against real files instead of a tmpdir fixture). Ran twice against independent live 3-node clusters: **both runs failed** — the floor segment protecting the killed target was trimmed both times. Root-caused (not guessed) to **bug-1474**, a general HA connection-management race entirely unrelated to config-marker triggering: killing a standby frees its replicator slot and sets the retention guard; ~9s later the primary's routine heartbeat reconnect sweep spuriously treats an already-connected peer as "missing" (comparing indices under two different conventions — relative `votes_idx` vs. absolute `peer_idx`, per the reviewer's independent finding), spawns a redundant reconnect, and `repl_add_standby`'s first-empty-slot allocation reuses the just-freed slot, `memset`-clobbering the guard before trim ever runs. Reviewer independently re-derived this from `repl.c:74-133`/`ha.c:9219-9254` plus raw preserved logs (not the implementer's quoted excerpts) and confirmed via `git blame` the responsible code predates this spec by weeks and is reachable identically under the OLD periodic-checkpoint trigger — genuinely out of this spec's boundary, same class of finding as bug-1471/1472. Reviewer also confirmed no scoped adjustment to the round's chaos could dodge the race without suppressing the live heartbeat behavior Req 7.4 requires testing under, so BLOCKED (not a redesign) is the correct call. **Spec-wide status after this task**: every implementable task (1 through 8.5, 8.7, 8.8) is `[x]`; the only two remaining open items (8.6, 8.9) are both blocked on real, independently-verified, out-of-spec-boundary HA defects this spec's own testing effort discovered (bug-1472, bug-1474) — neither blocks any OTHER task in this spec, and neither requires further config-upgrade-snapshot work once fixed upstream.
|
||||
|
||||
- **Task 8.8 (unit-level, no live soak needed — deliberately excludes 8.6):** proves Req 4.4/7.2 (a second marker committing before the first's install completes must cleanly supersede it, never conflate) via two independently-built `t42`/`t43`-family fixtures in `test_config_repl.c`: `c_control` commits+applies+checkpoints marker A then marker B normally; `c_raced` holds the shared `checkpoint_in_progress` CAS across *both* markers' commit+apply (inlining `t43_commit_marker`'s 4-step body rather than changing its signature), so both markers' forced-checkpoint attempts hit `-EBUSY` and skip — genuinely "still installing." Mid-race assertions (captured before releasing the CAS) prove `mc->cluster_config` already reflects marker B while `cfgref_present` is still false and the fence is set; post-resolve assertions (after releasing the CAS, calling `persist_checkpoint()`, then an explicit `upgrade_materialization_verify()`) prove the settled state reflects only B. Discovered mid-implementation that whole-file `snapdiff_paths` byte-identity between the two fixtures' final checkpoints does NOT hold — `c_control`'s marker-A checkpoint physically consumes one extra WAL LSN (`persist_write_checkpoint_marker`, persist.c:1684) before marker B is appended, so B lands at a different absolute LSN in each fixture, and `write_snapshot`'s sentinel header embeds `checkpoint_lsn` at byte offset 8. Adapted to a field-aware comparison (full `memcmp` from byte 16 onward) plus `snapdiff_paths` diagnostic corroboration that offset 8 is the *only* divergence — a more precise proof than blind byte-identity, reviewer independently re-derived and confirmed this against `write_snapshot`'s actual layout rather than trusting the claim. One RED→GREEN cycle (`bug-1473`, test-construction gap not a production bug): the bare `persist_checkpoint()` resolve call doesn't itself trigger `upgrade_materialization_verify()` in a delaymgr-less unit fixture — production's self-healing (`schedule_upgrade_materialization_retry`, fired from `ha.c:2541-2542` on every commit-advance) only runs on a live cluster; the fixture needs the manual call, mirroring the existing 6.1-c fence tests' pattern. Reviewer's only findings were non-blocking style notes (could have used the public `upgrade_materialization_retry_fn` wrapper instead of calling `upgrade_materialization_verify` directly — functionally identical here since the fence was already independently confirmed set). Suite: 1027 → 1028 tests, 0 failures.
|
||||
|
||||
- **Task 8.7 (both scenarios, unit-level, no live soak needed):** design.md's Testing Strategy was explicit this task should use deterministic fault injection rather than chaotic-soak timing probability, and both hooks needed already existed from earlier tasks (`persist_marker_checkpoint_fault_hook`, task 8.1; `xce_upgrade_marker_committed_hook`, task 7) — task 8.7 was purely additive test work, zero production code touched. **Scenario 1** (`test_marker_checkpoint_fault_hook_crash_then_restart_resumes_materialization`, `test_upgrade_snapshot_apply.c`) builds on the existing seam test (which only proved the fault hook fires and gates the checkpoint call) by clearing the fault, tearing down without deleting the on-disk WAL, and booting a genuinely fresh `persist_t`/`cluster_t` via `persist_recover()` — a real second-process boot-replay simulation, deliberately not a same-process re-call (which the test's own comment argues would be a weaker, trivially-passing proof). **Scenario 2** (`test_leadership_change_mid_materialization_completes_via_same_generic_apply_drain`, `test_config_repl.c`) proves Req 5.4 by building a promoted member and a non-promoted control member from byte-identical committed-but-unapplied markers, draining each via the respective real production path (`persist_apply_pump_run_sync`, which production promotion — `raft_cb_become_leader`, `ha.c` — calls unconditionally right after the role flip; vs. plain `persist_apply_pump_run` for the control), and asserting the resulting checkpoints are byte-identical via `snapdiff_paths` (task 8.4's tool). Reviewer independently verified (not trusted) both design claims by reading `ha.c` directly (`upgrade_materialization_fenced` only refuses an *already*-fenced member; the sync-drain call is unconditional and unguarded by role-specific logic) and specifically interrogated whether this was a tautology (calling the same underlying function twice) — concluded it genuinely is not, since the real variable under test is role state at drain time plus the production entry point, and the test separately asserts promotion alone has zero effect on the fence. Also confirmed via direct comparison against the existing 6.2/8.3 fence tests that this is new *continuation* coverage (fence unset → promotion succeeds → drain completes → config activates), not a duplicate of the existing *gating* coverage (fence set → promotion refused, never reaching the drain).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue