feat(config-upgrade-snapshot): live config-upgrade soak infra + fix bug-1468/bug-1469 (task 8.6)
Build the live 3-node chaotic-multi-kill config-upgrade soak scenario (run-ha-soak.sh --config-upgrade) required by Requirements 4.1, 4.3, 6.1, 6.4. This needed genuinely new client-side infrastructure: a hand-rolled capnp send path for OP_upgradeConfig (the auto-generated encoder is a no-op for its Data fields, worked around the same way xce_replconfigpush_req_raw already does), plus cmd_hostctrl/cmd_upgradeconfig CLI commands. Building and running this live surfaced two real production bugs, both fixed here: - bug-1468: an unrelated prior commit inverted OPCode_upgradeConfig's primary/standby dispatch guard in cluster.c, making config upgrades unreachable from any network client. - bug-1469: apply_committed_entry's WAL_CONFIG_MARKER case never called config_activate_committed(), so the node that originates a config upgrade never refreshed its own in-memory state as part of committing it. Killing that node before some later, unrelated event happened to refresh it left its checkpoint permanently stale -- the first live, empirical reproduction of bug-857's originally hypothesized failure mode. Fixed by activating before the marker-triggered checkpoint write, since checkpoint content is built from live runtime tables that only reflect the reload after activation runs. A separate, narrower checkpoint divergence (bug-1471, not config-related) remains open and still blocks a clean soak PASS -- task 8.6 stays in-progress until it's resolved.
This commit is contained in:
parent
2bd44e8e4d
commit
959b0cdacb
13 changed files with 1022 additions and 10 deletions
|
|
@ -266,6 +266,83 @@ invariant is asserted with an immediate `fail()` inside
|
|||
`run_array_chaos_retry_round()`, so a violation aborts the run before this
|
||||
line is ever reached.
|
||||
|
||||
### `--config-upgrade` mode: live configuration upgrade under chaotic failover (config-upgrade-snapshot task 8.6)
|
||||
|
||||
Drives a real `OP_upgradeConfig` RPC (the new `upgradeconfig` `xce_test`
|
||||
client command, backed by `xcecli_upgradeconfig` →
|
||||
`xce_upgradeconfig_req_raw`) against the live primary, submitted in the
|
||||
**background** and interleaved with a bounded chaotic kill/restart sequence
|
||||
(`XCE_HA_CONFIGUPGRADE_KILLS` rounds, default 3) — relying on the client's own
|
||||
transparent failover (`XCECLI_FAILOVER_CALL`) to survive a primary change
|
||||
mid-call, exactly like `--array-chaos-retry` relies on it for a job-array
|
||||
submit. See `run_config_upgrade_round()`'s header comment (`run-ha-soak.sh`)
|
||||
for the full mechanism trace.
|
||||
|
||||
Sequence:
|
||||
|
||||
1. Joins a synthetic worker host (wire-level `join`, same pattern
|
||||
`--factstream` uses — no real `xcecd` process) and admin-closes it via the
|
||||
new `hostctrl -c` command, giving Req 6.4 a concrete admin bit to prove
|
||||
survives the upgrade.
|
||||
2. Submits an initial batch of jobs — committed state the upgrade must
|
||||
preserve.
|
||||
3. Builds a second, "upgrade-target" `cluster.bin` from a scratch copy of the
|
||||
**whole** `src/configs/` tree (not just `queues.yaml` — `xceadmin config
|
||||
build` needs the full directory) with one deliberate, observable change:
|
||||
the `night` queue's description text. Deliberately an **update to an
|
||||
existing queue**, not an added one — see `run_config_upgrade_round()`'s
|
||||
own comment and `.wolf/buglog.json` bug-1469: a live (non-restart) config
|
||||
upgrade has no working mechanism today to create a *new* runtime queue
|
||||
object in a member's `c->qmgr` (`build_queues()`, queue.c, only ever runs
|
||||
once at process boot); that is a genuine, deep gap in tasks 1-7's
|
||||
materialization logic, confirmed live and flagged for their owners, but
|
||||
out of this task's scope to fix. An update to an existing queue's
|
||||
description exercises exactly what Req 4.1-4.3/6.1/6.4 ask this scenario
|
||||
to prove without depending on that separate, unrelated mechanism.
|
||||
4. Submits the upgrade in the background, then runs the chaotic kill/restart
|
||||
sequence — restarting the previously-downed node WITHOUT waiting for full
|
||||
reconvergence before the next kill (like `--chaotic`), so the upgrade
|
||||
genuinely has a chance to interleave with an in-progress recovery, not just
|
||||
complete cleanly before or after every kill.
|
||||
5. After the cluster reconverges 3-of-3 and the background submission
|
||||
confirms success (retried once against the settled cluster if it did not
|
||||
report success while kills were still landing), hard-asserts (`fail()`,
|
||||
not count-and-tolerate — a definite correctness claim, like
|
||||
`--force-snapshot`):
|
||||
- **materialized-state identity** across all 3 members — byte-diff
|
||||
(`snapdiff`) of each member's latest on-disk checkpoint file;
|
||||
- **active-configuration identity** — `configcompare` (task 8.4's tool,
|
||||
its first live use here) across all 3 members' `conf/current`;
|
||||
- the synthetic host's **admin-CLOSED status specifically survives**,
|
||||
checked DIRECTLY per member (not merely inferred from the two identity
|
||||
checks above) by rotating primary through all 3 nodes in turn
|
||||
(`force_elect_node`) and asserting `workerlist` shows it CLOSED while
|
||||
each one serves — proving a standby-turned-primary materializes the
|
||||
admin bit correctly too, not just whichever node happened to be primary
|
||||
when the upgrade committed.
|
||||
|
||||
Single dedicated round, not a loop — like `--force-snapshot`/`--array-soak`.
|
||||
|
||||
```bash
|
||||
SKILL_DIR=.claude/skills/xce-test-ha
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --config-upgrade ~/tmp/xce-ha-configupgrade
|
||||
# tune kill-round count / synthetic host name
|
||||
XCE_HA_CONFIGUPGRADE_KILLS=5 XCE_HA_CONFIGUPGRADE_HOST=my-host \
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --config-upgrade ~/tmp/xce-ha-configupgrade
|
||||
```
|
||||
|
||||
Verdict line:
|
||||
|
||||
```
|
||||
[xce-test-ha] PASS (config-upgrade): 3 jobs, zero materialized-state divergence, zero configuration divergence, host admin-status confirmed across all members
|
||||
```
|
||||
|
||||
Like `--array-soak`/`--array-chaos-retry`, `result` is unconditionally
|
||||
`PASS` here — every hard invariant is asserted with an immediate `fail()`
|
||||
inside `run_config_upgrade_round()`, so a violation aborts the run before
|
||||
this line is ever reached; there is no `verdict_emit`/bug-924 exclusion
|
||||
tally for this mode.
|
||||
|
||||
### Verdict self-test (no cluster)
|
||||
|
||||
The verdict parser has a deterministic, pure-bash self-test that feeds fixture
|
||||
|
|
@ -361,7 +438,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
| Env var | Meaning | Default |
|
||||
|---|---|---|
|
||||
| `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, or `array-chaos-retry` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, `array-chaos-retry`, or `config-upgrade` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
|
||||
| `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
|
||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–50) | `6` |
|
||||
|
|
@ -381,6 +458,8 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
| `XCE_HA_ARRAYRETRY_ROUNDS` | `--array-chaos-retry` kill-during-submit rounds | `5` |
|
||||
| `XCE_HA_ARRAYRETRY_SIZE` | `--array-chaos-retry` element count of the array each round submits | `6` |
|
||||
| `XCE_HA_ARRAYRETRY_KILL_DELAY` | `--array-chaos-retry` seconds slept after backgrounding the submit before killing the primary | `0` |
|
||||
| `XCE_HA_CONFIGUPGRADE_KILLS` | `--config-upgrade` kill/restart rounds interleaved with the upgrade submission | `3` |
|
||||
| `XCE_HA_CONFIGUPGRADE_HOST` | `--config-upgrade` name of the synthetic worker host joined and admin-closed before the upgrade | `cfgupgrade-host` |
|
||||
|
||||
## What the driver does (maps to the requested steps)
|
||||
|
||||
|
|
|
|||
|
|
@ -32,6 +32,25 @@
|
|||
# own log that a snapshot install actually happened (not
|
||||
# just a claim), then asserts zero walcompare divergence
|
||||
# and zero job loss. Single dedicated round, not a loop.
|
||||
# --config-upgrade config-upgrade-snapshot task 8.6 (Req 4.1, 4.3, 6.1,
|
||||
# 6.4): admin-closes a synthetic worker host, then submits a
|
||||
# live configuration upgrade (OP_upgradeConfig, via the new
|
||||
# `upgradeconfig` xce_test command) CONCURRENTLY with a
|
||||
# bounded chaotic kill/restart sequence (XCE_HA_CONFIGUPGRADE_
|
||||
# KILLS rounds), relying on the client's own transparent
|
||||
# failover (XCECLI_FAILOVER_CALL) to survive a primary
|
||||
# change mid-call. After the cluster reconverges, hard-
|
||||
# asserts (fail(), not count-and-tolerate — like
|
||||
# --force-snapshot, this is a definite correctness claim):
|
||||
# (1) materialized-state identity across all 3 members via
|
||||
# byte-diff of each member's latest on-disk checkpoint file
|
||||
# (snapdiff); (2) active-configuration identity via
|
||||
# configcompare (task 8.4's tool, its first live use); (3)
|
||||
# the admin-closed host's status specifically survives,
|
||||
# checked DIRECTLY per member (not inferred from (1)/(2))
|
||||
# by rotating primary through all 3 nodes and querying
|
||||
# workerlist against each. Single dedicated round, not a
|
||||
# loop.
|
||||
# --factstream raft-apply-decoupling task 7.2 (Req 10.2/10.4): drives a
|
||||
# continuous WORKER-FACT stream (jstart/jfinish/jfail — real
|
||||
# wire-protocol worker reports, distinct from job-submission
|
||||
|
|
@ -154,6 +173,14 @@
|
|||
# (default 0 — immediate, to maximize "chaotic"
|
||||
# uncertainty about whether the WAL append/
|
||||
# commit happened before the kill)
|
||||
# XCE_HA_CONFIGUPGRADE_KILLS kill/restart rounds interleaved with the
|
||||
# config-upgrade submission (default 3,
|
||||
# independent of the general --chaotic ROUNDS
|
||||
# knob — this is a single dedicated round, not
|
||||
# a soak loop)
|
||||
# XCE_HA_CONFIGUPGRADE_HOST name of the synthetic worker host joined and
|
||||
# admin-closed before the upgrade, used to prove
|
||||
# Req 6.4 (default "cfgupgrade-host")
|
||||
set -u
|
||||
|
||||
# ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─
|
||||
|
|
@ -174,6 +201,7 @@ while [ $# -gt 0 ]; do
|
|||
--factstream) MODE="factstream" ;;
|
||||
--array-soak) MODE="array-soak" ;;
|
||||
--array-chaos-retry) MODE="array-chaos-retry" ;;
|
||||
--config-upgrade) MODE="config-upgrade" ;;
|
||||
--selftest) exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;;
|
||||
*[!0-9]*|'') ARGS+=("$1") ;; # non-integer ⇒ deploy dir
|
||||
*) ROUNDS_CLI="$1" ;; # bare integer ⇒ rounds
|
||||
|
|
@ -1016,6 +1044,276 @@ run_force_snapshot_round() {
|
|||
walcheck_count "${#ALL_JOBIDS[@]}"
|
||||
}
|
||||
|
||||
# ── Config-upgrade primitives (config-upgrade-snapshot task 8.6) ────────────
|
||||
|
||||
# Locate the xceadmin binary — same search list deploy-ha.sh uses (meson
|
||||
# build output under whichever profile built it, or a nix-installed prefix).
|
||||
# Needed here because this mode builds a SECOND cluster.bin itself (the
|
||||
# upgrade target), unlike every other mode which only ever consumes the one
|
||||
# deploy-ha.sh already built into ctrl-1/conf during Step 2.
|
||||
find_xceadmin() {
|
||||
local candidate
|
||||
for candidate in \
|
||||
"$REPO/src/xce.build/xceadmin/.venv/bin/xceadmin" \
|
||||
"$REPO/src/build.release/xceadmin/.venv/bin/xceadmin" \
|
||||
"$REPO/src/build.dbg/xceadmin/.venv/bin/xceadmin" \
|
||||
"$REPO/src/result/bin/xceadmin" \
|
||||
"$(command -v xceadmin 2>/dev/null || true)"
|
||||
do
|
||||
if [ -n "$candidate" ] && [ -x "$candidate" ]; then
|
||||
echo "$candidate"
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Echo a member's latest (highest-LSN) on-disk checkpoint file
|
||||
# ($TOP/controller/<node>/data/snap/snap-<lsn hex>.capnp — persist.c's
|
||||
# p->snap_path, "%s/data/snap" under the node's workdir/name), or fail with
|
||||
# rc!=0 if none exists yet. Filenames are fixed-width %016 hex, so
|
||||
# lexicographic sort == LSN-numeric sort.
|
||||
latest_snapshot_file() {
|
||||
local node="$1" f
|
||||
f="$(ls -1 "$TOP/controller/$node/data/snap"/snap-*.capnp 2>/dev/null | sort | tail -1)"
|
||||
[ -n "$f" ] || return 1
|
||||
echo "$f"
|
||||
}
|
||||
|
||||
# config-upgrade-snapshot task 8.6 (Req 4.1, 4.3, 6.1, 6.4): drive a live
|
||||
# configuration upgrade (OP_upgradeConfig, the new client built in this same
|
||||
# task — lib/request.c xce_upgradeconfig_req_raw / lib/xceapi.c
|
||||
# xce_upgradeconfig / lib/client.c xcecli_upgradeconfig, testsuite/test.c
|
||||
# cmd_upgradeconfig) while chaotically killing/restarting the primary, then
|
||||
# hard-assert (fail(), not count-and-tolerate — mirroring
|
||||
# run_force_snapshot_round's own convention: this is a definite correctness
|
||||
# claim, not a racy/sometimes-expected outcome class):
|
||||
#
|
||||
# - materialized-state identity across all 3 members (Req 4.1) — via
|
||||
# byte-diff (snapdiff) of each member's latest on-disk checkpoint file,
|
||||
# the SAME tooling design.md's Requirements Traceability row 6.1 cites
|
||||
# (snapdiff_paths, configcompare) — NOT live per-node joblist/workerlist
|
||||
# RPCs, because this system's read model is primary-only: OPCode_get
|
||||
# (which joblist/workerlist/reslist all use) is absent from
|
||||
# ha_dispatch_gate's CTRL_STANDBY allow-list (ha.c), so a standby cannot
|
||||
# answer it directly — confirmed by direct inspection, not assumed.
|
||||
# - active-configuration identity across all 3 members (Req 4.2/4.3) — via
|
||||
# configcompare (task 8.4's tool), its first live use.
|
||||
# - a host's administratively-set status specifically survives (Req 6.4),
|
||||
# checked DIRECTLY per member — not merely inferred from the aggregate
|
||||
# identity checks above — by rotating primary through each of the 3
|
||||
# nodes in turn (force_elect_node, already used for this script's
|
||||
# leadership-coverage step) and querying workerlist against whichever
|
||||
# node is currently serving, asserting the host shows CLOSED each time.
|
||||
#
|
||||
# One dedicated round, not a loop — like run_force_snapshot_round.
|
||||
run_config_upgrade_round() {
|
||||
local kills="${XCE_HA_CONFIGUPGRADE_KILLS:-3}"
|
||||
local host="${XCE_HA_CONFIGUPGRADE_HOST:-cfgupgrade-host}"
|
||||
local xceadmin scratch_dir scratch_bin
|
||||
|
||||
xceadmin="$(find_xceadmin)" \
|
||||
|| fail "config-upgrade: xceadmin binary not found (run /xce-build all or /xce-build xceadmin)"
|
||||
|
||||
# 1. Join + admin-close a synthetic worker host BEFORE the upgrade, so
|
||||
# Req 6.4 has a concrete admin bit to prove survives. Wire-level
|
||||
# synthetic worker join — the same pattern run_factstream_round
|
||||
# already uses (`set myname <host>` + `worker` + `join`); no real
|
||||
# xcecd process is ever started. do_hostcontrol_op (worker.c) requires
|
||||
# the host to already exist in the live worker table (-ENOENT
|
||||
# otherwise) — this join step exists precisely to satisfy that, since
|
||||
# hosts.yaml config entries do NOT auto-populate workermgr (a worker_t
|
||||
# is created only by a live join, requirements.md's own Boundary
|
||||
# Context note).
|
||||
log "config-upgrade: joining synthetic host $host"
|
||||
local jout
|
||||
jout="$(printf 'set masters %s\nset myname %s\nworker\njoin\nquit\n' "$MASTERS" "$host" | xtest)"
|
||||
echo "$jout" | grep -q "host join ok" \
|
||||
|| fail "config-upgrade: synthetic host $host failed to join: $jout"
|
||||
|
||||
log "config-upgrade: admin-closing $host via hostctrl"
|
||||
local hout
|
||||
hout="$(printf 'set masters %s\nset myname soak\nclient\nwait-primary %s\nhostctrl -c -h %s\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" "$host" | xtest)"
|
||||
echo "$hout" | grep -q "Operation on host $host done" \
|
||||
|| fail "config-upgrade: hostctrl -c on $host failed: $hout"
|
||||
log "config-upgrade: $host admin-closed"
|
||||
|
||||
# 2. Initial batch of jobs — committed state the upgrade must preserve.
|
||||
submit_jobs "$NJOBS" || fail "config-upgrade: initial submit failed"
|
||||
verify_no_loss || fail "config-upgrade: job loss before the upgrade started"
|
||||
|
||||
# 3. Build the upgrade-target config: a scratch COPY of the whole
|
||||
# src/configs/ tree (schema/ subdir included, exactly mirroring what
|
||||
# deploy-ha.sh itself passes to `xceadmin config build`) with one
|
||||
# deliberate, observable change — the "night" queue's description
|
||||
# text — so the post-upgrade identity check is a real assertion, not
|
||||
# a tautology.
|
||||
#
|
||||
# Deliberately an UPDATE to an EXISTING queue, not an ADDED queue
|
||||
# (bug-1469 finding, .wolf/buglog.json): a live (non-restart) config
|
||||
# upgrade has no working mechanism to create a NEW runtime queue_t
|
||||
# object in any member's c->qmgr — build_queues() (queue.c), the only
|
||||
# code that ever populates c->qmgr from parsed config, runs exactly
|
||||
# once at process boot (xce_config_init) and is unconditional/blind-
|
||||
# create, unsafe to re-invoke live. That is a genuine, deep,
|
||||
# confirmed gap in tasks 1-7's materialization logic (requires new,
|
||||
# carefully-designed production code plus a queue-removal-semantics
|
||||
# design decision this task is not chartered to make) — reproduced
|
||||
# live TWICE (before and after config-upgrade-snapshot task 6.2's own
|
||||
# activation-ordering fix), fully root-caused, and flagged for
|
||||
# tasks 1-7's owners, but explicitly OUT OF SCOPE for task 8.6 to fix.
|
||||
# An UPDATE to an existing queue's description exercises exactly what
|
||||
# Requirements 4.1-4.3/6.1/6.4 ask this task to prove (materialized
|
||||
# job/host state + active-configuration identity across members,
|
||||
# surviving chaotic failover) without depending on that separate,
|
||||
# unrelated, unfixed mechanism — collect_queue_frames' QueueAdminState
|
||||
# frame only ever carries name+admin-status (persist.c:1895), never
|
||||
# description/priority/etc., so this change is invisible to the
|
||||
# queue-entity section either way and is carried purely by the
|
||||
# checkpoint's embedded raw-config-bytes section (task 2.3,
|
||||
# read_active_config_bytes — always a fresh disk re-read, unaffected
|
||||
# by the qmgr-sync gap) and by configcompare's config-file-hash
|
||||
# comparison — both of which are exactly the "active configuration
|
||||
# identical" mechanisms Requirement 4.2/4.3 actually asks about.
|
||||
scratch_dir="$TOP/.cfgupgrade-src"
|
||||
scratch_bin="$TOP/.cfgupgrade-src.bin"
|
||||
rm -rf "$scratch_dir" "$scratch_bin"
|
||||
cp -r "$REPO/src/configs" "$scratch_dir"
|
||||
sed -i 's/description: "night queue"/description: "night queue (config-upgrade-snapshot task 8.6 soak marker)"/' \
|
||||
"$scratch_dir/queues.yaml"
|
||||
grep -q "task 8.6 soak marker" "$scratch_dir/queues.yaml" \
|
||||
|| fail "config-upgrade: sed did not find the expected 'night queue' description line to mark — src/configs/queues.yaml may have changed shape"
|
||||
"$xceadmin" config build "$scratch_dir" -o "$scratch_bin" \
|
||||
|| fail "config-upgrade: xceadmin config build failed for the upgrade-target config"
|
||||
log "config-upgrade: built upgrade-target cluster.bin with an updated 'night' queue description: $scratch_bin"
|
||||
|
||||
# 4. Submit the upgrade CONCURRENTLY with a bounded kill/restart
|
||||
# sequence — at least one kill/restart genuinely overlaps the upgrade
|
||||
# RPC. xcecli_upgradeconfig's XCECLI_FAILOVER_CALL wrapper (Part A) is
|
||||
# relied on to survive a primary change mid-call; xtest's own 60s
|
||||
# timeout bounds the background job regardless.
|
||||
local upgrade_log="$TOP/logs/config-upgrade-submit.out"
|
||||
: >"$upgrade_log"
|
||||
( printf 'set masters %s\nset myname cfgupgrade\nclient\nwait-primary %s\nupgradeconfig %s\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" "$scratch_bin" | xtest ) >"$upgrade_log" 2>&1 &
|
||||
local upgrade_pid=$!
|
||||
log "config-upgrade: submitted upgradeconfig in the background (pid=$upgrade_pid, log=$upgrade_log)"
|
||||
|
||||
local DOWN="" round PRIMARY PREV_TERM NEWP PID
|
||||
for round in $(seq 1 "$kills"); do
|
||||
log "config-upgrade: kill round $round/$kills"
|
||||
if [ -n "$DOWN" ]; then
|
||||
log "config-upgrade: restarting previously-downed $DOWN (no convergence wait — overlapping, like chaotic mode, so the upgrade genuinely interleaves with an in-progress recovery)"
|
||||
start_node "$DOWN"
|
||||
fi
|
||||
PRIMARY="$(primary_from_log)"
|
||||
if [ -z "$PRIMARY" ]; then
|
||||
wait_for_primary_log || log "config-upgrade: round $round: no primary elected yet"
|
||||
PRIMARY="$(primary_from_log)"
|
||||
fi
|
||||
if [ -z "$PRIMARY" ]; then
|
||||
log "config-upgrade: round $round: no primary to kill — skipping"
|
||||
continue
|
||||
fi
|
||||
record_primary "$PRIMARY"
|
||||
PREV_TERM="$(latest_primary_term)"; PREV_TERM="${PREV_TERM:-0}"
|
||||
PID="$(pgrep -f -- "$(node_pattern "$PRIMARY")")"
|
||||
if [ -n "$PID" ]; then
|
||||
log "config-upgrade: round $round: killing primary $PRIMARY (term=$PREV_TERM)"
|
||||
kill -9 $PID 2>/dev/null
|
||||
DOWN="$PRIMARY"
|
||||
else
|
||||
log "config-upgrade: round $round: primary $PRIMARY already gone"
|
||||
fi
|
||||
sleep 1
|
||||
NEWP="$(wait_for_new_primary "$PREV_TERM")"
|
||||
if [ -n "$NEWP" ]; then
|
||||
log "config-upgrade: round $round: new primary elected: $NEWP (term=$(latest_primary_term))"
|
||||
record_primary "$NEWP"
|
||||
else
|
||||
log "config-upgrade: round $round: no new primary within window (possible election stall)"
|
||||
fi
|
||||
done
|
||||
|
||||
# Restore full 3-of-3 before judging anything.
|
||||
if [ -n "$DOWN" ]; then
|
||||
log "config-upgrade: restarting last-downed $DOWN and settling"
|
||||
start_node "$DOWN"
|
||||
fi
|
||||
wait_for_full_convergence \
|
||||
|| fail "config-upgrade: cluster did not reconverge to 3-of-3 after the chaotic kill sequence"
|
||||
|
||||
log "config-upgrade: waiting for the background upgrade submission to finish"
|
||||
local waited=0
|
||||
while kill -0 "$upgrade_pid" 2>/dev/null && [ "$waited" -lt 65 ]; do
|
||||
sleep 1
|
||||
waited=$((waited + 1))
|
||||
done
|
||||
wait "$upgrade_pid" 2>/dev/null
|
||||
if ! grep -q "Config upgrade submitted" "$upgrade_log"; then
|
||||
log "config-upgrade: submission did not report success while interleaved with the kills — retrying once against the now-settled, reconverged cluster (prior log: $(cat "$upgrade_log"))"
|
||||
printf 'set masters %s\nset myname cfgupgrade\nclient\nwait-primary %s\nupgradeconfig %s\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" "$scratch_bin" | xtest >"$upgrade_log" 2>&1
|
||||
grep -q "Config upgrade submitted" "$upgrade_log" \
|
||||
|| fail "config-upgrade: upgradeconfig never succeeded, even after the chaotic sequence settled: $(cat "$upgrade_log")"
|
||||
fi
|
||||
log "config-upgrade: upgrade submission confirmed committed"
|
||||
|
||||
# Give every member a moment to apply the marker and materialize its own
|
||||
# upgrade snapshot (Req 1.1/1.5 — synchronous FROM EACH MEMBER'S OWN
|
||||
# apply-pump perspective, but a behind/just-restarted member's own apply
|
||||
# can still lag the marker's commit by a few seconds).
|
||||
sleep 5
|
||||
|
||||
# 5. Materialized-state identity (Req 4.1) + job survival.
|
||||
verify_no_loss \
|
||||
|| fail "config-upgrade: a committed job was LOST across the upgrade+chaotic-kill scenario"
|
||||
|
||||
local f1 f2 f3
|
||||
f1="$(latest_snapshot_file ctrl-1)" || fail "config-upgrade: ctrl-1 has no on-disk checkpoint after the upgrade"
|
||||
f2="$(latest_snapshot_file ctrl-2)" || fail "config-upgrade: ctrl-2 has no on-disk checkpoint after the upgrade"
|
||||
f3="$(latest_snapshot_file ctrl-3)" || fail "config-upgrade: ctrl-3 has no on-disk checkpoint after the upgrade"
|
||||
log "config-upgrade: comparing latest checkpoint files: $f1 vs $f2 vs $f3"
|
||||
local sdout
|
||||
sdout="$(printf 'snapdiff %s %s\nquit\n' "$f1" "$f2" | xtest)"
|
||||
echo "$sdout" | grep -q "IDENTICAL" \
|
||||
|| fail "config-upgrade: materialized-state DIVERGENCE ctrl-1 vs ctrl-2: $sdout"
|
||||
sdout="$(printf 'snapdiff %s %s\nquit\n' "$f1" "$f3" | xtest)"
|
||||
echo "$sdout" | grep -q "IDENTICAL" \
|
||||
|| fail "config-upgrade: materialized-state DIVERGENCE ctrl-1 vs ctrl-3: $sdout"
|
||||
log "config-upgrade: materialized state IDENTICAL across all 3 members (checkpoint byte-identity)"
|
||||
|
||||
# 6. Configuration identity (Req 4.2/4.3). configcompare_paths takes each
|
||||
# member's workdir/name conf-root DIRECTORY — $TOP/controller/ctrl-N,
|
||||
# NOT .../ctrl-N/conf — it appends "/conf/current" itself
|
||||
# (configcompare.c's read_member_config_sha256; verified by reading
|
||||
# the tool's source directly before this invocation).
|
||||
local ccout
|
||||
ccout="$(printf 'configcompare %s %s %s\nquit\n' \
|
||||
"$TOP/controller/ctrl-1" "$TOP/controller/ctrl-2" "$TOP/controller/ctrl-3" | xtest)"
|
||||
echo "$ccout" | grep -q "IDENTICAL across 3 members" \
|
||||
|| fail "config-upgrade: active-CONFIGURATION divergence across members: $ccout"
|
||||
log "config-upgrade: active configuration IDENTICAL across all 3 members"
|
||||
|
||||
# 7. Host admin-status survival (Req 6.4) — checked DIRECTLY per member,
|
||||
# not inferred from the aggregate identity checks above. Rotate
|
||||
# primary through each of the 3 nodes in turn (force_elect_node
|
||||
# no-ops cheaply if the target is already primary) and assert $host
|
||||
# shows CLOSED via workerlist each time — the only way to query a
|
||||
# SPECIFIC member's own live view directly, since a standby cannot
|
||||
# answer OPCode_get (workerlist) itself.
|
||||
local n wl
|
||||
for n in "${NODES[@]}"; do
|
||||
force_elect_node "$n" || fail "config-upgrade: could not force-elect $n to directly verify its own host admin-status view"
|
||||
wl="$(printf 'set masters %s\nset myname soak\nclient\nwait-primary %s\nworkerlist\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" | xtest)"
|
||||
echo "$wl" | grep -qE "^${host}[[:space:]]+CLOSED" \
|
||||
|| fail "config-upgrade: host $host is NOT admin-CLOSED while $n served as primary: $wl"
|
||||
log "config-upgrade: $host confirmed CLOSED while $n served as primary"
|
||||
done
|
||||
}
|
||||
|
||||
# ── Fact-stream primitives (raft-apply-decoupling task 7.2, Req 10.2/10.4) ───
|
||||
#
|
||||
# Drives a continuous stream of WORKER-FACT messages (jstart/jfinish/jfail —
|
||||
|
|
@ -1925,6 +2223,28 @@ if [ "$MODE" = "array-chaos-retry" ]; then
|
|||
exit 0
|
||||
fi
|
||||
|
||||
# ── Config-upgrade mode: config-upgrade-snapshot task 8.6 ────────────────────
|
||||
# Opt-in (--config-upgrade / XCE_HA_MODE=config-upgrade). See
|
||||
# run_config_upgrade_round's own header comment for the full mechanism trace.
|
||||
# Single dedicated round, not a loop — like --array-soak/--array-chaos-retry
|
||||
# (not --force-snapshot's verdict_emit-based counted-verdict style), every
|
||||
# hard invariant (job survival, materialized-state identity via snapdiff,
|
||||
# active-configuration identity via configcompare, host admin-status
|
||||
# survival) is asserted with an immediate fail() inside
|
||||
# run_config_upgrade_round() itself, so reaching the PASS line below already
|
||||
# means every check passed — no scan_bug924/V_BUG924 tally here, matching
|
||||
# array-soak/array-chaos-retry's convention (not force-snapshot/factstream's,
|
||||
# which feed V_BUG924 into a verdict_emit exclusion). Does NOT run the
|
||||
# sequential loop or leadership-coverage maneuver.
|
||||
if [ "$MODE" = "config-upgrade" ]; then
|
||||
log "MODE=config-upgrade — driving a configuration upgrade interleaved with chaotic multi-kill failover"
|
||||
run_config_upgrade_round
|
||||
log "PASS (config-upgrade): ${#ALL_JOBIDS[@]} jobs, zero materialized-state divergence, zero configuration divergence, host admin-status confirmed across all members"
|
||||
cleanup
|
||||
trap - EXIT INT TERM
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Chaotic mode: overlapping-kill rounds → isolated verdict ──────────────────
|
||||
# Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up
|
||||
# windows; outcomes are COUNTED (committed divergence + committed job loss) with
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue